It is one connected statement. That's why it can't be used to prove that there's a different criterion for divorce vs. remarriage. Your approach attaches the exceptive clause exclusively to the remarriage phrase. Put that in light of Deuteronomy 24, where divorce is assumed and regulated, and you can see why, where divorce is appropriate, remarriage is allowed.1. I'm not sure what this proves - that exceptive clause is followed by the conjunctive clause "and marry another." It is all one connected statement.
2. I do think of adultery as being exclusively sexual in nature - isn't it always so when referring to human-to-human relationships throughout Scripture? Is there a place in Scripture where it is not? As I've mentioned above, I believe there can be dispute over what constitutes adultery and I have suggested that there may be a "lower bar" than how it is commonly defined today. Again note Christ's words in Matt.5.32: "But I say unto you, whosoever shall put away his wife except it be for fornication [πορνεία = sexual promiscuity of any type] causeth her to commit adultery [μοιχάομαι = breaking the covenantal bond]: and whosoever shall marry her that is divorced, committeth adultery." That adultery can happen prior to marriage proper is discussed in WCF 24.5: "Adultery or fornication, committed after a contract, being detected before marriage, giveth just occasion to the innocent party to dissolve that contract. In the case of adultery after marriage, it is lawful for the innocent party to sue out a divorce, and after the divorce to marry another, as if the offending party were dead." To allow for a non-sexual definition of "adultery" or a broader usage of adultery to include "all grave infidelity to the marriage" seems as harmful as expanding the Biblical teaching on abandonment in I Cor. to include other forms of abuse or neglect. Considering how often marriage, adultery, and divorce are used to teach us about our relationship to God, we need to take care how we define such issues (consider Hosea or Romans 7.1-4).
There's a reason we distinguish fornication from adultery. The difference is not that fornication is okay, but that the addition of a covenant adds a particular character of unfaithfulness and betrayal to the commission of fornication. If the only item in a marriage covenant were a promise not to have sex with anyone else, then nothing else would be a breach of the covenant. However, that is not the only commitment made in a marriage covenant. That would be the basis for a metonymic use of adultery.
Speaking of "expanding" presupposes the correctness of your position. There's also a danger of artificial restriction. Someone could argue, "I'm perfectly willing to live with my wife" who nonetheless had diminished her food, her raiment, and her duty of marriage. Saying "I don't mind if she stays" is not actually the same thing as living up to the commitments made. And such a diminution meant even a slave-wife could go out free (Exodus 21:7-11). On the basis of that passage, we can identify "abandonment" as broader than simply physical departure.