It's interesting you say that, because I find that more often than not it is not non-Christians who feel it is nonbinding to them, but it is instead Christians who treat the Christian Sabbath as theirs and theirs only, to the neglect of providing it for others.
Me and a friend of mine were recently talking about the Lord's Day. She told me that before she was a Christian, one of her jobs in college was waitressing. And she distinctly remembers the crowd of church goers that would come in and they would often ask, "Why aren't you in church?" or themselves invite her to their church. And she would just be aghast, because the answer to that was clearly, "I can't because I'm waiting on you!" She is conscientious now about how she orders her affairs for the Lord's Day so that she doesn't return that burden to others.
I have seen the sentiment echoed by many of my non-christian friends or acquaintances, especially those who work in the service industry. In our day, the Sabbath can often become a day for the well-off, or white collar, or middle-class when it is the common laborers, blue collar, who often have less ability to keep the Lord's Day and are often most affected. It's interesting to note who in particular the command is directed to--those in authority have an especially greater responsibility to uphold the day of rest, as they often determine the ability for others to keep the Lord's Day.
I think it's something to think about as an area of witness.
We have divorced the Sabbath from its grounding in creation, it was present before the law was even delivered to the people of Israel. Just like we see the identity and relationship of one man to one woman before the commandment on adultery, so too do we see the pattern of work with a day of rest in the creation. I think that's profound to think about. It's something that was grounded in the beginning, rooted in the work of Christ, and is something to look forward to in eternity as we get a foretaste of that fulfillment on earth through our weekly observance.