I guess it depends on what it is and how it fits into the Sabbath. Giving up video games then playing them on the Lord's Day would surely be worldly. But giving up ice-cream and then eating it on the Lord's Day would be fine in my opinion.
So for me, I think the real question is "should we be doing this on the Lord's Day?" I could be wrong, this was just my thought, though.