Daniel,
When I'm having bigger concerns about a church, I become easily annoyed by little things that would not bother others very much. So I think you ask an excellent question when you wonder if this baby-passing is a real problem or if you're the only one who finds it annoying.
If I recall correctly, you are starting to have other concerns about your church. It sounds like some of those concerns may be very valid. But it's wise for you to guard your heart during this time so that you don't start coming to worship with a critical attitude, easily bothered by many things. Keep yourself humble before God and appreciative of what is good, so that you at least are worshipping with a right heart. How others worship may indeed be troublesome, but don't let that make you into a man who looks for what is wrong and is quick to condemn.
I have to say, the baby-passing you describe sounds to me like it is a problem, but one that also might have a good side (for instance, it's good to delight in little ones being a part of God's people). And when compared to the other concerns you've mentioned in past posts, people letting a baby distract them during services seems a relatively mild concern.
This is why I'm suggesting you examine your heart. I suspect you may be at a place in life where your concern for right doctrine and right worship can make you easily annoyed with your church. It's good to be concerned for right worship, but dangerous to become a person who's quickly annoyed.
In your case, if this is not what's happening in your heart, then ignore what I've said. Please don't feel accused; I'm only guessing at what might be happening. But I bring it up because most of us have, at some time, become frustrated by our church and then become easily annoyed. It's happened to me many times, and I often have to guard against it when I walk into a worship service, because if I let it happen I will start thinking myself better than others and that is not fitting for worship.