Of course, much depends on the vibe within the particular church, and the fear of being labelled a danger to others is understandable. But I am suggesting it would be good if such concerns could be openly discussed in a church with a charitable mindset, so that people could bear each other's burdens rather than immediately despise or dismiss burdens they think are unreasonable.
In your case, let's say you have a burden in that you believe getting vaccinated is an unwise personal health decision or a capitulation to an evil agenda (or whatever your reason might be). If you had a church where people were inclined to help bear your burden even if they suspected you might be overreacting, or where people would challenge you patiently and without shaming you if they thought you needed to think through the matter more, and would work to find solutions that that help everyone follow their conscience and still join in fellowship, etc.—wouldn't that be a good church? Likewise, if you had discussions with that other family and came to understand why they are more concerned about the virus than you are, so that even if you ultimately still disagree you know how you can help bear their burden and/or gently challenge them in loving ways—wouldn't it be good for you to face up to your responsibility to love them like that?
The alternative is to believe we are so right and others are so wrong (they are so obviously deceived that we can only have scorn for their decisions) that the best we can do when we encounter them at church is keep our head down and avoid any meaningful fellowship. Perhaps this preserves some peace sometimes, but it isn't really what we ought to desire in a church.
A personal story: Yesterday at my church, a family that has been absent for a year showed up. I noticed them as they walked up the street toward the front door, and I greeted them. I told them I was glad to see them. Happily, they took a chance and explained that they'd stayed away due to COVID but were coming back now that the adults were fully vaccinated. They asked about my COVID/church decisions, and I explained my thinking (I've been in church every week except once when we stayed away voluntarily because we'd been exposed elsewhere in town). They seemed to appreciate that, and I asked them more about their concerns. They have a strong social conscience and mom and dad both have public-service jobs that require contact with vulnerable people, plus there's an ailing grandfather in the picture, and they felt duty-bound to stop attending worship for a while. Now, that is not a choice I think I would have made, for reasons they and I might discuss sometime, but rather than silently judge them or scoff at their decision I now can understand them better. Plus I know how to help bear their burden. They're still a bit leery of spending the morning inside our building, and of sending their daughter to my Sunday school class where the kids of course remain unvaccinated, and if I can make them feel a bit more comfortable by reminding my students about their masks or something of that sort, I'm glad to do it. It's a little bit of burden-sharing, and a significant kindness to them, even if I personally suspect it won't make much difference in actual virus protection.
My point is that I've never been particularly close to that family even though I've taught their kids, but after yesterday's talk at church—which did not shy away from COVID despite some differing concerns—I feel closer to them than before. That's good. That should be happening when the church meets together.