With that being said, if you happened to be on a deserted island (or remote location) and shared the Gospel with someone and they wanted to place their trust in Christ and be baptized, I don't see the reason why you shouldn't. I know this really is a hypothetical situation, but the point is, I don't believe we need to hold to an absolute rule, like, ONLY pastors and such can baptize. If anything, the situation in Acts 8 shows that there can be extraordinary circumstances.
I think your pastor was wise. And to your point I think
Baptism is not required for entrance to heaven.
Therefore I would not baptize him until God in His providence shipwrecked another vessel with an ordained minister on it who preached a sermon from the word with several members present.
Else when men will not step up to be elders, women would be justified in doing so.
We have seen this kind of thinking lead to a liberal mess