It is too simplistic to just dismiss the former idea
I think you mean "the latter idea."
Perhaps we should consider those (relatively) early witnesses, considering our distance from Peter's day. However, it is worth considering that the lack of information about the deaths of most of the Apostles prompted not a little speculative hagiography. Whenever and wherever Peter died, we know from the book of Acts and Romans that Paul undoubtedly arrived in Rome, and Peter was not there nor had he been by then.
Paul is overlooked as the Apostle who established the church in Rome. He did not begin its presence or organize its believers, but presumably
established it, meaning he did what Peter and John did in Act.8:14ff, when the church came to the converts of Philip's ministry, officially recognized and enfolded them into their single organization under the apostolate, and imparted spiritual gifts to them (see Rom.1:11).
Whether or not Peter did come to Rome and was martyred there as tradition has it, the claim that HE was the founder or establisher of Rome--both as to its congregational existence and status as the principal see--is assuredly an exaggerated claim, and part of the web of claims that Rome used and uses to aggrandize itself. In Rome's argument, all must be true for any of it to be true. Jesus needed to have made Peter the "rock" to build the church on. Peter had to be the universal bishop before he even left Jerusalem. Peter has to be the first pope. Any evidence to the contrary must be discounted as impossibly contradicting the "accepted version" that leaves Rome in an unchallengeable position of supremacy.
As the first city of the political Empire, it was natural that the political claims were transferred to the church and ecclesiastical claims. History stood against Rome's original assertion of primacy, as there were numerous other, important cities in the Christian world, starting with Jerusalem. And just when Rome began to boast mightily, the secular emperor moved his capital a thousand miles to the east, away from Rome. Rome's bishop was left claiming seniority over the church in the Western Roman empire. His pretensions to seniority over the Eastern church was just that, and the Byzantines generally never accorded Rome the status she asserted.
These facts are the sort of items that are used to poke holes in Rome's self-regard. Perhaps Peter came to Rome on his own, but it was not before Paul. Perhaps he was brought there, and was martyred there. Still, he was not the "first pope," from which idea Rome starts its hegemonic boasts. As Calvin and Henry and others reason, Rome has crafted numerous falsehoods in order to protect itself from counterclaims to its vaunted authority, and scrutiny of its claims; so why grant them even this?