It is Virgil. It is Eclogue 2.65. I don't think it needs to be translated "lusts," in the (Biblical) sense that we would take that. The Loeb translation of T.E. Page is "Each on is led by his own liking." The context refers to animals. Here is the section:
The grim-eyed lioness pursues the wolf,
the wolf the she-goat, the she-goat herself
in wanton sport the flowering cytisus,
and Corydon Alexis, each led on
by their own longing. See, the ox comes home
with plough up-tilted, and the shadows grow
to twice their length with the departing sun,
yet me love burns, for who can limit love?
A footnote from the English translation of Terence's Phormio is: Ver. 454. "Quot homines, tot sententiae." This is a famous adage. One similar to the succeeding one is found in the Second Eclogue of Virgil, 1. 65: "Trahit sua quemque voluptas," exactly equivalent to our saying, "Every man to his taste."