This is my same philosophy, including for myself. My eyes have been heavy before in the afternoon and I lament for my own soul that my flesh is so weak I cannot tarry a few more minutes. When my attention drifts the problem is with me not the minister. It is the Lord's Day not the Lord's hour - what better way do I have to spend my time than sitting under faithful exposition of God's word, hearing Him speak to me? When we leave the church building, you know what I go home and do? Listen to more sermons.
WLC 160 is a helpful reminder and something I pray regularly for my household and those attending our worship services:
WLC 160: What is required of those that hear the Word preached?
Answer: It is required of those that hear the Word preached, that they attend upon it with diligence, preparation, and prayer; examine: What they hear by the Scriptures; receive the truth with faith, love, meekness, and readiness of mind, as the Word of God; meditate, and confer of it; hide it in their hearts, and bring forth the fruit of it in their lives.