Anselm addressed why man is responsible for his sins in Cur Deus Homo. I.24 contains a nice illustration for why man is held accountable for his debt to God even though he is incapable of paying the recompense:
“Suppose someone assigns his bondslave a task, and tells him not to leap into a pit from which he cannot by any means climb out, and that bondslave, despising the command and advice of his master, leaps into the pit which has been pointed out to him so that he is completely unable to carry out the task assigned to him. Do you think that his incapacity serves in the slightest as a valid excuse for him not to perform the task assigned him?”
This nicely illustrates your suggested answer to your original question. For Adam, there was ability to keep the law. However, Adam willingly sinned, and thereby plunged the human race into sin. Thus, while we do not have the ability to keep the law perfectly, we are still responsible to do so.