This is the kind of question there's more than one correct answer to.
Most trained linguists will tell you to do whatever is most natural or whatever everyone else does. They tend to be a descriptivist bunch, linguists, saying language is as language is used. And whom does seem to be dropping out of English.
Grammar sticklers will insist on "whom" except in the subject of a sentence or clause. That's the "official" rule. (Some examples: Who let the dogs out? The dogs were let out by whom? Mary, who I think is nice but whom I hate anyway, hasn't told me who is coming with whom to my party.)
So what should you do?
John Lawler, a linguist who is well known to many non-linguists interested in English usage due to his heavy activity online, now suggests never using whom unless you're sure it's right. After all, you can't go wrong with who, at least informally, but whom is just plain wrong in some sentences.
I think if you properly learn in which kind of sentence to use which word, you'll sound more correct. But be careful, because you may also sound more pretentious. (That's besides Lawler's concern that you may use whom where it doesn't belong and thereby sound wrong to those who know the difference.) My personal advice: learn the "official" rule and use it in school essays and the like, being careful to use it correctly, but say whatever comes naturally to you otherwise, especially in conversation.