That could be it, I wipe my hands on my pants sometimes. Of course, I don't notice the smell unless I smell-check by sticking my nose in them, but I figure that I tend to notice my smell less than other people, so I should wash them before I think they are stinky, just as a matter of courtesy.
I'm assuming that if you do anything that gets them dirty, you wash them, right? Like if you change your car's oil, go camping, or anything like that? Maybe my garage is an anomaly, but any cloth that makes contact with that floor gets a mix of auto fluid residue, sawdust, and dirt ground into it instantly.