Well, it's not even filth really, most of it is just dead skin particles and oils from your skin. (Unless you've been doing other things and are coated in dirt, grease, grime, whatever, then it's more than that obviously.) It doesn't usually bother me because I'm going to be wiping all of that off with my towel (which granted probably has a bit of bacteria on it because of previous baths/showers unless it's a fresh towel).
I mean, when you think about it, you're never truly "clean". And if you're not sterilizing the tub every single time, neither is the bathtub.