I tend to dislike most ska because its sound is very jarring to me - I feel like it's rearranging my heartbeat with all those damn trombones. I hate really energetic jazz for the same reason. [Naturally, my dad plays trombone. I am truly cursed.]
I prefer music that has a traditional pattern to it, I guess. Avant garde is great, but some rules are there for a reason - poetry that seems to end in the middle of a stanza just doesn't work, and neither does music that's just too random. In my experience most ska tends to jump all over the place, cutting off notes in the middle and switching to an entirely different rhythm. I would never voluntarily listen to the Bosstones, Goldfinger, or any of their ilk.
One of the reasons I wasn't terribly impressed with Sufjan Stevens was the strange beat of the title track. I just couldn't get into it, because it set my teeth on edge. (I do love "John Wayne Gacy Jr." though.)
The only ska song I have is Five Iron Frenzy's "My Evil Plan to Save the World," because it makes me smile - but even that has a sort of rhythm to it that kinda works. Still has that intensely irritating brass, though...
(Though, I'd never think of TMBG as ska...is that really their classification?)