The lyrics are part of it, Britt Daniel has a knack for writing lyrics that find a happy medium between the banality of your stereotypical Brit-rock songs and the supposed pretension of your art-rockers (viz. Of Montreal, Art Brut).
The songs while spare are not what I would call minimalist (Low, Destroyer, those I could see as minimalist) but I'll admit adhere to a garage-y simplicity bringing them it into the orbit of say The White Stripes or Black Keys but with... two more members. On a track say like
I Turn My Camera On, the composition is exceedingly spare, you essentially get through the song with 2 chords and a very basic drum line and it's clearly about Daniel's vocals more than anything.
But then next track on the album you've got
My Mathematical Mind and while you're still not in prog rock territory you've definitely got a lot more going on instrumentally, culminating in a full on free style by the end of the song. Which is why a few songs here and there probably aren't going to answer your questions of where the attraction comes from. Although the band's aesthetic and Daniel's voice are very consistent within each album, the song construction tends to vary quite a bit.
But what you really want to do is go back to A Series of Sneaks and listen to a song like The Minor Tough which I've always felt sounds like what my high school friends' bands would have sounded like if my life were a movie starring Mischa Barton. It's elemental rock with very slutty sounding guitar, and really, either slutty guitar and Daniel's voice does it for you or it doesn't. Everything else is gloss.
Anyway I'm not nor have I ever been a musician as is probably blatantly obvious by how poorly I express myself when I'm trying to break songs down, so I'm sure some of the other Spoon fans on the forum with better musical literacy will be a lot more helpful.
