12-tones stuff by Schoenberg is pretty bizarre.
Someone earlier posted about Trout Mask Replica, and I have to say, that album is pretty tough to crack (still haven't made sense of it)
But I'm surprised no one has mentioned any avant-garde jazz yet. John Coltrane, whom most know by his modal jazz stuff (My Favorite Things, A Love Supreme, Giant Steps) ventured into that territory near the end of his life ('65-'67). Or Cecil Taylor, who's 1966 "Unit Structures" can overwhelm even the experienced jazz listener. And a personal choice for me is Ornette Coleman's "Free Jazz: A Collective Improvisation" (1960), which upon first listen, I absolutely hated. But I can see through all the madness now, and enjoy listening to it on occasion. Whereas, only one year previously he released the much easier to listen "The Shape of Jazz to Come".
I listen to the latter a good deal more, but I have to say, if I were trapped on a desert island, I'd go with the "unaccessible", the "weird" music any day over the easy and simple music, just because there is so much room to explore, so many musical passages to unwind, break-down, and make sense of.