The deal with stoner metal, specifically Sleep and their ilk, is that a huge portion of the music is the volume and fuzz/distortion level. If you're not already grabbed by the tone, the other elements of the music (namely huge slow riffs and whatever vocalist is in the band you're listening to) are probably not going to draw you in too much. Generally, the way stoner metal songs are written plays directly off of the guitar tone that's being used, instead of vice versa ... so if you can't get into the bass-heavy fuzziness, there isn't much you're likely to find in the music.
My advice is to listen to any of Om's albums or any of Electric Wizard's albums (specifically Dopethrone or Come My Fanatics). If those don't do it for you, try Kyuss, and if that's not your deal you should probably avoid anything that has the word "stoner" in the description.
However, there is a very real chance that you might just not be smoking enough weed.