One other nuance I enjoy about using "vanilla ice cream" as a metaphor for "plain": many other ice creams have a bit of vanilla in them as well. By itself it's one of the finest flavors, but it serves as a baseline for other, perhaps experimental flavors that might not be to everyone's taste. (Sure chocolate, but also neopolitan, rocky road, strawberry, chubby hubby, pistachio, mint chocolate chip, etc)
So IMO the metaphor works best when the subject being compared is just fine in its "vanilla" version, and can become something spectacular when extended or enhanced by creativity, even if that spectacularity is only fully appreciated by a small minority.
I don't intend the use of the term as a pejorative but I can see where someone into a "non-vanilla" scene might think to use it that way to derisively to describe someone who has never bothered to expand their horizons, especially if the vanilla person was hostile to whatever kink was involved.