My understanding of festivals is that most people don't really plan most of their experience, they just drift around between the acts they actively pursue and will get caught up in whatever is playing in the meantime. People have this sort of reaction to a real punk rock attitude from performers. It's not because they're stupid, it's because they aren't in on the joke / they feel insulted. When I went to Toronto the whole gang and I went to see a show featuring a noise artist who put on a big act about being petulant onstage. I wasn't really into it, primarily because I could see the light coming out of the seams in his dickhead costume and aside from rude sax skronking I wasn't really feeling the performance. Remote contempt is not really a suitable substitute for actual enthusiasm in live performance.
As for the music itself, sorry to say it but if you're not already a a Zorn / Reed / Anderson fan, your reaction to the music is going to depend on your tolerance of massive exposure to self-regard. The argument from importance isn't nearly as effective as we music geeks would like it to be. A lot of people, the vast majority I would say, have a violent reaction to avant-garde improv music. And frankly, if the legends onstage ever gave a fuck about whatever scorn they would receive on the occasion of their Very Momentous Set, they had no business doing it.