That's different. The Japanese language, in its construction, is completely different from English/French/German/Spanish/etc. If you speak English, it's not hard to intuit the quality of someone's acting when they're speaking French because the language is so similar to your own. Also, with live-action films, you can see the actor's body language.
You're right; I don't really ever consider watching Asian live-action films and TV shows in English, because it takes away from the acting of the character. But anime doesn't have actors, it has VOICE actors, and the person doing the voice didn't draw the physical movements or facial expressions you're seeing. So you're listening to a language you don't understand and just assuming they're doing a good job. In all reality, they could be the Japanese equivalent of Pauly Shore for all you know.
I have no problem with people who prefer the actual sound of the voices on a case-by-case basis. As said above, the Japanese Spike sounds horrid compared to the English. But, on the other hand, I far prefer the Japanese voices of the actors in Risky/Safety. That doesn't mean I know or believe that they are better actors, just that they have a more pleasing voice.
Oh God why am I arguing with an animu.