Yeah, seriously. Using the built-in effects is what's killing your performance. It's resource-chompy enough without any effects, and then you go up in there with amp simulation, EQ, reverb, and drum tracks, and all of a sudden you've got more than your hardware can deal with. It adds up really fast, especially in GarageBand 2 where they figured, "Nah, fuck it, they'll eventually have all the resources they need, let's just make this shit as detailed as we feel like."
It's the same problem that the game LockOn had. Brilliant in theory, but it was around 4 years before hardware finally caught up with it. It eventually almost killed the game entirely.