Actually, even in pickup groups I have had good experiences with loot. The way I do it is just tell the group before instancing: "Hey, right now I am looking for Leather Armor with Agility buffs, and Daggers. If any of those drops, please do not roll for them" at which point everyone else declares what they are looking for in the instance. If there is a specific item I am looking for in there, I will even mention that as well. If anyone in the group disagrees to that, I either quit the group and find another, or I see if everyone else in the group is mature enough to kick that immature player out. I have no time to waste in instances with a loot-hungry group that will dive on every sparkling corpse only to grab the only item I came to the instance for when they can't even use it. Declaring my intent before hand has never given me any problems, and it is a good way to show the pickup group that I am a mature player and that I will respect them for items they will need as well. It is also, as mentioned before, a really good way to weed out the immature players so you don't end up getting your drops ganked.
As far as pickup groups not being able to take larger mobs, that is why I try my best to dictate the roles of each person in the group before hand as well. If you give a player their primary focus in the instance, and then maybe some individual task for a unique type of mob, they do their jobs much more efficiently, and don't get overwhelmed with things to do, only to end up just using their solo technique they are used to. Giving them a clear plan like "Your pet is going to be the tank, don't worry if he isn't doing a lot of damage, just make sure he aggros as many enemies onto him as possible so we can damage them" allows them to focus their ability on holding aggro, while we all perform our roles, be it healing, damage, debuffs, or catching runners.
The best example I have of this was the plan I came up with for my group in Gnomeregan. Anyone who has fought Thermaplug in that instance knows he is a bitch of an enemy to fight. I gave my entire group (only 4 people) their tasks and we beat the guy time and time again without issue, when it was only 1 and a half people (me and a VoidWalker) damaging him. Normally people would think "No way ! That would take too long and you would get smoked !" but what I did is told our hunter to just kill every one of those bombs he saw, to keep them off me and the voidwalker... with his range on his gun, and his pet across the room, he could take them all out without issue long before they got to me. Our Paladin, instead of fighting, was dedicated to healing. That is all he did was watch everyone's health and heal who needed it. The Warlock only had to make sure his VoidWalker was keeping the boss's aggro and keep his demon from dying, while running around the room pushing all those buttons. And me, the rogue, was tasked solely with doing as much damage to Thermaplugg as possible, and stunning him often enough to keep the Voidwalker alive. I don't think anyone in our group ever got below half health, and Thermaplugg was slain every time... because everyone knew their role, and stuck strictly to it.
Much like in football, if everyone just lined up at the line and did whatever they felt they could to win that down, the game would never go anywhere since people don't have their roles, and the organized team will whip the crap out of them every time.
