I disagree with this idea. There aren't to many spin offs, and nothing is semi-related. I believe it was always the plan to have an evolving Avengers roster, because that's the way the comic worked. Unlike other groups like the Fantastic Four, there hasn't always been a 'core group'. The membership fluctuated all the time as people have active memberships, then go off for a while to maybe return later, or go join other groups. It's always been Marvel's 'anthology group'. Where pretty much anybody can be a member, eventually.
We had the founding members and all of their movies in phase 1 and 2. By the end of Age of Ultron though most of the team had split and gone their own ways, leaving replacements behind. Now we've got the phase 2/3 movies like Ant-Man, Black Panther and Captain Marvel who are all going to be joining the Avengers in time for the big showdown with Thanos. The only movie so far that on the surface seems out of place is Guardians of the Galaxy, but we know that ties very strongly with the overall storline that's been going from the start with Thanos and the Infinity Gems. To be honest the only movie or character I've seen so far that I can't place directly in relation to the overall storyline and makes me wonder 'why are they doing this' is Doctor Strange.
Are there a lot of characters? Sure. That's the Avengers for you. They've had even more character than the thousand different X-Men books over the years. But whether they fit within the seperate arcs or the overall story all of the characters and movies so far have fit neatly into place.