The 4 player team is the defending team, and the other two teams are the attackers.
How it originally worked was that during halftime, who ever was winning would be the defending team, but they weren't able to choose Tri-Color; instead, if they do Opens they'd randomly be put into a Tri-Color match against the two other teams. This was suppose to be a way for the loosing teams to make a comeback. The changes to make Tri-Color how it currently is was due to people not being able to find matches, unless you were on a super un-popular team, then you'd get it almost every time.
There's probably no more than 8 players, because the Switch likely isn't able to handle it. Fun fact, Splatoon 1 has better tick rates and less input/motion delay, because the Switch isn't able the handle those things, despite the WiiU being able to.