Ranked mode too punishing?

[jsilva]

I've already posted how the matchmaking works in another thread, but here's a tl;dr. "Skill" is measured by kill count, death count, and coverage points. You're matched based on how your stats in these three areas measure up to others.

I'd have to disagree with this. Despite that it doesn't follow my experience in going from C- to S on three accounts (long story...), starting out in turf wars on the two new accounts suggested to me that they have more sophisticated metrics for determining 'skill'. I'm not certain what they are but I'm convinced it goes beyond kills, deaths, and turf inked.

Reading this thread reminded me of one important fact I hadn't considered in the other thread. The Splatoon player base/WiiU install base. I believe that's where the agenda is really coming from. The system is trying to create the image of being fair and competitive. But its real agenda is not being fair and competitive, it's load balancing the limited player base across all lobby types so it remains functionally populated. And that is the only agenda that matters to the system to keep the game running. If it's creating unfair pairings, I'm guessing it's intentionally trying to move people down in ranks, not for competitive purposes, but for lobby population distribution purposes. Meaning our ranks really aren't our ranks reflecting our real skill but a server assignment based on distribution need, given to us via rigged matches by handing us easy wins or easy losses to move us to the lobby the system needs us to be in. A clever disguise and fun "competition" to mask the fact that this is Splatoon, not COD or Halo, and there aren't (or at least previously weren't) enough players to go around to give them real skill brackets.

We CAN defy the odds. You can also defy the stacked house odds at the blackjack table. You can sink the basketball into the ever so slightly oval hoop a the carnival too. it's just that it won't happen often. Come to think of it, that's probably the situation in a nutshell. Too many modes, not enough players (or at least not before Christmas. The rollout of ranked and modes really revolved around distributing the player base as well now that I think about it.

I forgot about the mixing of different ranks in a lobby and how that indicates there are not sufficient players in any one rank to actually make a bracket. If there were enough C+'s for example they wouldn't need to mix B- and B players in a C lobby, or S players in an A- lobby. There's just not enough players in a given rank bracket at any time to make a real rank bracket. It can't be fixed without selling more copies if so. That makes me sad. It means the real problem with ranked is that it's actually an illusion to begin with. :( Though it also gives me hope that now that there's more players they may issue a fix they were intending all along once the player base is higher.

Interesting thought... :) I had been thinking the lack of a 'perfect' set of criteria in matching due to a limited amount of players simply led to errors in the agenda outcome (i.e. I've played games where it 'felt' like it wanted me to win, but the opposing team had a lucky push), but maybe you're right, maybe they're trying to balance the rank population. I feel like it's more than that though. In any case, I'm sure there is an agenda, whatever it is.

 

[jsilva]

Sauce and sauce of sauce

The first is a translated summary of the second.

As for how matchmaking in general works, here is an overview of how Microsoft's TrueSkill algorithm works. Warning: it is very long and Math heavy. Still, it's an interesting read if you really want to get an idea of how matchmaking works.

Here's a layman's summary. The game calculates your skill based on certain variables (k/d, w/l, shot accuracy, how many times you defuse the bomb vs let someone else do it, etc). It then looks at whether you win or lose consistently, or whether you bounce between wins and losses to establish a standard deviation (how uncertain the game is that you will win regardless of your skill level). It then assigns you both a skill variable (mu) and uncertainty variable (sigma).

The game then calculates a "skill chain" (beta). It takes every member in the lobby and arranges them from best to worst based on their mu. It then calculates how many points it must add to your mu to give you an 80% probability of beating the least skilled player (who has the lowest mu). Low beta values means a more evenly matched game based on player ability because the game has to add less points to gain the 80% probability. A high beta means luck is a bigger factor because the game had to add more points to reach 80%.

The game also calculates "partial play:" the amount of time a player was active during a match. This gives the game an idea of whether someone played for the entire match or just stood there.

It then takes all of this information, plugs them into complex formulas, performs several marginal calculations to get other values to plug into these formulas, and eventually assigns a team performance value to the players grouped together. The game then subtracts these two values to determine the probability of one team beating the other. With that established, it then determines how much a player's rank should increase based on whether the win was expected, slightly unexpected, or very unexpected (an upset win). This is the "K factor" from chess: it determines how much a win or lose impacts a player's rank. An upset will result in a higher K factor, thus a greater change in your rank.

Side Note: I suspect this is why S rank is so punishing. The algorithm may consider any loss by S rank to be unexpected or an upset, thus a heavier loss in points.

Anyway, this K factor also updates a player's mu. An upset win will cause a higher jump or drop in your mu value. These changes become less dramatic them ore a person plays. Someone just starting out who destroys the competition will have an almost vertical spike in their mu. Assuming Splatoon uses an algorithm like TrueSkill, this can explain why so many experienced players who start alts get grouped with competitive players. Early in the game, this can cause dramatic shifts in the caliber of players you face. Many talked about starting new accounts and only getting to play a few games before getting thrown in with very competitive players. This is probably because the kill or coverage variables in your mu cause it to jump to a specific range where the game considers you S tier in those areas even though you just started playing. Ranked limits the speed you can make this jump with the points system to prevent people from jumping straight to S rank from the get-go, but TW probably doesn't have this "safeguard."

What does all of this mean for Splatoon? In looking at the cited source, it means that your mu is weighted such that kills, deaths, and coverage points result in more drastic changes than other factors (like wins and losses). The game probably flags certain value ranges as "high kill," "high death," and "high coverage." If you're borderline, you may bounce between those value ranges and randomly get matched up with two very different types of players.

This is all hypothetical of course I don't know if Splatoon uses a derivative of the TrueSkill algorithm or uses the more primitive "dueling heuristic" Elon method where it considers two teams as one player vs another and compares their bell curves. With that said, this does fit with my own observations. When I get a good kill streak going, I find myself playing more aggressive teams. When I play passively, I find there are more players who are also passive. Ymmv though.

Thanks for the details about how the TrueSkill algorithm basically works. It's interesting at the very least.

It makes me wonder what the experience is with it. Do people observe they are repeatedly matched on the clearly weaker team (without a bad attitude about it, i.e. blaming everyone else), suddenly get on long losing streaks and drop multiple ranks (or the equivalent) when they're playing as well as other times? It seems to me this could only be an anomaly with an unbiased matching algorithm.

My experience with Splatoon makes those long losing streaks (or other things, such as being on the stronger team, then the weaker team, back and forth over and over, etc.) more common than could be reasonably accounted for as an anomaly. Perhaps they could be an anomaly in the sense that it can't figure out a player's style/skill/etc. adequately which results in a series of errors that mess up matching.

 

[BlackZero]

I'd have to disagree with this. Despite that it doesn't follow my experience in going from C- to S on three accounts (long story...), starting out in turf wars on the two new accounts suggested to me that they have more sophisticated metrics for determining 'skill'. I'm not certain what they are but I'm convinced it goes beyond kills, deaths, and turf inked.

I'm sure specific match rules have separate metrics (how many times you've carried the Rain Maker, how long you ride the tower, etc), but K/D/C are universal factors in any game mode. It makes sense that they would have a greater impact. There's no point in matching people based on how far they carry the Rain Maker if they are playing Splat Zones. I suspect RM and TC have unique variables that also factor into to matchmaking in those specific modes, but the bulk of a player's skill data would have to be built on stats that are a major factor in all game modes for it to be useful in matchmaking. Those three are the only ones I can think of, though there are probably other "universal" stats the algorithm considers. With that said, I think those three are the most weighted because they are important throughout the entire game.

My experience with Splatoon makes those long losing streaks more common than could be reasonably accounted for as an anomaly. Perhaps they could be an anomaly in the sense that it can't figure out a player's style/skill/etc. adequately which results in a series of errors that mess up matching.

I don't think it's an error. I think the matchmaking algorithm was designed for a more conventional shooter where K/D, shot accuracy, etc. are reliable metrics for player skill and the goal is more often simply to eliminate the other team, or killing the other team is useful for completing that goal. Splatoon is very different from those games in that killing the other team isn't always helpful, thus isn't necessarily a goal in and of itself. The algorithm is rough around the edges because it's expected to match players in a way that it probably wasn't intended to. Computers are machines. Like machines, each part of a video game is designed with a specific function and method of performing that function in mind. When you jury rig something, it's going to be a bit wonky compared to designing something from the ground up. I think the matchmaking was jury-rigged from a more typical shooter to work in Splatoon.

 

[Award]

Sauce and sauce of sauce

The first is a translated summary of the second.

This is all hypothetical of course I don't know if Splatoon uses a derivative of the TrueSkill algorithm or uses the more primitive "dueling heuristic" Elon method where it considers two teams as one player vs another and compares their bell curves. With that said, this does fit with my own observations. When I get a good kill streak going, I find myself playing more aggressive teams. When I play passively, I find there are more players who are also passive. Ymmv though.

Great, informative post. It's very interesting. However, I think we could disregard complex matchmaking like TrueSkill. I'm sure it does use it to a degree, and some of that probably is implemented, but I think that's more on the lines of what a lot of us are saying the SHOULD be using.

The key thing to remember is Nintendo is one of the worst offenders of WDIIH Syndrome (We Didn't Invent It Here.) They do not generally use outside processes, software, algorithms, etc. They invent their own in house. Sometimes they get superior results. Sometimes not. (Microsoft is also one of the worst offenders of WDIIH, both topped only by Apple.)

I also like the quote at the end "Of course it's not a perfect system, so once in a while players might notice and think "I've been ranked here?", but hopefully players will come to the realisation that they've been assuredly put into the same category as like-minded players". All that needs is a "Please Understand." following it. :D:(:(

Also interesting is from their quote, that's how they're determining who you're matched AGAINST, but doesn't determine who you're matched with. But they're also talking specifically about Regular Battles and said that Ranked has a different (unspecified) system. Which is interesting because prior to 2.4 they said Regular Battles was entirely random. I think it's safe to say they are not using that same rule in ranked. I've been paired often AGAINST teams that are hyper aggressive and generally dominant. That could fairly be stated to match my style (weapon pending.) But I'm only 1/4 of the team, and the other 3/4 seem very passive. Why would 3 passive players and one aggressive player be placed against 3-4 aggressive players? If they're actually applying sophisticated matchmaking that still implies they're either intentionally unbalancing it to throw the match, or the player population is sufficiently small, all the matchmaking in the world can't save the fact that your simply matched against whoever happens to be around. But then win/lose should randomly happen in equal measure. And too many of us have seen very lopsided losing streaks. And too many times even if we re-enter several lobbies, we still see the losing streaks as though it's stacking us, very intentionally to lose. That doesn't mean they're NOT using Truskill type matching, only that they're using it to achieve something other than balanced teams (or there's just a giant bug in the formula.)

Personally I believe they have their own in house formula, and also, they're lying through their teeth to maintain the secret formula. If it were broken, they wouldn't likely openly acknowledge it. If it's not broken and doing what it's doing by design, a.k.a. rigging matches to achieve a desired outcome, the could never admit so.

I can appreciate them needing to balance the population across ranks by making it difficult to gain rank, but that still doesn't explain the badly matched teams within a given rank that consistently sets players up to lose, or to win. The same bad matchmaking happens in TW, but then those great players that dominated get rotated out in TW.

I do hope they're oversimplifying what they do use with the 3 variables though. Splatoon's weapon distribution would make that disastrous if it's not taking more into account.

Thanks for the details about how the TrueSkill algorithm basically works. It's interesting at the very least.

It makes me wonder what the experience is with it. Do people observe they are repeatedly matched on the clearly weaker team (without a bad attitude about it, i.e. blaming everyone else), suddenly get on long losing streaks and drop multiple ranks (or the equivalent) when they're playing as well as other times? It seems to me this could only be an anomaly with an unbiased matching algorithm.

My experience with Splatoon makes those long losing streaks (or other things, such as being on the stronger team, then the weaker team, back and forth over and over, etc.) more common than could be reasonably accounted for as an anomaly. Perhaps they could be an anomaly in the sense that it can't figure out a player's style/skill/etc. adequately which results in a series of errors that mess up matching.

I see only a few possibilities:

1. Too few players means matches are essentially random.
2. A flaw in the system where certain players with certain playstyles can't be determined accurately and thus they get thrown into chaotic positions near constantly (a.k.a. a bug to fix.)
3. A flaw in the system where the calculation or matchmaking actually values incorrectly, not due to a playstyle being hard to determine but by simply generating wrong values (another bug to fix.)
   
4. Using ranks to load balance population (a.k.a stacking the deck in favor of the house.) Meaning ranks aren't earned but distributed via pretend games, like pretty much every casino game ever (The Konami Maneuver) designed to feed you just enough wins to feel like you're getting somewhere to keep you compulsively playing a game that's already determined how to make you loose a predetermined portion of games. This is the core of Game Theory (the other kind of gaming) upon which every slot machine, pachinko machine, card deck, roulette wheel, and pair of dice is built. You don't play the gambling table with an ability to win. You have carefully been designed to win only as much as is needed to play the required rounds for you to lose the predetermined amount. Luck plays a role and you can defy the odds, but even the amount of times that is allowed to happen have been statistically determined. This was hilariously revealed a few years ago when a casino, I forget where, had a video slot machine that awarded a jackpot. The casino refused to pay out citing that it was clearly software glitch in the machine as that was not supposed to be able to happen. No reason to believe Nintendo isn't doing that here to hide the effects of the not so secret low player base of Wii U in general. If it feels the result has been predetermined, odds are, that's because it has been.
   
5. "Teaching" attempts where it's designed to force players to need to push the objective and force them to lead the team or create opportunities for the team. This would not be so much benevolence in wanting to train you, but more a matter of needing to groom the player base to be fulfilling all needed roles.
   
6. A system that just doesn't work as intended, or wasn't fully thought through that needs an overhaul.

IMO, I've experienced the same, the losing streaks followed by win streaks followed by lose streaks with teams being clearly outmatched in either direction. It does seem, in ranked, too predetermined, as though they'll let you advance if you're good, but it's designed to hold people in their given rank for as long as it needs them, letting you narrowly advance if you grind through enough rounds, but is all too willing to force you to drop with little you can do about it by making you play far beyond the same skill level that it was before, or after. Every time I complained that I had awful matchmaking, I did actually end up with more points when I ended than when I started. But it took far too many rounds and far too many undeserved losses and having to play too many more rounds to regain those points, just to gain too few points when it would have made a lot more sense, in a fair game, to simply promote me via properly matched games to be closer to other players of similar skill a rank or two above. It makes no sense that players in the same rank bracket can be so mismatched that a spawncamp can even happen in ranked. Similarly matched players could not successfully hold a spawncamp against the other. Players with that wide a difference shouldn't be in the same rank bracket at all for matchmaking to be able to mess it up.

 

[Award]

I'm sure specific match rules have separate metrics (how many times you've carried the Rain Maker, how long you ride the tower, etc), but K/D/C are universal factors in any game mode. It makes sense that they would have a greater impact. There's no point in matching people based on how far they carry the Rain Maker if they are playing Splat Zones. I suspect RM and TC have unique variables that also factor into to matchmaking in those specific modes, but the bulk of a player's skill data would have to be built on stats that are a major factor in all game modes for it to be useful in matchmaking. Those three are the only ones I can think of, though there are probably other "universal" stats the algorithm considers. With that said, I think those three are the most weighted because they are important throughout the entire game.



I don't think it's an error. I think the matchmaking algorithm was designed for a more conventional shooter where K/D, shot accuracy, etc. are reliable metrics for player skill and the goal is more often simply to eliminate the other team, or killing the other team is useful for completing that goal. Splatoon is very different from those games in that killing the other team isn't always helpful, thus isn't necessarily a goal in and of itself. The algorithm is rough around the edges because it's expected to match players in a way that it probably wasn't intended to. Computers are machines. Like machines, each part of a video game is designed with a specific function and method of performing that function in mind. When you jury rig something, it's going to be a bit wonky compared to designing something from the ground up. I think the matchmaking was jury-rigged from a more typical shooter to work in Splatoon.

Turf inked isn't really a valid metric for TC or SZ though. Maybe RM since it requires full map traversal more like TW. But TC and SZ, the required inked paths are so situational depending on what's going on, that many maps need ONLY the most direct path to the objective inked and little else. Other times the map ends up as inked as TW due to attempts at flaking etc. And in TC most of the inking is going to happen accidentally when shooting from/at the tower. Applying weight to inking in TC effectively rewards my missing players that did nothing on or against the tower and just played TW.

Your second paragraph is probably the heart of it though. It's a hastily jury rigged solution borrowed from something else (probably Devil's Third. That might explain why they published that thing at all....) that doesn't fit well but they needed to get the game out the door. And thus the original concept applies: It's just really broken. However I can't totally ignore the experiences that jsilva and I have experienced that it doesn't seem like random brokenness as much as planned wins/losses very, very often.

 

[BlackZero]

However, I think we could disregard complex matchmaking like TrueSkill. I'm sure it does use it to a degree, and some of that probably is implemented, but I think that's more on the lines of what a lot of us are saying the SHOULD be using.

The alternative is a modified use of the Elo method, where the game records data on kills, deaths, and coverage and builds a 3D bell curve. It would then match that bell curve to those of other players to create a lobby, then divide the players into teams randomly. That, or it averages the variables of all players to create a single bell curve for the team and treat both teams as "single players." Both are very simple ways to do it, but horrendously flawed and not well suited to team matches.

This article says that the game matches based on playstyles in Turf, so it certainly tracks player habits in TW.

Nintendo Dream starts off by asking how matchmaking works. Amano mentions that rather than focusing on levels, the game pairs players with similar playstyles in Turf War. Levels are more of “an indicator for how long the person has played the game.”

Since Splatoon matches users together based on how they play, Amano notes that those who tend to splat enemies are likely to play together. Likewise, those who concentrate on painting rather than splatting are more likely to get paired up.

That's probably why you have to level to 10 before you enter ranked: it needs a baseline. In the Elo method, these are called "provisional matches," and are used to set a starting player skill value. Basically, the algorithm needs data to build a bell curve that shows your play style so it can compare it to those of other players. There's no point in using data from only a few matches because that isn't enough to create a valid base of comparison.

Edit: hyperlink fail. Added the quote to save time.

 

[Award]

The alternative is a modified use of the Elo method, where the game records data on kills, deaths, and coverage and builds a 3D bell curve. It would then match that bell curve to those of other players to create a lobby, then divide the players into teams randomly. That, or it averages the variables of all players to create a single bell curve for the team and treat both teams as "single players." Both are very simple ways to do it, but horrendously flawed and not well suited to team matches.

This article says that the game matches based on playstyles in Turf, so it certainly tracks player habits in TW. That's probably why you have to level to 10 before you enter ranked: it needs a baseline. In the Elo method, these are called "provisional matches," and are used to set a starting player skill value.

That makes sense, and, given the way Nintendo works, just going over their history, interviews, Iwata Asks session (I really miss those :( ) that would be more consistent with Nintendo. They like to reinvent the wheel but they generally don't see the need for overly complex systems when a simple one will do. The TrueSkill system is astoundingly complex, and given Microsoft's nature was not purely about gaming, but more about AI research in general, with a practical application in gaming. Nintendo generally goes for "functional but cheap" and then tweaks it. They also see themselves as "toy makers" (Miyamoto specifically referred to himself that way a number of years ago) and have said regarding Smash that they don't really want to participate in competitive play so much as simply fun play. Given all that, I don't think they'd have put an excessive amount of effort into a matchmaking system (unless they just ripped Devil's Third's because it was there), when a simple Elo curve method would do. Broken as it is.

Though I STILL won't ignore the fact that sometimes it seems less random, more rigged. There's something to that....

AFAIK, pre-2.4 TW matched playstyle/habit. Post-2.4 it now matches skill as well. I can't help but wonder if that's groundwork/experimentation for broader improvements in ranked.

 

[BlackZero]

AFAIK, pre-2.4 TW matched playstyle/habit. Post-2.4 it now matches skill as well.

From a matchmaking perspective, they are the same thing. The computer can't assess things like experience, "gamer senses," or "advanced tactics," and you wouldn't want it to anyway. That would be crippling for the player's own creativity if the computer added/deducted points from your skill level for using certain strategies. It can only assess skill based on measurable parameters the developers define. If you focus on kills, the game doesn't know or care how you got those kills, it simply records the kill count and adds that to your player "skill" data.

 

[Zombie Aladdin]

I don't know if I'm anomalous or not, but over time, my rank (including the number within the rank) fluctuates wildly. I'd go up 70 to 90 points or so (and maybe promoted while I'm at it), then I'd go down about 60 to 80 (and sometimes a demotion). This is sometimes during the same session. Because of that, Ranked play to me is like selling stocks: I try to predict when I peak, then drop out when that happens.

From what I get from other people, their rank, including number, slowly but steadily climbs up and eventually flattens out somewhere. That is, their is something like a jagged mountainside. Mine is a rather unsteady sine wave with a gradual rise. It's part of why I feel like I am strongly pushed forward or held back by teammates, even though I feel like I'm contributing a lot by constantly and aggressively pursuing the objective.

Does anyone think ranked mode is too punishing? Like suppose you gain ten points by winning a match and lose ten points when you lose the next match, you basically lost a match worth of progress. Shouldn't it be that when you lose a match you only lose half the points that you would have got if you won the match? That way you still have a chance to gain back points plus a little more even if you won a match only to lose the next. What do you guys think?

Since it's supposed to be a reflection of how often you win or lose, I think it works as it is. In particular, the C-ranks have a higher point gain than loss. Psychologically speaking, this compels people to keep trying because it pretty much guarantees promotion out of C- and C, at least as long as you maintain a 50/50 win rate or more. (It's also why I'd keep finding incompetent people with B- or B ranks, probably--someone who knew what they were doing likely carried them up there.)

I'm not even going to touch how unfair the system is when it comes to disconnections. They only account for DCs before the match. If a teammate disconnects after the start, you still lose the same amount of points and that's hardly fair when you're outnumbered.

Oh yes, definitely. However, I think the points lost should be proportional to how long they were in the match before they get disconnected. If they get disconnected right near the end, then they'd pretty much count as a fully active team player, whereas if they get disconnected about halfway through, you'd maybe lose half the points you normally would. That is, I believe the game should keep track of the amount of time elapsed when a player disconnects, then how log it takes for the match to end, and figure out a ratio for that. That ratio would then become the multiplier for points lost when you lose.

A ranked format that favors going up the ranks would mimic something like Mario Kart's VR ranking system, where consistent effort (and not so much winning) could net a higher rank. What's different between Mario Kart and Splatoon however, is the fact that Splatoon is heavily team based, which means that your team's performance heavily matters, and therefore requires every player to be around the same skill level (ignoring the fact that matchmaking can be pretty lousy at times). If you had a ranking system that favored effort over skill, then you would just be mixing more unskilled players in higher ranks and that no doubt, will give everyone a better reason to complain that they're getting "useless teammates" all of the time.

I want to point out that Mario Kart Wii's Battle Mode is team-based and also runs on a strictly zero-sum system where you gain about as many points for doing better than normal as you'd lose for doing worse than normal. A major difference, though, is that you gain or lose BR based on how well you do in proportion to all other players, and not together as a team, which became a problem as people quickly decided they'd play for themselves and themselves alone.

 

[Award]

From a matchmaking perspective, they are the same thing. The computer can't assess things like experience, "gamer senses," or "advanced tactics," and you wouldn't want it to anyway. That would be crippling for the player's own creativity if the computer added/deducted points from your skill level for using certain strategies. It can only assess skill based on measurable parameters the developers define. If you focus on kills, the game doesn't know or care how you got those kills, it simply records the kill count and adds that to your player "skill" data.

In this case I don't think they mean it to be the same thing. I don't think they meant that they used to match style of tactics versus general calculated skill. I think they mean it was essentially random (a lv1 noob that shoots the walls and a lv50 S+ rank were not treated differently) which is how they discussed TW in the past, however the detail they added was that they did matchmake playstyle. I don't think they meant flaners/assaults/support-defense. I think they just only distinguished those that play TW as a paintathon versus those that play it similar to ranked. Essentially random, but isolating the people that like to paint and not kill versus more intense players, but not subdividing from there. 2.4 introduces ranked style matchmaking based on the calculated skill, however it may be calculated.

I do wonder how they treat players that have never played ranked, though. I've told bad players that haven't yet touched ranked not to, out of fear the nice easy lobbies they're used to in TW will become harder and then they'll lose interest in the game.

 

[jsilva]

I'm sure specific match rules have separate metrics (how many times you've carried the Rain Maker, how long you ride the tower, etc), but K/D/C are universal factors in any game mode. It makes sense that they would have a greater impact. There's no point in matching people based on how far they carry the Rain Maker if they are playing Splat Zones. I suspect RM and TC have unique variables that also factor into to matchmaking in those specific modes, but the bulk of a player's skill data would have to be built on stats that are a major factor in all game modes for it to be useful in matchmaking. Those three are the only ones I can think of, though there are probably other "universal" stats the algorithm considers. With that said, I think those three are the most weighted because they are important throughout the entire game.

I don't think it's an error. I think the matchmaking algorithm was designed for a more conventional shooter where K/D, shot accuracy, etc. are reliable metrics for player skill and the goal is more often simply to eliminate the other team, or killing the other team is useful for completing that goal. Splatoon is very different from those games in that killing the other team isn't always helpful, thus isn't necessarily a goal in and of itself. The algorithm is rough around the edges because it's expected to match players in a way that it probably wasn't intended to. Computers are machines. Like machines, each part of a video game is designed with a specific function and method of performing that function in mind. When you jury rig something, it's going to be a bit wonky compared to designing something from the ground up. I think the matchmaking was jury-rigged from a more typical shooter to work in Splatoon.

I still disagree with you, but let me ask you how your viewpoint would account for the scenario I experienced tonight:

After using the Luna Blaster Neo exclusively for few weeks (I wanted to get good at it using my second account) I decided to go back to the Tentatek. The Octoshot was my main previously.

Maybe because of new skills I've learned, once I got used to the Tentatek again I was playing better than I ever have. And I won several games to start out with. And after getting back up to S after a ridiculous losing streak before Christmas I was at about S 54.

And then the losing streak started, again. Pretty much consecutive losses down to S 1. And then the flip flopping—win, loss, win, loss, S 1, S 6, etc.

So then I decided to do a test. I decided not to play the objective and just ink around. I'd splat people if I came across them or splat players off the tower if I happened to be nearby. Did it make any difference in the game outcome? Barely. My team still won the first game at S 1, lost the game at S 6, then just barely lost the next game at S 1.

And then ... I proceeded to sink down to A+ 34 even though I started playing properly again. This is all while I'm playing better than I ever have, and I've been able to get three accounts up to S, two of them by around level 26, and as high as S 68 or so, so I don't think I'm a slouch of a player.

To me this sounds a lot like a matching system with an agenda beyond matching player skill, or at least a matching system that is severely broken. No?

Edit: Oh yes, and I forgot to mention that my second game as an A+ was 3 A+'s and an A on my team and three S's and an A+ on the other team, and I lost 12 points...

 

[Holidaze]

@jsilva I like to think that it doesn't take place immediately, cause if it is true, that the game decides your 'skill' level by k/d, turf inked, etc, then it requires a large number of games to be played, to find out what's good, and what's not so good.

So more data can be found out of, say, 200 average games, vs 30 very good/very bad games. And you won't be able to see immediate effects, since it is done in such a way.

 

[BlackZero]

I still disagree with you, but let me ask you how your viewpoint would account for the scenario I experienced tonight:

After using the Luna Blaster Neo exclusively for few weeks (I wanted to get good at it using my second account) I decided to go back to the Tentatek. The Octoshot was my main previously.

Maybe because of new skills I've learned, once I got used to the Tentatek again I was playing better than I ever have. And I won several games to start out with. And after getting back up to S after a ridiculous losing streak before Christmas I was at about S 54.

And then the losing streak started, again. Pretty much consecutive losses down to S 1. And then the flip flopping—win, loss, win, loss, S 1, S 6, etc.

So then I decided to do a test. I decided not to play the objective and just ink around. I'd splat people if I came across them or splat players off the tower if I happened to be nearby. Did it make any difference in the game outcome? Barely. My team still won the first game at S 1, lost the game at S 6, then just barely lost the next game at S 1.

And then ... I proceeded to sink down to A+ 34 even though I started playing properly again. This is all while I'm playing better than I ever have, and I've been able to get three accounts up to S, two of them by around level 26, and as high as S 68 or so, so I don't think I'm a slouch of a player.

To me this sounds a lot like a matching system with an agenda beyond matching player skill, or at least a matching system that is severely broken. No?

Edit: Oh yes, and I forgot to mention that my second game as an A+ was 3 A+'s and an A on my team and three S's and an A+ on the other team, and I lost 12 points...

As Holidaze said, 20-30 matches in one night aren't going to make a huge difference in your bell curve if you've played 500 or so. The longer you've played, the more "stable" your bell curve becomes and the more games you'll have to play to make a significant change. Idk the specific calculation, but I'd guess that you would need to change to a coverage-focused play style for a few weeks (or at least several days) if you've focused on kills in most of your Ranked matches. There is also plain old bad luck. I know that's not the answer anyone likes to hear, but it is also a factor even when dealing with computer algorithms.

With that said, I think you're right: the answer is probably much simpler. After some introspection, my new theory is that the game uses Elo matchmaking where it assembles a lobby based on kill, death, and coverage stats. Then it teams the players up based on the combination of averaged player values that gives both teams as close to a 50% chance of winning that it can. Since not everyone is equally skilled, you are going to get a range from best-worst in every lobby. If the worst are bad enough, it may put the best player on their team and leave the mid-tier players in the lobby on another team so that the numbers work out to 50/50. This doesn't account for individual ability, rather the average of all the members of a team. Ergo, these match-ups look fair to the computer based on the math, but are horribly imbalanced in practice because only one person on Team A can outplay all of Team B, but all of Team B can reliably outplay three people on Team A. It may seem like there is an agenda behind this if you're the best person in the lobby because you will likely get grouped with the worst players in the lobby in order to keep the 50/50 balance, though there is also the chance that keeping the best player on a team with the mid-tiers and moving the second best to the weaker team will balance things out. This is why some match-ups are really one-sided while others are more even.

I originally dismissed Elo matchmaking because it is designed for 1v1 matches and is terrible at team matches; what Nintendo dev in their right mind would use this as a matchmaking algorithm for a team shooter in 2015? Anyway, I guess the takeaway from all this is, always try to be middle of the road in every lobby you're in so you don't get stuck as the equalizer when the computer is putting together teams.

 

[jsilva]

As Holidaze said, 20-30 matches in one night aren't going to make a huge difference in your bell curve if you've played 500 or so. The longer you've played, the more "stable" your bell curve becomes and the more games you'll have to play to make a significant change. Idk the specific calculation, but I'd guess that you would need to change to a coverage-focused play style for a few weeks (or at least several days) if you've focused on kills in most of your Ranked matches. There is also plain old bad luck. I know that's not the answer anyone likes to hear, but it is also a factor even when dealing with computer algorithms.

With that said, I think you're right: the answer is probably much simpler. After some introspection, my new theory is that the game uses Elo matchmaking where it assembles a lobby based on kill, death, and coverage stats. Then it teams the players up based on the combination of averaged player values that gives both teams as close to a 50% chance of winning that it can. Since not everyone is equally skilled, you are going to get a range from best-worst in every lobby. If the worst are bad enough, it may put the best player on their team and leave the mid-tier players in the lobby on another team so that the numbers work out to 50/50. This doesn't account for individual ability, rather the average of all the members of a team. Ergo, these match-ups look fair to the computer based on the math, but are horribly imbalanced in practice because only one person on Team A can outplay all of Team B, but all of Team B can reliably outplay three people on Team A. It may seem like there is an agenda behind this if you're the best person in the lobby because you will likely get grouped with the worst players in the lobby in order to keep the 50/50 balance, though there is also the chance that keeping the best player on a team with the mid-tiers and moving the second best to the weaker team will balance things out. This is why some match-ups are really one-sided while others are more even.

I originally dismissed Elo matchmaking because it is designed for 1v1 matches and is terrible at team matches; what Nintendo dev in their right mind would use this as a matchmaking algorithm for a team shooter in 2015? Anyway, I guess the takeaway from all this is, always try to be middle of the road in every lobby you're in so you don't get stuck as the equalizer when the computer is putting together teams.

I follow your train of thought and I'd like to agree with you. However, I've had plenty of experiences which challenge it, for instance, the flip-flopping I talked about.

During these flip-flopping games, one game I'm matched with players who can pretty much hold their own against the four players on the other team, and in the next game I'm matched with players who are overall dominated by the other team. Repeatedly. I had played 5 or 6 of those games last night before I did my test, and if I had played that last S 1 game properly I probably would have won that game and the cycle probably would have continued. I can remember leaving and entering a different lobby once during one of these and it still continued.

Of course there are other experiences as well.

Additionally, part of the reason I created a new account was because my main account was suffering from these things and I was hoping (in vain) that a new account might escape it. I was up to level 50 and rank S in my original account and had been playing on that account since Splatoon's release and I avoided rank until I was past level 20 (and I pretty much always was at the top with ink turfed), so there was plenty of stats to round out any curve by the time I was up to S.

 

[Award]

I decided to go back to the Tentatek. The Octoshot was my main previously.

OT, but what made you choose octoshot before, and what made you choose tentatek now? It's the same gun, and it seems like such a privilege to have access to octoshot....but honestly ttk just looks more fun all around!

And then ... I proceeded to sink down to A+ 34 even though I started playing properly again. This is all while I'm playing better than I ever have, and I've been able to get three accounts up to S, two of them by around level 26, and as high as S 68 or so, so I don't think I'm a slouch of a player.

To me this sounds a lot like a matching system with an agenda beyond matching player skill, or at least a matching system that is severely broken. No?

Ultimately I think BlackZero's explanation finally nails perfectly what's going on (and answers why some players have the "oh, come on, just play better and learn to win!" mentality, as their general placement never places them as the equalizer player who is set up to lose time and time again. They can't see the problem because they don't experience it. They might get unbalanced teams but they're rarely the computer assigned "guaranteed loss" role.

However, I still would not discount load balancing the lobbies as a "secondary" factor. Wii U doesn't have a large install base. Splatoon's one of the few things that has massive attach rate, but compared to most shooters the player base is a problem that Nintendo had to get creative to resolve. Nintendo is very very good at making problems look like strengths and hiding real world issues behind gameplay systems. That's their defining philosophy behind creating lower powered hardware is that it forces creativity through exactly that need. It seems likely to me, load balancing a small population through manipulations in the matchmaking, adding drama and suspense in the process, is something they'd have high likelihood of doing. It would be a "Very Nintendo" solution.

@jsilva I like to think that it doesn't take place immediately, cause if it is true, that the game decides your 'skill' level by k/d, turf inked, etc, then it requires a large number of games to be played, to find out what's good, and what's not so good.

So more data can be found out of, say, 200 average games, vs 30 very good/very bad games. And you won't be able to see immediate effects, since it is done in such a way.

True, but if if it were a well thought out system, it wouldn't be a pure average. Recent matches would be weighted, possibly heavily in the average since it would need to take account of skill improvements in progression.

"Well thought out system" being the key word here. Thinking statistics out doesn't seem to be the strength of the Splatoon team. Let's just be glad they don't make SRPGs. I wouldn't want to touch a Fire Emblem game made by this group

I originally dismissed Elo matchmaking because it is designed for 1v1 matches and is terrible at team matches; what Nintendo dev in their right mind would use this as a matchmaking algorithm for a team shooter in 2015? Anyway, I guess the takeaway from all this is, always try to be middle of the road in every lobby you're in so you don't get stuck as the equalizer when the computer is putting together teams.

I think your whole description is spot on and finally gets to the bottom of what some of us have been experiencing for so long. As for what Nintendo dev would do that, it goes back to "Nintendo does't understand online", "Nintendo doesn't care about competitive play", and Nintendo ALWAYS goes for cheap (but usually still surpasses in quality versus those that spend spend spend.) We all know they were caught off guard by the costs and time of HD dev, Splatoon came late to the party and Wii U NEEDED a must have title. And it needed it fast. Netcode was a little rough around the edges and the teleporting characters problems tell us it still is. I think they got caught up on problems with performance in general and started taking shortcuts to get the game out before E3 since it's pretty much the only major title released in that 12 month period from E3 to E3 short of Smash. The so-so netcode and Elo systems are "functional" and fast to implement. Sure they've had plenty of time to tweak it, but I imagine as soon as Splatoon shipped, other than mandatory fixes and rebalances most of the team has probably been pulled into other projects to speed them along (Zelda U, SMTxFE,, SMM, XBCX, NX stuff.) So the "it works well enough for few to notice" Elo issues wouldn't be a priority to fix.

The unfortunate reality of that system is that the 50% of players in the middle will see no problems at all, 40% low end will know they're bad, and will ultimately drop in rank, and the 10% of us that are at the high end of our ranks will have a miserable time and get stuck in that rank for a long time.

The ripple effect is that the best players of a given rank are not promoted or are promoted slowly, while the middle of the road players are promoted quickly, while the bottom tier of a rank hangs onto the rank too long by occasionally getting carried and sometimes advanced if they get lucky. That continues the stratification that the best players of a given rank are generally going to be better than average players of the next rank or two up, and lose more. That might explain why @Holidaze found it easier to get out of the A's than the B's. He was one of the best B's and therefore assigned to constant failure. But in the A's he was only average, and therefore got to S quickly. A system can't get more broken than that. It's the opposite of how it should work. Wouldn't it be funny if I ended up at S by playing much worse in A than B? :p

 

[Award]

I follow your train of thought and I'd like to agree with you. However, I've had plenty of experiences which challenge it, for instance, the flip-flopping I talked about.

During these flip-flopping games, one game I'm matched with players who can pretty much hold their own against the four players on the other team, and in the next game I'm matched with players who are overall dominated by the other team. Repeatedly. I had played 5 or 6 of those games last night before I did my test, and if I had played that last S 1 game properly I probably would have won that game and the cycle probably would have continued. I can remember leaving and entering a different lobby once during one of these and it still continued.

Of course there are other experiences as well.

Additionally, part of the reason I created a new account was because my main account was suffering from these things and I was hoping (in vain) that a new account might escape it. I was up to level 50 and rank S in my original account and had been playing on that account since Splatoon's release and I avoided rank until I was past level 20 (and I pretty much always was at the top with ink turfed), so there was plenty of stats to round out any curve by the time I was up to S.

I do think BlackZero's spot on about the matchmaking itself and what's wrong with it. But that doesn't have to be the only issue. I'm pretty confident that the load balancing issue is a secondary "weighted" factor in the matchmaking. There's also random luck. You're not ONLY going to be the equalizer, as it assembles potential teams it can EITHER sort a low team with an equalizer OR a balanced team, depending on who's online. It sees them as the same thing (at least when it's not actively rigging you downward or more likely rigging the other team upward.)

I think the curve also gets worse because of point distribution. C- and C give more points than they take, so everyone's guaranteed to get out of C- if they grind and likely get out of C if they grind. C+ is the first rank you have to be a little more skilled and win. By B, B+ you start start having to win more often than not. This inherently funnels most low skills into B- by virtue of easily passing them along but also funnels skilled players downward by virtue of penalties. The "natural" distribution would make the upper ranks fairly barren, the upper mid extremely barren (top players would go up, lesser players would go down and few would remain) and the bottom ranks barren to all except newbies (which would be barren during low sales periods.)

If you take a disastrously broken matchmaking system, use it to arrange matches to yield rank points on an inherently asymmetrical ladder, and start with a small population to begin with, the ONLY solution is to artificially distribute that population. And why not use the oldest gaming (gambling) trick in the book to make an artificial determination system look natural: stack the deck and favor the house, making it look like a genuine game! Better yet the bad matchmaking system helps obscure that it's a rigged game, plus the fact that the truly exceptional can still beat the odds and make it look honest (unwitting ringers.)

The more I think about it if they were not artificially distributing ranks, given the current asymmetric ladder scoring and the somewhat low population base (now 4M which is great, but not all are actively playing I'm sure) some ranks, particularly the A's, or maybe B+ through A would find it almost impossible to find a lobby it would be so empty. They'd have to distribute. And unless they fix the point system on the ladder, even if they don't fix matchmaking, they really have no choice but to do so.

 

[BlackZero]

During these flip-flopping games, one game I'm matched with players who can pretty much hold their own against the four players on the other team, and in the next game I'm matched with players who are overall dominated by the other team. Repeatedly. I had played 5 or 6 of those games last night before I did my test, and if I had played that last S 1 game properly I probably would have won that game and the cycle probably would have continued.

First, I suspect matchmaking has two phases: lobby building and team building. In the first part, assuming the article I cited was telling the truth, consists of the game comparing things like kills to determine whether you should be put in a high-kill lobby or a high coverage lobby. Once it establishes that, it moves on to the actual Elo calculations, which are designed for one purpose: to calculate the probabilities of winning. I suspect this phase of matchmaking pays very little attention to K/Ds, etc. I was half-right when I said the Elo system uses this data to make teams. The Elo system itself doesn't use this information, the aforementioned lobby builder probably does.

This is precisely why an Elo based matchmaker is terrible for these games. It doesn't properly weigh a player's actual ability, it only uses an assigned player value as a means of calculating the probability of winning, and this problem is exacerbated when four different players are combined into a single figure that is supposed to represent how Team A stacks up against Team B. Elo doesn't know or care if you played the best match of your career. It simply plugs your player value into an equation to determine the probability of your team winning with a certain combination of players with different player values. At this point, skill doesn't enter into the equation. I was mistaken when I said it did.

So things like kills and coverage are used to put you into a certain pool of players with similar play styles based on your kill values or coverage values falling within a certain range. From there, it probably selects 8 people at random and runs some calculations on what team composition comes the closest to a 50/50 match up. In Turf Wars, this could be a mixed bag and you may end up with competitive players or you may end up with people who are absolutely terrible. In ranked, it's the same situation, but the player value dispersion is much less. So you will still be put into the high kill group, but the 8 players the game draws for the match will all be S+. There is still a lot of variation in player competence in S+ rank, so your lobby may have top tier super elites or it may draw the bottom of the S+ barrel. It's more likely that you'll be thrown in with some good S+ and bad S+. At that point, the Elo system takes over to decide how many good and bad players need to be on each team to make it a 50/50 toss up. If you're one of the best, you may get thrown in with lower caliber S+ players. If you're at the bottom, you may get matched up with a team of great players, or may get stuck on the team with one bad *** and three weaker players. Once again, the "safe zone" appears to be in the middle, where you are theoretically less likely to get stuck on a team in order to balance the equation.

So, in short: the matchmaking system is both random and skill based. Skills put you in a certain pool from which players are drawn to create a lobby. From there, it's random whether the game selects the best of the best, the worst S+ has to offer, or a mixture of everyone. At that point, Elo kicks in to balance the teams the best it can given the variation of player values.

 

[Award]

First, I suspect matchmaking has two phases: lobby building and team building. In the first part, assuming the article I cited was telling the truth, consists of the game comparing things like kills to determine whether you should be put in a high-kill lobby or a high coverage lobby. Once it establishes that, it moves on to the actual Elo calculations, which are designed for one purpose: to calculate the probabilities of winning. I suspect this phase of matchmaking pays very little attention to K/Ds, etc. I was half-right when I said the Elo system uses this data to make teams. The Elo system itself doesn't use this information, the aforementioned lobby builder probably does.

This is precisely why an Elo based matchmaker is terrible for these games. It doesn't properly weigh a player's actual ability, it only uses an assigned player value as a means of calculating the probability of winning, and this problem is exacerbated when four different players are combined into a single figure that is supposed to represent how Team A stacks up against Team B. Elo doesn't know or care if you played the best match of your career. It simply plugs your player value into an equation to determine the probability of your team winning with a certain combination of players with different player values. At this point, skill doesn't enter into the equation. I was mistaken when I said it did.

So things like kills and coverage are used to put you into a certain pool of players with similar play styles based on your kill values or coverage values falling within a certain range. From there, it probably selects 8 people at random and runs some calculations on what team composition comes the closest to a 50/50 match up. In Turf Wars, this could be a mixed bag and you may end up with competitive players or you may end up with people who are absolutely terrible. In ranked, it's the same situation, but the player value dispersion is much less. So you will still be put into the high kill group, but the 8 players the game draws for the match will all be S+. There is still a lot of variation in player competence in S+ rank, so your lobby may have top tier super elites or it may draw the bottom of the S+ barrel. It's more likely that you'll be thrown in with some good S+ and bad S+. At that point, the Elo system takes over to decide how many good and bad players need to be on each team to make it a 50/50 toss up. If you're one of the best, you may get thrown in with lower caliber S+ players. If you're at the bottom, you may get matched up with a team of great players, or may get stuck on the team with one bad *** and three weaker players. Once again, the "safe zone" appears to be in the middle, where you are theoretically less likely to get stuck on a team in order to balance the equation.

So, in short: the matchmaking system is both random and skill based. Skills put you in a certain pool from which players are drawn to create a lobby. From there, it's random whether the game selects the best of the best, the worst S+ has to offer, or a mixture of everyone. At that point, Elo kicks in to balance the teams the best it can given the variation of player values.

You've done an amazing job analyzing all this and pegging how it's working. I have no idea how, but you really did it!

It also paints a pretty gloomy picture. If you take a playstyle pool, through it through a disastrous equation to make the team split "balance" no matter what, that's bad enough. But then when you, on top of that, have a bad ladder points distribution system that widens the ranked pool to include players of very mismatched skill, the result is amplified more. And that ignores the high liklihood of an artificial distribution system to rig matches in favor of moving a percentile of players between ranks to better balance it which takes what's already broken and breaks it more for good measure.

Do you think it pools players differently that haven't played ranked at all? I know it's aware of it because it doesn't show ranks on miiverse posting inklings in the lobby. But do you think it pools differently a player that has not played any ranked battles and hasn't "unlocked" anything to do with ranked (status lines on character screens etc.) yet? I'm trying to figure out if it's "safe" to tell a bad player to try out ranked or not for fear of screwing up their "easy lobbies" in TW.

 

[BlackZero]

You've done an amazing job analyzing all this and pegging how it's working. I have no idea how, but you really did it!

WE did an amazing job. It was a group effort. Still, this is only a theory: there's no guarantee this is how it actually works. It's entirely possible Nintendo hired people make sure players get tortured with with bad match ups.

It also paints a pretty gloomy picture. If you take a playstyle pool, through it through a disastrous equation to make the team split "balance" no matter what, that's bad enough. But then when you, on top of that, have a bad ladder points distribution system that widens the ranked pool to include players of very mismatched skill, the result is amplified more. And that ignores the high liklihood of an artificial distribution system to rig matches in favor of moving a percentile of players between ranks to better balance it which takes what's already broken and breaks it more for good measure.

I believe the term is FUBAR.

Do you think it pools players differently that haven't played ranked at all? I know it's aware of it because it doesn't show ranks on miiverse posting inklings in the lobby. But do you think it pools differently a player that has not played any ranked battles and hasn't "unlocked" anything to do with ranked (status lines on character screens etc.) yet? I'm trying to figure out if it's "safe" to tell a bad player to try out ranked or not for fear of screwing up their "easy lobbies" in TW.

No, I don't think that makes a difference. I'm fairly convinced now that TW does a random sampling of whatever gameplay style you fall under.

 

[Award]

WE did an amazing job. It was a group effort. Still, this is only a theory: there's no guarantee this is how it actually works. It's entirely possible Nintendo hired people make sure players get tortured with with bad match ups.

That would make the first Splatoon related team where everyone participated! :p We still had our S and the scrubs though ;)

It's a very solid theory though. The system describes matches the real experience very well, including explaining why some people don't seem to see a problem with it !

I doubt Nintendo needed to hire someone to torture players with impossible odds. Miyamoto's been the worldwide expert on that for decades! :D

I believe the term is FUBAR.

LOL! I agree! :(

No, I don't think that makes a difference. I'm fairly convinced now that TW does a random sampling of whatever gameplay style you fall under.

That would be good. I'm thinking a player that sees more beginners and few lv40-50, who is a bad player that mostly inks but sometimes splats if someone turns up (and gets splatted often, though sometimes has positive k/d.) is paired with lower levels, then, not because it's keeping them with unskilled noobs, but because lower leveled players are more likely to focus entirely on inking?


Thinking about this in depth, it makes it a lot more complicated to figure out how to fix it. That Elo & lobby system would be core to the entire system, they're not going to change it. It must remain broken. But they have to find a way to mitigate the disastrous results it comes up with so you don't get penalized for their problems. The only way I can see them doing that is to just change the ladder scoring so that it's harder to get trapped in ranks. Despite that everyone wants team scoring, maybe it needs to award points based on the differential of high & low k factor rather than rank & team win? The only way to imrpove it short of not using elo at all, and assuming they can't apply k factor to the lobby sorter would be to prevent the ranks from being so divergent within a rank so that it CAN'T mess up by too much. And the only way to do that is fix rank sorting.

There is the other possibility though, which is applying k factor to the lobby building sequence. I mention that because of the the 2.4 patch notes for Turf Wars:

• In Regular Battle and Splatfest, players of similar skill will be matched more often.

IF the system works the way we think, that would imply they have introduced k into the lobby builder, at least in TW. Maybe they can apply that to ranked (or already have?) as well if so. Though if ranked worked correctly you wouldn't need to as rank would be a better indicator anyway.