NCAA Ditches RPI for New Ranking System

The NCAA issued a statement this morning that they will not longer use RPI as “the primary sorting tool for evaluating teams during the Division I men’s basketball season”. Instead, a new ratings system, the NCAA Evaluation Tool (NET) will be used to evaluate teams. According to the NCAA:

The NCAA Evaluation Tool, which will be known as the NET, relies on game results, strength of schedule, game location, scoring margin, net offensive and defensive efficiency, and the quality of wins and losses. To make sense of team performance data, late-season games (including from the NCAA tournament) were used as test sets to develop a ranking model leveraging machine learning techniques. The model, which used team performance data to predict the outcome of games in test sets, was optimized until it was as accurate as possible. The resulting model is the one that will be used as the NET going forward.

Those metrics seem reasonable enough, though it’s not clear whether they’re the best or even give a complete picture. Nonetheless, their overall methodology sounds good. We’ve often talked on AoG about how a key element of a good ratings system would be how well it can predict a winner, and I’m glad to see that they used that as part of their training of the NET. I’m a little worried about how many tournament games they used and how they were weighted. The nature of tournament games are so different that I’m not convinced that they may be skewing their results a bit.

The cynic in me has a lot of questions about what their tolerances were. There’s always a certain amount of ‘acceptable’ mistakes from an algorithim, and I’m curious about what they were and how they determined what mistakes were tolerable. Something tells me that they were far more concerned about how traditional powers fared when the algorithim made a mistake with Duke versus when it made a mistake with Vanderbilt.

Another part of the statement that caught my eye was this:

Of key importance, game date and order were omitted to give equal importance to both early and late-season games. In addition, a cap of 10 points was applied to the winning margin to prevent rankings from encouraging unsportsmanlike play, such as needlessly running up the score in a game where the outcome was certain.

At least the NCAA learned some things from the BCS system in football. I like the idea of weighing all the games evenly, and minimizing the human effect of giving too much (or too little) weight to a team that was over/underrated. I also like that there’s a limit to the margin-of-victory bonus, though I still see a lot of games where coaches are going to push for the maximum bonus.

Finally, I cannot find any implication that the actual machinations of the algorithim itself will be made public, which is bound to make coaches upset. Like the BCS showed us, hidden formulae and computer driven rankings will inevitably lead to arguments both legitimiate and illegitimate. RPI wasn’t perfect, but at least you knew EXACTLY how to maximize it as best you could.

We’re just going to have to wait and see how well the NET does. I’m sure there will be endless angry debates from the usual suspects about it, and it’ll dominate the entire season regardless of how good or bad it is. I doubt this is going to lessen any of the usual complaining we hear after selection Sunday.