are NCAA football rankings any good?

I missed all of this weekend’s NCAA football games, but I did not miss out on checking the AP and USA Today rankings.  There aren’t too many surprises there, since the rankings contain the historically strong teams.  I was only surprised to see Boise State in the top 5 on both lists, since Boise State is not from a major conference.  I am always curious about how good the rankings are, particularly the preseason rankings.  Rankings can always be improved, but it’s harder to forecast without any data from the games.

I looked at last year’s rankings, comparing the 2009 preseason rankings to the final rankings.  I looked at this in two ways.  First, I looked at the cumulative number of teams in the first X final rankings that were included in the top 25 teams in the preseason rankings.  Ideally, the cumulative number of teams is equal to X, as X increases from 1 to 25, which means that all X teams were somewhere in the top 25 preseason rankings.  This is pretty generous, since it does not take preseason over-ranking or under-ranking into account.  A more restrictive case looks at the cumulative number of teams in the first X final rankings that were included in the top X teams in the preseason rankings.  Ideally, the cumulative number of teams is equal to X, as X increases from 1 to 25.  In the first case, we get a so-called perfect solution of X if the teams in the top 25 are all included but the rankings are wrong, but in the second case, we get the perfect solution of X if the teams in the top 25 are right and we get the rankings right.

The following figure contains the two results for both the AP and USA Today polls, with perfect ranking shown as a reference.  It shows that the top 7 teams in the final polls were all in the top 25 in the corresponding preseason polls and that 13 of the top 14 teams in the final polls were in the top 25 in the corresponding preseason polls.  Not too surprising.  It also shows that 2 teams in the top 7 and 4 teams in the top 16 were ranked better or the same in the preseason ranking as compared to the final ranking.  That is, the top teams at the end of the season are often under-estimated at the beginning of the season.

Another way to look at this is to compare the teams that are in both the final and the preseason top 25.  The difference in these rankings captures the degree to which a team is over-ranked or under-ranked at the beginning of the season.  We see nearly all negative numbers in the top 15 final teams, which again indicates that most of the best teams were under-estimated early in the season (with the exception of Texas at #2 and Florida at #3).  We see nearly all positive numbers for the teams at the bottom of the top 25 list, which suggests that many of the top teams at the beginning of the season had disappointing seasons and ended up near the bottom of the top 25 rankings.

Of course, none of this so far has taken human elements into account.  The polls are created by people who vote.  I have noticed a lot of anchoring and biases when looking at rankings from week to week.  This mainly occurs with the teams in the preseason top #25, who the voters are loathe to drop out of the top 25.  There are conference biases, too.  But I’ll have to explore these topics another day.

Related posts:

• Let’s replace the BCS system…using OR
• The passing game in the NFL

Link:

• How Stuff Works podcast on college football rankings.  This was an enjoyable 33 min podcast about how the polls work, with a few tangents taken.  Warning: The podcast is made by Georgia alums who slam Tech at one point.
• NSF sponsored project on the science of NFL football

Which teams will get a BCS bowl bid?  Will any teams in the top 10 preseason rankings drop out of the top 25 by the end of the season?