I recently looked at how each of the recruiting services has performed in terms of NFL draft status as a measure of success. That analysis is here:
Now, I will look at each of them in turn in terms of how predictive they are of on-field success. As I have already found, recruiting and winning is generally non-linear, especially once you start including teams without much a blue-chip percentage.
In this first look, I used Rivals recruiting class ranking data from 2017 through 2019. I took the average ranking for the top 100 teams for those 3 years. Teams that averaged outside of the top 25 were dropped. 23 teams remained. The SEC was heavily represented by 9 teams. Here is how each of the conferences counted up:
Using a simple linear regression to determine how much of 2019 win % was predicted by average Rivals recruiting ranking, I found the correlation was fairly high at 63% (inverted because ‘higher’ rank numbers are actually worse). The graph below shows that lower (i.e. better) the ranking was, the higher the winning percentage in a pretty linear way.
Drilling down a little further to the team level and creating quadrants on average score allowed for some evaluation. In the graph below, the teams in quadrant 1 (upper left) had a) winning percentage above average in 2019 and b) recruiting classes above average as ranked by Rivals from 2017 through 2019. This is the good quadrant. Quadrant 4, directly below quadrant 1, is the bad one. These teams had good recruiting but bad winning %.
All of the teams in Q4 have negative distance scores. The percentage next to each team’s logo is how far away they were from the expected (theoretical) win % as predicted by Rivals class rankings. Florida’s 2019 win % was 12% over expected. FSU was the worst at -27.8%. For the stats people, the linear equation is:
Win perc 19 = -0.021 * Avg Rank + 0.961
So, to take UF for example. The Gators’ expected 2019 win percentage would be
-0.021 * 11.33+0.961
-0.234+0.961=0.726. Florida’s actual win percentage was 0.846, which is how we get 12% over. Below is a table of each team’s data.
Ultimately, using Rivals class rankings is not very predictive of success (margin of error is 31%). As a descriptive measure, however, it is interesting to look at. The overall regression model was significant (a<0.001) with a medium effect size (.406). So all in all, I would say Rivals did a pretty good job. I just wouldn’t fret too much about next year’s win percentage going by Rivals’ rankings alone.
All of the Other Services plus Aggregate Measure
Instead of creating new posts for each service, I decided to just put everything here. The main reason is that there isn’t that much of a difference between the services. But there are some interesting points so here is each service’s data and outcome with an aggregate summary.
When looking at each of the service’s rankings and how that relates to on-field production, they are all pretty equal. Analysis of variance indicated no statistically significant difference among any of the four groups (in terms of variance). As such, I wouldn’t infer that any single service is better than another (because the data doesn’t indicate that) but instead look at this analysis as like a report card for the 2019 season. It doesn’t tell us how good (or bad) a service is overall, just how they did this year. That being said, here is how things turned out:
The above table is the overall accuracy of each service. This statistic is the correlation between each service’s average 3-year class ranking and that team’s on-field win % in 2019. When combining all 4 services into one aggregate “service”, the accuracy is 74.3%, so still not as good as the composite here. Other insightful statistics such as effect size, standard error, and confidence intervals were all better for the composite (than the aggregate) by very slim and insignificant margins.
Relative to some other metrics, however, these did pretty well. SB Nation’s blue-chip ratio (https://www.bannersociety.com/pages/blue-chip-ratio-2019 was not very strong at predicting 2019 outcomes. The correlation for BCR was 44% with only 19.5% of that correlation attributable to BCR. BCR was a quasi-significant predictor of outcome (p = 0.08). To be fair to BCR, however, only a sample (the data that was on the above-linked post) was used so with larger n, it may do better
The final scatter was a bit more thorough in placing teams in their respective quadrant. Ultimately, the Seminoles were the furthest off of expectations overall.
Below is a table that outlines how each team did in the aggregate.
The above table is the most interesting to me. The right-hand column displays how many games above or below expected each team won according to the combined rankings for all 4 services. LSU, VT, UNC, Clemson, and Oregon all won 2 (when rounded) games more than expected. ND, The Florida Gators, Penn State, and Ohio State all won 1 more game than expected.
Texas, Nebraska, Texas A&M, Miami, and USC all lost 1 game more than would be expected while South Carolina and blue-ribbon loser FSU lost 2 more games than they theoretically should have.
Below, the data are broken down by each service provider for posterity: