Recruiting services evaluation: Rivals

I recently looked at how each of the recruiting services has performed in terms of NFL draft status as a measure of success. That analysis is here:

https://thefaircatch.com/2020/02/01/reviewing-the-recruiting-services-how-do-they-stack-up/

Now, I will look at each of them in turn in terms of how predictive they are of on-field success. As I have already found, recruiting and winning is generally non-linear, especially once you start including teams without much a blue-chip percentage.

In this first look, I used Rivals recruiting class ranking data from 2017 through 2019. I took the average ranking for the top 100 teams for those 3 years. Teams that averaged outside of the top 25 were dropped. 23 teams remained. The SEC was heavily represented by 9 teams. Here is how each of the conferences counted up:

Rivals performance by conf

Using a simple linear regression to determine how much of 2019 win % was predicted by average Rivals recruiting ranking, I found the correlation was fairly high at 63% (inverted because ‘higher’ rank numbers are actually worse). The graph below shows that lower (i.e. better) the ranking was, the higher the winning percentage in a pretty linear way.

Rivalsinitialscatter

Drilling down a little further to the team level and creating quadrants on average score allowed for some evaluation. In the graph below, the teams in quadrant 1 (upper left) had a) winning percentage above average in 2019 and b) recruiting classes above average as ranked by Rivals from 2017 through 2019. This is the good quadrant. Quadrant 4, directly below quadrant 1, is the bad one. These teams had good recruiting but bad winning %.

performance vs expected.rivals

All of the teams in Q4 have negative distance scores. The percentage next to each team’s logo is how far away they were from the expected (theoretical) win % as predicted by Rivals class rankings. Florida’s 2019 win % was 12% over expected. FSU was the worst at -27.8%. For the stats people, the linear equation is:

Win perc 19 = -0.021 * Avg Rank + 0.961

So, to take UF for example. The Gators’ expected 2019 win percentage would be

-0.021 * 11.33+0.961

-0.234+0.961=0.726. Florida’s actual win percentage was 0.846, which is how we get 12% over. Below is a table of each team’s data.

rivals.table

Ultimately, using Rivals class rankings is not very predictive of success (margin of error is 31%). As a descriptive measure, however, it is interesting to look at. The overall regression model was significant (a<0.001) with a medium effect size (.406). So all in all, I would say Rivals did a pretty good job. I just wouldn’t fret too much about next year’s win percentage going by Rivals’ rankings alone.

All of the Other Services plus Aggregate Measure

Instead of creating new posts for each service, I decided to just put everything here. The main reason is that there isn’t that much of a difference between the services. But there are some interesting points so here is each service’s data and outcome with an aggregate summary.

Aggregate Summary

When looking at each of the service’s rankings and how that relates to on-field production, they are all pretty equal. Analysis of variance indicated no statistically significant difference among any of the four groups (in terms of variance). As such, I wouldn’t infer that any single service is better than another (because the data doesn’t indicate that) but instead look at this analysis as like a report card for the 2019 season. It doesn’t tell us how good (or bad) a service is overall, just how they did this year. That being said, here is how things turned out:

service table

The above table is the overall accuracy of each service. This statistic is the correlation between each service’s average 3-year class ranking and that team’s on-field win % in 2019. When combining all 4 services into one aggregate “service”, the accuracy is 74.3%, so still not as good as the composite here. Other insightful statistics such as effect size, standard error, and confidence intervals were all better for the composite (than the aggregate) by very slim and insignificant margins.

Relative to some other metrics, however, these did pretty well.  SB Nation’s blue-chip ratio (https://www.bannersociety.com/pages/blue-chip-ratio-2019 was not very strong at predicting 2019 outcomes. The correlation for BCR was 44% with only 19.5% of that correlation attributable to BCR. BCR was a quasi-significant predictor of outcome (p = 0.08). To be fair to BCR, however, only a sample (the data that was on the above-linked post) was used so with larger n, it may do better

The final scatter was a bit more thorough in placing teams in their respective quadrant. Ultimately, the Seminoles were the furthest off of expectations overall.

all services expected vs win

Below is a table that outlines how each team did in the aggregate.

all_svc_table

The above table is the most interesting to me. The right-hand column displays how many games above or below expected each team won according to the combined rankings for all 4 services. LSU, VT, UNC, Clemson, and Oregon all won 2 (when rounded) games more than expected. ND, The Florida Gators, Penn State, and Ohio State all won 1 more game than expected.

Texas, Nebraska, Texas A&M, Miami, and USC all lost 1 game more than would be expected while South Carolina and blue-ribbon loser FSU lost 2 more games than they theoretically should have.

Below, the data are broken down by each service provider for posterity:

247

247.byConference

247 table

performance vs expected.247

ESPN

Espn.byconference.barchart

espn.table

performance vs expected.espn

Composite

comp_class by conference

composite table

performance vs expected.composite

 

 

One thought on “Recruiting services evaluation: Rivals

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s