RESEARCH: Evaluating the Starting Pitcher Report

"Which pitcher do I start?"

These are probably the most commonly asked words during the fantasy baseball season, whether wondered silently to oneself, asked to a friend, or posted on a message board. In the never ending chase for the elusive win, we all struggle with the decision of who to start and who to bench.

Fortunately, we've got an app for that. 

No, put your smart phone down—this app is the Starting Pitching Report, found here.

About the Starting Pitcher Report

The Starting Pitcher Report (which updates nightly for all games over the next seven days) assigns to each scheduled starting pitcher a rating from -5 to 5 based on that SP's average PQS score (using home/away splits). The report also lists the opposing team rating, which is the average opposing pitchers' PQS scores against that opponent (using both home/away splits and RHP/LHP splits). Thus for this measure, a higher rating means that opposing pitchers have had more success against this team. 

You can also use the report to search for the opposing pitcher's rating on the relevant date, as well as for the opposing's SP's opposing team rating, i.e., the team rating for your SP. This might serve as a reasonable proxy for the amount of run support your SP can expect to receive that game. Again, a higher value indicates opposing pitchers have done better against your SP's offense.

Data Set and Variables

The Starting Pitching Report was first introduced in mid-2010, so we now have access to two partial years of data: from June 28, 2010 through the end of the 2010 season; and from May 1, 2011 through the end of the 2011 season. While the data set covers only 8 months of baseball, we have 5918 individual observations which is a very healthy sample size.

We first evaluated the report's effectiveness at predicting the probability that any given SP would earn a win in a game that they start. From a theoretical perspective, it seems reasonable that the likelihood that a starting pitcher will record a win may depend on the following variables found in the Starting Pitcher Report:

  1. The skill of the SP (measured through "Rating")
  2. The quality of the opposing offense (measured through "Opp Team Rating")
  3. The skill of the opposing SP (found by looking for the opposing pitcher's Rating); and
  4. The quality of the SP's own offense vis-a-vis run support (found as the opposing pitcher's Opp Team Rating).

Analysis and Results

We evaluated the model by applying the probit model, which is a type of statistical regression frequently used to measure the likelihood of a binary event (such as a SP getting a win or not) occurring. The chart below shows the 95% confidence interval for each of the four independent variables listed above:


Not surprisingly, the SP's own PQS rating had the strongest relationship with the likelihood of getting a win; simply put, better pitchers win more games. Furthermore, this variable was statistically significant, so we can reject with confidence the hypothesis that pitcher wins are purely random.

The third variable—the skill of the opposing SP—was also statistically significant. Again, the model demonstrates that the intuitive result holds true: the better the quality of the opposing pitcher, the less likely our SP was to get the win. Notice though the standardized coefficient was only about half of that for the SP's own rating. The implication is that the quality of your SP is more important than the the quality of the opposing pitcher.

The take-away from this is that you should generally not bench a pitcher with a favorable start just because the opposing SP is also likely to throw a good game. Instead, your decision should mostly depend on the quality of your own SP, with the quality of the opposing SP serving more as a tiebreaker.

The other two variables—the quality of your SP's own offense and the quality of the opposing offense—were not statistically significant; what this means is that we can't reliably demonstrate that they influence the likelihood of a win. However, the results were directionally sensible—the stronger your SP's offense and the weaker the opposing offense, the more likely it is for your SP to win the game.

Although mathematically rigorous, the output of a probit egression is not all that intuitive. So, as an alternate way to demonstrate the predictive quality of the SP rating, we can also show how the results break out across different cohorts of Ratings:

Rating     Count   Wins   Win %  Avg PQS   DOM%    DIS%
=======    =====   ====   =====  =======   ====   ====
  >3         95     47     49%     3.7      68%    12%
 2 to 3     939    375     40%     3.5      60%    13%
 1 to 2    2362    896     38%     3.2      51%    17%
 0 to 1    1674    554     33%     2.9      42%    22%
-1 to 0     462    140     30%     2.6      39%    29%
-2 to -1    268     81     30%     2.5      29%    27%
  < -2      118     23     19%     2.1      28%    41%

Here, it's easier to see the strong relationship between a SP's Rating and the results, both in terms of the pure quality of their start as well as their likelihood of getting the win. 

Monthly Analysis

One of the limitations of using the Starting Pitcher Report very early in the season is that there just isn't enough data to generate useful predictions; you don't want to base a decision on the sample size of just one or two starts. It was for this reason that the report wasn't even available in the 2011 season until May 1. 

But once we cross this hurdle, does the report become more accurate as the season progresses?

When looking at the correlations between each SP's rating and their actual PQS result on a monthly basis, the report does not appear to get any more predictive as the season goes on:

Month    Correlation
=====    ===========
May          14%
June         26%
July         19%
August       15%
September    17%

An explanation for this may be that PQS, as a predictive measure, stabilizes quicker than you might otherwise expect. A few years back, we found that year-to-date PQS average was more accurate than the 5 game rolling average, but only marginally so. (This article isn't currently available online, but it can be found on page 65 of the 2011 Baseball Forecaster.) Given that many SP may have 10 or more starts by June, it is not surprising that the report's accuracy doesn't dramatically increase as the season progresses. 

Implications and Concluding Thoughts

The Starting Pitching Report is a powerful tool that can help you determine who to start and who to be bench. The higher the rating for your SP and the lower the rating of the opposing SP, the more likely it is that your SP will earn a win. 

But be careful to note that there are no guaranteed results—even the most favorable start will still only result in a win less than half the time. However, if you exercise patience and use the report to guide your decision making over the entire season, you can put yourself in a good position to pick up a few additional wins, which can sometimes be the difference between a championship finish and the bittersweet taste of second place. 

Click here to subscribe

  For more information about the terms used in this article, see our Glossary Primer.