THE CHALKBOARD
Myth Busters: Dear
Nate Silver: The NFL Is NOT Resistant To Analytics
Let's get this straight all you SABR-metricians: The NFL embraced objective,
data-intense, computer-based analytics before Moneyball author Michael
Lewis entered elementary school and still does today.
Yet, at the same time pitchers and catchers were reporting for spring training
this year, ESPN The Magazine published an article,
"The Search for Intelligent Life," in
which Nate Silver claimed the NFL "has failed for more than 20 years to
adopt some of the most basic tenets of analytics."
The truth is the NFL adopted the basic tenets of analytics decades before MLB
and the NBA even knew of the existence of a device known as a
"computer," as is made indisputably clear in Tex Maule's January 29,
1968 Sports Illustrated article on Dallas Cowboys general manager Tex
Schramm's pioneering adoption of analytics,
"Make No Mistakes About It."
In America's Game, Michael MacCambridge succinctly summarized how
Schramm toiled with analaytics.
“At the 1960 Olympics in Squaw Valley, Tex Schramm had been fascinated
with the information generated by an IBM computer, which calculated statistics,
medal standings, and record times," MacCambridge wrote in America's
Game. "Frustrated during his time with the Rams about how long it took
to wade through the avalanche of data on college seniors, Schramm was convinced
that computer technology could be used to organize the scouting data, and more
reliably isolate the most salient information….
"By 1964, Schramm was ready to implement a system that was a synthesis of
past and future. It would incorporate the old-school notions of bird-dogging
and beating the bushes with the Rams-inspired thoroughness that had become the
norm in the league, while adding a third element, a new mind-set that attempted
to quantify every opinion."
How far ahead of MLB was Schramm and the NFL in adopting analytics? Consider
that in 1964:
* Oakland A's GM Billy Beane was a 2-year-old toddler.
* Moneyball author Michael Lewis was a 4-year-old pre-schooler.
* Bill
James was a decade way from penning the first Baseball
Abstract.
* Steve Jobs and Steve Wozniak were a dozen years away
from inventing the Apple I computer.
Clearly, the NFL adopted the basic tenets of analytics decades before MLB did
so.
But how does the NFL compare to the NBA?
********
Houston Rockets general manager and Sloan Sports Conference founder Daryl Morey
was still eight years from birth when Schramm sprung analytics on the
NFL at the 1964 NFL Draft.
In his ESPN article, Silver lauds NBA teams for increasing their application of
an established play--the corner 3-point jump shot--in response to a rule change
that improved the play's risk/return ratio. Obviously, given it's distance
relative to other 3-point attempts, the corner jump shot is the shortest
distance to the basketball goal that returns 3 points, as opposed to 2
points.
But adopting the highest percentage shot at the goal is nothing more than the
hoops equivalent of what Bill Walsh started in Cincinnati in the 1970s and
perfected in San Francisco in the 1980s in response to new NFL rules that made
pass protection easier and made pass coverage more difficult.
"[B]y shortening—and timing—the passing game, Walsh reduced its
two biggest risks: interceptions and incompletions," Lewis wrote in his
book The Blind Side. "‘Our argument was that the chance of a
completion drops dramatically over twelve yards,’ said Walsh. ‘So, we
would throw a ten-yard pass. Our formula was that we should get at least half
of our passing yardage from the run after the catch.’”
While Walsh may not have utilized traditional mathematics per se to
reinvent NFL football, the first quarterback to run the new, new operating
system--the Bengals' Virgil Carter--deeply immersed himself in probabilistic
football analytics that were decades ahead of their time.
"Carter has set a literal new standard for the shopworn phrase 'student of
the game,' for he is a computer analyst whose diligent research into football
has led to findings that dispute some of the most sacrosanct coaching
theories," Ron Reid wrote in an October 2, 1972 Sports Illustrated
article, "Handy Pair of Brainy Bengals."
While working on his master's degree at Northwestern in 1970, Carter coded 53
variables in 8,373 plays from 56 games played during the first half of the 1969
season. It added up to over 440,000 bits of data that Carter then fed into the
computer.
"Among other things, the study showed that in some cases field position
was more important than ball possession, and that the laws of probability could
determine an offensive team's scoring chance from any spot on the field,"
Reid wrote. "Moreover, probability could dictate an expected value to the
offensive team's field position, which could be expressed in points."
NFL coaches eventually copied everything Walsh designed.
NFL coaches discarded Carter's probabilistic analytics.
Clearly, the question is ... Why?
Why did NFL coaches treat Walsh's play designs as super valuable
products worthy of infinite copying and discard Carter's analytics as so
much byproduct waste?
********
The answer rests in Silver's limited understanding of what he calls
"elective choices--options that opponents have little or no choice of
preventing."
The greatest play designers in football history became so by embedding
elective choices in their plays. For example, in his famous power sweep, Vince
Lombardi embedded elective choices known as ... wait for it ...
option blocking, which Lombardi developed at West
Point as an assistant to Red Blaik.
At the college level,
Bill Yeoman's veer and Darrell Royal's and Emory Bellard's
wishbone embedded three elective choices in a single play known as
... wait for it ... the triple option (dive, keep and pitch).
Likewise, Walsh embedded three elective choices in his short passing game
(primary, alternate and outlet receivers) and the most famous play he ever
designed, "The Catch," was called ... wait for it ...
"sprint right option."
The elective choices embedded in today's "spread designs," such as
the designs Chip Kelly brought with him to Philadelphia from Oregon, are
direct descendants of the embedded elective choices
in the designs of Lombardi, Yeoman, Walsh, et al.
Silver and others who allege the NFL has been resistant to analytics make no
distinction between an embedded or
endogenous elective choice--which can be imposed
upon the opponent on first, second, third or fouth down from inside the
play after the ball has been snapped--and far less powerful
exogenous elective choices that can only be made
from outside the play before the ball is snapped, e.g., a
fourth-down choice.
But the difference between an endogenous embedded elective choice that is
delayed until after the ball is snapped and an exogenous elective choice that
must be made before the ball is snapped is enormous because the longer the
elective choice is delayed the less time and opportunity the opponent has to
counter-adapt and prevent the manifestation of the consequences of the choice
before the whistle sounds and ends the play.
In other words, the longer a coach waits to impose an elective choice on an
opponent, the more powerful that choice will be.
Macro-economist Paul Romer essentially captured this principle in his seminal
paper on change, Endogenous Technological Change, when he found
"patience is always undervalued."
Paul Romer is of no relation to and should not be confused with macro-economist
David Romer,
who has written that NFL coaches should almost always eschew
patience when making elective choices on fourth down.
Like David Romer, Silver criticizes NFL coaches who refuse to eschew patience
on fourth down.
Silver characterizes NFL coaches as collectively "gutless" on fourth
down, which is an absurd characterization on its face, not to mention more than
a little condescending, rude, and mean-spirited.
Silver speculates without a shred of data that NFL coaches stubbornly
refuse to eschew patience on fourth down because the average NFL team has been
owned by someone other than the coach for 34 years. Apparently, coaches of NBA
teams that are owned by others for only 15 years are not saddled with this
debilitating lack of impatience although Silver did not provide any data to
support that conclusion either.
"This [NFL] is a culture that fosters extreme risk aversion," Silver
wrote. "Going for it on fourth down is risky twice over: in the micro
sense of staking more on the result of one play, and in the macro sense of
defying custom and tradition."
Are NFL coaches really that risk averse?
********
Let's first talk macro.
It is true on the micro level that going for it on fourth down is risky in that
it stakes more marginal risk on an individual play than on first, second, or
third down because the offense risks losing possession without losing the ball
on a turnover, but on the macro level it is Silver who is clinging to custom
and tradition when he insists on judging NFL coaches solely on their exogenous
fourth down elective choice-making.
Think about it, before they became legends, both Lombardi and Walsh defied
custom and tradition.
Did the NFL punish Lombardi and Walsh for doing so? To the contrary, at the end
for their respective Hall-of-Fame careers, the NFL named the Super Bowl trophy
after Lombardi and adopted the video of "The Catch" as the video
accompanying the
NFL's copyright disclaimer.
Did the NFL punish Chip Kelly when the Philadelphia Eagles signed him to a
$32.5 million contract before he ever designed a
play in the NFL?
As the title of his seminal paper suggests,
Paul
Romer brought change "inside" the field of macro-economics
by constructing mathematical representations in which change is the result of
the intentional actions of people, such as research and development, and the
new knowledge that results therefrom.
Like the designs of Lombardi, Walsh and Kelly, Paul Romer's model broke with
and sought to supplant well-established conventional thinking:
Nobel Prize winner Robert Solow's exogenous model of
change.
While new knowledge--the engine of change and more efficient production--lay
outside Solow's model, it was segregated in such a way that the contribution of
knowledge to the growth of productivity could be measured. But, there were
plenty of economists who figured that the riddle of change and growth
"hadn't been solved by Solow's model with its not-my-table exogenous
technology variable doing almost all of the work,"
David Warsh wrote in Knowledge and the Wealth of
Nations.
Enter Paul Romer.
According to Warsh, "[w]hen [Paul] Romer began thinking about what exactly
might constitute a new and valuable piece of knowledge worth spending money on,
he soon decided it must indeed be something like a new good, a successfully
differentiated product."
The Lombardi sweep? Yeoman's triple option? Walsh's short passing game? Kelly's
manic spread?
Each of these play designs was at one time "a new and valuable piece of
knowledge worth spending money on." And the returns to the teams that
stiff-armed custom and tradition and implemented the new knowledge on the field
were enormously positive.
********
In contrast, Silver's own analysis confirms an NFL coach's exogenous elective
choice on fourth down is of miniscule importance on the macro level.
"I estimate ... that the typical NFL team sacrificed about half a win over
the course of the 16-game regular season," Silver wrote. "That might
not sound like much. But 0.5 wins over a 16-game football season is equivalent
to five wins over a 162-game baseball season."
Here Silver makes a fatal macro mistake: He fails to account for the
indisputable fact that a win, like a piece of knowledge (or a turnover), is an
indivisible good.
As David Warsh pointed out in Knowledge and the Wealth of Nations, a
good is said to be indivisible if there is a minimum size at which it is
unavailable.
"You could have half a box of cereal, but not half a bridge over a
river," Warsh explained.
Obviously, the minimum size at which a win is available is one. Thus, if NFL
fourth down strategy is "costing" teams half of a win per year, it
really isn't costing them anything because the average fourth down error rate
is below the threshhold of divisibility.
That, according to Silver's accounting, half a loss in football is
"equivalent" to 5 full losses in baseball, a game in which the
most important decisions such as who will pitch (the right-hander or the
left-hander?) are exogenous elective choices, is irrelevant because the
so-called NFL loss never reaches the divisibility threshhold and thus probably
can be safely ignored at the macro level.
The reason Silver ignores the critical concept of indivisibility is probably
because he is unaware of it, but maybe it is because indivisibility tends to
wreak havoc on neat and tidy math because mathematically an indivisible good is
a nonconvexity.
One of the greatest economists in history, Kenneth Arrow, found this out in the
early 1960s. After Solow developed his exogneous model of growth in the late
1950s, Arrow tried to bring knowledge inside economics by using what we call
today "the learning curve," a model of knowledge accumulated through
experience.
"Then it turned out that the mathematics was too much," Warsh wrote.
"Arrow's model didn't have the well-behaved qualities expected of a
reliable guide to action. It was unstable. Little changes through it off. So
the learning-by-doing model failed to become part of the underground
economists' tool kit."
And from the time Arrow's math proved unstable in the late 1960s until Paul
Romer started to rethink the economics of change in the late 1980s, the role of
knowledge as the engine of change remained quietly outside the field of macro
economics.
********
Interestingly, Paul Romer's path to bringing knowledge "inside"
macro-economics began
at a ski resort on a chair lift. After publishing a
paper in 1987 on congestion in chair lift lines, economist
Tyler Cowen and others pointed out Romer's results
had a lot in common with other club
goods, such as highways, which also can exhibit congestion
problems.
At the 2010 Sloan Sports Conference,
Brian Skinner, a physicist at the University of
Minnesota, presented essentially the same concept in the context of basketball
in a paper entitled,
"The Price of Anarchy in Basketball." In
2013, Professor Skinner applied the same approach to NFL football and Peyton
Manning.
In
"The Price of Anarchy in Football,"
Professor Skinner noted "[i]n football, one can think that the goal of the
offense is to acquire yards. Every time the ball is snapped, the offense looks
to move the ball as far down field as possible. To do so, that offense has a
number of distinct [... wait for it ...] options."
More than 50 years ago, in defiance of custom and tradition, Lombardi began
devoting all his attention to a subset of these options--embedded
endogenous elective choices.
It's not that the NFL coaches retract into the fetal position when they
consider defying custom and tradition.
It's that some of the analysts, like Professor Skinner, are finally starting to
catch up to the NFL coaches who have been decades ahead of the analysts in
defying custom and tradition and practicing growth theory since about the same
time Robert Solow invented it.
That NFL coaches have been unaware they have been practicing growth theory is
irrelevant.
But it is something that an analyst might want to know.
********
We've talked macro (growth theory; endogenous vs. exogeonous elective
choices).
Now, let's talk micro.
"A computer will never make coaching decisions.
The idea is ridiculous," none other than Virgil
Carter himself said in 1972.
While the idea that a computer could make coaching decisions does not sound so
ridiculous today, on the micro level, an NFL coach that listens to a computer
and stakes more on the result of one play and loses may get himself fired,
particularly when he does it with a team that is better than its opponent.
For example, in Week 3 of the 2009 season, Jim Zorn's Washington Redskins (1-1)
travelled to Detroit (0-2) to meet the Lions who had lost 19 games in a row and
25 of their last 26, including an unprecedented all 16 during the previous 2008
season.
On the first possession of the game, Washington stymied the Lions. Redskins
quarterback Jason Campbell then led a brisk drive that stalled at the Detroit
1-yard line. Rather than be patient and accept a near certain field goal (3
points), Zorn went for the touchdown (7 points) and failed.
After Washington eventually lost by 5 points, 19-14, those missing 3 points
hung around Zorn's neck like a cement block when the Redskins' final drive
ended just one 10-yard pass short of field goal range in a desparate series of
improvised, chaotic and unsuccessful laterals. The hapless Lions would go on to
lose 12 of their next 13 games, the only victory being a wild 38-37 win over
the downtrodden Cleveland Browns (5-11).
Washington owner Daniel Snyder responded swiftly.
Just 10 days after Zorn eschewed patience, the
Redskins pulled Sherman Lewis out of a bingo hall
and a couple of weeks later publicly emasculated Zorn by announcing
Lewis would be the new playcaller in charge of
elective choices. At the end of a dismal 4-12 season, Zorn mercifully received
his pink slip.
This example is telling. It may be uncivil, but accurate, to call a person
gutless who fails to do the right thing based on a fear that never
materializes. But it is simply incorrect to apply the gutless label when the
feared consequences of the action are potentially real.
"Returning to the world periodically in search of relevant examples of the
phenomena he was describing was not 'dumbing down,' Romer said," according
to Warsh. "'Once you've erected the mathematical machinery and understood
how it works, you've got to cut back to the world to see how it works, you've
got to see whether you've cut to the essence of it."
********
So what, you say. Daniel Snyder is no Dan Rooney when it comes to football
knowledge.
True, but if you closely examine Zorn's decision using the type of analytics
that Silver views as the only way to evaluate fourth down decisions, you
can see that choosing the field goal was as justifiable a choice as going for
the touchdown.
In a blog post entitled
"Zorn is My Hero 2" following the debacle
in Detroit, Brian Burke, a Silver favorite, calculated that Zorn's decision to
eshew patience and go for the first quarter touchdown was worth 4.6
"expected points." A synonym for "expected points" is
"pretend points."
Simply, Zorn traded 3 real points for 4.6 pretend points.
Let's try it this way. What would you call a person who traded 3 real apples
for 4.6 pretend apples? You probably would call him hungry.
But analysts like Silver and Burke--who called Zorn's lack of patience "a
slam-dunk good decision" because according to the data 4.6 is greater than
3--simply ignore the different units of measurement, pretend points as
opposed to real points.
As Warsh stated in Knowledge and the Wealth of Nations, there is a big
difference between kowledge and data.
"Information consists of facts, not necessarily dependable, that may or
may not be gleaned," Warsh wrote. "There's something intrinsically
loose about information, even in its plural form, which is data...."
Zorn's fateful 2009 fourth down decision in Detroit is a perfect example.
While 4.6 is undeniably greater than 3, there is one other mathematial problem
with Zorn's lack of patience: Even if you ignore the different units of
measurement, 4.6 pretend points still would not have been enough for Washington
to beat the Lions because the Redskins lost by 5 points. So, even if the
Redskins had received 4.6 real points for Zorn's lack of patience, they still
would have lost 19-18.6 if Zorn's decision to go for it is isolated.
Furthermore, if Zorn had simply taken the pretend 2.6 points that Burke
calculated the field goal attempt to be worth, Washington would have only
trailed 19-16.6 and could have won, 19.6-19, if the Redskins could have moved
into position for a last play of the game field goal.
So, any reliance on prentend points in the first quarter leaves Zorn
needing 3 more real points on the last play of game to win.
To "tighten up" the loosness in Burke's data, you have to subject
Zorn's decision to the rigorouos test of isolation.
If one does so, one sees that as Zorn contemplated his fourth down choice in
Detroit, he confronted a subjective choice pregnant with multiple equilibria.
He could choose to further his expectation interest and go for
touchdown. In the alternative, he could choose to further his reliance
interest and send in the field goal team.
Neither choice could really be considered better than the other. If Washington
had scored the touchdown, it would have received a windfall above the 4.6 value
of the expectation interest that would have been enough to secure victory.
On the ther hand, if the Redskins had made the field goal, they would have been
in much better position to win the game with a field goal at the end of
regulation than they were as a result of the failed fourth down attempt.
Such multiple equilibria are signature characteristics of a monopoly and, as
Silver pointed out, NFL coaches possess a monopoly over exogenous elective
choice-making because the opponent has no way of preventing the coach from
making the choice.
Consequently, there simply was not any objective mathematical formula that
could tell Jim Zorn how to make his fourth down decision in Detroit. It was a
pick 'em choice.
In 2011, Professor Skinner presented a principled approach to risk-based
choice-making at the Sloan Sports Conference entitled
"Scoring Strategies for the Underdog: Using Risk as an
Ally in Determining Optimal Sports Strategies."
Under Professor Skinner's approach, "an underdog should be willing to
sacrifice from its expected final score in order to increase the variance. A
favored team should be willing to sacrifice from its expected final score in
order to reduce the variance."
In other words, generally, if the opponent is better than your team on a given
NFL Sunday, "sell short" by buying risk and go for it on fourth down.
If your team is superior to your opponent, then "bet long," send out
the kicking team, and patiently wait for your team's superiority to assert
itself.
But an NFL coach still might go against Professor Skinner's approach under
unusual circumstances. For instance, in the 1967 NFL Championship Game--the
famous "Ice Bowl"--Vince Lombardi found himself with the superior
team at the Dallas 1-yard line and 0:16 to play, but trailing the Cowboys,
17-14, and without any timeouts. Professor Skinner's approach would counsel to
"lengthen the game" and play conservatively for the field goal and
overtime.
But Packers' quarterback Bart Starr told Lombardi that he could wedge the ball
into the end zone on the frozen field.
Lombardi did not hesitate to go with his gut and told Starr, "Then run it
and let's get the hell out of here."
The rest, of course, is history. Green Bay guard Jerry Kramer pried Dallas
defensive tackle Jethro Pugh out of the way and Starr collapsed into the end
zone for the touchdown and a 21-17 Packers' victory.
It's not that the outcome of Lombardi's gamble was successful and Zorn's was
not that makes Lombardi, not Zorn, a legend and a hero. That difference is
mostly the result of Lombardi's relentless deployment of endogenouse elective
choice-making in the power sweep design as opposed to his single exogenous
elective choice in the Ice Bowl.
But the outcome of Lombardi's Ice Bowl elective choice sure didn't hurt
Lombardi's legacy either.
********
Today, the NFL clearly continues to embrace the analytic approach pioneered by
Dallas' Tex Schramm and to follow Lombardi's lead in the search for designs
that contain embedded endogenous elective choices.
Reportedly, at least six NFL teams have purchased
Paul Bessire's "Prediction Machine"
simulation analytics to help them in the NFL Draft. About a third of the NFL
reportedly subcribes to
Pro Football Focus' player grading analytics.
This off-season, the NFL general managers have refused to shower big money on
the top free agent running backs because analytics have proven that running
backs do not exert much influence over wins and losses.
"When it comes to RBs, teams have already learned their Moneyball
lesson: Don't overpay for talent at an injury-prone position that doesn't
necessarily correspond to wins,"
Ilan Mochari wrote on Inc.com.
Outside the NFL, at Football Perspective,
economics professor Andrew Healy used win probabiliy
analytics to identify "the twenty plays that shifted the probability of
the eventual Super Bowl winner the most."
Professor Healy's findings are eye-opening. In summary, he found that:
* 35% of the plays were "big" offensive plays, i.e., passes of 25 or
more yards (7 of 20 plays)
* 35% of the plays were "big" defensive plays, i.e., turnovers (7 of
20 plays)
* 15% of the plays were "little" defensive plays, i.e., incomplete
passes (3 of 20 plays)
* 10% of the plays were "little" offensive plays, i.e., runs,
including Starr's TD in the Ice Bowl (2 of 20 plays)
* 5% of the plays were special teams breakdowns (Buffalo kicker Scott Norwood's
missed field goal in SB25).
Only 10% of the plays were fourth down plays and only 10% of the plays probably
can be categorized as true exogenous elective choices: Starr's TD in the Ice
Bowl and John Riggins' fourth-and-1 run on "70
chip" that morphed into a tide-turning 43-yard TD in SB17, the
longest TD run in Super Bowl history at the time.
Professor Healy's new contribution to the stock of football knowledge might
help an NFL team design its approach to the game better by zeroing in on
players and plays capable of making big plays.
Somebody inside an NFL team has probably already read Professor Healy's blog
post and is wondering if there is anything that can be learned from the
distribution of impact plays Professor Healy has contributed.
********
Nate Silver probably has not read Professor Healy's post and still doesn't know
that only 10% of the plays that shifted the probability of the eventual Super
Bowl winner the most were exogenous elective choices.
The
logo of Silver's fivethirtyeight.com web site depicts a
fox as an allusion to a phrase originally attributed to a Greek poet
(not "Jimmy the Greek"): “The fox knows many things, but the
hedgehog knows one big thing.”
In time, Silver may live up to what that logo is intended to represent.
But right now, he doesn't seem to know enough about NFL football to badger its
coaches about their approach to analytics or their elective choices on fourth
down.
(Archives
Home)
|
