Nate Silver can’t hit a fastball, but he’s an ace at dissecting baseball stats. And Major League Baseball has taken notice.
Nate Silver, age 30, is a pundit—one who espouses opinions based almost exclusively on numbers. Like a meteorologist he mines data patterns to predict what the future may bring, be it the presidential election results or the Chicago White Sox’s win-loss record. Sitting with him in the White Sox’s U.S. Cellular Field three weeks into the season is a bit like visiting a clairvoyant.
“Who’s he?” I ask when Daric Barton, the Oakland A’s rookie first baseman, steps to the plate. “He was one of the minor leagues’ best hitters,” Silver replies. “He will be a good hitter—eventually.” Silver notes that the Player Empirical Comparison and Optimization Test Algorithm, or PECOTA—a system he developed to project baseball players’ career arcs by comparing them to similar players from the past—suggests Barton has a 44 percent chance of having a breakout season. That means he’ll increase his equivalent runs, or the number of runs he produces, by at least 20 percent. As Silver highlights Barton’s 73 percent chance to better his 2007 statistics—.347 batting average, four home runs, and eight runs batted in only 18 games—he strikes out. Still, Silver says, Barton could emerge as a “John Olerud type,” referring to the former Toronto Blue Jay and New York Met known for his keen batting eye and ability to draw more than 90 walks a season. On the other hand, he cautions, Barton could be a bust: after all, he has an 8 percent chance of collapse, meaning his performance would decline by at least 25 percent relative to his baseline performance over the previous three seasons.
Regardless of Barton’s success and the fact that the A’s lineup looks “really shitty,” the A’s are smart, Silver explains, because they’ve committed to rebuilding, and “you might as well finish in fourth place if you’re going to have to spend an extra $20 million to finish in third.”
Silver, AB’00, is the managing partner of Baseball Prospectus (BP). It’s a think tank devoted to the statistical analysis of the sport, or sabermetrics—a term derived from the acronym SABR, which stands for the Society of American Baseball Research. Producing research, articles, and books, BP sets the standard for independent baseball research and analysis, according to general managers like the A’s Billy Beane and the Cleveland Indians’ Mark Shapiro. Regularly cited in newspapers, sports magazines, and blogs, BP and its writers routinely appear on NPR and BP’s own radio podcast.
Silver essentially serves as the company’s chief executive officer. Standing about five feet nine, he wears his scribble of brown hair pushed forward and extends a flimsy handshake. He walks with a deliberate gait, speaks in tangents that often make him forget where he started, and analyzes nearly everything he encounters. He lives with a roommate in a disheveled, dorm-like Wicker Park apartment where the sink is perpetually full of dishes and the worn, black-leather couch cushions are cluttered with papers. He usually wakes up around 10:30 a.m., gulps down coffee or Coke, and plops on the sofa with his black Hewlett-Packard laptop, an “ironic” Apple logo stuck on its lid. He slogs away at work until 2 or 3 p.m., when he frequents one of the 19 taquerias—he’s counted—within a five-mile perimeter. Last July he began scrutinizing the restaurants’ tortillas, salsas, burrito proportion (the degree to which every component is “in balance”), décor, and overall experience, on his blog the Burrito Bracket.
But since November the bracket has been sidetracked by another numbers fixation—the presidential race. Posting on the liberal-leaning political blog Daily Kos as “Poblano” (a friend’s Daily Kos handle he borrowed, not expecting to gain recognition), he’s parsed polls and deconstructed the reasons behind long-assumed constructs like the Iowa Bounce, the momentum gain for candidates who win the Iowa caucuses. In March, as Poblano, he launched his own site, FiveThirtyEight, named for the 538 Electoral College members. The site features a primary- and general-election forecasting system that weights polls based on an individual pollster’s track record, the survey’s sample size, and the date of the poll (surveys have a half-life of 30 days, he says, based on empirical evidence derived from the 2000, 2004, and 2006 election cycles). The system correctly predicted the outcomes of the Indiana and North Carolina Democratic primary races, earning Poblano mentions in the New York Times and the National Journal’s Web site.
The pseudonym allowed Silver to maintain separate identities for baseball and politics. But as the site drew nearly 30,000 unique visitors a day, including influential journalists like the Atlantic’s Marc Ambinder and Andrew Sullivan and the National Review’s Mark Blumenthal, he decided to reveal himself as Poblano. Since disclosing his identity June 1 in the New York Post, he’s started writing “Strength in Numbers,” a weekly column for Britain’s Guardian newspaper. “In polling and politics,” he noted in the Post, “there is nearly as much data as there is for first basemen.”
At Baseball Prospectus Silver considers himself a writer, but managerial duties make up about 60 percent of his role. On a typical spring or summer day he may answer e-mail (he gets one or two a year from players he’s written about, a few more from players’ parents), analyze statistics or the company’s budget, and turn to his ever-present Blackberry to make calls about business strategy or an e-mail-server crisis. Each week he writes a “Lies, Damned Lies” column for the Baseball Prospectus Web site, applying sabermetric techniques to topics like the effect of performance-enhancing drugs (there is no statistical smoking gun, he argues) or which players won’t meet expectations this season (San Diego Padres catcher Josh Bard’s statistics, Silver says, will fall back in line with the minor-league track record he has outperformed in his first few years in the big leagues). He often writes until at least 2 or 3 a.m. From the end of the World Series through December, his days grow longer, as Silver and his fellow BP contributors write their annual guidebook, which contains more than 1,600 players’ statistics from the previous season and projections for the season ahead.
All of BP’s 13 writers work from their homes scattered across the country. With such independence comes “a lingering feeling that you could do more,” Silver says. “It’s almost like college; there are a lot of very bright people we interact with, there are a lot of things we want to do, and we almost always run out of time.” Nonetheless, since he became managing partner in 2004, BP’s revenues have grown about 200 percent. While his job is often unglamorous, Silver can’t imagine anything else he’d rather be doing, including working for a Major League team.
In 1995 Gary Huckabay, then an MBA student at the University of California, Davis, founded Baseball Prospectus to fill what he saw as a “pretty big gap in the market.” Part of a rich pool of smart people already analyzing baseball in a Usenet newsgroup, he recruited four fellow Usenet users, including Chris Kahrl, AB’90, to help him launch BP. The company’s trademark would be innovative statistical analysis, presented in an entertaining, humorous style. “There are plenty of boring analysts out there,” says Kahrl (who has been living as a woman since 2003 and now uses the byline Christina Kahrl). “And lots of people can put together a spreadsheet, but not everyone can make it sing.”
That first year, BP charged $20 for a statistics guide produced on a photocopier. It printed around 300 copies and sold about 170 to fellow statheads, even though the book was missing the St. Louis Cardinals. “It was terrible,” recalls Kahrl, “but it nevertheless didn’t discourage us.” Within a few years Brassey’s Inc. published the guide, which grew to about 3,000 copies. By 2007 it reached the New York Times bestseller list, topping 70,000 copies at $21.95 a pop.
Soon after Baseball Prospectus launched an online presence in 1997, primarily to present original research and promote the guide, Huckabay realized that the site was the best way to establish BP as the clearinghouse for baseball analysis. In 2003 it became one of the first sports-related sites to place content, including forecasts and statistical databases, in a premium-subscription section. A year later the company posted what amounts to a mission statement: “We feel that this greatest of games is so compelling that we want to know everything about it. We always want to improve our understanding of the game; each player, each play, each pitch, each throw, each hit—what does it really mean? Those arguments that take place in bars about the relative merits of different players? We really want to know the definitive answer to those questions. But we don’t want to kill the joy of the game while we’re looking.”
From baseball’s earliest days, fans have sought to understand the sport by examining its statistics. More than a decade before the first professional team, the Cincinnati Red Stockings, emerged in 1869, newspapers printed box scores—tables listing players’ names, how many runs each player scored, and the number of hands lost, or outs, each player caused. “It was not particularly sophisticated,” says Alan Schwarz, a New York Times sports writer and author of The Numbers Game: Baseball’s Lifelong Fascination with Statistics (Thomas Dunne Books, 2004), “but it was the first attempt to say what happened in the game.” By the turn of the 20th century the sport evolved into some semblance of modern-day Major League Baseball—team owners formed the National League; pitchers began throwing overhand standing 60 feet, 6 inches from the plate; batters walked after four balls and struck out after three strikes; and fielders wore gloves. Statistics also evolved, and newspapers started tracking pitchers’ innings pitched, hits, earned runs, walks, and strikeouts; and hitters’ batting averages, hits, and runs scored. In 1920, when Major League Baseball began requiring umpires to replace balls at the first sign of wear, fans and reporters reveled in the flurry of history-making records that stemmed from higher-scoring games.
The fascination with stats kept growing. In 1952 the Bowman Gum company and Topps Chewing Gum printed their first baseball cards, featuring players’ data on the flip side—info kids promptly memorized. As Schwarz wrote in The Numbers Game, “Topps baseball cards joined box scores as the major pipelines that fed statistics into young boys’ brains.” In 1961, when Bucknell University mathematics student Hal Richman compiled statistics into Strat-O-Matic, a tabletop game where players make strategic and personnel decisions, he spurred a generation of children and adults to think critically and scientifically about baseball.
That kind of thinking came naturally to Silver, a numbers fanatic by age three. “Today I’m a numbers machine,” he announced as his dad, a Michigan State University political-science professor, dropped him off at preschool. He then proceeded to count. And count. Two hours later, he was still counting. Within a year he learned to add and subtract negative numbers. When his parents gave him the children’s magazine 3-2-1 Contact, which featured readers’ submissions of computer programs written in BASIC, Silver promptly learned the code, consulted a manual, and taught himself to embellish one of the magazine’s programs into a Pong-like hockey game.
In 1984 the Detroit Tigers started the season 35–5. Silver was six. When the Tigers went on to win the World Series, he was hooked. He loved watching games and playing catch, so his dad signed him up to play T-ball in the East Lansing youth league. The next year he reached the Midget division, where batters faced live pitchers rather than a ball on a tee. He was, he realized, a lousy hitter. “But I also knew the pitchers couldn’t pitch better than I could hit, so I wouldn’t swing,” he recalls. “I’d walk. Then I’d steal. It was a cheap way to play, but hey, it worked.” The strategy, he says, yielded a pedestrian batting average of around .250 but an otherworldly on-base percentage that “was probably about .833.”
With Silver’s computer prowess and aptitude for baseball strategy, one of his father’s MSU colleagues suggested he try playing Scoresheet baseball, a Strat-O-Matic–like computer simulation using statistics from the current week’s games. The colleague was right; Silver loved it. He pored over the statistics in his baseball-card collection, read Baseball Digest and tomes like Total Baseball, and organized the information into spreadsheets to better manage his teams. When he got Internet access, he started reading and posting in Scoresheet discussion groups. By the time he was in the College, in addition to writing about sports for the Chicago Maroon, he analyzed players and strategy as a Scoresheet expert for fantasy-baseball guru Ron Shandler’s Web site BaseballHQ.com.
After graduating with an economics degree, Silver worked at the consulting firm KPMG International. He found some projects—like examining classroom computer usage for an Ohio school district—interesting, but most bored him. After work he’d unwind by playing Yahoo!’s free online poker game. Poker revolves around pattern detection and swiftly parsing through information. Silver was a natural and figured he could make real money gambling online. With poker’s popularity beginning to crest, Silver surmised that the influx of new players would allow “a mediocre player to be supported a lot longer than usual because they feed off the bad players.” Considering himself far better than mediocre, he’d play up to four tables at once. “You’re bombarded by information, and it can be exhausting,” he says. “You exploit small edges. You almost have to play unconsciously, or almost by instinct. I mean, if you play four tables online for four hours, you’re playing something like 16,000 hands. You’d have to sit in a casino for 24 straight hours to get that many hands.”
Between games, he kept posting in Scoresheet discussion groups, meeting Huckabay and other BP writers. And he began concocting PECOTA’s forecasting formula. His idea was that different types of players develop and age differently, and there was no way to predict definitively how each would perform. “I wanted to develop something probabilistic so that I would know that there’s a 14 percent chance that a guy will hit 14 home runs. And I wanted to be able to compare a present-day player like Derrek Lee to a Hall of Famer like Eddie Murray. I had intuitively made those comparisons for years, but I wanted to be accurate when I did it.”
Working in bits and pieces, a few hours a week at home, a few at the office, he developed PECOTA using spreadsheets that resembled those he created for KMPG clients. The spreadsheets included every player in the major leagues since 1946 (reflecting the end of World War II, considered the start of the game’s modern era). He then devised a convoluted formula to match players based on statistical performance levels including walk rate and isolated power (a hitter’s ability to generate extra-base hits), as well as such variables as position, body size, and handedness. He used correlation coefficients, or the linear relationship between two variables, like a player’s strikeout rates across consecutive seasons, to develop PECOTA’s predictions. He also included a measure that considered some statistics’ predictive relationships with others, such as strikeouts and batting average. For example, once a ball is put in play, there is an element of luck that is independent of a batter’s skill. But if the batter fails to put the ball in play, he has no chance of a hit. So a player with a low batting average but low strikeout rate has a good chance of seeing his average improve, while one with the same batting average but a high strikeout rate can expect his struggles to continue.
By 2002 Silver deemed his 100-megabyte Excel file ready. He sent a copy to his dad, who shared it with his Scoresheet-playing colleague. Then Silver showed it to Huckabay, who told him he wanted it to replace BP’s forecasting system. In 2003 he sold PECOTA to BP for an equity share in the company. Still working at KPMG, he started writing a few BP articles a week. In April 2004 he quit his $55,000-a-year job, figuring that he could earn $100,000 annually playing poker to supplement his BP earnings. A few months later Huckabay left BP to do consulting inside and outside of baseball. The BP staff decided Silver should take over as managing partner.
Baseball lends itself to quantification, says the New York Times’ Schwarz, because it has discrete points where the action stops, making individual contributions easy to assess. And because the regular season spans 162 games, “there is less importance placed on winning and losing. It’s much more nuanced, and the storylines are told in numbers.” For nearly a century those numbers concentrated on “counting things that people thought were important like home runs, hits per time at bat, protecting a lead.”
Enter Earnshaw Cook. In 1966 Cook published Percentage Baseball. Questioning the commonly accepted metrics, he argued against strategies such as placing a speedy runner at the top of the batting lineup. Rather, Cook suggested, teams should put their best hitter first, the second-best second, and so on to maximize how often the best players reach the plate. Using a mechanical calculator, slide rule, and colored pencils, Cook developed the scoring index, a measure that sought to distill the essence of a player’s worth. Three years after Percentage Baseball appeared, brothers Harlan and Eldon Mills rented time on an IBM 1620 mainframe to create the win average, a number that delineated a player’s contributions to his team’s victories.
Both Cook and the Mills brothers challenged baseball’s accepted norms, but their ideas received scant attention until 1977, when Bill James self-published his first Baseball Abstract, an annual guide that combined statistics and analysis with riveting writing. James examined whether defense was properly measured, whether stolen bases were meaningful, and how stadiums affected players’ performance.
James “didn’t invent a whole lot,” says Schwarz, “but he calibrated it in a different way.” He influenced the Usenet discussion-group members who helped spark BP, as well as a host of front-office types like Boston Red Sox general manager Theo Epstein, a Yale University graduate who joined the team in 2002, and San Diego Padres special assistant of baseball operations Paul DePodesta, a Harvard University alumnus who began as an A’s executive in 1999. In 2002 the Boston Red Sox hired James—who stopped producing the Abstract in 1988 but continued applying scientific thinking in baseball-related historical and reference books—as an adviser. Before he was hired the team had not won the World Series since 1918. Since his hiring they’ve won it twice.
Armed with more information from James, BP, and other sources, teams and fans have gotten smarter about the game, says John Zajc, executive director of the Society for American Baseball Research. “Five years ago it would be a rare broadcaster or sportswriter who mentioned on-base percentage. Today I think the bell curve has turned over to the other side—few don’t know what it is.” He points to teams like the A’s, who have combined sabermetric and economic principles to exploit marketplace disequilibrium: teams paying a premium for pitchers with a 98-mile-per-hour fastball, for instance, while ignoring their ability to prevent batters from hitting the ball. Sabermetrics, as Michael Lewis wrote in Moneyball: The Art of Winning an Unfair Game (W. W. Norton & Company, 2003), helped the A’s pay about $500,000 per win from 2000 to 2002, when they reached the playoffs each season. The Baltimore Orioles, meanwhile, paid nearly six times as much to finish each season in fourth place. Moneyball woke up some teams, and especially fans and reporters, to the benefits of predictive modeling, says Schwarz. But “it would have happened anyway,” he notes. “Theo Epstein was a GM before Moneyball. The book just pushed these ideas forward faster.”
In the decades after the early 1900s, football increased a touchdown’s value from five to six points, hockey adopted offside and icing rules, and basketball added the 24-second shot clock and three-point line. But baseball has remained remarkably consistent. “The game has always been the same,” says Zajc. “Every game starts with one pitch. Once the ball leaves the pitcher’s hand, it can be taken for a strike, it can be taken for a ball, and every action changes the strategy and plans for that one-on-one battle between the batter and pitcher.”
Because of that consistency, and because baseball is steeped in tradition and romance, not everyone is enthusiastic about the sabermetric approach. “The numbers guys get lost in their own statistics,” says the New York Times’ Murray Chass, who wrote a column in 2007 suggesting “new-age” statistics threaten to undermine fans’ enjoyment of the game. “I think they lose sight of the fact that baseball is played by human beings and statistics can’t explain human factors like a player being unable to take the pressure of being a Major League player and playing before 50,000 fans.” Similarly, Chicago Tribune columnist Rick Morrissey wrote in a March critique of PECOTA forecasting the White Sox winning only 77 games: “Computers have no use for heart, or [at] least they can’t quantify it.”
These critics miss the point, Silver says: sabermetrics has enhanced the quality of play. He points to a measure he calls the efficiency index, which compares the league’s best backups to its worst regulars. The idea is that if some teams’ backups outplay other teams’ regulars, the league is inefficiently distributing talent because of disparities among teams’ wealth or player-development systems. For the past five years, he says, talent distribution has been more efficient than at nearly any other time in history. Sabermetrics has also enlivened baseball history, Silver argues, providing perspective into past players’ feats. And the stats produce objective data that can debunk long-assumed constructs, like the existence of the clutch hitter, a player with a supposed knack for getting big hits. The numbers show that the skill either doesn’t exist or is so rare that it is hardly worth tracking.
Although sabermetrics is complex, baseball is basically simple, says Silver during the April afternoon game at U.S. Cellular Field. “If you take a team that wins half its games and add a couple guys who get hot at the right time or outperform expectations,” he says, “you can win.” He cites two examples: the 2005 World Series champion White Sox, whose pitching staff threw four straight complete games in the American League Championship Series, and the 2007 National League champion Colorado Rockies, who won 13 of the last 14 regular-season games and then won seven straight playoff games to reach the World Series. “If [White Sox pitchers] John Danks and Gavin Floyd are better than we think”—PECOTA predicts five wins each in 2008—“then the Sox could win the division,” he says. Just then Danks leaves the game after seven and two-thirds shutout innings, en route to his first win of the season. In baseball, as in presidential elections, “anything can happen.”