Our columnist explains why the thinkers who took data science mainstream now have serious reservations about its overuse.
Every Wednesday morning, artnet News brings you The Gray Market. The column decodes important stories from the previous week—and offers unparalleled insight into the inner workings of the art industry in the process.
This week, another cautionary tale from the land of numbers…
People have been debating how to value artworks for centuries. For most of written history, the central problem has been qualitative. Recent decades, however, have expanded quantitative possibilities beyond all previous imaginable measure, so that numbers threaten to swallow qualitative concerns whole. The payoff, proponents say, is that data-driven alternatives may reveal something deeper, truer, and more objective.
But I’m more convinced than ever that the quantitative revolution is marred by a fundamental misunderstanding about the value of data. This dose of skepticism came to me from the new season of “Against the Rules,” the podcast series from bestselling author Michael Lewis. In an episode titled “Field of Ignorance,” Lewis returns to correct the record about what may be his most famous book, Moneyball, chronicling how a cash-strapped, bottom-dwelling Major League Baseball franchise began outperforming the game’s richest, most glamorous teams by embracing a statistical rebellion.
The secret weapon was Bill James, a Kansas-based baseball outsider whose thinking obliterated 150 years’ worth of orthodoxy about what made players valuable. The details of his methods aren’t important for our purposes. (If you’re interested, read the book, watch the movie, or at least peep the trailer.) What matters is that they came from a slew of empirical data channeled through mathematical formulas of his own design, which he began publishing and selling annually in a tome called The Bill James Baseball Abstract.
A baseball signed by Babe Ruth that was put up for auction in October 2016 at Christie’s New York. (Photo by Spencer Platt/Getty Images)
That all of his findings were quantifiable, transparent, and replicable made James a wonky heretic in a game long ruled by former players who made decisions based on experience, instinct, and other factors often indistinguishable from bullshit. Yet his ideas turned a perennial loser into a championship contender almost overnight. The success of James’s metrics on baseball’s biggest stage transformed Moneyball from a bestselling book into a philosophy that permeated all of Major League Baseball within a few seasons of the book’s publication in 2003.
Since then, “Moneyball” has become shorthand for any quantitative approach to any other industry. Innumerable pitch decks from self-styled experts in a wide range of fields have touted their ability to upend the status quo with advanced metrics. The funniest—and most mortifying—sequence in “Field of Ignorance” memorializes this frenzy by editing together clips of news anchors breathlessly introducing stories about “Moneyball for the restaurant industry”; “Moneyball candidate” for president Marco Rubio (lol); and law enforcement’s data-driven pursuit of serial killers as “Moneyball for murder.”
The art market got in on the act, too. Almost 20 years since Lewis published his book, the industry is deeper in numbers than ever before. But “Field of Ignorance” renewed my fears that something important has been lost in the rally, just as James feared it could be lost in baseball.
Red Grooms, Home Run Sculpture at Miami Marlins Park during a preseason game against the New York Yankees in April 2012. Courtesy of Mike Ehrmann/Getty Images.
Contrary to popular belief, valuing artworks by alternative metrics was already an old practice before Moneyball’s release. Lee Seldes’s book The Legacy of Mark Rothko mentions 1955 Fortune magazine articles that categorized different groups of artists as different classes of stock bounded by specific price parameters and return potentials. European Old Masters were considered “gilt-edged” securities; early Modernist works priced up to $1 million were seen as “blue-chip” shares; and living artists whose works were available for $3,500 and under were once recast as high-risk, high-reward “venture capital” targets.
The broader quantitative revolution of the 2000s chimed with an increasing interest in art market data. Although Artnet launched its auction-price database in 1989, it took roughly a decade for the resource to become essential to the industry. Its adoption helped pave the way for the emergence of targeted analytics reports, offered by Artnet and a crop of competitors such as London-based ArtTactic, founded by former J.P. Morgan quant Anders Petterson in 2001. Four years later, the European Fine Art Foundation (TEFAF) commissioned Clare McAndrew to publish the first of her annual art-market reports.
The art industry’s drive to leverage advanced statistics made another step-change after the global financial crisis of 2008. The years since Lehman Brothers’s collapse have seen the creation of dozens of art-tech startups, each premised on wielding data science for goals such as quantifying artworks’ formal traits, optimizing client relationships, visualizing sales and viewership trends, and others.
The Holy Grail of art-market analytics has been the “true” market value of works on offer. The most infamous attempt to deliver it was Sell You Later, a startup launched in 2014 that placed artists onto a handful of action-oriented lists ranging from “buy now” to “liquidate” with the help of a proprietary algorithm. When founder Carlos Rivera later rebranded the company as Art Rank, the lists matured and multiplied to include “buy now under” various price ceilings, the slightly gentler “sell / peaking,” and the tantalizing “undervalued blue chip,” among others, before the company ceased operations in 2016.
More recently, a new phalanx of startups has attempted to perfect price discovery by hyping their use of artificial intelligence. In practice, the change is finer-grained than it sounds, “artificial intelligence” most often being window dressing for the same type of data science used earlier by Art Rank, only executed via different algorithms fed a few more years’ worth of data.
These competitors are still crunching numbers with an eye toward capturing an artwork’s value in a single master datapoint. That datapoint could be translated into a ranking, or it might become an alternate valuation in your currency of choice. Either way, the goal is to distill a swarm of complex information into a simple indicator that guides a definitive action.
The obvious appeal might make you think a similar master valuation powered Bill James’s ascension to sports godhood. If so, you would be dead wrong.
John Baldessari, Ext. Baseball Diamond on Carillo – Day Wayne It’s Calumny (2017). Image © John Baldessari, courtesy of the artist, Marian Goodman Gallery, and Sprueth Magers.
Despite calls from his growing audience of outsider wonks, James staunchly refused to develop an all-in-one rating for players. He explained why in the third edition of his annual Bible: “The Baseball Abstract, for three years, has reverentially avoided rating people. It has been almost a point of honor, for a rating is a form of opinion. If it is ground out of a formula, it becomes an opinion expressed in numbers.”
“An opinion expressed in numbers”—that’s the exact opposite of how our culture has come to understand advanced statistics (or even, for that matter, classical statistics). No matter the industry, these valuations are almost invariably shrouded in claims about “objective” and purportedly unbiased results.
But what James understood is that any quantitative analysis is rife with subjective decisions. What matters? How should it be measured? What amount and mix of data will make the results trustworthy? How should the different statistical variables be weighted? What is the best mathematical expression of each variable’s relationship to the others?
These choices are only ever judgment calls made by a group of human beings who happen to think best in numbers, not words or images. The outcomes might be valuable, James argued, but they were always and forever debatable, too. Every question they answered just opened up two or three more.
The problem was that more and more of James’s acolytes thought his work could be used to settle the debate rather than open it up. They thought an all-in-one player rating would therefore be the alpha and omega of value, which was exactly what James feared. Worse, as the Moneyball approach spread to industries far and wide, a core oversimplification spread alongside it. Lewis described the broader misunderstanding like this:
“We now quantify and then move on, as if there’s nothing more to say, when there’s often a lot more to say… The problem is not the numbers, obviously. It’s how people use them. The numbers start out as tools for thinking. They wind up replacing thought.”
Lewis’s highest-stakes example of the danger is Value at Risk (VAR), “a single number dreamed up by financial quants in the late 1980s so that a Wall Street CEO could supposedly see how much money traders might make or lose in a single day.” He contends that CEOs “began to trust VAR over their own traders,” partly because the metric’s creators claimed it was 99 percent accurate. That worked out fine for everyone for about 20 years, until overreliance on VAR played a starring role in triggering the 2008 financial crisis—a causal link supported by both academic research and the reporting in Lewis’s later bestseller, The Big Short.
A similar tension emerged in baseball analytics, too. After James rejected calls to develop a master rating for players, other wonks went ahead without him. The resulting metric, Wins Above Replacement (WAR), quantifies how many games per season a single player can win for his team in comparison to a perfectly statistically average (aka “replacement-level”) player at his position.
Many years after its introduction, WAR is now so deep in the game’s bloodstream that, during MLB’s recently resolved labor dispute, league officials and the players’ union considered using the metric as a means to determine bonus pay. Rather than cheering, however, leading quants at the three public analytics websites most frequently relied on to calculate WAR said basing players’ compensation on the metric was “problematic” at best and “a terrible idea” at worst.
Why? Because there is no standardized formula for computing WAR. Each database uses its own algorithm, which can be tweaked at any time—including retroactively, if the data scientists find a way to improve the accuracy of results from prior seasons.
In short, the quants’ objection to basing actual pay on WAR is that WAR is just an opinion expressed in numbers. Harry Pavlidis, the director of research and development at the highly regarded online database Baseball Prospectus, said it best: “It’s great that people understand that this is something that’s valuable and useful, but if they understood the limitations of it they wouldn’t be coming up with this kind of idea.”
Pavlidis’s caveat applies to data-driven, all-in-one valuations in the art market, too. Personally, I’m all for their development and consideration, as long as we also keep their subjectivity and imperfections front of mind. If we don’t, then we’re just accepting quantified opinions as inarguable facts, and shutting down the discourse with an instrument best used to open it up. The mistake won’t blow up the global economy like VAR did, but it can do tangible damage on a smaller scale all the same.
That’s it for this week. ‘Til next time, remember: whether building an algorithm or an art world, what’s left out is just as important as what’s welcomed in.
