What's Inside the NYT's Data Black Box?
This past Sunday, the New York Times ran a piece by Lorne Manly and Robin Pogrebin that Ice Bucket Challenged one of the new dogmas of the contemporary art market: that a nascent tribe of collectors is flipping artwork by emerging talent at a blistering pace, fundamentally changing the way business is done in the industry.
Titled “Barbarians at the Art Auction Gates? Not to Worry,” the story concludes that quick and frequent resales are vastly less prevalent in the market now than perceived, and that the ‘flip rates’ are broadly comparable to those in the recent past. The authors base their conclusions on two separate analyses of post-war auction results–one by a Brussels-based art advisory called Tutela Capital S.A., the other by fellow New Yorkers Beautiful Asset Advisors.
I’ve seen the article shared all over art world social media in the days since it was first published, usually with either no comment–which I interpret as tacit agreement–or with enthusiastic support–something like “The good guys are winning after all!"
So I was eager to see what all the excitement was about. And while I expected to harbor at least a few critiques of the piece, I didn’t expect to leave it in disbelief that The Times would even allow the piece to run at all. Yet that’s precisely where I exited the experience, and I’m a little surprised the criticism hasn’t been more pronounced.
Why? To me, there are severe problems with the entire presentation of the data–not because I disagree with the numbers or the methodology, but because the authors bar their readers from looking at either of those components at any reasonable magnification.
To give a better sense of the issue, I went through the story and aggregated the authors’ description of all the actual findings below:
Yes, for the past few years, postwar and contemporary art has been reselling at auction faster than, say, a decade ago, the data show. But the pace last year was only slightly faster than it was in the mid-1990s…
…[T]he data indicate that contemporary works appearing at auction within three years of their creation are not coming to auction faster than in the past, and that such flipping remains very much the exception, not the rule. Though more works come up for sale each year, the percentage of these works was essentially the same last year, less than 2 percent, as in 2007, Tutela Capital found.
Beautiful Asset…reached a similar conclusion: While, historically, the percentage of works resold within five years is higher now than it was, say, two decades ago, that percentage has been decreasing since 2008…
Beautiful Asset found that contemporary and postwar artworks resold at Christie’s and Sotheby’s last year had been held by their owners an average of 11.2 years, the highest level in six years and a reflection of behavior similar to that exhibited in the late 1990s.
All of which raises the following questions to me, just for starters:
What were "the past few years” during which contemporary works “were reselling at auction faster”? How much faster? In what quantities or by what percentages? And what qualifies as “slightly faster” than the mid-90s resale pace?
How is it a “similar conclusion” that one study found no significant change in the percentage of works flipped within 3 years of purchase when comparing the years 2007 and 2013, but the other found an annual decrease in the percentage of works flipped within 5 years of purchase from 2008-2013?
In the study that found an advancing decline in flipped lots, how much of a decline are we talking about year on year? And what was the percentage at its peak?
What do the authors mean when they state that the average 11.2 year sales turn-around at the two major houses was “similar” to that exhibited in the late 1990s? How similar? And why are we only given a data point for this metric by one of the two analysts?
Apart from the frequency at which the works are being resold, what about the profitability of that decision? Were 2013’s subsequent resales on average more lucrative, less lucrative, or roughly similar in comparison to the allegedly comparable past periods?
Given how much emphasis is being placed on the period from 2007-2013, did either research firm try to adjust for the fact that the most brutal macroeconomic shockwave since the Great Depression hit smack in the middle of that range?
And in all cases, what was the reasoning behind each firm’s choices about the data?
The authors address none of these questions. Which is fine in itself. I’m used to coming away from conclusions about data with questions, then going back into the data myself to figure out what I want to know. But that’s precisely where this situation starts sprouting the coarse hairs.
This is all the authors are willing to say about their analysts’ procedures:
At the request of The Times, Tutela Capital and Beautiful Asset Advisors…reviewed art market data from 1995 through 2013 to see if there had been a noticeable shortening in the time owners held onto art.
Beautiful Asset tracks sales at Christie’s and Sotheby’s, which by themselves account for about three-quarters of the value of the auction market worldwide but a smaller amount of the volume. Tutela Capital collects data from more than 2,500 auction houses. Both use the provenances supplied at the time of auction to track prior sales of a work, including those, in the case of Beautiful Asset, that occur at houses where data are not routinely collected.
Essentially, all of the above distills to “The analysts used auction data from 1995 to 2013.” I may still (and possibly forever) be a numbers novice, but the level of opacity at work there strikes me as bizarrely high.
Stranger still, rather than reveal the methodology, the authors try to bolster confidence in their analysts by trotting out a pair of experts to nod their heads but, notably, keep their mouths sealed:
The analysts’ methodologies were reviewed by two experts, Stephen T. Ziliak, professor of economics at Roosevelt University, and Alan F. Karr, director of the National Institute of Statistical Sciences and a professor of statistics and biostatistics at the University of North Carolina, Chapel Hill. Both experts found the methods sound.
So in summary, these are the facts we’re left with:
Two different studies by two different analysts used two different auction house data sets to arrive at their conclusions.
The authors believe those conclusions can be interpreted as broadly similar in one respect–discounting the prevalence of flipping in the art market–despite being markedly different in others.
The authors feel no need to explicitly try to account for the discrepancies between the two studies.
The authors feel no need to reveal either the underlying methodologies of the analysts or any starting-to-intermediate data that led to the conclusions.
However, two purported experts who theoretically know statistics–but who have no obvious connection to the peculiarities of the contemporary art market–“found the methods sound."
Those experts either offered no qualms whatsoever about the analysts’ procedures, or the authors felt no need to provide those qualms in print.
And apparently, there’s a chance I’m the only one in the art world left squirming by all this.
My guess is that Tutela Capital and Beautiful Asset only agreed to run the numbers on the condition that their approaches were kept confidential, likely on account of my favorite data trope, the "proprietary algorithm.” If so, then personally, I think The Times should have gone elsewhere for their analysis.
To inspire trust in data and the conclusions taken from it, it’s generally not a great strategy to simply state that someone else agrees with the approach. Far better to present the numbers and at least some sense of the methodology so readers can examine them on their own. Refusing to do so–especially without even explaining the decision–automatically casts a shadow over the reliability of the results.
After all, if there was nothing to hide, why not put it all out in the open?
Does The Times’s secrecy mean that Tutela Capital and Beautiful Asset are wrong? Not necessarily. But to feel comfortable relying on their conclusions, I personally need to know much, much more than what the authors and their editors gave us. I need to know what’s inside the black box, not just what it produces.
That opinion may put me in solitary confinement in the art world, but I’m fairly certain that it would be shared by most, if not all, financial journalists and data journalists. I believe the same standards those writers observe should apply in quantitative art market analysis, too. Otherwise, ironically, the coverage commits the same sins of opacity as the industry it’s meant to (finally) help clarify with hard facts.