Jeff Smith on Reinhart and Rogoff

Ace labor economist Jeff Smith’s office is only a few doors down from mine. Hearing what he had to say about Reinhart and Rogoff in a hallway conversation, I suggested it would make a good blog post on his blog Econjeff. I was right: Jeff’s May 1, 2013 post about Reinhart and Rogoff is excellent. (That was aftermy April 20 mea culpa about relying on Reinhart and Rogoff, but well before my two columns with Yichuan Wang about Reinhart and Rogoff on May 29 and June 12.) Jeff kindly gave me permission to reprint that post here. 

My thoughts on the Reinhart and Rogoff coding error:

  1. I liked the piece by Greg Mankiw.
  2. I liked the piece by cyniconomics, which was linked to on MR.
  3. I liked the piece by my colleagues  Betsey Stevenson and Justin Wolfers 
  4. Journalists and other economists need to have a reasonable prior here. The Journal of Money, Credit and Banking replication exercise, summarized in this (gated) American Economic Review article and now more than two decades old, probably provides the best evidence. The authors of the summary article conclude “our findings suggest that inadvertent errors in published empirical articles are a commonplace rather than a rare occurrence.”  Some enterprising journal should repeat the exercise so that priors can be updated.
  5. It seems to me professional courtesy that if you replicate someone’s paper that has gotten a lot of media attention, and you find something that is likely to draw more media attention, you should send your findings to the authors first and give them a week or two to digest and respond before going to the media yourself. My understanding is that the UMass-Amherst folks did not do this. In my view, they should have.
  6. My final (and quite serious) suggestion is that Dan Hamermesh write one of his famous, and very useful, advice papers on the topic of the professional etiquette of replication. Economists do not replicate as much as they should. Part of that is too-low rewards, but part of it is also, I think, that things may go sour in a very public way, as they have in several high profile replications in recent years. Perhaps a formal statement of norms of good behavior in this domain would clarify expectations all around and also encourage more social pressure on both the replicators and the replicated to behave in ways that enhances the scientific credibility of the discipline.

Thanks to Miles Kimball for some encouragement.