Replicating economic research

Felix Salmon at portfolio.com has a post about a recent academic paper that documents how hard it is to replicated academic research - specifically in economics.

The paper he references, by B. D. McCullough and Ross McKitrick is pretty damning arguing that:

"In practice it is rare for scientists to make their data and code accessible, and it is rare for scientists to replicate one another's work, in part because it can be so difficult to get the data"

Although this all seems pretty bad for academic economics, several of the specific cases that are cited are well known. For example, one that is cited is the Boston Fed study on mortgage red lining. This study led to an expansion of mortgage lending, but turns out to be based on very flawed analysis. This case has been well known for a while and is often cited as one of the causes of the current subprime mess.

But the problem extends to other academic papers such as those appearing in the American Economic Review and the Journal of Money Credit and Banking. In both cases the authors found it pretty hard to replicate key studies, largely because the original authors wouldn't divulge programs or data.

So why don't economics (or finance) researchers just post their code on their websites in addition to their working papers?

The answer is simple, a significant amount of the research process is devoted to writing the programs that crunch the numbers. For a complex paper that appears in a top journal, there can be hundreds of hours devoted to coding. This code has the potential to be used for future research projects. For an empirical researcher, the statistical software analysis code that he or she writes is basically what pays the bills.

Having said this, I personally am always willing to help others replicate or extend my papers, because, among other reasons, that new author will cite my work. But its pretty rare that I will just hand out programs - they are my intellectual capital.