Replicação em economia


John Cochrane soltou um post bacana sobre replicação em economia. Vale a pena conferir.

On replication in economics. Just in time for bar-room discussions at the annual meetings.

“I have a truly marvelous demonstration of this proposition which this margin is too narrow to contain.” -Fermat

“I have a truly marvelous regression result, but I can’t show you the data and won’t even show you the computer program that produced the result” – Typical paper in economics and finance.

The problem 

Science demands transparency. Yet much research in economics and finance uses secret data. The journals publish results and conclusions, but the data and sometimes even the programs are not available for review or inspection.  Replication, even just checking what the author(s) did given their data, is getting harder.

Quite often, when one digs in, empirical results are nowhere near as strong as the papers make them out to be.

I have seen many examples of these problems, in papers published in top journals. Many facts that you think are facts are not facts. Yet as more and more papers use secret data, it’s getting harder and harder to know.

The solution is pretty obvious: to be considered peer-reviewed “scientific” research, authors should post their programs and data. If the world cannot see your lab methods, you have an anecdote, an undocumented claim, you don’t have research. An empirical paper without data and programs is like a theoretical paper without proofs.

(continue lendo no blog do Cochrane)

Links diversos: Discriminação de preços em passagens, vídeo Piketty na USP e replicação dos códigos dos artigos.


Seguem alguns links interessantes da semana:

– Vai comprar passagens? Então não busque somente no “site em português” da companhia aérea, mas também nas versões estrangeiras. Algumas vezes o preço pode ser mais barato, bem mais barato.  Para verificar se isso ocorre por aqui, simulei hoje uma passagem Brasília – Vitória para a virada do ano, tanto na versão em português quanto na versão em inglês do site da TAM. Resultado: no site em inglês a passagem está quase R$1.000,00 mais cara. Neste caso, o gringo que estiver no Brasil pode economizar bastante apenas mudando a linguagem da página, mas já houve relatos de o inverso acontecer (a passagem no site em inglês estar mais barata).

– Lembra que o Piketty estava pelo Brasil? Pois bem, para quem não conseguiu estar presente, agora o  vídeo do debate que ocorreu na USP, com André Lara Resende e Paulo Guedes, está disponível (via Prosa Econômica).

–  Desde 2005, o Quarterly Journal of Political Science solicita aos autores os dados e códigos necessários para a replicação de seus papers. Com isso, o periódico faz uma revisão bem básica: apenas roda o que foi enviado pelos autores – as is – e verifica se os resultados são os mesmos apresentados pelo artigo. Este processo simples tem valido a pena? Segundo Nicholas Eubank, sim:

Experience has shown the answer is an unambiguous “yes.” Of the 24 empirical papers subject to in-house replication review since September 2012, [1] only 4 packages required no modifications. Of the remaining 20 papers, 13 had code that would not execute without errors, 8 failed to include code for results that appeared in the paper, [2] and 7 failed to include installation directions for software dependencies. Most troubling, however, 13 (54 percent) had results in the paper that differed from those generated by the author’s own code. Some of these issues were relatively small — likely arising from rounding errors during transcription — but in other cases they involved incorrectly signed or mis-labeled regression coefficients, large errors in observation counts, and incorrect summary statistics. Frequently, these discrepancies required changes to full columns or tables of results. Moreover, Zachary Peskowitz, who served as the QJPS replication assistant from 2010 to 2012, reports similar levels of replication errors during his tenure as well. The extent of the issues — which occurred despite authors having been informed their packages would be subject to review — points to the necessity of this type of in-house interrogation of code prior to paper publication.

Fica a pergunta: quantos journals brasileiros fazem isso?

(via Dave Giles)