The Economist evaluates the randomized evaluators

One of the bon mots I remember from The Economist, from the days when I used to read it, is their definition of a radical: "someone we disagree with." Well, by that standard, I am less and less of a radical with almost every issue of the magazine.

The magazine's latest Economics Focus column reports on the Brookings development conference that I had to miss, but to which I contributed a paper. Their bottom line on the contribution (and limitations) of randomized field experiments follows the arguments in my paper fairly closely--down to the example of anti-malaria bed nets that I used in my paper to illustrate the problems of extending results in one specific setting (Western Kenya) to others.

The article quotes Abhijit Banerjee, who says

"the quality of the evidence that informs much of the macro-growth debates is significantly worse than the quality of the data that bears on many of the micro-policy questions”. He adds: “The beauty of randomised evaluations is that the results are what they are.” In other words, they provide hard evidence, resting on a solid empirical base.

The trouble is that the moment you take the experiment from Western Kenya and want to use it to inform policy in another setting, you need to make all kind of additional, not rigorously tested assumptions (about how similar or dissimilar the settings are and how these affect the likely outcomes).  By the time the evidence is used, it has become as "soft" as many other kinds of evidence that development economists traditionally rely on.

(And before someone writes to ask how I know about this article if I do not read The Economist, the reason is that I was asked by the author to check on a quote prior to publication. Plus, I regularly Google myself of course...)

Comments

[Gabriel]

I've never understand this argument against randomized trials. The argument you seem to be making is that when it comes to what's relevant to making policy, hard evidence doesn't exist at all--that all evidence is the same level of "softness." It's hard to believe you really think this--if all evidence was equally soft, how would you choose which evidence to look at?

Of course applying results from randomized trials to larger populations in the real world always requires some use of assumptions and softer information, so your solid-as-can-be evidence is transmuted into something softer, say, from granite into sandstone. This is true for social experiments and randomized medical trials as well. For this reason, the value of a single randomized trial is limited. But with a series of randomized trials in different circumstances that all show results in the same direction, you can have confidence that the measure will work in other circumstances as well.

In the case of conditional cash transfers, for example, there are now numerous randomized experiments from several countries that show these are an effective way to increase school attendance among the poor.

There are so many development programs that have been implemented with no evidence whatsoever that they'll do any good (I could name a half dozen that I've seen up close). At least with these programs that have been subjected to randomized trials, we have at least SOME hard evidence that the programs can work at least in some circumstances.

[OS]

Quite right, Gabriel. Take the deworming example. Randomized experiments have yielded similar findings in Kenya, India and the US South in the 1920s. So we have evidence from several contexts, plus plausible theories to underpin the empirical findings. Why is this still deficient relative to other approaches to development research? Certainly individual experiments shouldn't be hyped as absolute truth. But the naysayers seem to be applying the external validity critique 'ad absurdum'.

As for macro, as Ricardo Hausmann says about the growth literature, if your mountain of facts (ie, there is only data for 120 countries) is smaller than your mountain of theories (well over 120 growth theories), you have no more degrees of freedom.

I think Banerjee's observation about micro vs. macro is right on target...

[ben]

Gabriel, I think your arguments are quite right. However, the backlash against randomization is not arguing that randomization is not a good way to generate evidence, it's just that its proponents have taken it too far. There is a real sentiment out there that is gathering steam that only a RCT can produce "hard" evidence, and any other information is secondary. That kind of thing is what the backlash is about.

[Qingdao]

I would like to comment on the previous Project Syndicate article: I agree with the thrust of the article, but the sentence that jumped out at me was the reference to China. Certainly China has achieved impressive reductions in poverty, but most of that was done in a very short time in the late 70s , early 80s. Yasheng Huang focuses on this period in his new book (I have only read the summary), and argues that old fashioned microeconomic principles were the "secret." It certainly appears that now China is regressing.

[Qingdao]

I would like to comment on the previous Project Syndicate article: I agree with the thrust of the article, but the sentence that jumped out at me was the reference to China. Certainly China has achieved impressive reductions in poverty, but most of that was done in a very short time in the late 70s , early 80s. Yasheng Huang focuses on this period in his new book (I have only read the summary), and argues that old fashioned microeconomic principles were the "secret." It certainly appears that now China is regressing.

[BB]

dani's comment about extrapolation, at least within a country or a reasonably definable unit, is really just about ensuring that the experimental intervention is done on a sample representative of the population of interest. yes, to date most interventions have been done on quite selected samples for many reaons but that is not how they should be done.
as long as you have a random sample from the population of interest, you can always extrapolate your answers to that population. so really, dani's comment can equally be interpreted as "let's throw even more money and resources at the randomistas so that they can generate truly random samples on which to try their interventions"

[Justin Rietz]

Dani,

First, and most important: not read the Economist? What do you read instead?

Second, your position here is in stark contrast to your reliance on econometric models. In that case, it appears you are willing to overlook the inability to account for a myriad of factors that are either too numerous to include, impossible to accurately quantify and/or just plain not known (the "unknown unknowns").

[Dominic]

Dani,

Regularly googling yourself is so inefficient. You should set up a Google News Alert instead!!

This way, every day (or instantly if you prefer) you get an email notifying you whenever you name appears online.

Just google yourself using google news, then scroll down to the bottom, and click the link about google news alerts.

This technique also covers any time your name appears in a blog on line or in a web page. Sweet.