The Economist evaluates the randomized evaluators

One of the bon mots I remember from The Economist, from the days when I used to read it, is their definition of a radical: "someone we disagree with." Well, by that standard, I am less and less of a radical with almost every issue of the magazine.

The magazine's latest Economics Focus column reports on the Brookings development conference that I had to miss, but to which I contributed a paper. Their bottom line on the contribution (and limitations) of randomized field experiments follows the arguments in my paper fairly closely--down to the example of anti-malaria bed nets that I used in my paper to illustrate the problems of extending results in one specific setting (Western Kenya) to others.

The article quotes Abhijit Banerjee, who says

"the quality of the evidence that informs much of the macro-growth debates is significantly worse than the quality of the data that bears on many of the micro-policy questions”. He adds: “The beauty of randomised evaluations is that the results are what they are.” In other words, they provide hard evidence, resting on a solid empirical base.

The trouble is that the moment you take the experiment from Western Kenya and want to use it to inform policy in another setting, you need to make all kind of additional, not rigorously tested assumptions (about how similar or dissimilar the settings are and how these affect the likely outcomes).  By the time the evidence is used, it has become as "soft" as many other kinds of evidence that development economists traditionally rely on.

(And before someone writes to ask how I know about this article if I do not read The Economist, the reason is that I was asked by the author to check on a quote prior to publication. Plus, I regularly Google myself of course...)

Comments

[Gabriel]

I've never understand this argument against randomized trials. The argument you seem to be making is that when it comes to what's relevant to making policy, hard evidence doesn't exist at all--that all evidence is the same level of "softness." It's hard to believe you really think this--if all evidence was equally soft, how would you choose which evidence to look at?

Of course applying results from randomized trials to larger populations in the real world always requires some use of assumptions and softer information, so your solid-as-can-be evidence is transmuted into something softer, say, from granite into sandstone. This is true for social experiments and randomized medical trials as well. For this reason, the value of a single randomized trial is limited. But with a series of randomized trials in different circumstances that all show results in the same direction, you can have confidence that the measure will work in other circumstances as well.

In the case of conditional cash transfers, for example, there are now numerous randomized experiments from several countries that show these are an effective way to increase school attendance among the poor.

There are so many development programs that have been implemented with no evidence whatsoever that they'll do any good (I could name a half dozen that I've seen up close). At least with these programs that have been subjected to randomized trials, we have at least SOME hard evidence that the programs can work at least in some circumstances.

[OS]

Quite right, Gabriel. Take the deworming example. Randomized experiments have yielded similar findings in Kenya, India and the US South in the 1920s. So we have evidence from several contexts, plus plausible theories to underpin the empirical findings. Why is this still deficient relative to other approaches to development research? Certainly individual experiments shouldn't be hyped as absolute truth. But the naysayers seem to be applying the external validity critique 'ad absurdum'.

As for macro, as Ricardo Hausmann says about the growth literature, if your mountain of facts (ie, there is only data for 120 countries) is smaller than your mountain of theories (well over 120 growth theories), you have no more degrees of freedom.

I think Banerjee's observation about micro vs. macro is right on target...

[ben]

Gabriel, I think your arguments are quite right. However, the backlash against randomization is not arguing that randomization is not a good way to generate evidence, it's just that its proponents have taken it too far. There is a real sentiment out there that is gathering steam that only a RCT can produce "hard" evidence, and any other information is secondary. That kind of thing is what the backlash is about.

[Qingdao]

I would like to comment on the previous Project Syndicate article: I agree with the thrust of the article, but the sentence that jumped out at me was the reference to China. Certainly China has achieved impressive reductions in poverty, but most of that was done in a very short time in the late 70s , early 80s. Yasheng Huang focuses on this period in his new book (I have only read the summary), and argues that old fashioned microeconomic principles were the "secret." It certainly appears that now China is regressing.

[Qingdao]

I would like to comment on the previous Project Syndicate article: I agree with the thrust of the article, but the sentence that jumped out at me was the reference to China. Certainly China has achieved impressive reductions in poverty, but most of that was done in a very short time in the late 70s , early 80s. Yasheng Huang focuses on this period in his new book (I have only read the summary), and argues that old fashioned microeconomic principles were the "secret." It certainly appears that now China is regressing.

[BB]

dani's comment about extrapolation, at least within a country or a reasonably definable unit, is really just about ensuring that the experimental intervention is done on a sample representative of the population of interest. yes, to date most interventions have been done on quite selected samples for many reaons but that is not how they should be done.
as long as you have a random sample from the population of interest, you can always extrapolate your answers to that population. so really, dani's comment can equally be interpreted as "let's throw even more money and resources at the randomistas so that they can generate truly random samples on which to try their interventions"

[Justin Rietz]

Dani,

First, and most important: not read the Economist? What do you read instead?

Second, your position here is in stark contrast to your reliance on econometric models. In that case, it appears you are willing to overlook the inability to account for a myriad of factors that are either too numerous to include, impossible to accurately quantify and/or just plain not known (the "unknown unknowns").

[Dominic]

Dani,

Regularly googling yourself is so inefficient. You should set up a Google News Alert instead!!

This way, every day (or instantly if you prefer) you get an email notifying you whenever you name appears online.

Just google yourself using google news, then scroll down to the bottom, and click the link about google news alerts.

This technique also covers any time your name appears in a blog on line or in a web page. Sweet.

[nike shox sale]

Nike Shox|Nike Shox Sale is special for you with a very pretty design

[Ugg london]

Don't waste your time on a man/woman, who isn't willing to waste their time on you.

[chi flat iron]

CHI flat iron by Farouk system. Direct from the manufacturer, this genuine Chi ceramic iron comes with valid, one year warranty!

[rado watches]

Why is everyone just montblanc watches willing to accept power cuts? Don't you think that patek philippe watches having continious power is your right? If people aren't going to demand rado watches 24X7 power, don't expect anything zenith watches to change. The government needs to look at other sources parmigiani watches of power generation. The only solution is more power panerai watches production. Nothing less.
http://www.watchvisa.com/montblanc-watches.html
http://www.watchvisa.com/patek_philippe-watches.html
http://www.watchvisa.com/rado-watches.html
http://www.watchvisa.com/tudor-watches.html

[kobe]

Thank you! This article gives me inspiration!

Hope in the study to you! Sorry, the

advertisement!
http://www.christian-louboutin-sandals.com
http://www.jp-brand-jp.com
http://www.showjordanshoes.com

[rolex watches]

People usually say :"Seeing is believing." http://www.tt88times.com
Each attempt has a corresponding gain, in part or obvious, or vague. At least we have the kind of satisfaction After I bought this watch ,in a sense,it means a great deal to me. http://www.fashionhairfu.com

[ghd straighteners]

GHD straighteners was kmown as ghd flat iron, which was authorized online seller provides all kinds of hair straighteners,pink ghd,purple ghd,babyliss. By visiting ghd australia , you will find what you want and made yourself more beautiful.If you miss it ,you miss beauty.Buy a piece of ghd for yourself.Come and join us http://www.ghdoutlet-au.com/ to win the ghd iv styler.

[birkenstock]

Birkenstock was Made in Germany since 1774 . Check out our Birkenstock sandals and Birkenstock shoes including the Birkenstock gizeh,at the lowest regular outlet prices, free shipping and when you put on Birkenstocks. you will feel very comfortable.

[Bell & Ross]

Amazing stuff,Thanks so much for this!This is very useful post for me. This will absolutely going to help me in my projects .

[who called me]

Your post gives useful and sensible information for someone who is thinking of venturing into home based business enterprise.thanks for your great work

[Acronym List]

I haven't really noticed much of this new intellectual climate here amongst development practitioners in Africa. But perhaps that is because there aren't many intellectuals here.

[free web directory]

DMOZ4U is a quality Free Web Directory, human edited Free Web Directory with quality internet resources. Submit free link, or find websites of your choice.

[Cheap Louis Vuitton Handbags]

In any case I just wanted you to know that I, like others, think you've already breathed new life and new challenges into in particular Mark Thoma's, Brad DeLong's, and David Altig's bloging. Not that they weren't doing a great job before.

[Cheap Louis Vuitton]

In any case I just wanted you to know that I, like others, think you've already breathed new life and new challenges into in particular Mark Thoma's, Brad DeLong's, and David Altig's bloging. Not that they weren't doing a great job before.

[lachou]

I like these articles , democratic countries formulate policies are both team and the interests of the people of power, but the undemocratic countries rarely for the interests of the people, http://www.jerseysky.com it is the sorrow of many democracy, I like the western and northern state policy.
http://www.cheapsaleing.com
http://www.nikeairmaxshoe.ocm

[Moncler]

I am a senior student too, I Love this post as i love study english, This article is to my feelings, thanks for your sharing, hoping to see more useful content, refueling

[Moncler Clothing]

I am a senior student too, I Love this post as i love study english, This article is to my feelings, thanks for your sharing, hoping to see more useful content, refueling

[jordan shoes]

This is a wonderful site. The things mentioned are unanimous and needs to be appreciated by everyone.

[jordan shoes]

This is a wonderful site. The things mentioned are unanimous and needs to be appreciated by everyone.

[cheap jerseys]

hey buddy,this is one of the best posts that I’ve ever seen; you may include some more ideas in the same theme. I’m still waiting for some interesting thoughts from your side in your next post.

[Moncler]

am a senior student too, I Love this post as i love study english, This article is to my feelings, thanks for your sharing

[AF]

Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming.

[NFL Football Jerseys]

If I speak in the Cheap Authentic NFL Jerseys tongues of men and of angels, but have not love, I am only a resounding gong or a clanging cymbal. If I have the gift of prophecy and can fathom all mysteries and all Official NFL Jerseys knowledge, and if I have a faith that can move mountains, but have not love, I am nothing. If I give all I possess to the poor and surrender my USA Jerseys Shop body to the flames,but have not love, I gain nothing.Love is patient, NFL Jerseys love is kind. It does not envy, it does not boast, it is not proud. It is not rude, it is not self-seeking, it is not easily angered, it NHL Jerseys keeps no record of wrongs. Love does not delight in evil but rejoices with the truth. It always protects, always trusts, always MLB Jerseys hopes, always perseveres.Love never NHL Hockey Jerseys fails. But where there are prophecies, they will cease; where there are tongues, they will be stilled; where there is knowledge, it will pass away. For we know in part and we prophesy in MLB Baseball Jerseys part, but when perfection comes, the imperfect disappea. When I was a child, I talked like a child, I thought like a child, I reasoned like a child. When I became NBA Basketball Jerseys man, I put childish ways behind me. Now we see but a poor reflection as in a mirror; then we shall see face to face. Now I know in part; then I shall know fully, even as I am fully known.And now these three remain: faith, hope and NFL Football Jerseys love. But the greatest of these is love. http://www.jersey-usa.com/sitemap.html LIJ

[viagra online]

I just came across this and loved your idea of focusing on a different aspect for each day of the week! Thank you so much for sharing this and allowing us to copy it as well! I think this will be a lovely addition to our morning prayer time as well!

[cheap clothing]

Although all gets out of bed incapable consonant in has according to may look up takes medicine, the vowels fell into the weekend horridness, the even meteorological observatory also participated in a prediction: tonight to tomorrow, will have a consonant storm.

[pandora-bracelets-australia]

I really enjoyed this too!

[pandora-bracelets-australia]

It's in the question itself. Of course, people can't afford if the price is too high for the minimum food they eat.

[buy pandora]

There is however no compromise on emailing and web surfing capabilities and can connect to cellular broad band wireless providers.

[Account Deleted]

raf

Thanks.

[Ohioguide]

columbus injury lawyer
I always enjoy reading your posts and I look forward to your next one.

[Mbtmbtshoes]

"the quality of the evidence that informs much of the macro-growth debates is significantly worse than the quality of the data that bears on many of the micro-policy questions”. He adds: “The beauty of
http://www.mbtusafootwear.com

[Account Deleted]

http://www.hermesfantasy.com/

[Account Deleted]

http://www.ghdstyle-au.com/ghds.html
http://www.chihairflatirons.us
http://www.ghdsoutlet.com
http://www.googleghdhair.com

[Account Deleted]

Many places and centers offer business and trade promotions to both buyers and supplier.What about the differences in skill intensities across industries? The job losses in the relatively unskilled-labor intensive battery industry should have little effect on the relatively skilled-labor intensive machinery
sexshop
sexyshop
sexshop online
alongador peniano

[Account Deleted]

You made some good points .I did a little research on the topic and found that most people agree with your blog. Thanks and appreciate to you for your Brilliant effort.

organic seo service

[Account Deleted]

http://seotoolinfo.com/2012/06/creating-history-creating-history-together-with-arenabetting-piala-eropa-2012/

http://seotoolinfo.com/2012/06/agenliga-agenliga-com-sebagai-agen-bola-online-untuk-euro-2012/

[Account Deleted]

Awesome, Very interesting issue that after a very successful. Nicely written article good work ! Thanks for the great piece of sharing. Enjoy to read your write .
Creating History Together With ArenaBetting Piala Eropa 2012 http://seotoolinfo.com/2012/06/creating-history-creating-history-together-with-arenabetting-piala-eropa-2012/
Agenliga Sebagai Agen Bola Online Untuk Euro 2012 http://seotoolinfo.com/2012/06/agenliga-agenliga-com-sebagai-agen-bola-online-untuk-euro-2012/

[Rastia Rigie]

Bank Mandiri Bank Terbaik di Indonesia
Bank Mandiri Bank Terbaik di Indonesia
Promo Bonus 100% Sbobet, Ibcbet dan Casino Online Dalam Rangka HUT BAVETLINE Yang Kedua
Masterbet88 Grand Opening Promo Bonus 50% Sportsbook dan Casino Online