The tyranny of methodological consensus in development economics

by Murat Iyigun, guest blogger

Listening to Angus Deaton’s dead-on critique of the current state of the literature on health and development at an AEA session this past weekend, I could not help but think about the costs and benefits of methodology fads in our profession in general, and growth & development economics in particular.

As Dani has blogged about in the past, there has been fascinating progress in the field of development in the past two decades. In the late-1980s and 1990s, macroeconomics-oriented, cross-country growth analyses ruled the day (see here and here, for instance). But due to various credible methodological concerns about simultaneity, reverse causality and omitted variables biases, this strand began to recede into the dusty, public-access portion of JSTOR. These days, the economic development literature is dominated by the microeconomics-based, randomized experiment studies (to take two very imaginative and useful applications, see this and that).

On the one hand, such cycles and transitions are a normal and required component of the process of scientific advance: we make observations, formulate hypotheses, adopt a methodology and test ideas. As scholarly scrutiny exposes the deficiencies, we revise, modify and retest. On the other hand, I am of the view that there are significant and dangerous intellectual costs associated with the compartmentalized and disjoint manner in which development economics has evolved.

Reading the current literature in the field, one could be forgiven to think that there is nothing to learn from a study unless it brings microeconomic evidence at the most disaggregated level (the title of my current project: “Corruption, Institutions and Growth: Evidence from Grandma’s Bingo Circuit”). Of course, fads come and go because there are decreasing returns in many things. However, the challenge for the existing randomized-experiment, micro-development literature runs deeper than that.

Part of the problem is a fundamental incompatibility in what makes this literature really valuable and the quirks of academia and publication recognition: Top-quality journals publish research that has broad and general implications, whereas the real benefit of the “evidence from this—evidence from that” and “Look ma! No hands” studies is the implicit recognition that “results may vary” depending on local conditions. The proponents of this school are typically quick to argue that, as evidence accumulates, we shall be able to reach more robust and general conclusions. But as research accumulates, we are more often than not likely to end up with credible arguments and evidence on both sides of the debate (see the current health and development literature). And as village-by-village evidence trickles in, does your benevolent social planner promise to hold structural change at bay ?

In this regard, why are country case studies not as valuable as the randomized experiment studies in development economics and policymaking? The success of the randomized-experiment literature in becoming so popular and successful in academia can only be explained by the perception that it lends itself to generalizations.

Specialization and costly, time-consuming investment in tools and methods are an integral part of academic research. But such specialization and vested interests should not blind us to the pros and cons of multiple approaches (even if some of those approaches have revealed their flaws and others are a hard sell). While the incentive system in academia makes this a difficult proposition, I do not think we can keep ignoring it in development economics policymaking.

Comments

[Chris Blattman]

One could reframe this comment as the presumed trade-off between internal and external validity--the tension between identifying a precise and accurate relationship and identifying something that is generalizable.

The quest for internal validity has admittedly become something like a fetish among academic economists. In part this is because it offers a chance to demonstrate cleverness and technical sophistication (we all want to be seen as clever!), but also because accuracy is a valuable thing in and of itself.

It is also a reaction, I think, to the (historic) lack of emphasis on internal validity in much of the cross-country and case study literature of the 1990s. I have not written off either method, and indeed practice both (in spite of concentrating on the more micro-level analysis). When sufficient attention and care is given to the challenge of understanding causation and bias in the case study and cross-country literature, the findings are useful and highly regarded.

To bring back these methods into the mainstream, I think it's incumbent upon the practitioners to take greater care with their research design; to explicitly discuss any challenges with causal inferences; to improve the quality of data; and to discourage or even shun the work that does not do so. In short, we must raise the bar.

For example, in cross-country work, more attention could be paid to identifying exogenous sources of variation in a single (well-measured) independent variable of interest and examining its effects, rather than regressing the variable of interest on a kitchen sink of endogenous proxies. Where such research has been done, the general profession has consumed it with relish. The AJR (Acemoglu-Johnson-Robinson)triumvirate is just the best known example. there are many more.

In case studies, care can also be given to counterfactuals and other aspects of research design. Dan Posner's work with arbitrary borders cleaving ethnic groups in two comes to mind.

Another promising move in this direction has been work by scholars like Gary King, Alberto Abadie, Thad Dunning, and others to develop approaches and methods to improve statistical inference in the case study.

In short, validity does not have to be a zero sum game. More obvious effort in this direction in the cross-country and case literature will make the pendulum swing back towards generalizable research faster (and better than before).

[sherwood]

I'd be very interested in reading the specific criticisms of Deaton. Is there a written piece where he states these critiques? Or could you paraphrase for us what he said?

The current fad with randomized experiment seems to go too far. It's virtually impossible to get anything pass the editors of JDE if you don't have some sort of an experiment. Or ask anyone who tried to submit a paper to last year's NEUDC with an empirical microeconomic focus without an experiment.

[Murat]

Dear Sherwood:

I have not checked if Deaton has anything related posted on his website, but others in the session mentioned that he had made similar comments elsewhere. In essence, he made the observation that we are bound to get mixed evidence (sometimes, not because the studies or the analysis is wrong, but because they are contextual). He also remarked that sometimes theory ahead of measurement can help to sort things out first.

Here is a related paper by Heckman:

http://www.stat.columbia.edu/~gelman/stuff_for_blog/heckman2.pdf

[conundrum]

well said, but it's telling that there is no link to a 'good' country case study.

murat, could you give us an example of a case study that should be published along the micro-papers that you cite? which are the criteria such a case study should satisfy?

[Dani Rodrik]

I don't think identification can ever be as clean in macro or case studies as in micro/experimental studies. But the relevant question is whether the former type of study can move our priors about important questions in economics. Anyone who thinks not has not read widely. I would point to the country studies in my "In Search of Prosperity" as an example. So the kind of tyranny Murat complains about does prevent useful knowledge from accmulating.

[Sebastian]

I do not think that there is a methodological consensus in development economics. I do not find true the assertion about the JDE. This is factual. Just look at the papers published there.
More importantly, it is true that we do not want a collection of results. We need to start to generalize from the experimental agenda. We need to learn more about causal surfaces as proposed by Rubin. However, that does not mean that these papers are not useful. Much to the contrary, some of them (like in all literatures) are very useful.
It is true that we should not request randomization as a bar to publish a paper. That is silly in many ways. There are lots of causing forces that are not likely to be randomized in practice. Sometimes randomized studies distort the causing factors in ways that the study ends up not being easily generalizable, and other identification strategies might led us to learn more than in such randomized trials.
Still, we need to keep searching for sources of identification of the parameters of interest. A randomized trial is an important tool. IV and longitudinal studies are other important tools. And, case studies or analytical narratives could also be useful tools when masterfully used.
One last comment: It is unfair to say that randomized trials are not useful because the collection of them leads to different results. Is not that always the case? Just pay a look at the superb book on Labor Demand by Dan Hamermesh.

[Peter]

This discussion interests me right now because I am in the final stages of a large-scale commissioned lit review on a particular topic in development econ. I have plowed through upwards of 150 papers, a large percentage in the form of natural experiments, quasi-experiments, and very clever pseudo-experiments. Of course, they come to very different conclusions. I am supposed to make sense of all this, and that's not really possible.

More than ever, I think economists are misled by what they think they should be looking for, which is ultimately attributable to a mistaken notion of the form that "science" (natural science) generally takes.

Wrong expectation: to find general "laws" or relationships that hold in the absence of countervailing circumstances -- generalizations to be modified by local conditions. This reflects an orientation to the physical sciences, which largely work this way (exception, perhaps: geology).

Right expectation: to find mechanisms, channels of causation by which factors within a system influence each other. This would provide the basis for going from a description of a local configuration to a set of analytical expectations -- an inductive rather than deductive methodology. In the world of science, think of the biological sciences, to which economics has the closest kinship.

I'm supposed to weigh the evidence and come down on the side of certain generalizations, but I don't intend to do this. I want to provide analytical tools for people who are interested in drawing out the implications of specific social-economic-political-cultural situations.

[conundrum]

Dani,

Thank you for the pointer towards the country case studies. I agree, those case studies can definitely move priors and that's what research is ultimately about. However, I think it is an issue that there is no clear set of methodological rules that allows an (economics) referee to determine the quality of a case study. As Chris points out, that's an active area of research in econometrics but we don't seem to have a consensus yet. I'm not an editor, but I guess that this lack of consensus on methodology is a real hurdle for such papers.

[Ugg london]

Don't waste your time on a man/woman, who isn't willing to waste their time on you.

[chi flat iron]

CHI flat iron by Farouk system. Direct from the manufacturer, this genuine Chi ceramic iron comes with valid, one year warranty!

[chi hair straightener]

CHI flat iron by Farouk system. Direct from the manufacturer, this genuine Chi ceramic iron comes with valid, one year warranty!

[cheap music cds]

There are many cd shops around. The collection makes people to attract and buy the cheap music cds available here.

[rolex watches]

People usually say :"Seeing is believing." http://www.tt88times.com
Each attempt has a corresponding gain, in part or obvious, or vague. At least we have the kind of satisfaction After I bought this watch ,in a sense,it means a great deal to me. http://www.fashionhairfu.com

[lachou]

I like these articles , democratic countries formulate policies are both team and the interests of the people of power, but the undemocratic countries rarely for the interests of the people, http://www.jerseysky.com it is the sorrow of many democracy, I like the western and northern state policy.
http://www.cheapsaleing.com
http://www.nikeairmaxshoe.ocm

[jordan shoes]

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing.

[Account Deleted]

raf

Thanks for article.

[Account Deleted]

On the one hand, such cycles and transitions are a normal and required component of the process of scientific advance: China Wholesale Beadswe make observations, formulate hypotheses, adopt a methodology and test ideas.

[Account Deleted]

Raf
raf

Thaks for comment.

[Account Deleted]

Raf
raf

Thanks to you ...

[Account Deleted]

Raf
raf

Thanks for this ...

[Account Deleted]

Many places and centers offer business and trade promotions to both buyers and supplier.What about the differences in skill intensities across industries? The job losses in the relatively unskilled-labor intensive battery industry should have little effect on the relatively skilled-labor intensive machinery
sexshop
sexyshop
sexshop online

[Account Deleted]

Amazing!I also wish him good luck to defend his gold medal. I like to share it with all my friends and hope they will also encourage him.

organic seo service

[Account Deleted]

fix powerpoint

Good Post.It's really helpful & interesting

[SellHouseNash]

Thanks for this!

SELL YOUR HOUSE NASHVILLE