« Trade and wages | Main | Second-best institutions »

January 07, 2008


Chris Blattman

One could reframe this comment as the presumed trade-off between internal and external validity--the tension between identifying a precise and accurate relationship and identifying something that is generalizable.

The quest for internal validity has admittedly become something like a fetish among academic economists. In part this is because it offers a chance to demonstrate cleverness and technical sophistication (we all want to be seen as clever!), but also because accuracy is a valuable thing in and of itself.

It is also a reaction, I think, to the (historic) lack of emphasis on internal validity in much of the cross-country and case study literature of the 1990s. I have not written off either method, and indeed practice both (in spite of concentrating on the more micro-level analysis). When sufficient attention and care is given to the challenge of understanding causation and bias in the case study and cross-country literature, the findings are useful and highly regarded.

To bring back these methods into the mainstream, I think it's incumbent upon the practitioners to take greater care with their research design; to explicitly discuss any challenges with causal inferences; to improve the quality of data; and to discourage or even shun the work that does not do so. In short, we must raise the bar.

For example, in cross-country work, more attention could be paid to identifying exogenous sources of variation in a single (well-measured) independent variable of interest and examining its effects, rather than regressing the variable of interest on a kitchen sink of endogenous proxies. Where such research has been done, the general profession has consumed it with relish. The AJR (Acemoglu-Johnson-Robinson)triumvirate is just the best known example. there are many more.

In case studies, care can also be given to counterfactuals and other aspects of research design. Dan Posner's work with arbitrary borders cleaving ethnic groups in two comes to mind.

Another promising move in this direction has been work by scholars like Gary King, Alberto Abadie, Thad Dunning, and others to develop approaches and methods to improve statistical inference in the case study.

In short, validity does not have to be a zero sum game. More obvious effort in this direction in the cross-country and case literature will make the pendulum swing back towards generalizable research faster (and better than before).


I'd be very interested in reading the specific criticisms of Deaton. Is there a written piece where he states these critiques? Or could you paraphrase for us what he said?

The current fad with randomized experiment seems to go too far. It's virtually impossible to get anything pass the editors of JDE if you don't have some sort of an experiment. Or ask anyone who tried to submit a paper to last year's NEUDC with an empirical microeconomic focus without an experiment.


Dear Sherwood:

I have not checked if Deaton has anything related posted on his website, but others in the session mentioned that he had made similar comments elsewhere. In essence, he made the observation that we are bound to get mixed evidence (sometimes, not because the studies or the analysis is wrong, but because they are contextual). He also remarked that sometimes theory ahead of measurement can help to sort things out first.

Here is a related paper by Heckman:



well said, but it's telling that there is no link to a 'good' country case study.

murat, could you give us an example of a case study that should be published along the micro-papers that you cite? which are the criteria such a case study should satisfy?

Dani Rodrik

I don't think identification can ever be as clean in macro or case studies as in micro/experimental studies. But the relevant question is whether the former type of study can move our priors about important questions in economics. Anyone who thinks not has not read widely. I would point to the country studies in my "In Search of Prosperity" as an example. So the kind of tyranny Murat complains about does prevent useful knowledge from accmulating.


I do not think that there is a methodological consensus in development economics. I do not find true the assertion about the JDE. This is factual. Just look at the papers published there.
More importantly, it is true that we do not want a collection of results. We need to start to generalize from the experimental agenda. We need to learn more about causal surfaces as proposed by Rubin. However, that does not mean that these papers are not useful. Much to the contrary, some of them (like in all literatures) are very useful.
It is true that we should not request randomization as a bar to publish a paper. That is silly in many ways. There are lots of causing forces that are not likely to be randomized in practice. Sometimes randomized studies distort the causing factors in ways that the study ends up not being easily generalizable, and other identification strategies might led us to learn more than in such randomized trials.
Still, we need to keep searching for sources of identification of the parameters of interest. A randomized trial is an important tool. IV and longitudinal studies are other important tools. And, case studies or analytical narratives could also be useful tools when masterfully used.
One last comment: It is unfair to say that randomized trials are not useful because the collection of them leads to different results. Is not that always the case? Just pay a look at the superb book on Labor Demand by Dan Hamermesh.


This discussion interests me right now because I am in the final stages of a large-scale commissioned lit review on a particular topic in development econ. I have plowed through upwards of 150 papers, a large percentage in the form of natural experiments, quasi-experiments, and very clever pseudo-experiments. Of course, they come to very different conclusions. I am supposed to make sense of all this, and that's not really possible.

More than ever, I think economists are misled by what they think they should be looking for, which is ultimately attributable to a mistaken notion of the form that "science" (natural science) generally takes.

Wrong expectation: to find general "laws" or relationships that hold in the absence of countervailing circumstances -- generalizations to be modified by local conditions. This reflects an orientation to the physical sciences, which largely work this way (exception, perhaps: geology).

Right expectation: to find mechanisms, channels of causation by which factors within a system influence each other. This would provide the basis for going from a description of a local configuration to a set of analytical expectations -- an inductive rather than deductive methodology. In the world of science, think of the biological sciences, to which economics has the closest kinship.

I'm supposed to weigh the evidence and come down on the side of certain generalizations, but I don't intend to do this. I want to provide analytical tools for people who are interested in drawing out the implications of specific social-economic-political-cultural situations.



Thank you for the pointer towards the country case studies. I agree, those case studies can definitely move priors and that's what research is ultimately about. However, I think it is an issue that there is no clear set of methodological rules that allows an (economics) referee to determine the quality of a case study. As Chris points out, that's an active area of research in econometrics but we don't seem to have a consensus yet. I'm not an editor, but I guess that this lack of consensus on methodology is a real hurdle for such papers.

Ugg london

Don't waste your time on a man/woman, who isn't willing to waste their time on you.

chi flat iron

CHI flat iron by Farouk system. Direct from the manufacturer, this genuine Chi ceramic iron comes with valid, one year warranty!

chi hair straightener

CHI flat iron by Farouk system. Direct from the manufacturer, this genuine Chi ceramic iron comes with valid, one year warranty!

cheap music cds

There are many cd shops around. The collection makes people to attract and buy the cheap music cds available here.

rolex watches

People usually say :"Seeing is believing." http://www.tt88times.com
Each attempt has a corresponding gain, in part or obvious, or vague. At least we have the kind of satisfaction After I bought this watch ,in a sense,it means a great deal to me. http://www.fashionhairfu.com


I like these articles , democratic countries formulate policies are both team and the interests of the people of power, but the undemocratic countries rarely for the interests of the people, http://www.jerseysky.com it is the sorrow of many democracy, I like the western and northern state policy.

jordan shoes

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing.

Account Deleted


Thanks for article.

Account Deleted

On the one hand, such cycles and transitions are a normal and required component of the process of scientific advance: China Wholesale Beadswe make observations, formulate hypotheses, adopt a methodology and test ideas.

Account Deleted


Thaks for comment.

Account Deleted


Thanks to you ...

Account Deleted


Thanks for this ...

Account Deleted

Many places and centers offer business and trade promotions to both buyers and supplier.What about the differences in skill intensities across industries? The job losses in the relatively unskilled-labor intensive battery industry should have little effect on the relatively skilled-labor intensive machinery
sexshop online

Account Deleted

Amazing!I also wish him good luck to defend his gold medal. I like to share it with all my friends and hope they will also encourage him.

organic seo service

Account Deleted

fix powerpoint

Good Post.It's really helpful & interesting


Thanks for this!


Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)