That is the monumental task that Doug North, John Wallis, and Barry Weingast have taken on in their forthcoming book "A Conceptual Framework for Interpreting Human History." The book is being discussed today and tomorrow at the Encounters with Authors series of Harvard's Institute for Quantitative Social Science.
The book's focus is on two fundamental modes of social organization, which the authors call the Natural State and the Open-Access Order. The first of these appeared about 10,000 years ago and characterizes not only most of recorded human history, but also most of today's countries. It is a system where a small group of elites--the authors emphasize that the multiplicity of elites is important--reach a modus vivendi whereby they agree to create and distribute rents among themselves.
Only about 25 countries and 15 percent of the world's population live in open-access societies, according to the authors. Open-access orders are based on competition--both in economics and in politics--and use the threat of entry to regulate social, economic, and political relations.
The authors emphasize that much of social science today goes astray because it tries to fit all countries under the same theoretical construct. Instead, they argue, you must understand that the logic of these two systems are fundamentally different.
I am not quite through with the manuscript, but what I am missing in the book so far (and what the first six chapters do not seem to contain--there is a seventh chapter which I have not seen) is a good discussion of what this framework explains that other theories do not--especially as regards current political and economic developments. In other words, why did we really need this book and its conceptual framework?
I am also uncertain about the validity of the sharp distinction that North et al. draw between their two modes of social order insofar as we are meant to apply this distinction to contemporary realities. I know that most developing societies have features that are different from those in the advanced democracies, but I am not sure I see a fundamentally different way of organizing society (instead of gradations). Their historical argument has also come in for a bit of criticism, notably by Niall Ferguson in his discussion of North et al.'s chapter on the evolution of property rights in medieval and early modern England.
What the authors do get right, I think, is that otherwise similar institutional arrangements will induce different behaviors and outcomes in different social orders. Doug North has been making this point for quite a while, and it is an especially important one to keep in mind in light of the current obsession with governance reforms and best-practices.