Scholarship vs. bloggership

Some years ago I served on a committee at the Kennedy School charged with evaluating the quantity and quality of the School's published output. One of the exercises I remember doing is to correlate the scholarly output of individual faculty members with their op-ed and similar publications. My hypothesis was that if we were doing our job right those who wrote the most op-eds would be also the most prolific scholars. Otherwise what authority would the op-eds really have?

I will not reveal the results of that exercise, but it recently gave me an idea for a similar analysis of the economics blogosphere. The question is: what is the relationship between a blogger's popularity and his scholarly impact? (As far as I can tell we are all males strangely enough.) So I asked my assistant Dana Brudowsky to do some research.

We obtained rankings of blog popularity from Aaron Schiff's site, which are in turn based on Technorati data. We based our rankings of scholarship on Google citation data, using Publish or Perish. Publish or Perish is a software that retrieves Google Scholar citation data and computes a number of indices based on them. We used the simplest among them, namely the total number of citations to an economist's total body of work, to rank economist bloggers in terms of impact of scholarship. These citation totals range from 35,835 (in the case of Gary Becker) to a low of 3. (Yes, I am aware of all the problems with this... Believe me, that is not how we make promotion decisions at Harvard, even though we certainly look at the numbers.)

We limited our search to economists with a Ph.D. and who seem to have a university affiliation. When a blog has multiple authors, we gave each author the same blog rank (but of course different scholarship ranks based on Google citation data). Obviously, we could not include blogs that are anonymous. By the end of the day, we had identified about 100 blogging economists with footprints in Google scholar.

And the result? Going into this, my expectation was that blog popularity and scholarship would have little (or perhaps even a negative) correlation. After all, the skills of a blogger (writing quickly and well, working for short-term results, spending a lot of time reading and digesting others' work) are not necessarily those that a scholar who wants long-term impact needs to have. Plus, there is the time spent on the blog--which does mean less time for research. Remember the Acemoglu response: I am too darn busy writing research papers... And one can certainly be an excellent and popular blogger--providing stimulating commentary on others' work--without having large scholarly output or high impact.

And yet the correlation between how well one does on bloggership and on scholarship turns out to be positive and statistically highly significant. The rank correlation between the two is 0.27, and it is significant at the 99% level of confidence. Here is the scatter plot:

Wait a minute, you say, this does not look like a very strong positive relationship--and you are right. Even though it is statistically significant, the quantitative significance is not that impressive. In fact, if one excludes the top 10 scholars the correlation is no longer statistically significant at conventional levels. Here is the data for those who want to play with it, or check for (and report) errors.

One way to interpret the results is to say that high-impact scholarship appears to be a sufficient but not necessary condition for successful bloggership. Once one leaves out the very top scholars, there is very little relationship between scholarly impact and popularity as a blogger. (I should add, to cover myself against the statistics police, that these statements are conditional on having a blog.) Cyberspace creates its own pecking order. 

And now to the real puzzle that this little research project left me with: why are there (apparently) no women economics bloggers?

UPDATE: OK, I think I have overlooked two women economist bloggers who fit the criteria: Lynne Kiesling and Chiara Lombardini-Riipinen. I am sure that I have overlooked many other male ones as well (including Craig Newmark, who submitted a comment below). Craig Newmark also points to some anomalies in the Google Scholar citation data.  Some of these are readily explained. The number of papers seems inflated because Google Scholar treats different versions of each paper as a separate "publication." So a paper circulated under two working paper series counts as two papers.  On the other hand, I have no idea what "years" (which comes from Publish or Perish) is referring to.  In any case, I believe the citation data that I use is unaffected by either of these, since it is the sum of citations to individual "publications," with no overlap. 

UPDATE2: See further clarification on Google Scholar and Publish or Perish here.

Comments

[Dainius from Lithuania]

I am just starting to dig into scholarly work on blogging for my phd., but it seems there is general trend in political and filter (providing mainly links to other sites) blogs to be written by white, highly educated males living in US.

But articles finding this trend is based on 2004-2005 data.

[robertdfeinman]

Two comments:
1. "I will not reveal the results of that exercise,..."

Why not? The only conclusion to be drawn is that the results put much of the faculty in a bad light.

2. You don't explain your methodology in detail, but I've observed that what passes for "scholarly" publication in economics differs from what is expected in other fields. Specifically there is a great deal of non-refereed material published. To my mind this should not be given the same weight as work that has been properly vetted. Even refereed material seems to have problems since there seems to be some partisanship at work in various journals as to what gets accepted. Papers that go against the journal's ideology get refused.

I did some professional work on citation analysis at one time and there are many pitfalls. In highly specialized areas the number of links tend to be very small, but they are all highly relevant. In other areas people all tend to cite some key papers, but the connection is rather weak and the citations are mostly padding or acknowledging the history of the discipline.

What this all means is that counting the number of times a paper is cited doesn't tell you much about its relevance.

As to the connection to blogging, it has been my observation that the more scholarly and fact-based a blog entry is, the less traffic it generates. Blogging is the electronic version of standing on a soap box. Rhetoric and timeliness are favored more than well reasoned postings. Bloggers who can provide reassurance to their adherents are also favored. People don't like their beliefs challenged and gravitate to sites where their views are supported.

[ZH]

There is one well known female economics blogger, Megan McArdle, although she is not an academic economist with a PhD. However, there is Lynne Kiesling of Northwestern (Knowledge Problem) if you are looking for an academic. Still, two female econ bloggers is a very small percentage of econ blogs.

[Idaho_Spud]

Scholarship and informational and/or writing style for entertainment value are not one and the same.

[anonymous]

Most status rendering research in academia has nothing to do with anything that would show up in an Op-Ed piece, not to mention a blog. And I'd even say that _most_ status rendering research done by economists or Kennedy schooers has little to do with the genuine advancement of social science -- and contributes little of practical value in terms of general knowledge. Most economics involves further development of degenerate research programs, i.e. failed science. And research shows that most of it is little read or never read again after 5 years or so.

>>My hypothesis was that if we were doing our job right those who wrote the most op-eds would be also the most prolific scholars. Otherwise what authority would the op-eds really have?<<

[anonymous]

Most status rendering research in academia has nothing to do with anything that would show up in an Op-Ed piece, not to mention a blog. And I'd even say that _most_ status rendering research done by economists or Kennedy schooers has little to do with the genuine advancement of social science -- and contributes little of practical value in terms of general knowledge. Most economics involves further development of degenerate research programs, i.e. failed science. And research shows that most of it is little read or never read again after 5 years or so.

>>My hypothesis was that if we were doing our job right those who wrote the most op-eds would be also the most prolific scholars. Otherwise what authority would the op-eds really have?<<

[Craig M. Newmark]

There seem to be some gross errors in the data. (I downloaded your spreadsheet at 12:10 p.m. Eastern, 9/24.)

Two examples:

There are 14 entries in the "Years" column greater than or equal to 85 (including one, for George Borjas, at an astounding 291 years).

There are 10 authors listed as having more than 300 papers, including you, Professor Rodrik, at 820. (And how does Posner rank only fourth? I don't believe it.)

I was going to complain about being omitted--my blog, Newmark's Door, has been ranked around 44th for at least several weeks--but given these apparent errors, I'm not as upset. :-)

Craig M. Newmark

[paine]

nice
nimble number knocking

the best mad science
is done with numbers
not greek letters

[Phil]

I've always wondered whether high profile academic bloggers (this one excepted, of course) get help from their graduate students in writing their blogs, or coming up with ideas. Those higher profile academic bloggers also likely have more graduate students available to them. Plus working on a blog entry is probably a nice distraction to working on a thesis ...

[paine]

PHIL:
MAYHAPS WE SHOULD
TRY TO DETECT

is this post by the master
or merely
from the studio of dani R

[Stephen Gordon]

Huh. For some reason, you weren't able to get any information about my publications. I do have some. Really.

[Anne-Wil Harzing]

Hi,

The years field in the Publish or Perish software refers to the number of years an academic has been active (years since first publication).

This can be incorrect for at least three reasons:

1. There is more than once academic publishing under the same name and the person doing the analysis with PoP has not separated them.

2. Some references to the academic's work are incorrect (typos or other mistakes) and refer to the wrong year.

3. Google Scholar's automatic parsing mechanism has parsed the year field incorrectly and lists another number as the year.

Please note that whilst no. 3 is Google Scholar specific, problems 1 and 2 occur with fee-based databases such as Web of Science and Scopus as well.

For more information about the various metrics and possible inaccuracies, please read the extensive PoP help file.

For a comparison of Google Scholar and Web of Science, please see my white paper "Reflections on Google Scholar"

http://www.harzing.com/pop_gs.htm

[Barkley Rosser]

If one is an outlier, does this mean that one should publish less (or at least get cited less) or blog more (or better)?

[Chris Jeffords]

Given my time constraint(s), I do not have the resources to grind out what I would like to with this fairly simple data set. However, I have a few ideas which may prove meaningful in this context. First, minimize the errors in the data as pointed out in previous posts. Then create a reliable data set with accurate information relative to papers, citations, etc. and begin:

a. Isolate the cases where there is more than one author to a blog. Then, controlling for academic institution, analyze the relationship between authors of the same blog relative to papers, citations, and number of authors per paper. The quality of the papers in this case is also an issue one would have to control for as well as the number of years a given author has had available to publish. There are many implications this line of analysis could yield.
b. Do the same as in (b), but allow academic institution to vary. Compare to the results found in (b).
c. Consider each blog as an entity and combine (obtain) the data on a per blog basis rather than per author. Do some more analysis.
d. Combine (obtain) the data by academic institution. Again, do some more analysis.

That is all I can think of for now – good post.

-Chris

[Paul Turpin]

I have been making the argument (in admittedly small parochial circles) that blogging, op-eds, etc., should definitely count towards promotion and tenure, but they should count as service, as in *public* service, rather than as research. Service, alas, has little weight, especially public service, but I admire those who do it nevertheless. If for no other reason, than that they find the time for it! So it helps to be tenured, sayeth one who is not (yet).

Paul Turpin

[Andrew Leigh]

Neat exercise. Why don't you come back to the data in a year, and ask the question "do economist bloggers publish more in the period before they start blogging than the period after they start blogging?".

Incidentally, I can't help noticing that your spreadsheet has me publishing for 31 years. I think I must have done my best work at age 4.

[rüya yorumları]

thanks good post

[Successful Researcher: How to Become One]

Interesting!

[Term Paper Writing]

I found this blog really good, this is a best way to share information to increase knowledge and learning capabilities. good work