Political Entrepreneurs

The Economic Engine of Political Change

Ranking Think Tanks: The Measurement Problem

March 1st, 2013 by Edward Lopez

In an initial post last week, I argued that the proliferation of think tanks over the past 50 years is likely to be a good influence on human affairs, and so is the emerging competition to measure (rank) the performance of think tanks. In this follow-up post and a planned third, I will question the usefulness of the measures used in these rankings.

Why measure think tank effectiveness with data like social network impact, web traffic, publication counts, and academic citations? Well, like the man who looks for his lost keys under the street lamp where the light is good, the answer is: because these are the best, easily available data.

Yet the ultimate measures of a think tank’s success are more complex and obscured, for three sets of interrelated reasons.

1). The ultimate goal of a think tanks is not merely to influence the climate of ideas and public opinion, but to change social institutions in the direction of its guiding philosophies. Web traffic, publication counts, and so forth aren’t capturing outcomes, but instead are capturing proxies that can be related to outcomes but clearly are one or more steps removed.

Ideally a think tank’s performance would be measured by its marginal effect on the power of ideas to overcome interests that defend a status quo. What are the marginal effects of a think tank on the evolution of public opinion, where madmen in authority make their appeal? What are its marginal effects on the set of proposals that are “politically feasible” at any time and place? And what are its marginal effects on the set of proposals that ultimately are adopted? Answers to these would capture truer measures of a think tank’s effectiveness because these would capture its unique contributions to political change.

It is not difficult to conceive of such measurements, but executing them would be both costly and imperfect. And this is one reason why (for now) it still makes sense to use proxies.

2). Long and uncertain periods of time separate a think tank’s everyday activities from the eventual, ultimate outcomes that it seeks to influence. The framework in Madmen identifies the stages of production that generate political change.

Outcomes in society are the summation of individual actions that are guided by incentives. In turn, incentives are shaped by institutions, and institutions are shaped by the flow of ideas doing battle with status quo interests. It is in this top segment of the triangle where think tanks compete – in the marketplace of ideas. To put it slightly differently, a think tank’s business model is one of investment in ideas, and the returns to investment materialize as institutional change over long and uncertain periods of time.

How might we measure and rank the efforts of think tanks today in terms of their payoff in the future? For example, London’s Institute of Economic Affairs (IEA) toiled for two decades before Margaret Thatcher rose to power and began sourcing from IEA many of her proposals for change (HT: Brad DeVos). It would require some clever measures to have IEA ranked highly during its first 20 years. Again, those measures would be costly and imperfect.

3). Unlike market competition, political and intellectual entrepreneurs lack the discipline of clear success and failure metrics. What are a think tank’s deliverables? How does it demonstrate its rate of return to its investors/donors? This is not as straightforward as a corporation referring stockholders to its profit & loss statement, or even a newspaper showing advertisers its circulation numbers. A market entrepreneur is guided by relatively clear feedback mechanisms—profit and loss denominated in units of the currency. By comparison, a political or intellectual entrepreneur is guided by more nuanced, even murkier criteria.*

These interrelated reasons might be sufficient to argue that measuring think tank effectiveness is difficult, and perhaps prohibitively so. But data and measurement techniques could (and probably will) get better. And the framework above could lead to specific theories and testable hypotheses that improve our understanding of long-term investment in ideas. Till then, it will still make sense to use proxies instead of truer measures of effectiveness—i.e. to look for our keys under the street lamp, where the light is good.

For the onlooker (e.g., the analyst seeking to measure performance), this means the performance picture is relatively bright in market competition, but relatively dim in political and intellectual competition. For the think tank, this means having to consider carefully its various strategies, and having to take different sorts of risks compared to market entrepreneurs. This will be the starting point of the third post in this series.

*Note: I raise the issue of market vs. political entrepreneurs’ feedback mechanisms on Marginal Revolution, and Wayne elaborates in his entry “Learning From Failure” here on PE.

Update: Corrected for typo. Here is a link to the next (third) entry in this series, “Ranking Think Tanks: The Challenge of Specialization

2 thoughts on “Ranking Think Tanks: The Measurement Problem”

Comments are closed.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.189, ch.7)

The most successful entrepreneurs know what they do well, they know the market and the opportunities within it, and they choose those activities that create the most value. This is true in economic as well as political markets.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.178, ch.7)

[W]hen the right elements come together at the right time and place and overwhelm the status quo, it is because special people make it happen. We call them political entrepreneurs.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.176. ch.7)

While we started this book with Danny Biasone saving basketball, we end it with Norman Borlaug saving a billion lives. These stories are not that different. Both faced vested interests, which were reinforced by popular beliefs that things should be a certain way—that is, until a better idea came along.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.174, ch.6)

Because there was a general belief that homeownership was a good thing, politicians found the public with open arms.... Everybody was winning—except Alfred Marshall, whose supply and demand curves were difficult to see through the haze of excitement at the time, and except Friedrich Hayek, whose competition as a discovery procedure was befuddled... In short, once politicians started getting credit for homeownership rates, the housing market was doomed.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.166, ch.6)

Everyone responded rationally to the incentives before them. In short, the rules that guided homeownership changed over time, which in turn changed the incentives of these actors. And bad things happened.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.153, ch.6)

They understood the economics. The ideas had already won in ... the regulatory agency itself. All that remained to be overcome were some vested interests and a handful of madmen in authority.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.146, ch.6)

If the idea for auctions of spectrum use rights had been part of the public debate since at least 1959, why didn’t the relevant institutions change sooner? What interests stood in the way?

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.121, ch.5)

When an academic scribbler comes up with a new idea, it has to resonate well with widely shared beliefs, which in turn must overcome the vested interests at the table. Many forces come together to explain political change, even though it may seem like coincidence of time and place.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.120, ch.5)

It’s the rules of the political game that deserve our focus, not politicians’ personalities or party affiliations.

From the Pages of Madmen, Intellectuals, and Academic Scribblers (p.119, ch.5)

In short, ideas are a type of higher-order capital in society. Like a society that is poor in capital and therefore produces little consumer value, a society that is poor in ideas and institutions will have bad incentives and therefore few of the desirable outcomes that people want.

©2023 Wayne A Leighton & Edward López • Web Design by Barrel Strength