The agency paradox

Or: Why development researchers cannot tell policy-makers what to do, only how to think about what they do

The promise of policy-relevant research is the ability to influence policy-making through the supply of evidence for or against specific interventions. Development studies as an academic community is a perfect illustration of this aspiration: a significant part of its research is directly or indirectly funded by government, many of its researchers have also worked in policy as consultants or civil servants, and the field itself is organized around policy issues and not intellectual boundaries, attracting scholars from economics, political science, or sociology who are more interested in practical problems that disciplinary agendas.

But there is a fundamental conceptual obstacle between what policy researchers can offer and what policy-makers often demand: agency, understood as the ability of purposeful actors to change the world that they live in. Social science research –of the kind that development studies pursue- does not deal very well with purpose. For the most part it does not know what to do with change, either. No matter what the ontological, epistemological or methodological school a researcher may adhere to, the vagaries of social research are likely to push her towards trend, not exception, and towards stasis, not change.

A cautionary tale in two acts

In September 2006 the American political scientist James Fearon testified before the US House of Representatives in order to provide his expert advice on how to deal with the mounting chaos of post-Saddam Iraq. Fearon had spent decades studying war, and his “Ethnicity, Insurgency and Civil War” (co-authored with David Laitin)[1] is the most viewed article of the American Political Science Review[2]. His testimony, however, may not have been the sort that US lawmakers were hoping for. According to his testimony[3], civil wars “typically last a long time” and “usually end with decisive military victories (at least 75%)”, power-sharing succeeds in “one in six cases, at best”. That’s what research told Fearon, and that’s what he told Congress. Given the dominant trend against peaceful resolution by power sharing, his only real advice to the US government was to get out.

The remarkable thing about Fearon’s testimony is that it was based on probabilistic statements elicited from cross-country research evidence across all cases (“typically”, “75%”, “one in six”…). Fearon was not an expert in Iraqi culture and politics, a counter-insurgency expert, or a military strategist. Because for social-scientific purposes that kind of expertise is irrelevant once the data are good enough. But this also limits severely what a researcher can say with confidence about the probability of producing policy change.

Right around the time that James Fearon was testifying before Congress, a political movement had already begun sweeping disparate Sunni tribes opposed to al-Qaida in Iraq, turning them into a coherent movement in what would later be known as the “Anbar Awakening”. Seeing an opportunity to weaken extremist insurgents, US leaders quickly seized the chance to ally with and support the “Awakening”. The following year, moreover the US government increased the number of ground personnel in the country –“the Surge”- guided by the counter-insurgency ideas of General David Petraeus, inspired by the tactics of the French in Algeria in the 1960s and the US in Vietnam in the 1970s. The contribution of a new counter-insurgency policy and a new structure of alliances contributed to a significant decrease of violence throughout the country.

Change –fleeting, reversible, but change nonetheless- had taken place due to the ideological and political activism of a few individuals and groups who refused to be swept away by the same structural trends that James Fearon saw as inevitable.

Exceptions and change are bad for social science

Social science methodologies are based on stringent scope conditions and assumptions of unit homogeneity that ensure that we are comparing comparable things; non-comparable things have to be left out for our analysis to make any sense. Qualitatively-minded researchers ensure comparability by carefully selecting cases; experimentally and statistically-minded researchers do so by engineering random selection from a population and inferring only probabilistic relations (“if A then % B”).

This means that researchers tend not to like exceptions, variously called outliers or deviant cases, because we cannot actually explain why one case falls into the dominant trend whereas other does not. History, at the end of the day, is contingent[4]. As Abhijit Banerjee and Esther Duflo acknowledge at the end of their famous book Poor Economics:

Economists (and other experts) seem to have very little useful to say about why some countries grow and other do not. . . . In retrospect, it is always possible to construct a rationale for what happened in each place. But the truth is, we are largely incapable of predicting where growth will happen, and we don’t understand very well why things suddenly fire up[5].

By making causal claims only about a constrained set of cases or only in probabilistic terms, researchers insulate their work from the typical criticism that “your model does not apply in country X”. In the process of ensuring causal validity, social science tends to develop a dire case of what we can call “dominant trend bias”.

Together with an aversion to exceptions and “deviant cases”, social scientists tend to have a penchant for stasis in social processes. Whether we study macro-processes or micro-behaviour, at the end of the day we are happiest when the world is not malleable, but stably variable.

Consider research dealing with large-scale political issues, like state building, poverty, or corruption. Most of the time researchers approach these issues by constructing country-level concepts, slotting cases into typologies, and conducting some form of causal analysis on the determinants of macro-social outcomes. Countries are democratic or authoritarian, they have liberal or managed economies, functioning or failed states, strong or weak civil societies. But since these factors are used as either causes or consequences, comparative analysis demands that they remain constant for the purposes of the study. Because they are dealing with complex, macro-social phenomena, social science researchers often tend to simplify and assume where these stable phenomena come from.

Alternatively, consider research dealing with individual-level decisions and strategies, like voting or asking for bribes. The problem is that for models of micro-behaviour to work we also need very stable assumptions about which options the actors prefer and why; moreover, the only way to isolate the causal impact of these preferences is to have clearly structured models or games in which we can funnel them through specific decisions or choice points. Preferences and models are all based on assumptions which construct manageable social actors, not realistic ones. In fact a realistic model is an oxymoron, as the most faithful model would be as complex as life itself.

Ultimately, most research into macro- and micro-social questions, from the largest-N statistical analysis to the most descriptive case study, usually deals with comparative statics: actors with given preferences (be them profit, perceptions, or norms), institutions with given incentives (be them coercive, monetary or customary), and stable independent and dependent variables (causes and outcomes). Even research that privileges causal mechanisms and in-depth analytical frameworks falls prey to this stability bias, to the extent that a mechanism can only be tested by comparing initial conditions and ultimate consequences. Regardless of their methodology, this severely limits what policy researchers can offer by way of actionable propositions: invariably, suggestions tend to follow the structure “given A, then (surely or probably) do B”. Hence James Fearon’s grim testimony.

Squaring the agency-structure circle

The combination of the dominant-trend and stasis biases generates a remarkably conservative policy spectrum in which “givens” are virtually impossible to change: some policies work in democracies but not autocracies, some moral economies support independent bureaucrats but others don’t, and so on. At best, institutional and political equilibria adapt to changes in resource endowments, so that a purposeful actor should in principle be able to change policy by introducing or removing enough resources. At worst, these stable patterns of behaviour are based on prevalent norms and ideas, which leaves us none the wiser about how to alter them without somehow getting into people’s heads.

And yet policy processes usually evolve with the rise and fall of specific individuals and organizations. These actors espouse ideas about how things work and ought to work, build alliances and networks to promote their agendas, and wield whatever resources they can gather in order to influence decision-makers and institutions. Even though day-to-day policy implementation may be the job of those who prefer predictability and continuity, policy-making is the realm of the activist, of the purposeful agent of change. And it is this purpose that most of the time makes change political and not merely technocratic. As Kaushik Basu puts it in his critique of policy advice by economists, government is endogenous to policy change [6].

Social thinkers have been struggling for decades with the question of agency and structure: Anthony Giddens’s theory of structuration is but an attempt to solve this question[7], as is Pierre Bourdieu’s concept of habitus [8]; they both attempt to reconcile continuity and change by claiming that individuals are the ones who reproduce ideas and resources, and that these individuals can be fallible or creative, an idea that William H. Sewell has formulated more cogently[9]. Of the fathers of modern social science, only Max Weber fully acknowledged the role of inspiration and personality by invoking a form of legitimacy and rule based on charismatic individuals, whose rise was ultimately unpredictable[10].

But every major modern theory of social action –from microeconomics or bounded rationality to evolutionary institutionalism and symbolic interactionism—fails to deal with human agency understood as the power to change ideas or the meaning of resources, to change the very rules of the game, to challenge the burden of history. And that is precisely what NGOs, development agencies, or policy entrepreneurs seek to do; which means that social science research cannot give them the answers they seek.

This disparaging conclusion inevitably raises the following question: What –if anything- can policy research do?

Policy research as a map

The world is a complex place. There are too many factors, too many actors, too many patterns, too many dynamics. In other words, the world is uncertain. Not in the sense that policy-actors face risks that they have to assess and respond to, but rather in the sense proposed by economist Frank Knight: the policy environment is subject to risks which are simply immeasurable[11]. To paraphrase John Maynard Keynes, this uncertainty is not the one that one finds in the game of poker, but the one that is linked to the market value of extant shares twenty years from now or how technologies that are yet to be invented will affect daily life [12]. The pursuit of certainty in social phenomena is at best illusory; at worst, Karl Popper would argue, it is the path to totalitarianism[13].

Crises are particularly stark illustration of uncertainty, and the only way that policy-actors can navigate their way out of them is by following some guiding principle or ideology. As much as any other policy area, development has managed uncertainty through ideologies: the big push, import-substitution industrialization, structural adjustment, good governance, poverty reduction, or co-production are just some of the untested core concepts animating policy in the last five decades. These ideas were espoused in turn by policy entrepreneurs armed with some empirical evidence –of varying quality- but above all with a clear philosophical and ethical purpose. They were agents of change, precisely of the kind that social science research is so bad at explaining.

Social science cannot tell agents of change what works and what doesn’t beyond a few intuitive propositions that any observer can readily discover; it cannot tell them when innovations will take place or new ideas take root, when policy entrepreneurs will succeed for certain. But it can supply the alternative to ideology in the pursuit of change: it can translate immeasurable risk –uncertainty- into measurable risk through the use of analytical frameworks which order complexity into manageable and purposeful problems. It can supply entrepreneurs with a map for navigating the world, a map based as much on the weight of accumulated –if often confounding- evidence as on the basis of the expert judgment of those who have invested decades in understanding complexity.

Causal analysis cannot solve the agency problem without jettisoning many of the assumptions that researchers work with; but theory does have a role as a toolkit for policy agents, highlighting windows of opportunity, potential champions and spoilers of reform, and mechanisms of persuasion and change. Social science cannot tell decision-makers how to change public attitudes or social expectations; but theory can point to them the most likely cognitive entry points, material incentives for behaviour, and opportunities for normative updating. The point is not to determine the validity of one model across many static cases, but to assess how useful different models are for tackling one process of change.

This notion of social science as a map goes back to the only father of modern social theory who allowed a role for individualism and uncertainty, Max Weber, for whom social research was not just an investigation into what the world is like, but also a thought experiment about what the world could be like[14]. In the end, he thought, it was the capacity and will to take a stance towards the world that lent it meaning. This may not be a philosophy of science compatible with the incentives of academia today, but it is certainly compatible with the needs of decision-makers, and it could begin to address the agency paradox that keeps much of social science research irrelevant for policy debates.

What development researchers can offer policy-makers

Better descriptions: Beyond country- and issue-specific expertise, development researchers can help policy-makers understand the challenges they face through the use of better description: despite decades of study we are still debating what key factors separate a democracy from a dictatorship, how to identify a weak state, or what makes a political regime developmental or pro-poor. Indicators and categories have analytical roots, and as they become increasingly prominent in the policy agenda –as a way to determine goals or discriminate between interventions– the depth and quality of such roots become all the more pressing. “Mere description”, as John Gerring puts it, remains a central task of social science[15]; a task perhaps even more important for policy than causal analysis with its subordination to controlled environments.

Better models: Policy-makers are accustomed to dealing with complexity on a day-to-day basis, not through some superhuman intellectual ability to comprehend a myriad relevant factors and the interactions among them, but through the use of policy models. Sometimes these models are based in science, sometimes in ideology, but more often than not they are based on persuasion: the right combination of evidence and message, cold truth and emotion. As Stephen Krasner has argued, politicians tend to be surrounded by solutions looking for problems, advisors and organisations “selling” a particular policy option[16]. By developing a willingness to engage with this type of communication, development researchers too can become a source of models. The cost is a modicum of investment in advocacy; the benefit, however, is the use of analytically sound models –and not ideology- as the basis for policy-making.

Better questions: Finally, sometimes researchers are uniquely positioned –by virtue of being outsiders- for providing a critical or alternative interpretation of policy success and failure. Development scholars can ask aloud the questions that practitioners only discuss in private; this freedom to analyse and criticise can lead to better approaches to seemingly intractable policy dilemmas. Research from this standpoint becomes not a repository of evidence, but a “framing process”[17]. Development scholars are already embedded in a variety of epistemic communities[18] involving practitioners: that puts them in a privileged position to make policy-makers rethink and reframe policy problems and their potential solutions.

Through the advocacy of better descriptions, better models, and better questions development researchers can escape the “evidence trap” that puts undue pressure on evolving methods and contested findings which are seldom compatible with agency, contingency and change. Instead of telling policy-makers what to do, development scholars can tell them how to think about what they do. At the end of the day, overcoming the agency paradox in policy research might require social scientists to become precisely that which they understand the least: agents of change.