2 Randomized Experiments
This week we will review the logic that underpins a research design that has become a mainstay of political science research: randomised experiments. We will focus on why randomisation is such a powerful force for making causal inferences (spoiler: internal validity), and will discuss the trade-offs implicit in experimental research (spoiler: external validity). In learning how to analyse experimental data, we will review the t-test and also cover regression as a tool for analysing experiments.
The main motivation for using randomized experiments is that when treatments are randomly assigned, both observed and unobserved potentially confounding factors will be balanced across treatment and control conditions in expectation. That is, randomization solves the selection bias problem that we outlined last week. The intuition behind this result is nicely described in both the Mostly Harmless book (Chapter 2), and also in the Mastering ’Metrics book (see, in particular, Chapter 1). As Angrist and Pischke put it, “Random assignment works not by eliminating individual differences but rather by ensuring that the mix of individuals being compared is the same.” (MM, p. 16)
For a review of statistical inference (sampling distributions, t-tests, standard errors, etc) the Masterin’ Metrics book has a nice appendix on pages 33-46. The Gerber and Green book (chapter 3) is also very useful, and very clear, though note it pays a good deal of attention to randomization inference (which we do not cover on this course), rather than classical statistical inference methods.
Chattopadhyay and Duflo (2004) is an excellent example of using the randomized nature of a real-life policy implementation to draw conclusions about an important political science question. We will be using data from this paper throughout today’s seminar. For the purposes of this course you can ignore the theoretical section of the paper (though it’s worth reading, as it’s an interesting model), but in short it concludes that we should expect there to be differences in policy outcomes between areas that are and are not governed by a female Pradhans (village chiefs). Instead you should focus on a) the detailed description of the randomization procedure, and b) what the authors did to ensure that randomization of treatment and control conditions was successful.
The Kalla and Brookman (2017) paper is an instructive example of using (several) field experiments that helps to overcome very tricky issues regarding selection bias in the context of a research question that focuses on the role of persuasion in politics. Another classic field experiment in political science is described in Gerber, Green and Larimer (2008) which in addition to being interesting in its own right also forms the basis of this week’s second part of the seminar task/homework. If it caught your interest in the lecture, you can also read the full paper by Banerjee et. al. (2015) which describes the results from 6 important experiments that aim to establish the causal effects of development aid on outcomes for the poor.
Finally, randomized experiments are playing an increasingly important role in policy-making, and it is worth having a look at the Test, Learn, Adapt paper produced by the Behavioural Insights Team and the Cabinet Office, which represents a call-to-arms for experimental methods in developing better public policy. In addition to situating the experimental methods we study in a broader policy-making context, this paper has a nice set of examples of successful public policy experiments that have been conducted over the past 20 years.