4 Selection on Observables (II)
This week we continue to consider methods that rely on a selection-on-observables assumption for making causal inferences from non-experimental data. In particular, this week we focus on assessing under which conditions linear regression can be used to make causal statements.
In the lecture this week we revisit the linear regression model, and discuss in particular the circumstances under which regression coefficients resemble the average treatment effects that are our core concern on this course. If you need a refresher on linear regression, then chapter 2 in the Mastering ’Metrics book is excellent. In addition to covering the basics of regression analysis, it provides a very accessible treatment of omitted variable bias (p. 68 - 74 and p. 89 - 90), as well as helpful summaries of regression with logged dependent variables (p. 93 - 94) and uncertainty in regression analysis (p. 95 -96). All of these topics are also covered in the MHE book (chapter 3), though at a higher level of mathematical detail.
We will spend some time describing the implicit conditional-treatment-weighting that is imposed by OLS when calculating treatment effects while conditioning on observed covariates. The MHE book provides a derivation and explanation of this result on pages 73 to 76, and there is a similar treatment in the Morgan and Winship book (chapter 6). One of the implications of this implicit weighting scheme is that the effective sample on which the results of any observational regression are based may be very different from the full population of interest; a result that is clearly and cleanly articulated in the Aronow and Samii, 2016 paper (we strongly recommend this paper!). I also recommend the paper by Ho et. al (2007) which gives an excellent (and intuitive) account of how combining matching with regression analysis can reduce model dependence in causal analyses, although in some ways it just shifts the model dependence problem to the matching decisions rather than the choice of regression model.
Both this an last week have the same core main message: the conditional independence assumption, i.e. selection on observables, is a very strong assumption, and in most applications hard to justify.It only holds when we have controlled for all confounders, but no post-treatment variables (mediators or colliders). The question then is, which variables should you, or should you not control for? This article by Cinelli, Forney & Pearl (2024) discusses all the ways in which you can have “good” and “bad” controls, using potential outcomes framework and causal diagrams (directed acyclic graphs (DAGs)) to show that controlling for more is very often not better.
Even though, as always, the identification assumption (in this case CIA) is untestable, it is possible to provide indirect evidence about how sensitive the estimated treatment effect is to potential unobserved confounders. Cinelli and Hazlett (2020) propose such a sensitivity analysis, whereby the core idea is to provide a metric of how strong would unobserved confounding have to be - with both treatment and outcome - to explain away the estimated effect. Their method is implemented in the sensmarkr package (you can find an applied introduction here).
Another way to assess the credibility of a given research finding is to check “for an association that should be absent if the assumptions underlying the design hold but might be present if those assumptions are violated in some relevant way” (Eggers, Tuñón & Dafoe, 2023, p.1106). While the idea of placebo tests has a long history (in particular in medical/pharmacological research), Eggers & coauthors develop a coherent framework to think cleary about what are good and bad/not useful placebo tests. While some of the front-end of the paper may be a bit hard to chew through, it lays out a number of intuitive ways to help choosing useful placebo tests with some good examples.
This week we will be learning how to use linear regression to estimate causal effects (under the selection on observables assumption) in R. We will also learn how to specify non-linear models for certain explanatory variables by using polynomials, and how to construct heteroskedasticity-robust standard errors. Finally, we will also learn some functions for interpreting the output of our models.