• COVID-19
• treatment outcome
• epidemiology

The COVID-19 pandemic has provided limitless opportunities to compare pandemic policies across countries and over time. When the aim is to assess the comparative success of these policies, the comparison requires thinking counterfactually about ‘what would have been’ in some unrealised hypothetical (counterfactual) scenario. Whether generating modelling projections,1 making data-driven comparisons across countries2 or attributing excess harms,3 causal inference often rests on counterfactual comparisons, even if those comparisons are only implicit. However, in the pandemic, counterfactual analyses that are overly simplistic, uninformative or outright flawed have been an epidemic in their own right. The examples I explore here are not the worst offenders and my aim is not to criticise them but to use them to illustrate cautionary lessons. By exploring the theory of counterfactuals and common problems with their use, we can learn to do better. Slow conceptual thinking is needed even in times of fast science.

Counterfactuals have played a central role in discussions of causation in philosophy4 and in the health sciences5 and social sciences6 over the past 50 years. According to a framework popular in these disciplines, an intervention causes some outcome if that outcome represents a difference between two hypothetical scenarios in which only the intervention differs. Because the scenarios are mutually incompatible, at least one of them is ‘counterfactual’—that is, contrary to what actually occurs or ‘counter to fact’. Philosophers sometimes think about a counterfactual scenario as an imaginary world that is perfectly identical to the actual world except that the intervention is miraculously altered or manipulated with surgical precision. For instance, if the number of COVID-19 cases would be greater in a possible world that is identical to the real world but in which …