Article Text

Purposes and duties in scientific modelling
Free
1. Eric Winsberg1,
2. Stephanie Harvard2
1. 1 Department of Philosophy, University of South Florida, Tampa, Florida, USA
2. 2 Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
1. Correspondence to Dr Eric Winsberg, University of South Florida, Tampa, Florida, USA; winsberg{at}usf.edu

## Abstract

More people than ever are paying attention to philosophical questions about epidemiological models, including their susceptibility to the influence of social and ethical values, sufficiency to inform policy decisions under certain conditions, and even their fundamental nature. One important question pertains to the purposes of epidemiological models, for example, are COVID-19 models for ‘prediction’ or ‘projection’? Are they adequate for making causal inferences? Is one of their goals, or virtues, to change individual responses to the pandemic? In this essay, we offer our perspective on these questions and place them in the context of other recent philosophical arguments about epidemiological models. We argue that clarifying the intended purpose of a model, and assessing its adequacy for that purpose, are moral-epistemic duties, responsibilities which pertain to knowledge but have moral significance nonetheless. This moral significance, we argue, stems from the inherent value-ladenness of models, along with the potential for models to be used in political decision making in ways that conflict with liberal values and which could lead to downstream harms. Increasing conversation about the moral significance of modelling, we argue, could help us to resist further eroding our standards of democratic scrutiny in the COVID-19 era.

• models
• theoreticaL
• COVID-19
• climate change
• public health

## Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.

This article is made freely available for personal use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.

https://bmj.com/coronavirus/usage

## Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

## Introduction

More people than ever are paying attention to epidemiological models, including philosophers of science. A result is a greater focus on philosophical questions about this type of modelling, such as its susceptibility to the influence of social and ethical values,1 its sufficiency to inform policy decisions under certain conditions,2 and even its fundamental nature—as Fuller3 recently asked ‘what are COVID-19 epidemic models modelling, anyway?’. Although Fuller’s question is multifaceted, an important part of it pertains to model purposes: indeed, there is considerable confusion about how exactly certain COVID-19 models are meant to be used. Are these models for ‘prediction’ or ‘projection’?3 4 Are they adequate for making causal inferences?3 Is one of their goals, or virtues, to change individual responses to the pandemic?5 In this essay, we offer our perspective on these questions, first by giving our own answers to them, then by placing them in the context of other recent philosophical arguments about epidemiological models.1 3 6 Specifically, we argue that (1) clarifying the intended purpose of a model and (2) assessing its adequacy for that purpose are ongoing moral-epistemic duties that must be upheld throughout the modelling process. In other words, certain modelling tasks are moral responsibilities, duties which pertain to knowledge but are moral nonetheless. The moral significance of modelling, we argue, stems from the inherent value-ladenness of models,1 7 8 along with the potential for models to be used in political decision making in ways that conflict with liberal values2 9 and which could lead to downstream harms. Increasing conversation about the moral significance of modelling, we argue, could help us to resist further eroding our standards of democratic scrutiny in the COVID-19 era.10

## Prediction, projection or possibly more?

Two models that have gained much attention during the COVID-19 pandemic are the Imperial College London (ICL) model11 and Institute for Health Metrics and Evaluation (IMHE) model.12 Briefly, the main goal of the ICL model was to represent the impact of various non-pharmaceutical interventions on COVID-19 deaths and hospital/intensive care unit admissions in Great Britain and the USA (although the authors note that results are ‘equally applicable to most high-income countries’(Ferguson et al, p4).11 The IMHE model aimed to describe the need for hospital beds, intensive care unit beds, and ventilators due to COVID-19 based on projected deaths for all US states.12 From a technical perspective, the two models are very different: the ICL is an individual-based simulation model, while the IMHE projects deaths using a statistical model. For the purpose of our discussion, these technical differences are of limited importance; we will note them as we go along.

### Prediction versus projection

Recently, Fuller3 and Schroeder4 have analysed the ICL and/or IHME models with the same question in mind: what kinds of predictions do these models make? Following Fuller3 and Schroeder,4 there are two possibilities to consider. The first is that these models make predictions about what will actually occur, the second that they make predictions about what would happen under certain conditions. Both Fuller3 and Schroeder4 call the first type of prediction forecasting and the second conditional projection. As each writer gives just a brief run-down of the distinction, we will give our own, more detailed explanation of it following Winsberg.8 Winsberg8 describes how the distinction between forecasting (or just ‘prediction’) and projection is used in climate science, but, as we will show, his discussion can be fruitfully applied to epidemiology. Distinguishing between prediction and projection helps us to see that different types of models invite us to use different strategies to assess their adequacy for purpose, which is relevant to debates surrounding different conceptions of ‘model validation’ in the health sciences.13

As Winsberg8 explains, in climate science, the difference must often be stressed between weather models and climate models. Weather models are indeed used to tell us what will actually occur at a particular time and place, that is, to make spatially and temporally fine-grained predictions about states of the atmosphere. Weather models have three important features: (1) they take into account our best and most recent measurements of the current atmospheric conditions in the relevant region; (2) their predictions are based on a comparatively high degree of fidelity to our near-perfect understanding of causal relationships in weather systems; (3) the accuracy of their predictions is easy to test empirically: they make frequent predictions concerning the near future that we soon come to observe in great detail. When a model has these features like these, our strategy for assessing its adequacy for purpose can be relatively straightforward, focused on the match between our predictions and our later observations.

Climate models, on the other hand, do not make spatially and temporally fine-grained predictions about states of the atmosphere. The chaotic nature of the atmosphere makes this impossible to do on the century-long time scales that concern climate science. Rather, climate models project global or wide-regional averages of the variables describing these states, averaged over timespans of 30 years or longer. Furthermore, rather than forecast what will actually happen in the future, climate models are generally used to explore what possible evolutions in the climate could be triggered by different external forcings, defined as perturbations that are outside the climate system but capable of pushing it beyond its normal range of variation (eg, ozone depletion, C02 emissions, other greenhouse gases, deforestation). Climate models themselves have three important features: (1) they model climate variables in the form of coarse-grained statistical summaries of weather variables (eg, averages, degrees of variation in weather variables) and do not take account of current atmospheric conditions; (2) their projections are based on a representation of the dynamics of the atmosphere and ocean that, in order to be computationally manageable, has a much lower degree of fidelity to our best physics than weather prediction models; (3) the accuracy of their predictions is much less testable than those of weather models. This reduced testability is because we only get one run of the planet’s climate evolution and we will not see it until it is too late: although climate models are meant to project the effects of many different possible emissions pathways (eg), we will only ever see the outcome of one of these. When a model has features like these, assessing its adequacy for purpose is a far more complicated enterprise.

According to Schroeder,4 the purpose of the IHME model should be understood not as forecasting but as conditional projection. He argues this on the grounds that IHME model documentation includes alongside its estimates caveats like ‘assuming social distancing measures are maintained’ and excludes arguments to the effect that all US states will (or are likely to) institute such measures (Schroeder, p3).4 Fuller3 suggests that it is reasonable to interpret the IHME model results as forecasts (since the model made a single prediction after most states had implemented ‘lockdowns’) and to interpret the ICL model results as conditional projections (since the model’s multiple estimates corresponded to different policy options). At the same time, Fuller3 questions whether the forecast/unconditional projection distinction is cogent in epidemiology: he notes that epidemic models always make assumptions and wonders whether we should think epidemic models ever make unconditional predictions (see section 3).

To richen this discussion, we should point out that ‘projection’ is not a homogeneous purpose to which models can be put, nor one that stands opposite ‘prediction’ in a binary conception of possible model purposes. To be sure, the ICL and IHME models do not have the features of weather models: they do not incorporate precise, local measurements, their predictions are not based on a near-perfect understanding of causal relationships, and the accuracy of their predictions is not easy to test empirically. Rather, the ICL and IHME models have some features in common with climate models: they model coarse-grained statistical summaries of variables (eg, social contact and mortality rates), the models are not high-fidelity representations of their target systems (indeed, the IHME model does not even directly represent infectious disease transmission dynamics), and the accuracy of their predictions is very difficult to test (see section 2.3). However, we should be clear the ICL and IHME models do not make conditional projections in quite the same way as climate models do.

## Duties in modelling

In the introduction, we suggested that people involved in scientific modelling have the moral-epistemic duties to establish what purposes a given model aims to serve and to continually assess the model’s adequacy for those purposes. Our claim builds on the idea that scientists have a moral responsibility to avoid foreseeable harms,25 taking into account that using models that are inadequate for purpose can lead to such harms, including the endorsement of false claims and the unjust omission of information,20 which are significant in the context of decision making. Before we go on, it is important to stress that model purposes come from people, not from models themselves. In principle, any model could be used for almost any purpose—displayed as a work of art, incorporated into one’s spiritual practice, or used to make causal inferences or inform policy decisions—regardless of any relevant shortcomings. It is too much to ask for modellers to be responsible for ensuring that their models are never used in a way they do not intend. Rather, it is their duty to establish what they do intend and, thus, the scope of their responsibility.

When one is involved in scientific modelling for the purpose of informing decision making, moral and epistemic duties are not as distinct as philosophers often conceive of them26 but rather cross paths. We call modelling tasks moral-epistemic duties in order to underline that they have moral significance. The moral significance of modelling tasks stems in part from the inherent value-ladenness of modelling, including the representational decisions that are an intrinsic part of the process.1 20 Representational decisions, in terms of what to represent (eg, in the model structure) and how to represent it (eg, through parameterisation), are uniquely guided by the purpose of the model. Not only is there no such thing as a ‘factually correct’ way to model something, we often lack anything resembling ‘factually correct’ model inputs (eg, ‘one cannot hope to obtain an accurate, data-informed value of all parameters in contention’ in the ICL model (Edeling et al p128)).14 The result is that representational decisions are routinely informed not by assessments of what is empirically true, but of what is reasonable, sufficient or adequate given the purpose of the model. These are often morally charged decisions, such as whether it is reasonable to exclude certain costs (eg, unintended harms) or adequate to use a certain source of data (eg, from a faraway setting) when estimating the overall benefit of a health intervention.1 One recognises these decisions as morally significant as soon as one learns that different people would make them differently. Modelling decisions share this in common with all expert judgements: ‘If those making judgements share certain characteristics, such as gender, age, race, home ownership, or wealth, they may fail to recognise how the costs of policies (such as stay-at-home orders) are likely to affect those who do not share those characteristics; they might recognise those costs and consider them to some extent but they will not feel those consequences of their decisions’ (Moore et al, p1).27

The fact that modelling decisions are both value-laden and purpose-guided lends an obvious significance to a model’s intended purpose; to clarify this purpose is thus a primary moral-epistemic duty in modelling. As we have shown, this task is not as simple as it seems: disambiguating a model’s purpose from nearby possibilities requires detailed analysis along multiple axes.3 4 28 29 Furthermore, a model’s purpose is not something that can be easily settled once and for all at the beginning of a modelling project. On the contrary, a model’s purpose must continually be re-assessed and understood relative to its epistemic features, both its virtues and shortcomings—in other words, its adequacy-for-purpose. This assessment process involves a complicated weighting of epistemic (scientific) and non-epistemic (social) values that different scientists inevitably carry out differently. This is partly why a question like ‘do epidemic models ever make unconditional predictions?’3 is difficult to answer: epidemiologists with different values may use the same model for different purposes given their different assessments of the model.

Many contributors have documented the predictive failures of certain COVID-19 models,21–23 30 31 while others have pointed to the (at least relative) success of the same models.4 19 32 This discrepancy should remind us that there is no universally agreed-upon, objective basis on which to define the accuracy of model predictions. When building a prediction model, modellers’ moral-epistemic duties include setting a clear standard for predictive accuracy, not only in terms of precision but in terms of what will count as an accurate prediction. A model’s predictions can be accurate for reasons that are not rooted in epistemic virtues—if a model overestimates the impact of a virus on mortality, but also overestimates the efficacy of an intervention, it might end up making an empirically accurate mortality projection in an area where the intervention is adopted— and modellers should be clear about whether or not they value this sort of achievement. At the same time, there is a duty to establish what will count as a failure: often, a model whose purpose is to make conditional projections has an infinite number of built-in ‘escape routes’, reasons to which the model’s defenders can appeal to justify why projections have not panned out in the real world. It is sometimes possible to recognise a model at the outset of development as being unlikely to have strong predictive capabilities, particularly over the long term, given the complexity of its target system, known limitations in the relevant evidence, or other factors. In these cases, the model’s purpose should be understood as being to assist decision making under significant uncertainty. This involves special moral considerations, including whether to postpone the decision and pursue additional information.16

Beyond a certain degree of uncertainty, utilitarian frameworks for decision making, like those behind harm–benefit and cost-effectiveness modelling, will be of limited use. In such cases, caution is required: we should guard against models being used to justify existing political views by representing their favoured policies as the ones that ‘follow the science’. Otherwise, our standards of scientific and democratic scrutiny will suffer.

## Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.

## Acknowledgments

Stephanie Harvard gratefully acknowledges post-doctoral funding from the Michael Smith Foundation for Health Research.

## Footnotes

• Contributors We wrote this paper together with equal contributions.

• Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

• Competing interests None declared.

• Provenance and peer review Commissioned; externally peer reviewed.