Background Finding an association between a policy and an effect in an observational study is not enough to prove a causal relationship. Impact evaluations may be strengthened by developing an understanding of the causal explanation(s) behind an association. Here, we assess the feasibility of using process tracing (PT). While PT has been applied to a limited number of programme evaluations, we believe this is the first attempt to apply the method to a public health policy evaluation. Given evidence of a statistical association, can PT be usefully operationalized in a public health policy evaluation?

Methods We used the Barbados sugar-sweetened beverage (SSB) tax as a case study. We previously demonstrated an association between tax introduction and an observed decrease in SSB sales. According to dominant theory, price change is the sole mechanism through which SSB taxes dampen consumer demand. However, SSB taxes may also have a signaling effect, raising awareness and reducing demand. Following PT best-practice, we developed causal theories, pre-specified the evidence we would expect to find under each theory, operationalized tests to identify this evidence, and assessed the probative value of each test. We assessed prior confidence in both theories and described implications of each test.

Results We identified a range of potential tests (8 tests of the price change only theory, 8 separate tests of the signaling effect). For example, one test of the signaling effect could be an assessment of whether the public’s perception of ‘good’ vs. ‘bad’ drinks matches the pattern of change observed more than a categorization based on taxed vs. untaxed status. In this example, we propose to use print media to qualitatively identify how ‘good’ and ‘bad’ drinks were characterized (i.e. were sodas and juices portrayed differently?) and then use this categorization to re-analyze grocery store sales data using an interrupted time series. If this categorization explains the data more fully than an analysis based on taxed/untaxed status, this test would strongly favor signaling over the price change only hypothesis, making this a test with high probative value. We identified methods and data that could be used to empirically assess each test and assessed each test’s probative value.

Conclusion Further work will be needed to empirically conduct and critically assess as many of these tests as possible, prioritizing those with greatest probative value. However, this study has suggested that PT may be able to make a useful contribution to improving public health policy evaluations.