THEORY AND METHODS

Criteria for evaluating evidence on public health interventions

L Rychetnik¹, M Frommer¹, P Hawe² and A Shiell²

¹ Effective Healthcare Australia, School of Population Health and Health Services Research, University of Sydney, Australia
² Department of Community Health Sciences, University of Calgary, Canada and School of Public Health, LaTrobe University, Australia

Correspondence to:
Correspondence to:
Lucie Rychetnik, Effective Healthcare Australia, Victor Coppleson Building, DO2, University of Sydney, NSW 2006, Australia;
lucier{at}med.usyd.edu.au

Public health interventions tend to be complex, programmatic, and context dependent. The evidence for their effectiveness must be sufficiently comprehensive to encompass that complexity. This paper asks whether and to what extent evaluative research on public health interventions can be adequately appraised by applying well established criteria for judging the quality of evidence in clinical practice. It is adduced that these criteria are useful in evaluating some aspects of evidence. However, there are other important aspects of evidence on public health interventions that are not covered by the established criteria. The evaluation of evidence must distinguish between the fidelity of the evaluation process in detecting the success or failure of an intervention, and the success or failure of the intervention itself. Moreover, if an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently faulty (that is, failure of intervention concept or theory), or just badly delivered (failure of implementation). Furthermore, proper interpretation of the evidence depends upon the availability of descriptive information on the intervention and its context, so that the transferability of the evidence can be determined. Study design alone is an inadequate marker of evidence quality in public health intervention evaluation.

Keywords: public health intervention

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

• Freund, M., Campbell, E., Paul, C., Sakrouge, R., McElduff, P., Walsh, R. A., Wiggers, J., Knight, J., Girgis, A. (2009). Increasing smoking cessation care provision in hospitals: A meta-analysis of intervention effect. Nicotine Tob Res 11: 650-662 [Abstract] [Full Text]  
• Adams, J., Witten, K., Conway, K. (2009). Community development as health promotion: evaluating a complex locality-based project in New Zealand. COMMUNITY DEV J 44: 140-157 [Abstract] [Full Text]  
• Egan, M, Bambra, C, Petticrew, M, Whitehead, M (2009). Reviewing evidence on complex social interventions: appraising implementation in systematic reviews of the health effects of organisational-level workplace interventions. J. Epidemiol. Community Health 63: 4-11 [Abstract] [Full Text]  
• Tannahill, A. (2008). Beyond evidence--to ethics: a decision-making framework for health promotion, public health and health improvement. HEALTH PROMOT INT 23: 380-390 [Abstract] [Full Text]  
• deWilde, C. K., Milman, A., Flores, Y., Salmeron, J., Ray, I. (2008). An integrated method for evaluating community-based safe water programmes and an application in rural Mexico. Health Policy Plan 23: 452-464 [Abstract] [Full Text]  
• Bambra, C, Whitehead, M, Sowden, A, Akers, J, Petticrew, M (2008). "A hard day's night?" The effects of Compressed Working Week interventions on the health and work-life balance of shift workers: a systematic review. J. Epidemiol. Community Health 62: 764-777 [Abstract] [Full Text]  
• Tang, K-C, Choi, B C K, Beaglehole, R (2008). Grading of evidence of the effectiveness of health promotion interventions. J. Epidemiol. Community Health 62: 832-834 [Abstract] [Full Text]  
• Ramanathan, S., Allison, K. R., Faulkner, G., Dwyer, J. J. M. (2008). Challenges in assessing the implementation and effectiveness of physical activity and nutrition policy interventions as natural experiments. HEALTH PROMOT INT 23: 290-297 [Abstract] [Full Text]  
• Robitaille, Y., Gauvin, L. (2008). Fall prevention in older adults: towards an integrated population-based perspective. Inj. Prev. 14: 147-148 [Full Text]  
• Slade, M., Gask, L., Leese, M., McCrone, P., Montana, C., Powell, R., Stewart, M., Chew-Graham, C. (2008). Failure to improve appropriateness of referrals to adult community mental health services--lessons from a multi-site cluster randomized controlled trial. Fam Pract 25: 181-190 [Abstract] [Full Text]  
• Spencer, B., Broesskamp-Stone, U., Ruckstuhl, B., Ackermann, G., Spoerri, A., Cloetta, B. (2008). Modelling the results of health promotion activities in Switzerland: development of the Swiss Model for Outcome Classification in Health Promotion and Prevention. HEALTH PROMOT INT 23: 86-97 [Abstract] [Full Text]  
• Bambra, C., Egan, M., Thomas, S., Petticrew, M., Whitehead, M. (2007). The psychosocial and health effects of workplace reorganisation. 2. A systematic review of task restructuring interventions. J. Epidemiol. Community Health 61: 1028-1037 [Abstract] [Full Text]  
• Egan, M., Bambra, C., Thomas, S., Petticrew, M., Whitehead, M., Thomson, H. (2007). The psychosocial and health effects of workplace reorganisation. 1. A systematic review of organisational-level interventions that aim to increase employee control. J. Epidemiol. Community Health 61: 945-954 [Abstract] [Full Text]  
• Waters, E., Priest, N., Armstrong, R., Oliver, S., Baker, P., McQueen, D., Summerbell, C., Kelly, M. P., Swinburn, B. (2007). The role of a Prospective Public Health Intervention Study Register in building public health evidence: proposal for content and use. J Public Health (Oxf) 29: 322-327 [Abstract] [Full Text]  
• James, E. L., Fraser, C., Anderson, K., Judd, F. (2007). Use of research by the Australian health promotion workforce. Health Educ Res 22: 576-587 [Abstract] [Full Text]  
• McLaren, L., Ghali, L. M., Lorenzetti, D., Rock, M. (2007). Out of context? Translating evidence from the North Karelia project over place and time. Health Educ Res 22: 414-424 [Abstract] [Full Text]  
• Nicholson, P J (2007). How to undertake a systematic review in an occupational setting. Occup. Environ. Med. 64: 353-358 [Full Text]  
• Verhoef, M. J., Vanderheyden, L. C., Fonnebo, V. (2006). A Whole Systems Research Approach to Cancer Care: Why Do We Need It and How Do We Get Started?. Integr Cancer Ther 5: 287-292 [Abstract]  
• Kovats, R S., Kristie, L E. (2006). Heatwaves and public health in Europe. Eur J Public Health 16: 592-599 [Abstract] [Full Text]  
• Mallonee, S, Fowler, C, Istre, G R (2006). Bridging the gap between research and practice: a continuing challenge. Inj. Prev. 12: 357-359 [Full Text]  
• Whittaker, K., Sutton, C., Burton, C. (2006). Pragmatic randomised controlled trials in parenting research: the issue of intention to treat.. J. Epidemiol. Community Health 60: 858-864 [Abstract] [Full Text]  
• Stronks, K., Mackenbach, J. P. (2006). Evaluating the effect of policies and interventions to address inequalities in health: lessons from a Dutch programme. Eur J Public Health 16: 346-353 [Abstract] [Full Text]  
• Waters, E., Doyle, J., Jackson, N., Howes, F., Brunton, G., Oakley, A. (2006). Evaluating the effectiveness of public health interventions: the role and activities of the Cochrane Collaboration.. J. Epidemiol. Community Health 60: 285-289 [Abstract] [Full Text]  
• Wang, S., Moss, J. R., Hiller, J. E. (2006). Applicability and transferability of interventions in evidence-based public health. HEALTH PROMOT INT 21: 76-83 [Abstract] [Full Text]  
• Dooris, M. (2006). Healthy settings: challenges to generating evidence of effectiveness. HEALTH PROMOT INT 21: 55-65 [Abstract] [Full Text]  
• Oakley, A., Strange, V., Bonell, C., Allen, E., Stephenson, J., RIPPLE Study Team, (2006). Process evaluation in randomised controlled trials of complex interventions.. BMJ 332: 413-416 [Full Text]  
• Thomson, H., Atkinson, R., Petticrew, M., Kearns, A. (2006). Do urban regeneration programmes improve public health and reduce health inequalities? A synthesis of the evidence from UK policy and practice (1980-2004). J. Epidemiol. Community Health 60: 108-115 [Abstract] [Full Text]  
• Jackson, N., Waters, E., for the Guidelines for Systematic Reviews in Healt, (2005). Criteria for the systematic review of health promotion and public health interventions. HEALTH PROMOT INT 20: 367-374 [Abstract] [Full Text]  
• Ogilvie, D., Egan, M., Hamilton, V., Petticrew, M. (2005). Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?. J. Epidemiol. Community Health 59: 886-892 [Abstract] [Full Text]  
• Wong, T Y (2005). Cataract surgery programmes in Africa. Br. J. Ophthalmol. 89: 1231-1232 [Full Text]  
• Ogilvie, D., Hamilton, V., Egan, M., Petticrew, M. (2005). Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go?. J. Epidemiol. Community Health 59: 804-808 [Abstract] [Full Text]  
• Aoun, S. M, Kristjanson, L. J (2005). Challenging the framework for evidence in palliative care research. Palliat Med 19: 461-465 [Abstract]  
• Tooth, L., Ware, R., Bain, C., Purdie, D. M., Dobson, A. (2005). Quality of Reporting of Observational Longitudinal Research. Am J Epidemiol 161: 280-288 [Abstract] [Full Text]  
• Ogilvie, D (2005). Hospital based alternatives to acute paediatric admission: a systematic review. Arch. Dis. Child. 90: 138-142 [Abstract] [Full Text]  
• Rush, B., Shiell, A., Hawe, P. (2004). A census of economic evaluations in health promotion. Health Educ Res 19: 707-719 [Abstract] [Full Text]  
• Ogilvie, D., Egan, M., Hamilton, V., Petticrew, M. (2004). Promoting walking and cycling as an alternative to using cars: systematic review. BMJ 329: 763- [Abstract] [Full Text]  
• Petticrew, M., Whitehead, M., Macintyre, S. J, Graham, H., Egan, M. (2004). Evidence for public health policy on inequalities: 1: The reality according to policymakers. J. Epidemiol. Community Health 58: 811-816 [Abstract] [Full Text]  
• Power, R., Langhaug, L. F., Nyamurera, T., Wilson, D., Bassett, M. T., Cowan, F. M. (2004). Developing complex interventions for rigorous evaluation--a case study from rural Zimbabwe. Health Educ Res 19: 570-575 [Abstract] [Full Text]  
• Hawe, P., Shiell, A., Riley, T., Gold, L. (2004). Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J. Epidemiol. Community Health 58: 788-793 [Abstract] [Full Text]  
• Rychetnik, L., Hawe, P., Waters, E., Barratt, A., Frommer, M. (2004). A glossary for evidence based public health. J. Epidemiol. Community Health 58: 538-545 [Abstract] [Full Text]  
• Adily, A, Ward, J (2004). Evidence based practice in population health: a regional survey to inform workforce development and organisational change. J. Epidemiol. Community Health 58: 455-460 [Abstract] [Full Text]  
• Abbema, E. A., Van Assema, P., Kok, G. J., De Leeuw, E., De Vries, N. K. (2004). Effect evaluation of a comprehensive community intervention aimed at reducing socioeconomic health inequalities in the Netherlands. HEALTH PROMOT INT 19: 141-156 [Abstract] [Full Text]  
• Rychetnik, L., Wise, M. (2004). Advocating evidence-based health promotion: reflections and a way forward. HEALTH PROMOT INT 19: 247-257 [Abstract] [Full Text]  
• Jack, L. Jr., Liburd, L., Spencer, T., Airhihenbuwa, C. O. (2004). Understanding the Environmental Issues in Diabetes Self-Management Education Research: A Reexamination of 8 Studies in Community-Based Settings. ANN INTERN MED 140: 964-971 [Abstract] [Full Text]  
• Roberts, H, Curtis, K, Liabo, K, Rowland, D, DiGuiseppi, C, Roberts, I (2004). Putting public health evidence into practice: increasing the prevalence of working smoke alarms in disadvantaged inner city housing. J. Epidemiol. Community Health 58: 280-285 [Abstract] [Full Text]  
• Thomson, H., Hoskins, R., Petticrew, M., Ogilvie, D., Craig, N., Quinn, T., Lindsay, G. (2004). Evaluating the health effects of social interventions. BMJ 328: 282-285 [Full Text]  
• Tang, K C, Ehsani, J P, McQueen, D V (2003). Evidence based health promotion: recollections, reflections, and reconsiderations. J. Epidemiol. Community Health 57: 841-843 [Full Text]  
• Bull, S. S., Gillette, C., Glasgow, R. E., Estabrooks, P. (2003). Work Site Health Promotion Research: To what Extent can we Generalize the Results and what is Needed to Translate Research to Practice?. Health Educ Behav 30: 537-549 [Abstract]  
• Petticrew, M, Roberts, H (2003). Evidence, hierarchies, and typologies: horses for courses. J. Epidemiol. Community Health 57: 527-529 [Abstract] [Full Text]  
• McQueen, D V (2002). The evidence debate. J. Epidemiol. Community Health 56: 83-84 [Full Text]