Study objective: To assess current capacity to implement evidence based practice (EBP) in population health.
Design: Postal survey of a regional population health workforce in Sydney, Australia.
Setting: Division of Population Health, South Western Sydney Area Health Service.
Participants: 104 population health staff (response rate: 73%).
Main results: In the sample of regional population health practitioners, views about the current promotion of EBP were positive. Non-medical respondents with less that Masters degree were more likely to report “high self assessed need” to increase their capacity in EBP (p = 0.022). Confidence in understanding of EBP terminology was not associated with seniority but with highest level of education reached (p<0.000) and having medical qualifications (p<0.000). Occupational category was not associated with respondents’ self assessed “need for evidence”, “need for EBP skills” or “need to increase their capacity in EBP” in their current position. The proportion of participants “strongly” supporting implementation of a colorectal cancer screening programme whose benefit was expressed as relative risk reduction was greater than that so supporting a programme whose benefit was expressed as number needed to screen (p = 0.008). Most respondents referred to their immediate managers when seeking support for EBP.
Conclusions: The findings provide a quantitative baseline for capacity building through workplace programmes. Managerial commitment has been increased and performance development is now underway.
- evidence based practice
- population health
- needs assessment
- EBP, evidence based practice
- NNS, number needed to screen
- RRR, relative risk reduction
- ARR, absolute risk reduction
Statistics from Altmetric.com
- EBP, evidence based practice
- NNS, number needed to screen
- RRR, relative risk reduction
- ARR, absolute risk reduction
Evidence based practice (EBP) promotes the incorporation of “best available” evidence from research into practice.1 It is increasingly being discussed in health disciplines other than clinical medicine, including population health.2–4 Indeed, it specifically has been argued that applying evidence is potentially the most effective and efficient way to deal with the inevitability in health care of spiralling costs yet limited resources.5 Hence, “evidence-based health care has entered the consciousness, if not the decision making, of most clinicians and managers”.6
To improve the uptake of evidence into practice, the importance of research literacy among health professionals as well as policy makers has been emphasised.7,8 Each of nine core functions specified for public health practice in Australia invites an evidence based approach (box 1).9 Yet very little is known about the capacity for EBP specifically among professionals working in population health. What has been revealed through studies undertaken with related disciplines inspires little confidence. For example, general practitioners’ skills are lacking.10,11 Self assessment of skills for EBP is overly optimistic.12 When presented with hypothetical scenarios, epidemiologists disagree in their attribution of causality.13 Time constraints have been identified as a major barrier to EBP.11 Lack of organisational support also mitigates EBP.14
Box 1 Core functions of population health practice in Australia9
Assess, analyse, and communicate population health needs and community expectations
Prevent and control communicable and non-communicable diseases and injuries throughout risk factor reduction, education, screening, immunisation, and other interventions
Promote and support healthy lifestyle and behaviours through action with individuals, families, communities and wider society
Promote, develop, and support healthy public policy, including legislation, regulation, and fiscal measures
Plan, fund, manage, and evaluate health gain and capacity building programmes designed to achieve measurable improvements in public health status, and to strengthen skills, competencies, systems, and infrastructure
Strengthen communities and build social capital through consultation, participation, and empowerment
Promote, develop, support, and initiate actions to ensure safe and health environments
Promote, develop, and support healthy growth and development throughout all life stages
Promote, develop, and support actions to improve the health status of Aboriginal and Torres Strait Islander people and other vulnerable groups
New South Wales (NSW) is the most populous Australian state.15 Over the past decade, divisions of population health have emerged as efficient organisational structures to bring together services with a common vision of the “new” public health.6,16 In one region, namely South Western Sydney, key mortality and morbidity indicators confirm profound inequity in population health.17 In 2002, the Division of Population Health (DPH) serving this population commenced a review of its mission and strategic directions in response to organisational changes within South Western Sydney Area Health Service (SWSAHS). We undertook this needs assessment as a baseline measure of capacity for EBP in our workforce.
At the time of conducting this survey, the SWSAHS Division of Population Health comprised:
Drug and alcohol services (n = 37 staff)
Public health (n = 20 staff)
Health promotion (n = 16 staff)
Refugee health (n = 9 staff)
Community paediatrics (n = 8 staff)
Epidemiology unit (n = 4 staff)
Centre for Health Equity Training Research and Evaluation (CHETRE) (n = 4 staff)
Academic general practice unit (n = 3 staff)
Oral health (n = 2 staff)
Aboriginal health (n = 1 staff)
In August 2002, all eligible staff in the Division of Population Health other than the two authors (n = 104) first were sent a one page letter in advance of our survey to increase response rate.18 Five days later, questionnaires were mailed with covering letters and reply paid envelopes. Reminders were implemented 5, 24, 31, and 40 days after initial mail out. Data were kept strictly confidential.
Current familiarity with EBP and perceptions of organisational support
We asked respondents firstly to rate their level of need to access evidence; secondly, their level of need to understand descriptive evidence (that is, data about the extent of a population problem) and thirdly, their level of need to understand interventional evidence (that is, data about the effectiveness of strategies to make a difference to a population outcomes). We also asked about their levels of need in literature searching; critical appraisal, and biostatical terms. For each of these items, respondents were provided with a five point scale (“no need” to “high need”). Our next question asked respondents to self assess the extent to which they needed to increase their own capacity for EBP in their current position (“no need” to “high need”).
Respondents also were asked to indicate how often, in the past 12 months, they had referred to their immediate manager, a more senior manager, the Epidemiology Unit, external epidemiologists, their peers working in other Area health services, staff working in NSW Health (our central government office) and university based academics specifically for EBP. For each item, we provided five response options (“never”, “once”, “quarterly”, “monthly”, “weekly”).
Comparative views of self and staff
Participants were asked to indicate their own views towards current promotion of EBP in population health and, then, their perceived views about those of their colleagues, their immediate manager and senior management of the Division towards EBP, using a five point scale for each (“extremely cynical” to “extremely positive”).
EBP skills and “framing effect”
Respondents then were asked to rate their understanding of each of 21 EBP terms, using a five point scale (“not at all confident”, “a little confident”, “moderately confident”, “quite confident”, “very confident”).
We next included two case scenarios adopted from Fahey et al to assess whether our workforce was influenced by “framing effect”.19 For each scenario, respondents were asked to indicate how likely they would be to agree to implement each programme in our local region, using a five point scale (“strongly disagree”, “disagree”, “neither agree nor disagree”, “agree”, “strongly agree”). Our first scenario was based on published data of a meta-analysis about the effectiveness of colorectal cancer screening.20 Identical data for programme effectiveness were presented as relative risk reduction (RRR), absolute risk reduction (ARR), number needed to screen (NNS), and no reduction in all cause mortality (box 2, available to view on the journal web site http://www.jech.com/supplemental). Our second scenario was based on published data from a non-randomised trial to assess the effectiveness of a smoking cessation programme.21 Identical data were presented as RRR, ARR, percentage of event free patients (EFP) and number needed to treat (NNT) (box 2, available to view on the journal web site http://www.jech.com/supplemental).
Personal and professional details
We also collected demographic data, namely sex; age bracket; highest level of education reached, having medical qualification, years of being in population health workforce, employment status (casual, temporary part time, temporary full time, permanent part time and permanent full time), and occupational category (directors (level I), coordinators/senior managers (level II), and officers/project staff (level III)). Occupational category (I, II, or III) was used as a predictor variable to analyse data based on staff levels of decision making in their current practice. To determine functions of staff in their specific position, participants were asked to indicate the required level of each of 10 competency areas in population health in their current position, as specified by the NPHP22 (“negligible”, “low”, “moderate”, “high”). We added two more items to this list of competencies, namely “Teaching” and “Research”.
Firstly, descriptive statistics were calculated to describe responses to questionnaire items. In each table, cumulative percentages may not add to 100% because of rounding. When required, univariate analyses were performed to examine relations between predictor and outcome variables. Univariate relations were assessed using Pearson’s χ2 test. McNemar’s test was used to compare for paired proportions.
Three scales were constructed, namely
A “need for evidence” scale by summing across three component Likert scale items. Missing data were excluded (possible scores ranged from 3 to 15).
A “need for skills” scale by summing across three component Likert scale items. Missing data were excluded (possible scores ranged from 3 to 15).
An “EBP competency scale” by summing across 21 component Likert scale items. Missing data were excluded (possible scores ranged from 21 to 105).
Univariate associations with these scales were assessed using either Student’s t test or analyses of variance. All analyses were performed using SPSS 11.0 for Windows.
This study was approved by the South Western Sydney Area Health Service Human Research Ethics Committee in July 2002. Ethics approval also was obtained from the University of New South Wales Human Research Ethics Committee in August 2002.
From 104 eligible staff, we received 76 questionnaires (73% response fraction). There was no association between respondents’ sex and response rate (p = 0.51). Response rate also was not associated with occupational category (p = 0.25) or having medical qualification (p = 0.20). Response rates by services within the division ranged from 60% to 100% (mode = 100%, median = 100%). No other characteristics of non-responders were available to assess response bias.
Professional characteristics of respondents and views about EBP
Table 1 summarises demographic characteristics of workforce respondents. Directors (level I) comprised 13% of the sample. In response to our question about respondents’ required level of competency in their current position, “high need” was expressed by at least one quarter of respondents for each of 12 key competency areas in population health as follows: communication (78%), health promotion (38%), health policy (30%), teaching, (29%) research (28%), management (28%), healthcare evaluation (26%), information management (25%), risk assessment/management (22%), infectious diseases, (21%) and epidemiology and biostatistics (13%). None indicated health economics as “highly needed” in their current position. Four respondents (5%) added “implementation” or “intervention management” as additional competency areas.
Table 2 displays respondents’ perceptions of their current EBP needs. Respondents aged 40 years or over were significantly more likely to report “high self assessed need” to increase their capacity in EBP than younger respondents (69% v 45%) (p = 0.041). Non-medical respondents without a Masters qualification also were significantly more likely than others to report “high self assessed need” (p = 0.022). Sex, occupational category, employment status, and having medical qualification were not statistically associated with perceived need to increase EBP capacity.
The calculated range of scores for “need for evidence” was 3 to 15 (median = 13, mode = 15). There were no statistical association between “need for evidence” scores and occupational category. “Need for evidence” scores were significantly higher among respondents with high “self assessed need” to increase capacity in EBP than those with low “self assessed need” (55% v 42%) (p = 0.001).
Similarly, the calculated range of scores for “need for skills” scale was 3–15 (median = 12, mode 15). There were no statistical association between “need for skills” scores and occupational category. Respondents’ “need for skills” scores were not significantly associated with self assessed need to increasing capacity in EBP.
Table 3 summarises participants’ use of EBP support in previous 12 months. The most frequently cited source for support was their immediate manager (table 3).
As table 4 shows, responses of participating staff about EBP were positive. The proportion of respondents who indicated “extremely positive” views towards the current promotion of EBP in population health was significantly greater than the proportion of respondents indicating that this was the view of their colleagues however (21% v 8%) (p<0.006). In addition, the proportion of respondents who perceived the view of their immediate manager towards EBP as “extremely positive” was statistically greater than the proportion of respondents who indicated their own view was “extremely positive” (46% v 21%) (p<0.000).
Understanding of EBP technical terms
Table 5 summarises respondents’ understanding of 21 EBP technical terms. An EBP competency scale was calculated as explained in Methods. The range of scores was 21–105 (median = 57, mode = 81). “Highest level of education” was significantly associated with EBP competency scores (p<0.000). “Having medical qualification” also was significantly associated with higher EBP competency scores (p<0.000). Higher EBP scores were not associated with sex, age group, occupational category, employment status, and years of practice in population health. Higher EBP scores also were not statistically associated with respondents’ perceived need for increasing capacity in EBP.
Table 6 summarises responses about how worthwhile each colorectal cancer screening programme would be to implement locally. The proportion of participants who strongly agreed with implementation of programme A (RRR) was statistically greater than the proportion of respondents who strongly agreed with implementation of programme C (NNS) (p = 0.008). No other statistical differences were found. In response to this scenario, 12 (16%) respondents gave identical responses to each of the four options posed—that is, they were not influenced by “framing effect”. Yet 58 (76%) respondents changed their level of agreement from option to option (responses missing for six). Of those 12 not influenced by “framing effect”, all except one (n = 11, 92%) indicated “neither agree nor disagree” to implement three options. There were no statistically significant differences in characteristics of respondents influenced by framing effect, including EBP competency scores, compared with those who were not influenced (data available from authors).
Table 6 also summarises responses to four smoking cessation scenarios. The proportion of respondents who indicated “strongly agree” to implement programme A (data presented as RRR) was significantly greater than the proportion of respondents who indicated “strongly agree” to implement programme B (data presented as ARR) (p = 0.002). It also was higher than the proportion of respondents who indicated “strongly agree” to implement programme D (data presented as NNT) (p = 0.003). No other statistical difference was found. In response to this scenario, 14 (18%) respondents were not influenced by “framing effect”. Only seven of these had not also been influenced by “framing effect” in the first scenario. In this second scenario, there also were no statistically significant differences in characteristics of respondents who were influenced by framing effect compared with those who were not (data available from authors).
While there is considerable narrative rhetoric about EBP, our study is among the first to assess quantitatively the capacity of a population health workforce newly committed to EBP. Our high response rate shows interest about EBP among staff. Our findings also show unmet need for training and skills development. For example, more than half indicated “high self assessed need” to increase own capacity in EBP. “High self assessed need” was reported by significantly more respondents without a Masters qualification than those who had obtained at least this level of academic attainment. “High self assessed need” also was associated with respondents’ perception of their “need for evidence” in their current position. Yet there was no such relation between respondents’ self assessed need to increase their capacity in EBP and their self assessed need for development in skills such as critical appraisal.
We also demonstrated that “self assessed need” to increase capacity in EBP was not associated with “EBP competency scores”. This was unexpected. The level of understanding of EBP technical terms did not determine respondents’ perceived need to increase their capacity in EBP. Managerial seniority of respondents was not associated with their level of understating of EBP technical terms. However, respondents with higher level of tertiary education, specifically medical staff, were more likely to report greater confidence in understanding of EBP technical terms.
Capacity to understand and apply interventional evidence is especially valued by population health professionals.
Capacity for evidence based practice is more likely among those with postgraduate qualifications.
Staff need support from senior management to acquire skills necessary to apply an evidence based approach to public health.
Systematic needs assessment enables a thoughtful response to workplace training and skills development.
To further explore how respondents applied their current skills, we used two scenarios reflecting common challenges in decision making in public health. Most respondents (80% or more) were influenced by “framing” when interpreting the interventional evidence we provided. In both scenarios, highest support was indicated for that programme whose benefit was described as RRR rather than ARR. Similar susceptibility to “framing” has been shown in studies of health service managers,19 clinicians,23 and consumers.24 As respondents in our needs assessment disclosed a high need for interventional evidence, their susceptibility to “framing effect” when provided with such evidence behoves redress.
Our study also examined aspects of the workplace that could be improved through comprehensive organisational development. Although “need for evidence”, “need for skills”, and “EBP competency scores” were not statistically associated with respondents’ occupational categories, immediate managers were most commonly cited as the first reference point for EBP. Respondents indicated that the views of these immediate managers towards EBP were more positive than their own while the views of their colleagues were not.
This regional workforce is diverse in its roles and responsibilities with respect to core competencies proposed for public health practice.22 Our training and development programmes in EBP must accommodate heterogeneous needs, perceptions, and workplace demands. Our needs assessment also invites particular targeting of level II (coordinators/senior managers) and level III staff (officers/project staff) even though capacity at level I (directors) is not yet ideal. Respondents perceived that the attitudes of managers were more positive towards EBP than other staff. Therefore, all staff will probably benefit from strategies to change organisational culture and incentives to reward evidence based policy and practice.
We thank all staff members of SWSAHS Division of Population Health who participated in this study; Executive Directors of the Division for reviewing the questionnaire; Mr David Smith, Cancer Council New South Wales; Ms Mary Osborn, FPA Health and Ms Victoria Toulkidis, then at the Royal Australasian College of Physicians, for reviewing draft questionnaires; Ms Tracey Bruce for graphic design and Mr Mark Thornell, Business Manager, SWSAHS Division of Population Health for organisational support.
Conflicts of interest: none declared.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.