METHODS

Survey administration

At the time of conducting this survey, the SWSAHS Division of Population Health comprised:

• Drug and alcohol services (n = 37 staff)

• Public health (n = 20 staff)

• Health promotion (n = 16 staff)

• Refugee health (n = 9 staff)

• Community paediatrics (n = 8 staff)

• Epidemiology unit (n = 4 staff)

• Centre for Health Equity Training Research and Evaluation (CHETRE) (n = 4 staff)

• Academic general practice unit (n = 3 staff)

• Oral health (n = 2 staff)

• Aboriginal health (n = 1 staff)

In August 2002, all eligible staff in the Division of Population Health other than the two authors (n = 104) first were sent a one page letter in advance of our survey to increase response rate.¹⁸ Five days later, questionnaires were mailed with covering letters and reply paid envelopes. Reminders were implemented 5, 24, 31, and 40 days after initial mail out. Data were kept strictly confidential.

Survey instrument

Our self administered questionnaire included previous survey items^8,10,19,20 as well as items generated de novo.

Current familiarity with EBP and perceptions of organisational support

We asked respondents firstly to rate their level of need to access evidence; secondly, their level of need to understand descriptive evidence (that is, data about the extent of a population problem) and thirdly, their level of need to understand interventional evidence (that is, data about the effectiveness of strategies to make a difference to a population outcomes). We also asked about their levels of need in literature searching; critical appraisal, and biostatical terms. For each of these items, respondents were provided with a five point scale (“no need” to “high need”). Our next question asked respondents to self assess the extent to which they needed to increase their own capacity for EBP in their current position (“no need” to “high need”).

Respondents also were asked to indicate how often, in the past 12 months, they had referred to their immediate manager, a more senior manager, the Epidemiology Unit, external epidemiologists, their peers working in other Area health services, staff working in NSW Health (our central government office) and university based academics specifically for EBP. For each item, we provided five response options (“never”, “once”, “quarterly”, “monthly”, “weekly”).

Comparative views of self and staff

Participants were asked to indicate their own views towards current promotion of EBP in population health and, then, their perceived views about those of their colleagues, their immediate manager and senior management of the Division towards EBP, using a five point scale for each (“extremely cynical” to “extremely positive”).

EBP skills and “framing effect”

Respondents then were asked to rate their understanding of each of 21 EBP terms, using a five point scale (“not at all confident”, “a little confident”, “moderately confident”, “quite confident”, “very confident”).

We next included two case scenarios adopted from Fahey et al to assess whether our workforce was influenced by “framing effect”.¹⁹ For each scenario, respondents were asked to indicate how likely they would be to agree to implement each programme in our local region, using a five point scale (“strongly disagree”, “disagree”, “neither agree nor disagree”, “agree”, “strongly agree”). Our first scenario was based on published data of a meta-analysis about the effectiveness of colorectal cancer screening.²⁰ Identical data for programme effectiveness were presented as relative risk reduction (RRR), absolute risk reduction (ARR), number needed to screen (NNS), and no reduction in all cause mortality (box 2, available to view on the journal web site http://www.jech.com/supplemental). Our second scenario was based on published data from a non-randomised trial to assess the effectiveness of a smoking cessation programme.²¹ Identical data were presented as RRR, ARR, percentage of event free patients (EFP) and number needed to treat (NNT) (box 2, available to view on the journal web site http://www.jech.com/supplemental).

Personal and professional details

We also collected demographic data, namely sex; age bracket; highest level of education reached, having medical qualification, years of being in population health workforce, employment status (casual, temporary part time, temporary full time, permanent part time and permanent full time), and occupational category (directors (level I), coordinators/senior managers (level II), and officers/project staff (level III)). Occupational category (I, II, or III) was used as a predictor variable to analyse data based on staff levels of decision making in their current practice. To determine functions of staff in their specific position, participants were asked to indicate the required level of each of 10 competency areas in population health in their current position, as specified by the NPHP²² (“negligible”, “low”, “moderate”, “high”). We added two more items to this list of competencies, namely “Teaching” and “Research”.

Data analysis

Firstly, descriptive statistics were calculated to describe responses to questionnaire items. In each table, cumulative percentages may not add to 100% because of rounding. When required, univariate analyses were performed to examine relations between predictor and outcome variables. Univariate relations were assessed using Pearson’s χ² test. McNemar’s test was used to compare for paired proportions.

Three scales were constructed, namely

• A “need for evidence” scale by summing across three component Likert scale items. Missing data were excluded (possible scores ranged from 3 to 15).

• A “need for skills” scale by summing across three component Likert scale items. Missing data were excluded (possible scores ranged from 3 to 15).

• An “EBP competency scale” by summing across 21 component Likert scale items. Missing data were excluded (possible scores ranged from 21 to 105).

Univariate associations with these scales were assessed using either Student’s t test or analyses of variance. All analyses were performed using SPSS 11.0 for Windows.

This study was approved by the South Western Sydney Area Health Service Human Research Ethics Committee in July 2002. Ethics approval also was obtained from the University of New South Wales Human Research Ethics Committee in August 2002.

RESULTS

Response rate

From 104 eligible staff, we received 76 questionnaires (73% response fraction). There was no association between respondents’ sex and response rate (p = 0.51). Response rate also was not associated with occupational category (p = 0.25) or having medical qualification (p = 0.20). Response rates by services within the division ranged from 60% to 100% (mode = 100%, median = 100%). No other characteristics of non-responders were available to assess response bias.

Professional characteristics of respondents and views about EBP

Table 1 summarises demographic characteristics of workforce respondents. Directors (level I) comprised 13% of the sample. In response to our question about respondents’ required level of competency in their current position, “high need” was expressed by at least one quarter of respondents for each of 12 key competency areas in population health as follows: communication (78%), health promotion (38%), health policy (30%), teaching, (29%) research (28%), management (28%), healthcare evaluation (26%), information management (25%), risk assessment/management (22%), infectious diseases, (21%) and epidemiology and biostatistics (13%). None indicated health economics as “highly needed” in their current position. Four respondents (5%) added “implementation” or “intervention management” as additional competency areas.

Table 2 displays respondents’ perceptions of their current EBP needs. Respondents aged 40 years or over were significantly more likely to report “high self assessed need” to increase their capacity in EBP than younger respondents (69% v 45%) (p = 0.041). Non-medical respondents without a Masters qualification also were significantly more likely than others to report “high self assessed need” (p = 0.022). Sex, occupational category, employment status, and having medical qualification were not statistically associated with perceived need to increase EBP capacity.

The calculated range of scores for “need for evidence” was 3 to 15 (median = 13, mode = 15). There were no statistical association between “need for evidence” scores and occupational category. “Need for evidence” scores were significantly higher among respondents with high “self assessed need” to increase capacity in EBP than those with low “self assessed need” (55% v 42%) (p = 0.001).

Similarly, the calculated range of scores for “need for skills” scale was 3–15 (median = 12, mode 15). There were no statistical association between “need for skills” scores and occupational category. Respondents’ “need for skills” scores were not significantly associated with self assessed need to increasing capacity in EBP.

Table 3 summarises participants’ use of EBP support in previous 12 months. The most frequently cited source for support was their immediate manager (table 3).

As table 4 shows, responses of participating staff about EBP were positive. The proportion of respondents who indicated “extremely positive” views towards the current promotion of EBP in population health was significantly greater than the proportion of respondents indicating that this was the view of their colleagues however (21% v 8%) (p<0.006). In addition, the proportion of respondents who perceived the view of their immediate manager towards EBP as “extremely positive” was statistically greater than the proportion of respondents who indicated their own view was “extremely positive” (46% v 21%) (p<0.000).

Understanding of EBP technical terms

Table 5 summarises respondents’ understanding of 21 EBP technical terms. An EBP competency scale was calculated as explained in Methods. The range of scores was 21–105 (median = 57, mode = 81). “Highest level of education” was significantly associated with EBP competency scores (p<0.000). “Having medical qualification” also was significantly associated with higher EBP competency scores (p<0.000). Higher EBP scores were not associated with sex, age group, occupational category, employment status, and years of practice in population health. Higher EBP scores also were not statistically associated with respondents’ perceived need for increasing capacity in EBP.

Framing effect

Table 6 summarises responses about how worthwhile each colorectal cancer screening programme would be to implement locally. The proportion of participants who strongly agreed with implementation of programme A (RRR) was statistically greater than the proportion of respondents who strongly agreed with implementation of programme C (NNS) (p = 0.008). No other statistical differences were found. In response to this scenario, 12 (16%) respondents gave identical responses to each of the four options posed—that is, they were not influenced by “framing effect”. Yet 58 (76%) respondents changed their level of agreement from option to option (responses missing for six). Of those 12 not influenced by “framing effect”, all except one (n = 11, 92%) indicated “neither agree nor disagree” to implement three options. There were no statistically significant differences in characteristics of respondents influenced by framing effect, including EBP competency scores, compared with those who were not influenced (data available from authors).

Table 6 also summarises responses to four smoking cessation scenarios. The proportion of respondents who indicated “strongly agree” to implement programme A (data presented as RRR) was significantly greater than the proportion of respondents who indicated “strongly agree” to implement programme B (data presented as ARR) (p = 0.002). It also was higher than the proportion of respondents who indicated “strongly agree” to implement programme D (data presented as NNT) (p = 0.003). No other statistical difference was found. In response to this scenario, 14 (18%) respondents were not influenced by “framing effect”. Only seven of these had not also been influenced by “framing effect” in the first scenario. In this second scenario, there also were no statistically significant differences in characteristics of respondents who were influenced by framing effect compared with those who were not (data available from authors).

DISCUSSION

While there is considerable narrative rhetoric about EBP, our study is among the first to assess quantitatively the capacity of a population health workforce newly committed to EBP. Our high response rate shows interest about EBP among staff. Our findings also show unmet need for training and skills development. For example, more than half indicated “high self assessed need” to increase own capacity in EBP. “High self assessed need” was reported by significantly more respondents without a Masters qualification than those who had obtained at least this level of academic attainment. “High self assessed need” also was associated with respondents’ perception of their “need for evidence” in their current position. Yet there was no such relation between respondents’ self assessed need to increase their capacity in EBP and their self assessed need for development in skills such as critical appraisal.

We also demonstrated that “self assessed need” to increase capacity in EBP was not associated with “EBP competency scores”. This was unexpected. The level of understanding of EBP technical terms did not determine respondents’ perceived need to increase their capacity in EBP. Managerial seniority of respondents was not associated with their level of understating of EBP technical terms. However, respondents with higher level of tertiary education, specifically medical staff, were more likely to report greater confidence in understanding of EBP technical terms.

To further explore how respondents applied their current skills, we used two scenarios reflecting common challenges in decision making in public health. Most respondents (80% or more) were influenced by “framing” when interpreting the interventional evidence we provided. In both scenarios, highest support was indicated for that programme whose benefit was described as RRR rather than ARR. Similar susceptibility to “framing” has been shown in studies of health service managers,¹⁹ clinicians,²³ and consumers.²⁴ As respondents in our needs assessment disclosed a high need for interventional evidence, their susceptibility to “framing effect” when provided with such evidence behoves redress.

Our study also examined aspects of the workplace that could be improved through comprehensive organisational development. Although “need for evidence”, “need for skills”, and “EBP competency scores” were not statistically associated with respondents’ occupational categories, immediate managers were most commonly cited as the first reference point for EBP. Respondents indicated that the views of these immediate managers towards EBP were more positive than their own while the views of their colleagues were not.

This regional workforce is diverse in its roles and responsibilities with respect to core competencies proposed for public health practice.²² Our training and development programmes in EBP must accommodate heterogeneous needs, perceptions, and workplace demands. Our needs assessment also invites particular targeting of level II (coordinators/senior managers) and level III staff (officers/project staff) even though capacity at level I (directors) is not yet ideal. Respondents perceived that the attitudes of managers were more positive towards EBP than other staff. Therefore, all staff will probably benefit from strategies to change organisational culture and incentives to reward evidence based policy and practice.