Interest in methods of implementation monitoring in healthcare evaluation is growing, as a result of the recognition that the findings of an intervention trial are unhelpful unless a precise picture of what was evaluated is given.^1,2 However, information about the process of programme implementation is only part of the story in the interpretation of the findings of intervention trials. Contexts vary widely. The effectiveness of any particular programme is expected to vary according to a range of contextual factors, such as staff morale and competence, the level of resources a programme receives in different settings, and the support of other agencies in the community. This means that statements about programme effectiveness that are based solely on traditional methods of programme evaluation, in contrast with programme in context evaluation, may be misleading. A recent paper in this journal by Rychetnik and colleagues has highlighted this issue.³ They further pointed out that across the world none of the 17 checklists that are in common use to assess the quality of evidence for a public health intervention contain details on how the differential effect of context (however defined) could be taken into account.³ Based on a search of Medline and Embase since 1966 using the term “community trial” and “cluster randmomised community trial”, we could find no cluster randomised intervention trials in geographical communities that had attempted to assess systematically the extent to which the context into which a whole community intervention was introduced may have contributed to the differential uptake, success, or sustainability of the intervention.

In this paper, we describe a combination of qualitative and quantitative methods in place to track the unfolding of a large scale primary care and community development intervention in maternal health in Australia. In particular, we focus on the methods developed to describe the nature of the context before and during the three year intervention trial. We are taking context to mean the social, political, and organisational setting within which an intervention is implemented.³ Pooling insights across our multiple data sources will allow us to elucidate what aspects of context appear to matter and how the intervention could be adjusted accordingly.

PROCESS EVALUATION METHODS WITHIN INTERVENTION STUDIES

Methods of process evaluation in intervention studies typically focus on a number of core issues. These include the reach of the programme into the target group, the degree to which its various components are implemented, the acceptability of the intervention (satisfaction of parties involved), and the extent to which the intervention meets agreed quality criteria.^4,5 Although process evaluation methods are well established in the literature in education and social welfare programmes, the uptake of these methods in health care has been slow. As a consequence, when investigators think an intervention has worked, they have had inadequate data to account for what the intervention actually was or why it had its effects.⁶ Alternatively, when intervention studies have failed to produce expected effects, investigators are plagued by the thought that their results might have been explained by better process evaluation.^7,8 As well as explaining what actually happened, properly conducted process evaluation can alert investigators to early problems with programme implementation that can be rectified before proceeding to a full scale, expensive trial.⁴

A belated conclusion of the early intervention trials in cardiovascular disease prevention was that “communities are complex dynamic entities.”(page 582)⁸ Yet, for the most part cluster randomised, community intervention trials acknowledge only very simple differences in communities in the sense that pre-stratification or matching on cluster size or other simple potential confounding factors (such as urban/rural location) may be recommended.⁹ The recommended approach to the evaluation of the intervention’s unfolding has not changed, remaining for the most part steadfastly fixed on measuring the intervention mechanistically, as a “dose” of various components sent (for example, media messages broadcast, information kits distributed, professional education workshops provided) and received (for example, media messages recognised, information read and understood, appraisals of teaching processes in professional education components).¹⁰ Use of such methods to determine what has happened in interventions presupposes what has taken place (that is, looking for what one wants to see). While it is sensible to investigate whether what was “promised” actually happened, it is also prudent to explore more naturalistically how the intervention might vary according to the different community contexts into which it is placed. This would allow one to delve deeper into the intervention than the usual process evaluation methods. Such methods might also reveal the indigenous theory of the intervention.¹¹ By this, we mean the causal assumptions and principles that govern decision making in the intervention, which may or may not match the rhetoric of the theory claimed to inform the intervention design. The rest of this paper describes the methods we have developed to meet some of these challenges within a cluster randomised, community intervention trial to promote maternal health.

THE PRISM INTERVENTION TRIAL

PRISM, Program of Resources, Information and Support for Mothers is a coordinated and comprehensive primary care and community based strategy to promote maternal health after childbirth and reduce postnatal depression. The study involves 16 local government areas in the state of Victoria, Australia and about 22 000 women. It started in late 1998 and the first results are expected in 2004. The rationale for the intervention, and the evidence on which it is based are described by the PRISM designers, our collaborators.¹² The intervention is anchored and facilitated in each of the eight intervention communities by a full time community development officer (CDO) working with a local steering committee. The chief components are professional education and development for general practitioners and maternal and child health nurses, a mothers’ information and resource kit, schemes to promote social contact/”befriending” among mothers, and a scheme to encourage local business traders to donate service vouchers for mothers (for example, to encourage time out for a swim, a free cup of coffee, etc). PRISM also sets out to create mother friendly environments within local communities. Two part time coordinators (also trial investigators) took responsibility for overall management of the intervention and ongoing support to the CDO in each site.

EcoPRISM is a separately funded and administered “sister” research project to PRISM. Funds within the main PRISM trial are used to measure health outcomes in mothers (impact on health status and postnatal depression) and for traditional aspects of process evaluation (for example, satisfaction and implementation of key components). Funds within EcoPRISM are devoted to an economic evaluation and to detect broader intervention effects, reflecting an ecological systems theory/social ecological view of the intervention.¹³ In brief, this means that PRISM’s potential is seen at multiple, complex, interacting levels.¹⁴ Accordingly, if successful, its effects are hypothesised to be multiple and multiplied beyond the immediate target group across the community. This model would also suggest that the degree to which the intervention is taken up in each community is partially dependent on pre-existing local factors such as history of innovation, quality of the primary care team environments, the degree of prior collaboration among key agencies, and the presence and vitality of local PRISM “champions”.¹⁵ Compatibility of the intervention with the local context might also predict sustainability of the intervention after primary resources are withdrawn.¹⁶ As such, EcoPRISM research methods are designed to supplement the PRISM trial methods and provide insights into how complexities of local context illuminate our understanding of what the PRISM intervention is in practice and the levers and constraints on its evolution during the trial period.

PROCESS AND CONTEXT EVALUATION METHODS WITHIN THE ECOPRISM STUDY

The notion of context evaluation as an additional category to the more familiar process, impact, and outcome evaluation was highlighted in a comprehensive review undertaken by Israel and colleagues.¹⁷ If we take process evaluation to be confined to questions about the intended delivery of the intervention, context evaluation pertains to the naturally occurring events and influences in the setting or environment of the intervention that might act to contribute to or impede intervention success.

The methods we are using capture aspects of both intervention delivery as well as aspects of pre-existing context. We also describe methods that capture the immediate effects of the PRISM intervention in terms of the creation of mother friendly environments, events, and actions by local agencies to promote maternal health, policy changes, and so on. Strictly speaking, it could be argued that this is impact evaluation, but their accumulation over time represents important symbolic achievements and tangible change in the broader organisational context. This may in turn spawn enthusiasm and energy for the intervention among local participants (that is, these impacts provide the context for further actions). Our methods are also designed to capture the thinking that occurs within such dynamic contexts of practice, as a way of revealing intervention theory.

The four principal methods we are using to capture the context within which PRISM is unfolding and the thinking and practices that are tied to this are (a) inter-organisational network surveys conducted both before and after intervention, (b) prospective narratives from the CDOs throughout their employment period, (c) impact logs throughout the trial period, and (d) key informant interviews at the conclusion of the intervention. We end with the discussion of the particular challenges these methods bring.

Inter-organisational network analysis

The PRISM intervention engages local maternal and child health teams and local general practitioners in the better care of mothers. Maternal and child health teams are also nested within local government municipalities, whose community services and urban planning departments can also do much to make communities better places for mothers. PRISM requires collaborative activity across multiple partner agencies. Our first hypothesis is that PRISM will increase collaboration across agencies by creating new relationships and increase the strategic position of key agencies in the community. Our second hypothesis is that uptake of the PRISM intervention will be smoother in those communities where agencies are familiar with each other and have a history of collaboration. We also think that sustainability of the PRISM intervention (if viewed as successful) will be more assured in these communities.

Network analysis is a quantitative method to assess the nature and strength of relational ties between actors (in our case, agencies, or organisations) in a defined population.¹⁸ In community based health promotion, network analysis was first used by Wickizer and colleagues in 1993 to compare the degree of coordination among agencies across different types of programme areas (for example, comparing substance misuse agency networks with adolescent pregnancy prevention networks).¹⁹ Network analysis has subsequently been used in single community case studies. For example, to describe how HIV/AIDS service organisations collaborate to provide care within specific geographical regions²⁰ and to describe the dynamics of how inter-organisational coalitions work.²¹ In the PRISM trial, we are using inter-organisational network analysis to compare the degree of contact and collaboration among community agencies in each of the 16 communities before and after the intervention.

The method entails the construction of “bounded” samples of about 25 relevant agencies in each community.²² That is, the investigators fix the size and make up of the network and investigate the relational properties within it.²² To be eligible for the sample an agency has to meet certain criteria. In our case, this meant organisations most likely to be (1) directly involved in the intervention by the CDOs (for example, a family, child or maternal health service agency, a part of local government in involved in community services, or environment and planning), or (2) an organisation that has contact with new mothers (for example, Mothers Association), or (3) an organisation considered by key informants in the community to be a “mover and shaker” in the community, that is, part of the local machinery in addressing community needs (for example, Rotary and Lions clubs).

Network analyses provide two summary statistics. The density score of the network captures the number of ties between organisations as a percentage of all possible ties (so a high score is a strongly inter-connected community).¹⁸ The centrality scores tell us how central or marginal a particular agency is within the network (a high score means highly centralised indicating that this agency has direct ties with many others in the network).¹⁸ The method allows us to compare across communities whether or not, an organisation with the same roles and responsibilities (for example, the maternal and child health team) has the same strategic position in its respective network. The way in which PRISM is embraced in a community may depend on whether the maternal and child health team is very centred in its community network. Thus, the position of “key players” in each community’s networks is a contextual factor that might affect the intervention’s success.

We are assessing network ties in telephone surveys with representatives (mostly managers or directors) of organisations in each of the 16 intervention and control communities within the PRISM trial. To get an accurate picture of the agency’s (as compared with an individual worker’s) network, questions have been confined to organisational ties and ones that can be substantiated with documentary sources. As each agency is asked about each other, it is possible to follow up and correct any discrepant reporting about a tie as it occurs. In some cases, multiple informants may be drawn from an agency to complete the picture. Key relationships being assessed include whether or not organisations have contact with each other, coordinate activities together, sit on committees together, and engage in joint planning together. We end the interviews with an open-ended question to capture any other relational ties. Data from the baseline survey have been fed back to both intervention and control communities to give each organisation a picture of where they fit in the network and how these patterns vary across the rural and metropolitan communities in the PRISM trial.

We originally set out to see if the intervention had an impact on referral relationships across agencies but we dropped this measure at time 2. This was because our indicator was binary and too blunt (presence or absence of a referral relationship) and we found that many of the agencies in our sample (for example, Rotary clubs) were not dealing with “clients” anyway. Instead, at time 2, we took the opportunity to investigate a newly observed phenomenon from the qualitative data, namely the presence of important informal ties across agencies (for example, friendships among local workers).

Narratives of practice: fieldworker accounts of the intervention over time

The primary agent of the PRISM intervention in each community is the CDO. Over a two year period the CDO has been responsible for assisting the trial coordinators to develop information kits and other resource materials for mothers, engaging local organisations in activities and events to make the local environment more “friendly” for mothers (for example, parking spaces in shopping centres, baby feeding facilities), mobilising mothers to support each other, and providing support for the local steering committee. Throughout this time, each CDO has maintained a daily contact sheet to record to whom she has spoken on the phone or met face to face. In addition, each CDO has been responsible for keeping a written diary of her activities and strategies. The diaries are totally unstructured, the only instruction being to write thoughts, feelings, frustrations, plans, and any other issues that come to mind. Interwoven within the diary records are observations about the environment or context for this experience.

Participation by the CDOs was voluntary, refusal was possible, and the material provided was confidential to two members of the EcoPRISM team only. One EcoPRISM team member (TR) read the diaries as they were submitted on a monthly basis and used these to construct follow up regular interviews to explore the themes and issues raised. These confidentiality safeguards provided an opportunity for CDOs to “tell it like it was” in contrast with how they thought the intervention “was supposed to be”, as diaries were not being used to manage the intervention or supervise the CDOs’ work. Over a period of five months each CDO experimented with ways of keeping the diary (for example, hand written, electronic, weekly, daily) until the CDO and the EcoPRISM team were comfortable that the way of recording was minimally disruptive to work routines and able to provide a high quality, in depth account of how the CDO was thinking and what was happening from week to week. All narrative data have been transferred into electronic format and entered into a qualitative data software package, to assist with analysis. Each CDO field diary consists of about 40 000 words of verbatim reflection. At the end of the first four months of the PRISM trial an analysis of the entire qualitative dataset (diaries and follow up interviews) was conducted to (1) provide feedback to the CDOs and PRISM team on issues related to how the intervention was faring so that any issues or problems could be addressed; (2) practise data management and analysis procedures; and (3) demonstrate the utility of the data so as to encourage further cooperation and engagement by CDOs in its collection.

Analysis of the qualitative dataset is proceeding on three levels. A thematic, semiotic, and narrative analysis of the field diary and interview data is being undertaken.^23,24 The thematic analysis has allowed us to interrogate the data for thematic patterns (within the CDO dataset) across all the communities. In doing so, we explore more fully, via a comparative approach, the similarities and differences in intervention implementation. Themes include barriers to intervention implementation, the role of personal relationships in levering access to the community, and the extent of material support provided by the organisations within which the CDOs are located. The semiotic analysis is concerned with the words, signs and symbols that are applied to give meaning in language.²³ The analytical focus here is the way the project and its dynamics are described. That is, how the “official’ project rhetoric was interpreted, translated, subverted, or deployed by CDOs once they were mediating the various socioenvironmental contexts of their practice. Our interest is in how these interpretations create new PRISM language understandings and to assess whether and when the intervention came to be represented as locally owned, a factor known to affect sustainability after the withdrawal of initial funds.¹⁵ We can examine if the project is represented more as a unified technology or a package of resources, or as a change process. The narrative analysis is focused on why the story was told in a particular way, drawing attention to how local participants depict antecedents, causes, and consequences.²⁵ Among other things, this analysis is expected to illuminate the causative reasoning and decision making that governed local adaptation of the intervention. For example, while the trial investigators sought to standardise the components of the educational packages in the intervention, most CDOs felt local pressure to adapt these to local language style and literacy. As such, the diaries are permitting us to explore these usually private contexts of practice²⁶ and the theories that actually drove the intervention, as compared with the theories that the intervention purports to test, described in funding submissions and protocols.

While journaling or reflective writing is becoming increasingly used to monitor professional practice in nursing, teaching, and one to one counselling,^27–29 no large scale community intervention trials have reported using these methods to make sense of how a complex intervention develops and adapts to local context. With reflective personal writing, there is no alternative source to test the validity of what CDOs say (in contrast with how validity is traditionally investigated in epidemiology). However, we are able to interrogate the emerging themes in the one to one interviews that are held with the CDOs in between diary recordings. We are also comparing CDOs’ ideas about local context influences on the intervention in interviews being conducted with other residents and workers in each community. These are described below as the key informant interviews. The CDOs’ experience of writing and recording the project and extent to which they actually “told it like it is” is being investigated in exit interviews with CDOs conducted by an independent evaluator (that is, someone who is not a member of the PRISM or EcoPRISM research teams).

Impact logs: capturing actions, events, and results as they happen

Researchers at the University of Kansas provided access to their workbook for developing observational methods and a coding scheme for monitoring the outcomes of community development initiatives.^30,31 This instrument is used to track “events” in the history of a community based intervention such as obtaining resources, holding a programme launch, gaining media coverage, development of a policy, a change at an environment level, and so on. It allows community projects to track their progress across time. This has been shown to be useful both in detecting community level changes and in providing continuing motivation for community agencies to take part in community initiatives.³⁰ The latter is crucial when the final outcomes of these initiatives (for example, change in health outcomes) are long term. We asked CDOs to record “events” in PRISM in their own community (what happened, who was involved, with what result or impact). The log sets out a number of headings (date, contact name and organisation, purpose/aim of contact, result or impact) and the CDO is required to make a few dots points or write a few brief sentences about events as they happen. The logs are then coded by an EcoPRISM researcher into one or more of 22 descriptive codes that were derived from earlier analysis and refinements of the data. Codes include environment level changes (such as the relocation of parking spaces for mothers), policy level changes (such as the requirement by municipal councils for the inclusion of mother-baby rooms in commercial development applications), and awarding of resources to PRISM (in kind such as the loan of facilities for mothers’ activities or in cash as in awards for salary costs).

The resource codes in the logs fulfil two functions. By recording the additional resources that are drawn into PRISM by the actions of the CDOs and others, they inform efforts to cost the intervention. The availability of resources and the ease with which they can be accessed by the CDOs are also important aspects of context that might explain differences in implementation across the participating communities. In combination with narrative data from the CDO diaries, the event logs are being used to compile a picture of the efforts entailed in gaining access to the physical, financial, and human resources that each CDO needs to implement PRISM such as office supplies, reimbursement of expenses, use of the municipal government motor vehicle. Such access affects the efficiency of the CDO’s actions. It might also impact on intervention-effectiveness more directly, because of any symbolic importance conveyed by the easy availability of material support.

We have engaged in two forms of data quality control. Coding consistency is tested at different time points to ensure that there is consistency from year to year. We also verify the events and impacts that have been recorded by CDOs by checking with an independent source. This might be a second person who can confirm that a resource was provided, or the minutes of a meeting that record that particular transactions or events have taken place. Each year a random sample of 10% of items recorded over a four month period are chosen and verification reports sought from a documentary source or another person in the community.³⁰ For nine months after the CDOs’ employment ended, responsibility for completing the logs shifted to a local (paid) person from each community who had involvement with the project. This will allow us to examine how well local initiatives were sustained.

Key informant interviews—at the end of the intervention, looking back

We are conducting up to 30 interviews in each of the eight communities at the conclusion of PRISM to discover how local players viewed the intervention, factors working for and against its development in the local context and, for other purposes, the range of impacts they consider the intervention to have had. The sample consists of all the main agencies involved in the intervention in each community, and past and present members of the steering committees. The interviews will allow us to follow up and further explore contextual themes emerging from the other datasets and to test and confirm interpretations about the significance of varying contexts within which PRISM has been developed. We will also use the interviews as a way of assessing perceived local impact of the intervention.

It is not unusual to construct a picture of what happened in an intervention and the context within which it was delivered by interviews with key informants, or people in a position to comment on the intervention by virtue of their involvement (or choice not to be involved).^32,33 Qualitative methods are being seen more often within randomised trials.³⁴ What may be unique about our combination of methods is our capacity to compare the prospective data (the narratives and logs of events as they happen) with the retrospective accounts from the interviews. That is, we are exploring “meaning” from different observers and at different time points. Note that in neither the EcoPRISM study nor PRISM study are interviews being conducted with recent mothers. This was a part of the intervention development¹² but not part of the trial because of a concern that the research process might affect the ways mothers complete the outcome questionnaires. However, those mothers involved in steering groups or community organisations are included in our sample.

DISCUSSION

Our methods provide a unique opportunity to examine how community level factors affect intervention success. This is a relation we can explore quantitatively by regressing the health outcomes in each community against their inter-organisational network scores and socioeconomic characteristics of the community at baseline, and qualitatively in the CDO narratives and key informant accounts of the intervention/context dynamic. Our narratives provide insights of the dynamics of how an intervention unfolds that have not been previously described. While we have conducted this work as part of a RCT, our purpose is to draw attention to context level factors that might be thought of as effect modifiers³⁵ and assessed beforehand by way of preparation for a RCT.

The research approach has raised a number of key issues and exciting challenges in the way we think about complex intervention research.³⁶ The first issue is, are we assessing enough? Are our methods sufficiently sensitive? We have cast widely to detect local contextual influences, but it is clear that there are a number of factors in this domain that we are not addressing directly. For example, with the insights we now have from the qualitative data, we regret not inserting into the trial design a quantitative assessment of the work environment of the maternal and child health teams who carry much of the responsibility for the intervention. Our expectation is that the time, energy, quality, and professional confidence of maternal health workers may have as much, if not more, to do with health outcomes to mothers in the trial, than say, whether or not the information kits were distributed properly and rated satisfactorily by mothers. Other contextual factors that might affect the outcome of the trial include the experience and interest of the general practitioners.

The second issue is the impact our research has on the intervention itself. We refer here to the extent to which the evaluation methods may be producing their own effects (Hawthorne effect) and issues of burden for the fieldworkers. Our diary methods were carefully tailored to each CDO’s personal style to reduce burden. That is, some preferred to write long hand; others ended up communicating by long emails. Some stopped writing on occasions when things got too busy in the field and we “back filled” these data using interviews instead. But by creating a reflective writing/talking process in the trial, were we creating “unrealistic” practice? Some CDOs were not used to the process, but they said that they found it a useful professional development device for thinking through their work and planning their strategies. Indeed journaling is becoming much more common place in health promotion and field practice generally as part of an increased emphasis on professional development.^37,38 Our expectations on CDOs, may have been somewhat unusual for the time (1999 and 2000), but they created a professional practice environment that is more widespread now. We found it helpful at the time to talk through these issues with other fieldworkers and researchers and to offer our reasoning processes up for wider scrutiny.^39–41

The third concern is the ethical issues raised by potentially knowing more about an intervention as it happens than investigators who may use more cursory means to assess implementation. A particular concern was the anxiety caused to trial investigators who were called upon to respond to our data about implementation and to work through their position in relation to these. Our discussions have raised interesting new issues about the nature of harm arising in the context of intervention trials. These ethical challenges are presented in a separate paper.⁴² In addition to the widely accepted notion of (outcome) data monitoring committees, we suggest that community interventions require process monitoring committees with starting and stopping rules tied to process and “side effect monitoring” in communities.⁴² Such committees would include community representatives. We are indebted to colleagues in the Netherlands who have been willing to share with us their experiences about conditions when community level preventive interventions should be rethought or renegotiated.^43,44

The fourth issue is how we make sense of the data and the challenges it raises for intervention theory and trial designs of the future. Investigators who look more closely inside the “black box” of an intervention must be prepared for what they might find. The creation of our dataset has prompted the research teams (PRISM and EcoPRISM) to engage in dialogues about how particular data scenarios will be interpreted. This opportunity is provided because traditional process evaluation (focused on the prespecified intervention components) is being collected within the PRISM trial.¹² Questions we can address include, for example, is PRISM more than the sum of its parts (all the key elements combined)? Does the programme have symbolic value over and above its operational components? How is intervention integrity defined? If PRISM were to be replicated, at a minimum what would it be? Is there any basis for advising particular types of communities in the future that they are not yet “ready” for a PRISM? Based on their baseline collaboration patterns in the inter-organisational network analysis, for example? What degree of local resourcing and prior groundwork may be required to make PRISM effective? Seeing “PRISM” as a complex intervention in eight different contexts has prompted these questions and it behoves us, and other investigators in similar trials, to reflect on the theory and logic driving the intervention.⁴⁵ Without this degree of interrogation we cannot really begin to know “what” worked and why. Useful insights here come from Durlak⁴⁶ who asserts that “non-standard” interventions (interventions that look different in different sites) do not require the abandonment of the randomised trial design. But such interventions do require careful elucidation of the process and logic of the intervention so that quality control is not compromised.

It is time that community intervention trials consider these issues more carefully. Especially because, for the most part, outcomes of community intervention trials have been small or non-significant.⁴⁷ Part of this may be simply because investigators go into communities with intervention designs and theories too weak and unsophisticated for the scale of the change process they are seeking to bring about.⁴⁸ With better planning and theorising, “trial and error learning” in community intervention trials could be minimised.⁴⁹