Methodological considerations

Several significant considerations underlying the COPI design are noted here and discussed further in the following section. The study was designed primarily to inform the programme, providing the same degree of qualitative insight that analysis of participation in small-scale settings has done. However, given the scale of the programme and the considerable variations in community mobilisation across it, a new approach was required, combining qualitative and quantitative methods to capture data on complex and sometimes abstract processes of community mobilisation. This was achieved in part by breaking down the community mobilisation process into components (referred to in this account as parameters and indicators) relatable to concepts from participation discourse. These parameters and indicators could be measured and aggregated at the level of individual CBGs as well as state and national levels. Aggregation was facilitated through the use of a weighting mechanism, where weights represented an estimation of the relative importance of various aspects of CBG performance. In addition, a method was devised to quantify qualitative data, such as the level of CBG members' participation in activities, or the extent to which activities contributed to structural change. This standardisation enabled the progress of individual CBGs to be measured from year to year, providing insights on community mobilisation within and across CBGs while capturing a level of nuance that could prompt meaningful reflection and course corrections at the local level.

The COPI was designed as a repeating cross-sectional survey, conducted annually across 32 districts in six states, using an index-based assessment of CBGs' capacity to manage programmes. The initial assessment established a baseline, to be followed by four rounds of surveys from 2010 to 2013.

Two principles drawn from participatory development influenced the process by which the COPI was designed. First, members of the high-risk communities participated actively in all stages of the design, including helping to determine the relative weight to be given to the various dimensions of community mobilisation. Second, a process for sharing data with the CBGs was built into the survey design. The process of data collection and analysis across six states was rapid (taking <3 months in total), and data were initially shared with each CBG as soon as they had been gathered, and then later through linked activities that facilitated participatory learning and planning. The intention was that CBGs could respond to the data and make decisions about the future course of their organisations and activities, and that the experience of participating in the study should be empowering and serve the CBGs directly.

Conceptual roots and the study design process

The development of the COPI was accomplished through a participatory iterative process, in which facilitated discussions and focus groups with high-risk communities were key to each stage. The process included: a review of background material and theory as well as learning from the experiences of Indian CBGs working in HIV prevention; design of the study framework and related indicators and parameters; weighting of indicators; and development and pilot testing of the survey tools. At each of these stages, the participation of high-risk community members was supplemented by input from various academic disciplines (eg, statistician, sociologist, anthropologist, demographer and gender expert). This required careful planning and took more time than if the design had been left to a small team of experts. However, it reflected the desire to develop a survey that reflected the community's own understanding of what was important to effective mobilisation.

The ‘powercube’ model, developed in part by John Gaventa, which focuses on power relations among different stakeholders to assess the strength of communities, was used initially as the basis for a discussion with CBG leaders in order to understand power relations for CBGs within the Avahan Initiative.40 ,41 Based on these discussions, a core design team, including high-risk community members and experts, identified three broad categories of stakeholders whom CBGs need to engage to address issues of vulnerability: (1) the project (HIV prevention intervention), including the implementing NGOs and programme components such as clinics and drop-in centres; (2) the state, that is, governmental bodies and authorities; and (3) wider society, for example, neighbourhood residents, religious leaders, politicians and gatekeepers (gatekeepers might include owners of sex establishments, clients of sex workers, drug dealers, intimate partners and law enforcement authorities). Table 1 uses the stages of the Avahan community mobilisation logic model to illustrate the expected trajectory of CBGs in relation to these stakeholders.31 Over the 10-year period of the Avahan Initiative, CBGs were expected to move towards the more advanced outcomes of Stage 3, where they would take ownership of elements of the intervention, develop procedures and spaces to claim their entitlements from the state, and assert their identity and rights as citizens within wider society.

Further discussion with CBG members, together with a review of the literature on assessment of NGOs42 and self-help groups43 and consideration of the transformative potential of participatory processes,1 identified characteristics of a strong CBG. These included leadership and the ability to strategise independently about all issues affecting the high-risk community; addressing the immediate and long-term needs of the community; transparency of management and decision-making; communication skills; going beyond HIV prevention and care; the ability of the CBG to remain connected to the wider high-risk community and acceptable to it; and scope for community members to participate voluntarily in the CBG's activities, especially those relating to realising their rights and entitlements. Working with this information, the design team collaborated with CBG members and experts to develop the evaluation framework illustrated in box 1.

At the broadest level, the framework posits four dimensions considered essential to the transition readiness of CBGs: (1) leadership, governance and decision-making; (2) sustainability through resource mobilisation and networking; (3) project management; and (4) engagement with the state and wider society. These dimensions are elaborated into eight broad parameters, subdivided into 23 specific indicators (with between two and five indicators per parameter). Each indicator is a practical, operational expression of participatory concepts. Progress on each indicator is assessed through sets of questions. Each possible response to a question is precoded with a score; when the survey is conducted, the scores received for the questions relating to an indicator are totalled to arrive at an overall score for that indicator. Each indicator score is multiplied by the weight assigned to that indicator. The sum of the weighted indicator scores within each parameter yields a parameter score.

The scoring of questions was designed to assess multiple qualitative aspects of an activity. For example, instead of assigning the same score to each CBG where a particular activity takes place, a progressively higher score would be assigned if the planning for that activity was done by paid peer educators, by the CBG's leadership team working with peer educators or by the leadership team without peer educators. (The implicit rationale in this progression is that paid peer educators, although members of the high-risk community, are still part of the NGO-led programme and transition requires independence from the programme.) The same continuum applied to the execution of the activity. In some cases, the nature of the activity—informal, engaging a formal power structure or working for policy-level change—was also considered part of a progression in the levels of a structural intervention and was also scored on a continuum.

The procedure to validate the weights assigned to each of the 23 indicators is an example of the participatory approach to the design of the COPI. The numeric values expressing each indicator's relative significance to the overall process of community mobilisation in the CBG were debated in multiple forums, with experts including Gaventa and Chambers whose approaches have informed the programme, and with members of the high-risk communities, who negotiated together to determine the relative importance of indicators and parameters. In order to relate the parameter scores to transition readiness in a straightforward manner for discussion with the CBGs, a scale was used, dividing the maximum possible score of 100% into seven equally spaced bands (table 2). If a particular parameter score fell within one of the top two bands (‘Vibrant’), the CBG was considered to be ‘transition ready’ with respect to that parameter. (It was understood that the overall transition readiness of a CBG would depend on numerous contextual factors and involve negotiation with different actors, including the government body charged with taking financial responsibility for the intervention.)

Validation of the COPI

The predictive validity of the COPI was tested in several stages. It was not possible to test the validity before implementation, since the gold standard was not known. At that stage, only the content validity could be measured. First, a multiple-scenario analysis was carried out using the adopted design before the actual collection of the data, by inserting all possible types of potential answers into the cells. This analysis was helpful in checking the consistency and validity of possible responses to the questions, and whether the overall analysis aligned with the community mobilisation model for change.31 ,39 Second, a pilot test of the survey was conducted with four CBGs in the states of Andhra Pradesh, Maharashtra and Manipur, and when the data were scored using the COPI design, the results were found to be consistent with the current status of those CBGs as perceived by programme implementers. Third, the COPI tools were used to analyse the progress of one of the Avahan CBOs which had already demonstrated high achievements in most of the parameters. Further stages of validation are discussed later in this paper.

Implementation

Because of the limited resources available, it was not possible to conduct the assessment in all 83 districts where Avahan was working. A sample size of 32 districts (approximately 39% of the total) was fixed. The districts were selected using stratified random sampling and ensuring representation of: (1) all six states; (2) all seven state-level implementing partners; (3) all categories of high-risk group (FSWs, HR-MSM, TGs and IDUs) in each state where they were present; and (4) districts where Avahan was the sole HIV prevention intervention and districts where it was not.

In the majority of the districts selected, there was only one CBG or CBO operating at the district level. (There might be several smaller, informal CBGs at local levels, with formal representation in the district-level group.) Where there were two CBGs or CBOs at the district level, the one with the largest membership was selected for the assessment, and where there were three or more such CBGs or CBOs, the two with the largest memberships were selected. This method yielded a total of 36 CBGs and CBOs for assessment in the 32 districts.

Researchers were chosen based on several criteria: educational qualifications; past experience in field work; aptitude for field surveys and particularly for participatory approaches; tact and sensitivity in dealing with a stigmatised community; and commitment to the cause of HIV/AIDS prevention. The 25 selected researchers underwent intensive training to ensure that they understood how to implement the survey and the concepts behind the COPI design.

The assessment tool and methodology were discussed by Praxis's Internal Review Board, and a quality standards form was devised, incorporating questions related to the ethical dimensions of data collection. The researchers were required to complete this form each day that they were collecting data, and it served as both a self-monitoring system and a record for later appraisal. Praxis also formed a community-led ethics review committee representing FSWs, HR-MSM, TGs and IDUs. The committee reviewed the assessment tool and methods and served as an external body to ensure that the monitoring was conducted in a way that was ethically acceptable to the community.

The field work for the entire survey took place over a period of 3 months. The researchers, working in teams of two, conducted 5-day-long visits at each CBG. Questionnaires (some completed in written form ahead of time, but most completed through face-to-face discussion during the visits) were given to four groups of respondents, identified as the primary stakeholders mandated with preparing CBGs towards transition readiness: (1) the CBG's leadership team and office-holders; (2) members of CBG committees; (3) staff of the implementing partner (NGO); and (4) community members. The survey was scheduled so that respondents were not required to be present for more than 2 days to make it as easy as possible for them to participate.

Data collection at each CBG took place over three-and-a-half days. Typically, 1 day was spent with the leadership team (all members of the executive committee), half-a-day each with the office-holders (president, secretary and treasurer), NGO staff and committee members (crisis response and advocacy committees), and 1 day with two subgroups of site-level community members. The final one-and-a-half days were reserved for gathering any incomplete information and for a consultation with the respondents who were presented with a preliminary analysis of findings from the data collected. This feedback was intended to add value to the overall process by minimising sampling biases or errors and providing the CBG and implementing partner with immediate data to inform their plans and programmes. Contact with community leaders was maintained so that further data could be requested or submitted via post or email as needed. A report, translated into the local language, was later sent to community leaders for their review, and additional data use workshops were held.

Data collection proceeded simultaneously in all the sample districts, and data were entered directly in a computerised database developed by Praxis using Filemaker Pro (Santa Clara, California, USA) platforms, which allowed concurrent analysis of data for individual CBGs. This necessitated a thorough monitoring system at all levels of data collection and entry to ensure the completeness and quality of data and to promptly resolve any concerns. The researchers completed a checklist, consisting of an evidence sheet, probe questions and a tool sheet, to ensure that no requisite data were omitted.

State coordinators performed quality checks and cleaning of data at the end of each day of field work and informed the field researchers of any gaps. Data were then analysed by a team of state and national coordinators whose concerns were to ensure overall consistency in the gathering of data and other evidence.

Wherever possible, documentary evidence was collected to cross-check statements made by the respondents. This evidence included newspaper reports, audit reports, statutory returns filed, minutes of meetings and copies of statements made to the police or requests filed under the Right to information Act. Documentary evidence was reviewed first by the researchers and then by the state coordinators before data collection was considered complete.