Study objective: To assess the effectiveness of a telephone reminder in increasing responses to postal surveys and to calculate the differential costs per completed questionnaire.
Design: Randomised controlled trial.
Setting: Australian university and rehabilitation medicine practice.
Participants: The trial was conducted in 1999 among the 143 non-respondents to a questionnaire about work related neck and upper body disorders. The questionnaire was sent to two Australian female samples: 200 office workers (Sample A) and 92 former rehabilitation medicine patients (Sample B). A reminder letter, another copy of the questionnaire and a final letter were sent at two week intervals. Half of the non-respondents within each sample were randomly selected to receive a telephone reminder just after the second mailout of the questionnaire. All direct costs were calculated.
Main results: Responses were significantly higher among those who received the telephone reminder intervention (relative risk 2.54, 95% confidence intervals 1.43 to 4.52). Analysed by intention to phone, 47% of non-respondents in Sample A and 38% in Sample B returned a complete questionnaire after the intervention, compared with 21% and 10%, respectively, in the control groups. For the 112 women (combined samples) who returned completed questionnaires before randomisation, the average cost per respondent was AUD14. There was a higher total cost for the intervention groups (AUD851 versus AUD386 for controls), but the significantly higher number of additional completed responses (31 versus 12) resulted in a 15% lower marginal cost per completed questionnaire in those groups.
Conclusion: Telephone reminders are cost effective in improving responses to postal surveys.
- cost effectiveness
- reminder systems
Statistics from Altmetric.com
Mailed self administered questionnaires have a cost advantage over personal contact methods,1 but suffer from lower response rates.2 A number of studies have investigated methods to increase responses to postal health surveys. Techniques that have shown empirical evidence of a positive impact on the proportion of responses are follow up contacts, particularly letters or postcards,3–5 the inclusion of material incentives,3 and the use of stamps rather than business reply postage.6 Other techniques with a more ambiguous effect include the length of questionnaires4 and telephone reminders.7
Telephone follow up of non-respondents has long been used among market researchers on the assumption that it increases response rates.8–10 However, the merit of telephone reminders has not been conclusively established in epidemiological research, nor has its cost effectiveness been fully investigated.
The contribution here is the use of a randomised trial to examine the effects of a telephone reminder on response rates to a postal health questionnaire, with the addition of a comprehensive calculation of the direct costs involved.
The trial was carried out with two separate Australian female samples: current office workers at a university (Sample A) and patients seen by a consultant in rehabilitation medicine between January 1980 and November 1984 (Sample B). These samples were also used to test the acceptability and appropriateness of a questionnaire to assess past and existing symptoms of work related neck and upper body disorders, as well as the long term consequences of such disorders on the health, occupational and socioeconomic status of those women who had been affected (30% of Sample A and all of Sample B). Sample B was also used to assess accuracy of recall of onset of symptoms, of the body areas affected and of the diagnosis given by a doctor after more than 15 years. These studies were pilots for a long term follow up investigation of a large historical cohort of female office workers employed in the early 1980s during an “epidemic” of work related neck and upper body disorders. The studies were approved by the Australian National University Human Research Ethics Committee.
The first study sample (Sample A) comprised 200 women randomly selected from all current female office workers at an Australian university (n=399). A simple random list generator facility in the Epi Info software11 was used for the random selection. The second sample (Sample B), which consisted of women who had developed work related neck and upper limb disorders between January 1980 and November 1984, was selected from the records of a consultant in rehabilitation medicine with a practice in the Australian Capital Territory. A screening of 60% of all records kept by the consultant allowed 129 cases to be identified, but current addresses could be found for only 92 (71%) cases using telephone directories and the Australian Electoral Roll.
The questionnaire, which was sent to both samples in September 1999, contained 317 questions about musculoskeletal symptoms, sociodemographic characteristics, general health, symptoms of depression, health services utilisation, tobacco and alcohol consumption, social support, occupational history, and job satisfaction. Pre-testing showed that completion time varied substantially, with a mean of 1.5 hours (range 0.5 to 2.1 hours). The questionnaire was sent with a cover letter and a reply paid return envelope. Informed consent was considered to have been granted by the act of returning a completed questionnaire.
The follow up methods
Figures 1 and 2 show the design of the follow up protocol used and the response rates for Samples A and B. For both samples, a reminder letter was posted after two weeks and after another two weeks non-respondents were sent a second copy of the questionnaire and randomly allocated (within study samples) to one of two groups of approximately equal size. One group (n=72, combined samples) was randomised to receive a reminder telephone call soon after the second questionnaire was mailed, with a subsequent final reminder letter if they had not responded within two weeks. The other group (n=71, combined samples) received only a final reminder letter if they did not respond two weeks after the second questionnaire had been dispatched. No material incentives were used. Telephone numbers were obtained using an electronic telephone directory.
Two researchers using a standardised script implemented the telephone contact. The message focused on the relevance of the study for informing working women of the long term impact of work related neck and upper body disorders and on reiterating the importance of participation to enhance the study findings. Up to five attempts were made to reach the selected women at home. All calls occurred out of office hours, including weekends. Arrangements of the most suitable time to contact women were sought when the telephone was answered by a third party. Messages left on answering machines stated only the identity of the researchers. Cumulative response rates for each group (within samples) were analysed after the survey cut off date both by intention to phone and by telephone contact actually made to determine whether significant differences in responses could be detected.
Differences in responses were calculated using the χ2 statistic with Yates's correction. Relative risks for decrease in response rates were also calculated for each of the study samples. The Mantel-Haenszel weighted relative risk across the two samples was then calculated for response rates, and the Breslow-Day test for homogeneity was used to assess heterogeneity across the groups.
All direct expenses incurred in carrying out the survey protocol and telephone reminders were recorded. Fixed costs included the printing of the questionnaires and reply paid envelopes, postage, and labour involved in mailouts. Variable costs were associated with telephone charges and labour spent on telephone calls. Most calls were local because all women in Sample A and 60% of the women in Sample B were living in the city where the study was conducted. All costs were calculated using standard Australian rates. Infrastructure costs were not included.
The base fixed mailing costs before randomisation were divided by the number of respondents in Samples A and B. After randomisation, the fixed mailing costs incurred were calculated separately for each group, and the variable costs were calculated for the intervention group.
Respondents in Samples A and B were similar in age, marital status and educational level. Overall 63% were 45 years or older, 73% were married or in de facto relationships, and 59% had post secondary school qualifications. However, women in Sample B were more likely to be retired (40% compared with 0%) and to have a lower income (43% had a yearly gross family income of $A 30 000 or less compared with 6%). All of the women in Sample B who were still employed were also office workers.
The randomised groups did not differ with respect to age, the only variable known before randomisation. Among respondents in each sample there was no statistically significant difference at p=0.05 level in age, marital, educational, occupational and retirement status, or income between those in the intervention (telephoned) and control (not telephoned) groups.
Before randomisation, 96 (48%) of Sample A and 16 (17%) of Sample B returned completed questionnaires. At this stage, 19 (10%) and 18 (20%) questionnaires, respectively, were undeliverable or refused. For Sample A, the intervention group (those to be contacted by telephone) consisted of 43 women, with 42 women in the control group. Of the 43 women, 40 (93%) were successfully contacted. Analysed by intention to phone, the completion rates were 47% for the telephoned group versus 21% for the control group (relative risk 2.17, confidence intervals 1.12 to 4.21, p=0.027; table 1). Of those telephoned, 16% were definite refusals (rather than non-responses), compared with 10% for those not telephoned.
For Sample B, 29 women were randomly allocated to each group. Twenty six (90%) were reached by telephone. Analysed by intention to phone, the completion rates obtained were 38% for those telephoned versus 10% for those not telephoned (relative risk 3.67, confidence intervals 1.14 to 11.79, p=0.031; table 1). Forty one per cent of those telephoned were definite refusals, compared with 10% of those not telephoned.
For women, telephone reminders significantly increase completion rates in postal health surveys.
There is a significant improvement in response rates through telephone reminders in groups with both initially high and initially low participation rates.
The marginal cost per respondent is 15% lower when a telephone reminder is included in the follow up protocol for postal surveys.
A Mantel-Haenszel weighted relative risk across the two samples and groups was calculated, as there was no significant heterogeneity across these strata. The results were 2.54 (confidence intervals 1.43 to 4.52) for analysis by intention to phone and 2.67 (confidence intervals 1.53 to 4.65) for analysis by successful telephone contact (table 1).
Total costs and costs per respondent are shown in table 2. For the 112 women (combined samples) who returned completed questionnaires before randomisation, the average cost per respondent was $A14. For those who responded after the questionnaire was mailed a second time, the costs were as follows. For the group randomised to receive a telephone reminder, the marginal cost per respondent was $A27. For the control group, the marginal cost per respondent was $A32. The total cost was higher for the intervention group ($A851) compared with the control group ($A386), but the significantly higher number of completed responses (31 versus 12) resulted in a lower marginal cost per respondent.
These findings clearly show that a telephone reminder to non-respondents during the middle stage of data collection is an effective strategy to significantly improve response rates to postal surveys.
This was found in two populations recruited in different settings and with different completion rates (48% before randomisation for Sample A compared with 17% before randomisation for Sample B). If the data collection protocol had not included a telephone reminder at all, a 57% final completion rate could have been expected for Sample A, whereas with a telephone reminder to all women (with the same success in making contact), a 68% completion rate was likely to have been achieved. The analogous rates for Sample B are 24% and 41%. These estimations suggest that the improvement in response rate through a telephone reminder is larger in a group with a low initial response rate (Sample B) than in a group with a high initial response rate (Sample A), although there was no significant heterogeneity across the two sample strata. However, because of the small numbers in Sample B, further investigation of this hypothesis is warranted.
The unambiguously stated refusal rates, as compared with non-response rates, were also increased among the telephoned group, particularly for Sample B. This permits a clearer assessment of possible response biases, which must then be taken into account in the interpretation of the results. The high refusal rate in Sample B made us aware of the possible strength of feeling about their condition in this group, which is being further investigated.
The marginal cost per respondent was substantially lower (15% from the time of randomisation at the mailout of the second questionnaire) for the telephoned group. The precise costs will vary, of course, depending on the success of finding telephone numbers and of making contact.
A potential limitation to the generalisability of the results of this trial is that only women were recruited. It is not possible to know whether the results would also hold for men. The samples were drawn from office workers who are relatively well educated, with good English language skills. Again it is not possible to know how generalisable the results are to groups with different occupational, educational, and language status.
These results show that adding a telephone reminder soon after the second mail out of the questionnaire in a mail health survey is a cost effective way of significantly increasing response rates from relatively well educated female populations.
Marluce Salim Silva is a PhD candidate sponsored by the Federation of Industries and the Secretariat of Health of the State of Minas Gerais, Brazil. Valuable assistance was received from Dr Jim Butler, Dr Keith Dear, Ms Karen Lees, Dr John Corry, Dr Geoffrey Speldewinde and Ms Karen Alarcon.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.