You are here

Article Text

Article menu

Download PDFPDF

Short report
The effect of a direct payment or a lottery on questionnaire response rates: a randomised controlled trial
Free
  1. Paula-J Robertsa,
  2. Chris Robertsb,
  3. Bonnie Sibbaldc,
  4. David J Torgersond
  1. aSeven Brooks Medical Centre, 21 Church Street, Atherton M46 9DE, bUniversity of Manchester, Oxford Road, Manchester, cNational Primary Care Research and Development Centre, Williamson Building, University of Manchester, Oxford Road, Manchester, dDepartment of Health Studies and Centre for Health Economics, University of York, York
  1. Dr P-J Roberts

http://dx.doi.org/10.1136/jech.54.1.71

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Postal questionnaires are frequently used in research. One way of improving response rates is to use lotteries, although the evidence for their effectiveness is equivocal.1-4 An alternative, or complementary, approach to using lotteries is to make direct payments to survey responders. Few trials have evaluated direct payment compared with lotteries.3

    Methods

    Questionnaires about menopause services in the North West of England were sent to a random sample of 1000 women aged 40 to 65 during September to November 1997. The questionnaires included questions about use of hormone replacement therapy (HRT) and women's demographic characteristics.

    The sample of women were randomised in a 2 × 2 factorial trial to estimate the effectiveness of the two forms of incentive and their potential interaction within a single study. Unequal randomisation of 3 to 1 favouring no direct payment was used to minimise research costs, which led to unequal numbers in each of the four arms of the study (lottery alone, n =375; direct payment alone, n = 125; lottery and direct payment, n = 125; no lottery and no direct payment, n = 375). The direct payment consisted of £5 for each respondent while the lottery was for a prize draw of £50. Payment and/or inclusion in the prize draw were made on receipt of a completed questionnaire unless declined by respondents.

    Non-responders were sent up to two further mailings approximately one month apart. Women returning a blank questionnaire were not sent further mailings. To comply with the Data Protection Act, a Health Authority custodian attended with names and addresses for the three mailings. The investigators therefore had no information about the characteristics of non-responders.

    Analysis was by means of logistic regression using the SPSS statistical package.

    Results

    After three mailings 587 completed questionnaires were returned. Two questionnaires were returned by the post office as not being known at the address and these were removed from the analysis. The mean age of respondents was 51 range 40–65, 91% had one or more children and the mean number of children was 2.2.

    Table 1 shows the proportions of patients responding by study groups and the odds that financial incentives increased response when compared with non-payment. There was no interaction between lottery and payment (p=0.78) so it was possible to examine the effect of each intervention independently. Direct payments significantly increased the odds of response (odds ratio = 1.70, 95% CI 1.11 to 2.60, p = 0.013). The lottery did not significantly increase response rates (odds ratio = 1.22, 95% CI 0.91 to 1.62, p = 0.19).

    View this table:
    Table 1

    Response rates by study group and multivariate analysis

    After pooling the lottery and non-payment groups, the response rate was 67.6% in the payment group compared with 56.1% in the non-payment group (diff =11.5%, 95% CI 4.7% to 18.3%). 83% (140 of 169) of respondents completing questionnaires accepted payment, 29 respondents declined payment. Table 1 shows the cost per sampled woman and the marginal cost per completed response. The means by which the these costs were calculated is explained in the footnotes.

    There were no differences between the direct payment and non-payment groups in mean age (diff = −0.6, 95% CI −1.8 to 0.6 years, p = 0.32) or level of educational qualifications (Mann-Whitney U test p = 0.83). The direct payment group had lower ever HRT use compared with the non-payment group (diff = 8.5%, 95% CI of difference 0.0 to 16.9%, p = 0.056).

    Comment

    The use of a £5 direct payment incentive increases absolute response rates by 12%. However, like other studies, a lottery did not significantly increase response rates.1 4 The marginal cost per extra response using a £5 incentive was four times that of the control group. Whether this extra cost is worthwhile may depend upon whether increasing response rates leads to a more representative sample. In this study, more non-users of HRT responded to payment suggesting that payment had a larger impact among women for whom the questionnaire had a lower interest (that is, non-users of HRT).

    It seems that direct payments may be more effective than lotteries and payment may preferentially increase response rates among women with least interest in the subject of the survey. These findings were obtained in middle aged women and may not apply to other populations.

    References

    1. ↵
      1. Brown AP,
      2. Lawrie HE,
      3. Kennedy AD,
      4. et al.
      (1997) Cost effectiveness of a prize draw on response to a postal questionnaire: results of a randomised trial among orthopaedic outpatients in Edinburgh. J Epidemiol Community Health 51:463–464.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Spry VM,
      2. Hovell MF,
      3. Sallis JG,
      4. et al.
      (1989) Recruiting survey respondents to mailed surveys: Controlled trials of incentives and prompts. Am J Epidemiol 130:166–172.
      OpenUrlPubMed
    3. ↵
      1. Marrett LD,
      2. Kreiger N,
      3. Dodds L,
      4. et al.
      (1992) The effect on response rates of offering a small incentive with a mailed questionnaire. Ann Epidemiol 2:745–753.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Mortagy AK,
      2. Howell JBL,
      3. Waters WE
      (1985) A useless raffle. J Epidemiol Community Health 39:183–184.

    Footnotes

    • Funding: NHS NorthWest Region Research Practices Initiative.

    • Conflicts of interest: none.