You are here

Article Text

Article menu

Download PDFPDF

Theory and methods
National health surveys by mail or home interview: effects on response
  1. H S J Picavet
  1. National Institute of Public Health and the Environment, the Netherlands
  1. Dr Picavet, Centre for Chronic Diseases Epidemiology (CZE, pb 101), National Institute of Public Health and the Environment, PO box 1, 3720 BA Bilthoven, the Netherlands (susan.picavet{at}rivm.nl)

Abstract

STUDY OBJECTIVE To study the effect of using a mail questionnaire or home interviews on the size and the selectivity of response to national health surveys.

DESIGN The interview survey and the mail survey were both carried out in the same country (the Netherlands) using the same sample frame, the same study period (1998) and collected partly the same data on demographic, socioeconomic and health characteristics.

SETTING The Netherlands.

PARTICIPANTS Dutch non-institutionalised inhabitants aged 25 years and over.

MAIN RESULTS Response to the mail survey was lower (46.9%, n=3664) than to the interview survey (58.5%, n=6061). The mail survey gave higher response rates for women and lower response rates for persons with lower levels of education. Respondents to the mail survey reported lower rates of smoking but a slightly worse health status and higher figures on the use of health care services. No differences by method of data collection were found for age, marital status, region, household composition, work status and categories of body mass index.

CONCLUSION Although the response of the mail survey was lower than the home interview survey, respondents showed generally small differences, with exception of level of education.

  • mail surveys
  • interview surveys
  • non-response

https://doi.org/10.1136/jech.55.6.408

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    National health surveys are the main sources for data on many (public) health indicators. Data collection can be carried out by face to face interviews, telephone interviews, mail questionnaires or a combination of methods. In the Netherlands we often use face to face interviews carried out at home or mail questionnaires. The choice of the mode of data collection is determined by several factors, including available resources and response expectations. Face to face interviews surveys are much more expensive than mail surveys.1Several effects of mode of data collection on response are known.

    Firstly, it is generally thought that response rates are better for interview surveys than mail surveys,2 some evidence for elderly people is available.3 Secondly, those who respond to interview surveys can be different from those who respond to mail surveys. There is some evidence that lower socioeconomic classes are under-represented in mail surveys compared with interview surveys.4 Thirdly, people can respond differently to questions on paper than to questions asked by an interviewer. For instance, for some disability indicators it is known that systematically higher prevalences are found using self administered questionnaires compared with interviews.5 The same is found for other health indicators.6-8 In addition, questions that can be affected by social desirability, for example, alcohol consumption, using of car belts, are suspected to do better in self administered questionnaires than in face to face interviews. However, information on height, household composition, work status is considered not to be affected by mode of data collection.9

    For this paper we were able to study the response to a health mail survey and to a health interview survey, both using the same sample frame (population register), the same target population (the Dutch non-institutionalised population) and mainly the same topics and questions. The question of our study is: does using a mailed survey or an interview survey lead to different response groups? According to the (limited) data in the literature we expect (1) the response on the mail survey to be lower than on the interview survey, (2) that respondents to the mail survey with lower educational levels are underrepresented and (3) that population estimates on non-mode dependent questions such as work status, number of persons in the household, height and weight, are not affected by mode of data collection. (4) Our last hypothesis is that population-based estimates of health indicators based on mail survey will represent a less favourable health status compared with the interview survey.

    Methods

    Two health surveys carried out in the Netherlands were analysed (1) the Netherlands Health Interview Survey (NetHIS) of 1998, which uses face to face interviews at home carried out by trained interviewers combined with a paper questionnaire, and (2) the baseline of the Dutch Musculoskeletal Complaints and Consequences Cohort study (DMC3-study), which uses mail questionnaires. General characteristics of the two studies are presented in table 1. The NetHIS is a continuous survey started in 1981 and carried out by Statistics Netherlands. From 1997 the NetHIS is one module of the integrated system of face to face interviews of Statistics Netherlands. The DMC3-study is carried out by the National Institute of Public Health and the Environment in collaboration with Statistics Netherlands. For the two surveys the same sample frame (population register) and sample method were used, although for the DMC3-study this was a stratified sample. The sample frame provides us with data on date of birth, sex, marital status and address details of the persons in the sample. The surveys collected identical information for sociodemographic characteristics, health indicators, risk indicators and the use of health services. We have, however two exceptions: the questions on houshold composition and on educational level were much more detailed in the NetHIS than in the DMC3-study.

    View this table:
    Table 1

    Overview of survey characteristics

    The net response was calculated by dividing the number of respondents by the number of those actually approached excluding those who were known to be deceased or those whose address was unknown. The response figures are presented for both surveys: for the total group and by sample frame characteristics—that is, age group, sex, marital status and region. For NetHIS two response figures are given: one for the interview and one for questionnaire. After the interview, which is carried out by trained interviewers with laptop computers the interviewees were handed the paper questionnaire and asked to fill it in and send it back by free post return envelope.

    To compare the results of the surveys, both surveys were weighted. Weighting factors were constructed in such a way that the distribution of both surveys by age, sex, region and marital status was equal to that of the Dutch population of 1998. The surveys were then compared for (1) sociodemographic characteristics (household composition, education, work status), (2) health indicators (perceived health, limitations in daily life, chronic conditions), (3) risk factors (smoking, body mass index) and (4) the use of health care services.

    Household composition presents the number of persons living in the household. Level of education was measured as the highest level reached and then summarised in four groups: primary school, junior (vocational) education, secondary (vocational) education, vocational colleges/university. Work status is defined according to four categories: have paid work for more than 12 hours a week, does not have paid work for more than 12 hours per week but wants to have work for more than 12 hours a week, does not have nor wants work for more than 12 hours a week, and those who are work disabled or have a pension (at least everybody above 65 years of age). Perceived health was measured with the question, “How do you rate your health using a mark for a school report?” In the Netherlands these marks are between 1 and 10 with 10 as the best mark. The following groups were made according to other descriptions of health10: mark 1 to 5 (bad health), 6 to 7 (average health), 7.5 to 8.5 (good health) and higher than 8.5 (excellent health). Limitations in daily life were measured by a simple question “Do you have any limitation in your daily activities due to a health problem?” For the assessment of chronic conditions a list of chronic health problems was used. The majority of the descriptions of chronic conditions were identical for the DMC3-study and the NetHIS. These were: COPD (chronic obstructive pulmonary disease), sinusitis , coronary heart disease or other severe heart disease, hypertension, (consequences of) stroke, peptic ulcer, severe intestinal disorder, diabetes, thyroid disorders, epilepsy, dizziness and falling, migraine, severe skin disease and cancer.

    For smoking five fixed response categories were used: every day smokers, occasional smokers, former every day smokers, former occasional smokers and never smokers. “Occasional” smokers are those who smoke less than one cigarette a day. Body mass index (BMI) is calculated by dividing weight (kg) by squared length (m) and was categorised as < 18.5 (underweight), 18.5–25 (normal weight), 25–30 (moderate overweight) and > 30 (severe overweight).11 The use of health care services was measured identically in both surveys for contact with general practitioner (GP), medical specialist and physiotherapist. For the GP and the medical specialist two indicators were calculated: contact in the past two months and contact in the past year. For the physiotherapist only contact in the past year was measured.

    Differences in results between the surveys were calculated using the weighted percentages and the 99% confidence limits were calculated using the standard errors of the unweighted prevalences.

    Results

    key points

    • For many health topics carefully designed mailed surveys are probably an equally good alternative for the—much more expensive—interview surveys.

    • People with only primary school should receive extra attention in the health survey design.

    • Investigating non-response bias should be part of every survey regardless of the response rate.

    The response to the interview of the NetHIS was higher (58.4%) than the response of the mail questionnaire of the DMC3-study (46.9%) (table 2). Using a questionnaire as a second step after the interview survey gave some additional non-response, resulting in a net response to the NetHIS questionnaire of 47.9%. The mail survey gave higher response rates for women than for men whereas the NetHIS shows no differences by sex. The response patterns of the two surveys for age, marital status and region were similar. The response of persons of 65 years and over was slightly lower than of the other age groups. Those who were married showed the highest response and those who were not married the lowest. By region of living we found a slightly lower response in the west, which is the most urbanised region of the Netherlands, including Amsterdam, Rotterdam and The Hague.

    View this table:
    Table 2

    Response for DMC3-study (mail survey) and NetHIS (interview survey) by demographic characteristics available from the sample frame

    Table 3 presents the estimations of different characteristics of the Dutch population aged 25 years and over according to the two surveys. These estimations were almost identical for household composition and work status, but for level of education we found that those with only primary school were underrepresented in the DMC3-study with 14.5% compared with 21.1% in the NetHIS. If this figure is estimated on the basis of those who participated in the NetHIS and also returned the questionnaire, we get a percentage of those with only primary school of 19.5%. Of the NetHIS respondents who did not return the questionnaire 28.4% (p<0.01) had only primary school. For the other characteristics there was no difference between the respondents of NetHIS who did and who did not return the questionnaire (not shown).

    View this table:
    Table 3

    Estimations of sociodemographic and health and health related characteristics according to DMC3-study and NetHIS, both surveys weighted for the Dutch population of 1998, and the differences between the surveys, including 99% confidence limits (CL)

    For the health indicators we see that estimated proportions of the extreme values of subjective/perceived health (bad and excellent) were the same for both surveys but that there was a small shift to better health in the NetHIS. In contrast with that the NetHIS reported a higher prevalence of persons with limitations in daily life (16.6% versus 12.7%), suggesting a worse health for the respondents of NetHIS. The estimated prevalences of 14 chronic conditions were the same in both surveys or slightly higher in the DMC3-study. The latter was true for: sinusitis, severe intestinal disorder, dizziness with falling, and migraine.

    The DMC3-study gave a lower figure for every day smokers than the NetHIS, 24.2% versus 29.4% but the estimations of BMI categories were the same for both surveys. The use of health care services (GP or specialist) during the past two months and contact with the physiotherapist during the past year was more often reported by the respondents of the DMC3-study. The percentage of persons with contact with the GP during the past year was the same for both studies and the percentage with contact with medical specialist during the past year was relative lower in the DMC3-study.

    Discussion

    In this study we found that a health mail survey had a lower response rate than a health interview survey but that the differences in the respondents by mode of data collection were small with exception of educational level.

    Other studies also reported higher response rates for interview surveys than mail surveys.3 However, it is very difficult to make absolute statements about this because many other factors could have affected response such as study design differences. In our study one important drawback of the mail survey was the length of the questionnaire. The questionnaire was rather long, with 28 pages. However, the questionnaire was divided in a few parts indicated by coloured paper and if a screening question was negative many pages could be skipped. There is some evidence that long questionnaires affects response negatively,12 13 although one study did not found a difference in response by using a 4 page or 16 page questionnaire.14 We think that the response of the mail survey would have been higher if we had used a much shorter questionnaire but then the amount of information gathered would have been much less.

    In both surveys analysed in this paper, high risk groups for low response were: higher age groups (65 years and over), and those who were not married (anymore), and this was found before.13-17 Usually men were also underrepresented but we did not find this in the NetHIS. We found no differences between the mail survey and the interview survey with regard to age, marital status and region of living. Other studies also reported no difference by region18 although rural areas had a slightly higher response than urbanised areas.13 For the demographic characteristics, we conclude that our national mail and interview surveys with respect to health related topics have similar, slightly selective, response.

    An under-representation of lower socioeconomic groups in the mail survey has been found before, regardless the indicator being income levels17 19 or level of education.14 20However, also examples exist with no differences in response by income groups.21 Because level of education is in general such an important determinant of health22 and health related behaviour23 it was important to look for response bias attributable to educational level. Unfortunately there is no other source than population surveys for the information on the distribution by level of education in the Netherlands. So it is impossible to say whether or not the estimations of the surveys represent an underestimation or overestimation. This study indicates that those with only primary school were less likely to respond to a mail questionnaire than to an interview survey.

    For a health survey it was also important to identify response bias attributable to health. What are the health characteristics of respondents and non-respondents? In general it is suggested that respondents to health surveys are the “worried well”24: healthy people who see their doctor regularly and follow healthy lifestyle practices. Our study showed that the mail survey gave higher rates of health care utilisation and a lower prevalence of smoking compared with the health interview survey. The overestimation of the utilisation of health services by mail surveys was found before25 26 and the same was true for the underestimation of smokers.20 27 The mail survey gave a slightly more unhealthy picture of the population than the interview survey, based on subjective health and a few chronic conditions. However, in contrast with what we expected5 the prevalence of limitations in daily life was higher in the interview survey than in the mail survey. Examples of studies on non-response bias with respect to health characteristics are mail surveys among the very elderly and non-response bias was often found9 attributable to bad health and diminished cognitive functioning,28 affecting health related estimations more than factual information.9Elderly non-responders also had higher death rates.9 24In general we can expect that health surveys miss the very unhealthy part of the population and that respondents of mail surveys can be described as the “worried well”. It was important to note however that although some of the differences between the NetHIS and the DMC3-study are statistically significant the differences in terms of percentage points are small. A lack of differential non-response in comparing mail surveys and interview surveys was also found in other studies.21 29

    A limitation of this study is that both surveys have a rather low response. Surveys in the Netherlands are in general confronted with lower response rates than health surveys elsewhere in Europe,30 or compared with mail surveys published in medical journals31 despite intensive efforts to reduce non-response. General guidelines to increase motivation for participation32 were used where possible, including an interesting topic (health), confidentiality, a good reputation of the two organisations responsible for the surveys, approval by official institutions, use of advance letters in the home interview survey, repeated contacts (with a maximum of three), use of trained interviewers and attractive questionnaires. Systematic differences in health and health related topics between those participating and not participating in health surveys—whether or not by mail or interview—are possible. If such extreme groups exist we are never able to investigate them but they do not differ in terms of sociodemographic characteristics and they do hardly differ between a mail or interview survey.

    An additional limitation is the difference in time period of data collection between the NetHIS (all year in 1998) and DMC3-study (September to December 1998). In 1998 the response to the HIS per month varied from 55.7% (June) to 64.1% (October). The response to the NetHIS is somewhat lower in the summer months because of holidays. There is however no systematic variation in health prevalences by month or season.10

    Differences in sponsorship or themes of the surveys can also contribute to the response. Statistics Netherlands was involved in both studies and in the DMC3-study also the National Institute of Public Health. Both organisations are national government associated non-profit organisations. We do not think that differences in the perception exist or should have affected response rates. Because the focus of the DMC3-study was on musculoskeletal health problems, and the NetHIS was a general health survey, the response could have been higher for those with musculoskeletal health problems than for those without musculoskeletal health problems. Because musculoskeletal health problems are very common this would not have affected the response rate but an overrepresentation of musculoskeletal health problems can be expected. For limitations in daily life, however, we found a higher prevalence in the NetHIS than in the DMC3-study.

    Our comparisons showed that the response was lower on the mail survey than the interview survey, that responders with lower educational levels are underrepresented in the mail survey, and that estimates of questions such as work status, number of persons in the household, height and weight, are not affected by mode of data collection. Our fourth hypothesis, stating that population-based estimates of health indicators based on mail survey will represent a less favourable health status compared with the interview survey, is not confirmed for all health indicators. In general, we can conclude that the differences in respondents between interview surveys or mail surveys are no reason for great concern.

    Acknowledgments

    The author wishes to thank H W V van Gils from University of Maastricht and Statistics Netherlands for the analyses of NetHIS and Dr J S A G Schouten for help in the design of the study and colleagues for useful suggestions for the text.

    References

    1. ↵
      1. O'Toole BI,
      2. Battistutta D,
      3. Long A,
      4. et al.
      (1986) A comparison of costs and data quality of three health survey methods: mail, telephone and personal home interview. Am J Epidemiol 124:317–328.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Dillman DA
      (1978) Mail and telephone surveys. (John Wiley, New York), p 325.
    3. ↵
      1. Mallinson S
      (1998) The Sort-Form 36 and older people: some problems encountered when using postal administration. J Epidemiol Community Health 52:324–328.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Eastwood BJ,
      2. Gregor RD,
      3. Maclean DR,
      4. et al.
      (1996) Effects of recruitment strategy on response rates and risk factor profile in two cardiovascular surveys. Int J Epidemiol 25:763–769.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Picavet HSJ,
      2. Bos GAM van den
      (1996) Comparing survey data on functional disability: the impact of some methodological differences. J Epidemiol Community Health 50:86–93.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Weinberger M,
      2. Oddone EZ,
      3. Samsa GP,
      4. et al.
      (1996) Are Health-related quality-of-life measures affected by the mode of administration? J Clin Epidemiol 49:135–140.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Lyons RA,
      2. Wareham K,
      3. Lucas M,
      4. et al.
      (1999) SF-36 scores vary by method of administration: implication for study design. J Public Health Med 21:41–45.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. McHorney CA,
      2. Kosinski M,
      3. Ware JE
      (1994) Comparisons of the costs and quality of norms for the SF-36 health survey collected by mail versus telephone interview: results from a national survey. Med Care 32:551–567.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Hérbert R,
      2. Bravo G,
      3. Korner-Bitensky N,
      4. et al.
      (1996) Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. J Clin Epidemiol 49:373–381.
    10. ↵
      1. Sonsbeek J van
      (1996) Vertel me wat er aan scheelt. Betekenis en methodische aspecten van enquetevragen naar gezondheid. [Tell me what is the matter. Meaning and methodological aspects of health survey questions]. [PhD thesis]. (Statistics Netherlands, Voorburg, Nijmegen), p 300.
    11. ↵
      1. World Health Organization Expert Committee
      (1995) Physical status: the use and interpretation of anthropometry. WHO technical report series 854. (World Health Organization, Geneva).
    12. ↵
      1. Lund E,
      2. Gram IT
      (1998) Response rate according to title and length of questionnaire. Scand J Soc Med 26:154–160.
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Eager S,
      2. Bergstrom R,
      3. Bergstrom A,
      4. et al.
      (1998) Response rate to mail epidemiologic questionnaires: a population-based randomized trial of variations in design and mailing routines. Am J Epidemiol 147:74–82.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Hoffman SC,
      2. Burke AE,
      3. Helzlsouer KJ,
      4. et al.
      (1998) Controlled trial of the effect of length, incentives, and follow-up techniques on response to a mail questionnaire. Am J Epidemiol 148:1007–1011.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Côté P,
      2. Cassidy JD,
      3. Carroll L
      (1998) The Saskatchewan health and back pain survey, the prevalence of neck pain and related disability in Saskatchewan adults. Spine 23:1689–1698.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Pope D,
      2. Croft P
      (1996) Surveys using general practice registers: who are the non-responders? J Public Health Med 18:6–12.
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Dengler R,
      2. Roberts H,
      3. Rushton L
      (1997) Lifestyle surveys—the complete answer? J Epidemiol Community Health 51:46–51.
      OpenUrlPubMedWeb of Science
    18. ↵
      1. Barton J,
      2. Bain C,
      3. Hennekens CH,
      4. et al.
      (1980) Characteristics of respondents and non-respondents to a mail questionnaire. Am J Public Health 70:823–825.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Siemiatycki J
      (1979) A comparison of mail, telephone, and home interview strategies for household health surveys. Am J Public Health 69:238–245.
      OpenUrlPubMedWeb of Science
    20. ↵
      1. Bostrom G,
      2. Hallqvist J,
      3. Haglund BJ,
      4. et al.
      (1993) Socioeconomic differences in smoking in an urban Swedish population. The bias introduced by non-participation in a mail questionnaire. Scand J Soc Med 21:77–83.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Siemiatycki J,
      2. Campbell S
      (1984) Nonresponse bias and early versus all responders in mail and telephone surveys. Am J Epidemiol 120:291–301.
      OpenUrl
    22. ↵
      1. Mackenbach JP,
      2. Kunst AE,
      3. Cavelaars AE,
      4. et al.
      (1997) Socioeconomic inequalities in morbidity and mortality in Western Europe. The EU Working Group on socioeconomic inequalities in health. Lancet 340:1655–1659.
    23. ↵
      1. Droomers M,
      2. Schrijvers CT,
      3. Mheen H van der,
      4. et al.
      (1998) Educational differences in leisure-time physical inactivity: a descriptive and explanatory study. Soc Sci Med 47:1665–1676.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Paganini-Hill A,
      2. Hsu G,
      3. Chao A,
      4. et al.
      (1993) Comparison of early and late responders to a postal health survey. Epidemiology 4:375–379.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Etter JF,
      2. Perneger TV
      (1997) Analysis of non-response bias in a mail health survey. J Clin Epidemiol 50:1123–1128.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Reijneveld SA,
      2. Stronks K
      (1999) The impact of response bias on estimates of health care utilisation in a metropolitan area: the use of administrative data. Int J Epidemiol 28:1134–1140.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Macera CA,
      2. Jackson KL,
      3. Davis DR,
      4. et al.
      (1990) Patterns of non-response to a mail survey. J Clin Epidemiol 43:1427–1430.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Launer LJ,
      2. Wind AW,
      3. Deeg DJ
      (1994) Nonresponse pattern and bias in a community-based cross-sectional study of cognitive functioning among the elderly. Am J Epidemiol 139:803–812.
      OpenUrlPubMedWeb of Science
    29. ↵
      1. Bongers IM,
      2. Oers JA van
      (1998) Mode effects on self-reported alcohol use and problem drinking: mail questionnaire and personal interviewing compared. J Stud Alcohol 59:280–285.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Hupkens CLH,
      2. Berg J van den,
      3. Zee J van der
      (1999) National health interview surveys in Europe: an overview. Health Policy 47:145–168.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Asch DA,
      2. Jedriewski MK,
      3. Christakis NA
      (1997) Response rates to mail surveys published in medical journals. J Clin Epidemiol 50:1129–1136.
      OpenUrlFREE Full Text
    32. ↵
      1. Kessler RC,
      2. Little RJA,
      3. Groves RM
      (1995) Advances in strategies for minimizing and adjusting for survey non-response. Epidemiol Rev 17:192–204.

    Footnotes

    • Funding: the Dutch population-based Musculoskeletal Complaints & Consequences Cohort (DMC3-study) was financially supported by the Ministry of Health, Welfare and Sport of The Netherlands and the National Institute of Public Health and the Environment and carried out in collaboration with Statistics Netherlands.

    • Conflicts of interest: none.