Article Text
PDF
Abstract
Background Diaries are used in sexual behaviour research to reduce recall bias. Diary collection via mobile phone text messaging (SMS) has not been trialled previously in sexual behaviour research. This randomised controlled trial compared SMS, paper and online diaries on response rate, timeliness, completeness of data and acceptability. The correlation between behaviour reported in all three types of diaries and data collected in a retrospective questionnaire was also determined.
Methods Participants were recruited by telephone and randomised into one of three groups. They completed weekly sexual behaviour diaries for 3 months by SMS, online or paper (by post). An online survey was conducted at the end of 3 months to compare retrospective reports to the diaries and assess opinions on the diary collection method.
Results 72 participants were enrolled in the study, 24 in each group. Online diaries were more likely to be submitted late than SMS diaries (p<0.001). 3.9% of SMS diaries, 3.1% of paper diaries and 0.5% of online diaries were incomplete (p=0.001). Online data collection was the preferred mode for 51%. 65 participants completed the end point retrospective questionnaire. The correlation between the diary and questionnaire on sexual risk classification was substantial (κ=0.74) regardless of diary mode.
Conclusions SMS is a convenient and timely method of collecting brief behavioural data, but online data collection was preferable to most participants and more likely to be complete. Data collected in retrospective sexual behaviour questionnaires were found to agree substantially with data collected through weekly self-reported diaries.
- Sexual behaviour
- validity of results
- cellular phone
- data collection
- internet
- diaries
- survival me
Statistics from Altmetric.com
Introduction
In sexual behaviour research, most collected data are based on self-reported information. Sexual behaviour is generally not directly observable to researchers, hence this research is based on the assumption that participants are providing accurate information.1 Recall bias may affect the validity and reliability of behavioural information. When participants are required to provide a frequency or average frequency of a behaviour, they will often be unable to remember specific details and need to estimate or describe a modal or usual behaviour.1–3 Regularly reported behavioural information can be used to improve the validity of the data collected. Previous studies on sexual behaviour have compared information reported via regular ‘diaries’ with information reported in retrospective questionnaires to determine the validity of responses.4–6 The theory is that recall bias is minimised by decreasing the period of time covered by each diary.
Diary information can be collected through different modes. Changing the medium through which participants are asked to complete the diary can affect response rate, timeliness of diary completion, patients' opinions of the diary and reported behaviour.7–12
Traditionally, sexual behaviour diaries have been recorded on paper.4 13–16 However, when diaries are paper-based, participants may be concerned that their sensitive information could be found by a parent or partner.8 17 Furthermore, unless the diaries are collected immediately (which is rarely practical), there is no way to accurately determine when they were completed. In contrast, electronic methods can provide a time-stamp. A study using a secret electronic timer in a binder holding paper diaries found that only 20% of participants had opened the diary binder within 90 min of the time they reported completing the diary.16
Electronic devices such as personal digital assistants and pagers have been used to collect diary data.7 11 18 19 E-mail diary submissions have been used,5 as well as online web-based surveys.20–22 Online diaries can be designed to enforce completion of particular questions and can incorporate logical rules in the flow of questions, thereby reducing the occurrence of missing and contradictory data.
Telephone diaries have also been trialled, whether through voice contact,9 12 13 an automated questionnaire11 23–26 or, more recently, mobile phone text messaging (SMS).27 28 Person-to-person voice contact is time-intensive for interviewers, and telephone methods may be inconvenient for participants if they have to make or receive telephone calls at particular times.9 SMS methodologies are novel and, to date, have only been used in disease monitoring (eg, for recording asthma symptoms).27 Compared with other modes of data collection, SMS is convenient as diaries can be completed anywhere and at any time. There is growing interest in the possibilities of SMS in medical research.29 30 In Australia, use of mobile phones is particularly prevalent among young people aged 18–34, 87–90% of whom use mobile phones (overall 79% of Australians use mobile phones).31 SMS is therefore likely to be particularly useful for reaching young Australians.
The first aim of this trial was to compare a novel mode of diary collection (SMS) with diary collection using paper (collected by post) and online web-based surveys. The three modes of sexual behaviour diary data collection were compared in their response rates, timeliness, completeness and acceptability. A retrospective questionnaire was also designed as an alternative mode of sexual behaviour data collection. The second aim was to determine the validity of the retrospective questionnaire in comparison to all three modes of diaries for collecting sexual behaviour information.
We hypothesised that SMS diaries would be considered the most convenient mode of data collection for participants and have the best response rate, but have higher rates of missing data; paper diaries would have the lowest response rate and higher concerns regarding privacy; and participants would report reasonably similar behaviours in the diaries and questionnaires.
Methods
Participants were selected from a list of contacts who had previously taken part in a cross-sectional questionnaire about sex and drugs at a music festival.32 To be eligible for the original study, participants had to be aged between 16 and 29 years. Before being contacted, all participants on the list were assigned to one of the three modes of diary collection using random number allocation. Potential participants were telephoned between 3 and 17 April 2008. If interested in participating, they were sent further information and a consent form by either email or post.
The diary required participants to specify their number of sexual partners in the previous week and, for each partner, to specify: (1) the type of partner(s) (regular or casual); (2) whether that partner was a new partner or a previous partner; (3) the number of times they had sex that week; and (4) the number of times they used condoms that week. The paper group were also asked to specify the date on which they completed the diary.
At the beginning of the study period, 13 paper diaries and 13 reply-paid envelopes were sent to participants in the paper group. Reminders were sent to this group each Monday either by SMS or email, depending on individual preference. Additional reminders were sent to participants whose diaries had not been received by the following Monday. Each Monday, links to the online survey were sent to participants in the online group by email and/or SMS. SMS messages containing the diary questions were sent to participants in the SMS group each Monday. Participants in the SMS group completed the diary by replying to the message. Further reminders were sent each week to participants in the online and SMS groups who had not submitted their diaries by Tuesday afternoon.
The end point questionnaire was conducted online. Two days after the final diary was supposed to be completed, invitations to complete the questionnaire were sent to participants in all three groups by email and SMS. Reminders were sent periodically for the following 3 weeks to participants who had not yet completed the questionnaire. In the questionnaire, participants were asked the same questions about sexual behaviour that they were asked each week in the diaries; however, in the questionnaire these covered the entire 3-month time period. Additional questions elicited demographic information and opinions about the modes of diary collection.
Participants were reimbursed $2 per completed diary in the paper and online groups and $3 per diary in the SMS group (to cover the costs of sending the messages borne by participants). Participants who completed all diaries and the follow-up survey were also given a $25 music store voucher.
Analysis of data
To detect a difference of 1.5 points between the three groups in mean level of agreement with statements about the diaries with power of 80%, assuming a SD of 1.7 and allowing for the possibility that the agreement levels may not be normally distributed, a sample size of 25 participants was required per group.
All analysis was conducted in Stata 10. In order to determine whether there were demographic differences between groups, the distribution of age and gender in the three groups were compared using Kruskal–Wallis and χ2 tests, respectively. Response rate was measured by comparing the number of diaries submitted by each participant across groups using a Kruskal–Wallis test. Timeliness was measured by comparing the number of diaries that were returned late for each participant and, in total, in the online and SMS groups using Mann–Whitney rank sum tests. The effect of diary group on the number of days that each diary was submitted late was then quantified using negative binomial regression. Completeness of data was assessed by comparing the number of incomplete diaries per group using χ2 and Fisher exact tests. Acceptability was assessed by comparing levels of agreement with a series of statements about the diaries on the end point survey. Agreement was measured using an 8-point scale, ranging from strongly agree to strongly disagree. Levels of agreement were compared by group using Kruskal–Wallis and Mann–Whitney rank sum tests, adjusting for ties.
The validity of the questionnaire in relation to the diaries was assessed by comparing reported frequency of sex and condom use, number of partners and risk behaviours between the questionnaire and the diaries. To calculate total frequency of sex and condom use from the diaries, all entries from each participant were combined; no adjustments were made for missing diaries. Distinct partners were identified based on the number of partners reported each week, citing a new partner and changing from a regular to a casual partner. Participants were classified as at risk of sexually transmitted infections if they had multiple, new or casual partners during the study period and did not use condoms all of the time.32 Kendall τ coefficients were used to assess correlations between numerical data collected in the diaries and questionnaire; κ coefficients were used to assess correlations between binary categorical variables. Correlations were classified using the labels proposed by Landis and Koch: ‘almost perfect’ (>0.81), ‘substantial’ (0.61–0.80), ‘moderate’ (0.41–0.60), ‘fair’ (0.21–0.40) and ‘slight’ (0.00–0.20).33
Results
Participants and diaries
Of 243 potential participants, 79 (33%) could not be contacted during the recruitment period, 35 (14%) refused to participate and 129 (53%) agreed to receive further information; 72 (56%) provided written consent and were enrolled in the study, 24 to each group. Sixty-five (90%) participants completed the end point retrospective questionnaire, 23 from the SMS group, 22 from the online group and 20 from the paper group. Most (72.3%) were female, and most (73.8%) resided in metropolitan areas. The median age was 21 years and most participants were either enrolled in tertiary education or had completed a tertiary qualification (81.5%). 40% of participants lived with their parents, 27.7% with housemates and 27.7% with a partner. Very few (6.2%) had never had sex. The distributions of participant characteristics were similar across all three groups (p=0.66 and p=0.92, respectively) (table 1).
Participant characteristics
Response rates
All but one participant (from the online group) completed at least one diary. The maximum possible number of completed SMS, online or paper diaries was 13, and a median of 13 diaries were completed for each mode (p=0.61).
Timeliness
Of the diaries submitted, 233 of 291 (80.0%) SMS diaries and 179 of 284 (63.0%) online diaries were submitted on the correct day. The median number of diaries that each participant submitted late was 1 (IQR 2) in the SMS group and 1 (IQR 7) in the online group (p=0.04). Of the diaries submitted late, the median number of days late was 1 (IQR 1) in the SMS group and 2 (IQR 3) in the online group. Online diaries were less likely to be submitted on the correct day than SMS diaries (p<0.001) and were likely to be submitted 1.4 days later than SMS diaries (p<0.001). Based on participant self-report, 229 of 276 (83.0%) paper diaries were completed on the correct date. However, there were long time lags between the reported completion dates and receipt of the diaries by researchers (median number of days between self-reported completion and receipt of diary by the researchers: 3, IQR 5, range 53).
Completeness
One participant in the online group (4.1%), two in the paper group (8.3%) and four in the SMS group (16.7%) submitted at least one incomplete diary (p=0.33). Information about a sexual partner was missing for 12 of 191 partners (6.3%) reported by the SMS group, 5 of 170 partners (2.9%) reported by the paper group and 1 of 284 partners (0.4%) reported by the online group (p=0.001).
Acceptability
The diaries were rated as ‘very’ or ‘moderately’ convenient by 100% of the SMS group, 100% of the online group and 42% of the paper group. The diaries were perceived as very or moderately private by 95% of the SMS group, 95% of the online group and 89% of the paper group. When asked which method they would have preferred to complete, overall 51% preferred online, 38% preferred SMS and 8% preferred paper. Similar proportions of participants within the SMS group (70%) and online group (73%) preferred their assigned method.
A series of statements relating to the following themes could be rated from 1 (strongly disagree) to 8 (strongly agree): How easy was it to complete the diaries correctly and honestly each week? How enjoyable was it to complete the diaries? Did you feel that by completing the diaries you were contributing to important health research? The statements and levels of agreement are shown in table 2. Participants in the SMS group were more likely to express uncertainty about how to complete the diary than those in the online group (p=0.047). No other significant differences in levels of agreement between groups were detected.
Opinions of diaries by group
Comparison between diaries and retrospective questionnaire
Sixty-four participants (90%) reported sex at any time during the study period in the diaries and 56 (86%) in the questionnaire. In the diaries a total of 92 different partners were reported, and 80 were reported in the questionnaire.
There was almost perfect agreement between reports regarding partner type (regular or casual) in the diaries and retrospective questionnaire, substantial agreement on number of partners, frequency of condom use and frequency of sex, and moderate agreement on the proportion of partners who were new sexual partners (table 3). The diary collection mode did not affect the correlation.
Correlation between questionnaires and diaries
In the diaries, 24 participants (34%) were classified as at risk of sexually transmitted infections (see Methods section) whereas 20 (28%) were classified as at risk in the questionnaire. If only the questionnaire had been used and the diaries were taken to be the gold standard, two participants (2.8%) would have been misclassified as at risk and six (8.5%) would have been misclassified as not at risk. The correlation between the diaries and questionnaire on risk classification was substantial (κ=0.74).
Discussion
The SMS and online groups achieved higher response rates than the paper group and SMS was superior to online collection in terms of response time. Nevertheless, participants in the SMS group were significantly more likely to return incomplete diaries than those in the online group.
Online diary collection was the method preferred by most participants; however, among the SMS group, SMS was preferred over other modes. The paper group found the diaries inconvenient and, although they reported that their diary completion was timely, the long delays between the reported completion and receipt suggest that these reports were not valid. Participants in the SMS group were more likely to express uncertainty about how to complete the diary than those in the online group. In addition, although these differences did not reach statistical significance, the online group found the diaries more enjoyable, less boring and easier to complete and the paper group were more concerned that someone would find and read their diaries.
On most measures, online diaries were found to be superior to SMS and both were superior to paper; however, the online diaries were more likely to be completed late compared with the SMS diaries. In some situations SMS may be preferable for data collection. For example, if data were collected daily rather than weekly, SMS would probably be more convenient and timely than other modes. SMS, however, could not be used for longer or more complex surveys as messages are limited in character length. The timeliness of SMS must therefore be weighed against the lower completion rate of individual questions.
Previous studies have shown that sexual behaviour information collected regularly in diaries is more accurate than information collected in retrospective questionnaires. In our study the correlations between data reported in the diaries—encompassing all three diary modes—and retrospective questionnaire ranged from moderate to almost perfect. The weakest correlations regarded the number of sexual partners reported and the classification of partners as new or previous partners. The discrepancies regarding classification of partners as new or previous might be attributed to confusion about when the time period started and what constituted a new partner.
It was unexpected that the correlation between the diaries and retrospective questionnaire would be lower in reporting numbers of partners than in reporting the frequency of sex or condom use. In most diary studies, accuracy is greater for a less frequent event (such as partner change) than a high frequency event (such as having sex). Nevertheless, there was little variation in frequency of sex over time for most participants, which may explain why the data reported in the diaries and the retrospective questionnaire were more strongly correlated for this high frequency yet stable behaviour. As the calculation of number of partners reported in all three modes of the diaries was partly based on classifying a partner as new (which, as discussed above, may have been inaccurate), the calculated number of partners in the diaries may have been inaccurate.
This study shows that reported levels of risk are substantially correlated between diaries and retrospective questionnaires, with a slight tendency to under-report risky behaviour in the retrospective questionnaire. In the literature, some studies find over-reporting and some under-reporting, and the level of correlation ranges from strong to weak.3 5 8 22 25 34 However, overall it seems that retrospective self-reported behaviour—while not perfect—is sufficiently similar to data collected using diaries for research purposes where identifying risk behaviour is the primary outcome.35
This study was limited by its sample size, which was below 25 per group, the size required to achieve adequate statistical power. Given that initially only 24 participants were recruited to each group and, of these, only 20, 22 and 23 participants from the paper, online and SMS groups, respectively, responded to the end point questionnaire, the study was underpowered, particularly with respect to measuring outcomes relating to the end point survey.
The sample was convenience-based so the findings may not be generalisable to other young people. The participants were recruited from a list of participants from a previous study who had agreed to be contacted for further research. Given that only 30% of the 243 potential participants took part in this study, those who participated were probably not representative of the potential sample. This may have affected the diary response rates, the sexual behaviour of the participants and the correlation between reported behaviour in the diaries compared with the end point questionnaire. In particular, it may not be possible to replicate the intensive data collection used in the diaries in a less motivated group of participants.
In addition, completing the diaries throughout the 3-month period may have enhanced recall in the end point questionnaire. Sexual health behaviour studies often require retrospective reports about time periods longer than 3 months or the recall of events or periods in the more distant past. Further larger studies may be required to determine the length of time for which retrospective questionnaires remain comparable to diaries. Furthermore, the data are self-reported and participants may over-report or under-report sexual risk behaviour if they believe that researchers will form a better impression of them.36
In conclusion, this study found that SMS is a convenient and timely method of collecting brief behavioural data, but that online data collection is preferable to most participants and more likely to be complete. In addition, self-reported sexual behavioural data and classifications of participants' risk of sexually transmitted infections collected in retrospective sexual behaviour questionnaires were found to substantially agree with data collected through weekly self-reported diaries.
What is already known on this subject
Regularly reported sexual behaviour information collected in sexual behaviour diaries has been shown to be more accurate than information collected retrospectively through questionnaires. Compared with diaries completed on paper and posted to researchers, online and telephone sexual behaviour diaries have been shown to promote timeliness of response and to reduce concerns regarding confidentiality. Mobile phone text messaging (SMS) has previously been shown to be a convenient and timely method for monitoring disease (eg, to record asthma symptoms) but has not previously been used to collect sexual behaviour information.
What this study adds
This is the first study to trial the use of SMS as a mode of sexual behaviour diary data collection. It shows that SMS is more timely than online and paper-based methods. Moreover, like online diary collection, SMS is considered more private than paper diary collection. However, when compared with online diaries, SMS diaries are more likely to be incomplete, and online completion is preferred to SMS and paper diaries by most participants.
Acknowledgments
Tim Spelman and Maelenn Gouillou assisted with statistical analysis.
References
Footnotes
Funding Burnet Institute.
Competing interests None.
Patient consent Obtained.
Ethics approval Ethics approval for the study was obtained from the Victorian Department of Human Services and the trial was registered with the Australian New Zealand Clinical Trials Registry (No. 00082347).
Provenance and peer review Not commissioned; externally peer reviewed.