STUDY OBJECTIVE This study assessed several methodological aspects related to the quality of published controlled clinical trials (CCTs) in relation to the participation of an epidemiologist/biostatistician (E/B).
DESIGN Handsearch of CCTs published in four medical leading journals for 1993–1995.
METHODS Quality variables, abstracted from a review, were related to authors' specialties. Five hundred and ninety four CCTs were identified via a hand search. The department/unit membership was used to attribute authors' specialties. Of 594 CCTs identified, in 127 the authors' specialties could not be known, leaving 467 trials for analysis.
RESULTS E/B participation occurred in 178 trials (38.1%). This participation was more frequent in multicentric, bigger, and in those trials describing any funding agency. These factors were controlled for in the analysis. E/B participation was positively associated with pre-study sample size estimation (OR = 1.5, 95% confidence intervals (CI) 1.0, 2.3), with reporting the dates for starting/ending the study (OR = 2.1, 95% CI 1.4, 3.3), with using an objectively assessed outcome (OR = 2.4, 95% CI 1.2, 4.6) and with the intention to treat principle (OR = 2.0, 95% CI 1.3, 3.0). The overall quality score was higher in trials where E/B participated.
CONCLUSIONS The results suggest that E/B improve the quality (at least of reports) of clinical trials. Given that quality of research is frequently used to evaluate potential sources of heterogeneity between trials, these results are relevant for meta-analysis.
- clinical trials
Statistics from Altmetric.com
Quality of published research is an issue reinforced by the dissemination of meta-analysis. All researchers agree that the most standardised design is the randomised controlled clinical trial (CCT), permitting the quality to be ascertained using validated protocols.1 There has been a considerable debate about whether the quality of a study should be included in pooled estimates in quantitative meta-analysis.2-4
Nevertheless, most authors agree that differences in quality may help in explaining heterogeneity among studies.5 6 Although many reports have considered the items needed to adequately assess the quality of a clinical trial, the variables influencing quality have seldom been evaluated. Many textbooks recommend that an epidemiologist and/or a biostatistician participates in clinical trials from the beginning.7 8 We have not found any previous report analysing whether the contribution of an epidemiologist/biostatistician (E/B) actually improves the quality of a published controlled clinical trial and this is the main objective of our assessment.
The target population was CCTs published between 1993 and 1995 in the New England Journal of Medicine, theLancet, the Journal of the American Medical Association, and theBritish Medical Journal. We included all reports of experimental trials on humans that had two or more treatment groups (including placebo group as a treatment group); and those labelled by the authors as “clinical trials”, “field trials” or “randomised trials”. This search yielded 617 clinical trials. Of these, 23 had no control group and were excluded, leaving 594 CCTs.
Data were collected by a trained reviewer who was unaware of a future assessment of the contributions of certain specialty to the overall quality of a clinical trial. General information was abstracted from each CCT, including design type, number of participating centres, setting of the research, country, aknowledgement of funding, and authors' specialties. This latter item was assessed by department/unit membership. An epidemiologist and/or biostatistician was considered as coauthor if at least one of the authors belonged to a department/unit of epidemiology, clinical epidemiology, and/or biostatistics. In 127 CCTs the authors' specialties could not be ascertained and these articles were omitted for this report, leaving a study population of 467 CCTs.
Each CCT was evaluated according to published guidelines.9 10 Data on several methodological characteristics were gathered. We assigned one point to the correct answers for each of the items in the following list marked with an asterisk, in order to compute a quality score10:
- Population recruitment: pre-study sample size estimation*, existence of inclusion and/or exclusion criteria*, number of people asked to participate and the number who accepted*.
- Assignment of the intervention: use of randomisation*, whether it was blind*, and checking of baseline comparability of study groups.
- Data collection and reporting: reporting of both date of starting and date of ending, description of the treatments given to intervention and control groups*, masked assessment of outcome*, justification of unmasked procedures, whether a method to assess treatment compliance was described, assessment of treatment compliance, assessment of adverse effects, existence of detailed criteria to assess the main outcome* and whether these were objective*, and the number of losses (subjects who abandoned the study) during follow up.
- Statistical analysis: type of statistical procedure used*, reporting p values and/or confidence intervals*, application of the intention to treat principle*, use of multivariate procedures when needed (for example, if the assessment of baseline comparability between the study groups revealed differences), and estimation of statistical power if the results did not achieve statistical significance*.
The χ2 test was applied to compare proportions. The odds ratio and its 95% confidence limits (CI) were used to assess the degree of association between several methodological characteristics and the appearance of an E/B among the authors of a CCT. Several variables were related with both CCT quality and coauthorship by an E/B and could be confounding factors. They were controlled for using multiple logistic regression analysis. The mean quality score of CCTs according to authors' specialties and its 95% CI were estimated. Adjusted quality scores were estimated by analysis of covariance.
Participation of researchers belonging to epidemiology/statistics units increases with the number of participating centres, study sample size, and the existence of funding.
Clinical trials with authors belonging to epidemiology/statistics units report their methods and results better, even in multicentric studies.
Multicentric clinical trials score better in all quality related variables than clinical trials based on only one setting.
Researchers of epidemiology/statistics units improve the adherence to the intention to treat principle.
E/B were coauthors in 178 (38.1%) of all CCTs. This was more frequent in multicentric trials, in bigger trials, in trials assessing “death” as the main outcome, and in those that indicated their source of funding (table 1). There were no large differences according to the use of blinding procedures, treatment given to the control group, approval by an Institutional Review Board, reporting of informed consent, and type of design.
The relationships between authorship by an E/B and several characteristics related to the selection of the study population and the assignment of intervention are summarised in table 2. Overall, the crude frequency of these quality aspects was higher when an E/B was involved. In crude analyses, we found differences for pre-study sample size estimation only. After adjusting for the trial's size, number of participating centres and funding description, only pre-study sample size estimation remained statistically significant; a borderline association (lower limit of CI close to unity) was found for the assessment of baseline comparability of study groups.
Similar results were observed for characteristics of data collection during follow up (table 3). There were only four CCTs that did not describe sufficiently the treatments given to their study populations (results not shown in table 3). The only variable negatively associated (OR < 1) with the participation of an E/B was reporting the number of withdrawals during follow up. Positive associations after controlling for potential confounders were observed regarding the information about dates of beginning and ending the trial, and the use of objective methods for the assessment of outcome.
Regarding statistical analysis (table 4), a clear association was observed with the application of intention to treat principle. In the remaining variables, participation of an E/B scored better, although the results were non-significant.
A quality score was computed. The mean values for studies with and without an E/B were 10.9 (95% CI 10.7, 11.1) and 10.3 (95% CI 10.1, 10.5), respectively (p < 0.001). Analysis of covariance, controlling for the same variables as logistic regression analysis, did not change this difference (p = 0.008).
Finally, the above mentioned analyses were repeated for multicentric studies, although only the most relevant variables are displayed in table 5. Several variables are not shown because of low numbers (randomisation as there were only two non-randomised CCTs, reporting p values and/or CIs as all studies did notify them, and description of treatments given to the patients as all but one CCT did it). The multicentric character was considered one of the most important features associated with a high quality score; size of the study and funding lost relevance when it was taken into account. A trend to observe less strength of association (lower ORs) was appreciated and in most cases adjusting for funding and sample size did not change the estimates. Nevertheless, studies with participation of an E/B showed more frequently pre-study sample size estimation and blind randomisation, a higher application of the intention to treat principle, had explicit methods to assess treatment compliance more frequently, assessed the outcome more objectively, and the dates of starting/ending were more often reported. The mean quality scores for both studies with and without an E/B were higher than before, 11.2 (95% CI 10.9, 11.4) and 10.6 (95% CI 10.3, 10.8), respectively (p = 0.001). Adjustment for sample size and funding did not change this difference (p = 0.007).
This study may present several limitations. Firstly, we are aware that the attribution of specialty to authors is imperfect. It is possible that some authors belonging to clinical departments can be E/Bs as many departments/services have positions that need the qualification of an E/B. This misclassification would introduce a bias toward the null under the assumption that an E/B improves the quality of a clinical trial.
Secondly, the adjustment can be considered as questionable. Most of the studies included in this report were not launched by epidemiologists nor biostatisticians, but by clinicians. Given that clinicians look for collaborators of other institutions and for funding we consider that the variables controlled for usually precede the participation of an E/B. This does not preclude that E/Bs be responsible of multicentric CCT, although it is less common.
Thirdly, observer bias can be responsible of the results. Quality evaluation was not blind and an epidemiologist reviewer can favour studies with participation of colleagues. The reviewer was not an epidemiologist but a pharmacist (MRC) who did this task as part of his PhD thesis; the objective of his thesis was to focus on ethics in clinical trials.11 The key methodological questions of the quality protocol were distributed unevenly through the questionnaire. All the different authors' specialties were ascertained in the same group of questions. The reviewer was therefore unaware of a future assessment of the contributions of a certain specialty to the overall quality of a clinical trial.
Fourthly, is the quality of the report of a published trial related to the intrinsic (true) research quality? We cannot answer this question, as original authors were not requested to provide additional information on methodological details not adequately reported in an article. The contents of the Methods section of a paper are not an exclusive responsibility of authors, but also of the type of journal. Thus, editors and reviewers also influence them. It may be possible that epidemiologists and biostatisticians are more familiarised with the methodological standards to be mentioned in the Methods section, and, consequently, the articles signed by them receive a higher quality score, not only because of an actual higher quality of the study itself, but mainly for not omitting the description of these issues. This would have been overcome if reviewers and editors judge a submitted article according to established guidelines. Notwithstanding, there are some aspects12 that do not depend on “omissions” from the Methods section, such as the analysis following the intention to treat principle, which, according to our data, was clearly related with the participation of an E/B.
As epidemiologists and biostatisticians usually have a more comprehensive mathematical and methodological background, the results regarding pre-study sample size estimation, the higher frequency of statistical power estimation after a negative result, etc, were expected. It is interesting to observe that the dates of starting and ending a clinical trial (that is, the study period) were more frequently given when these professionals were included in the list of authors. This difference in reporting the study period can be useful for explaining heterogeneity in a meta-analysis if there is a period effect in the assessed intervention.
Other non-quantitative variables, such as the use of an objectively measured outcome, were also related with the inclusion of an E/B in the list of authors.
The results suggest that published CCTs with E/Bs as coauthors more adequately meet the required standards of clinical trials. If it were true that the quality of a published report adequately reflects the intrinsic research quality, a cautious recommendation would be to include an epidemiologist and/or a biostatistician in the team conducting clinical trials; this recommendation can be found in several textbooks.7 8 In any case, our results may be relevant for meta-analysis to help explain heterogeneity as differences in quality have been reported to partially explain heterogeneity6 10 13 and our findings show that differences in quality, in turn, are related to the composition of the research team.
Funding: this work was funded by a fellowship from the Association of Friends of University of Navarre.
Conflicts of interest: none.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.