Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews

Introduction

Identifying relevant studies is “the most fundamental challenge” when compiling a systematic review.1 Electronic databases, such as Medline, may detect only about half of papers identified by the gold standard of hand searching journals.1 Hand searching requires a focus, usually the specialist literature,2 which may not exist for newly developed fields or those that cross boundaries with other areas. We examined the usefulness of contacting other experts when searching for relevant references for a systematic review of a field where such a specialist focus does not exist.

Methods and results

As part of a systematic review undertaken in 1996, all published literature relating to “near patient testing” (any investigation performed in a clinical setting where the result is available without a sample being sent to a laboratory for analysis) in primary care was identified for 1986-95.3 Electronic databases were searched and secondary citations were collected from identified publications (see table). The search strategy is reported elsewhere.3 Indexes of abstracts from major international primary care scientific conferences were hand searched. We sent a questionnaire to 194 academics in the United Kingdom (heads of academic departments of general practice and clinical chemistry and researchers identified from the previously mentioned abstracts) and to 152 commercial companies known to have an interest in near patient testing. The questionnaire requested key references from journals, unpublished data, and names of other workers in the field.

Overall, 156 (45%) questionnaires were returned completed (103 (53%) of those sent to academics and 53 (35%) of those sent to commercial companies). No unpublished data were offered.

Articles that did not report original data, were not relevant to primary care, or were not in English were excluded. Remaining articles were then judged by both an external and internal reviewer against standard appraisal criteria,4 with discrepancies being adjudicated on by a third reviewer (BCD). A verifying search did not identify any further key words or references.

The searches yielded 1286 citations comprising 1057 unique references potentially eligible for inclusion in the review. The table shows a breakdown of the results by source—102 references were relevant, of which 29 were of high quality. Of the 102 unique eligible references, 50 (49%) were identified by one of the electronic databases, 40 (39%) by people working in the field, and 31 (30%) by hand searches. Each of these sources produced a similar proportion of high quality papers.

The 1057 potential references came from 418 different publications, of which only 48 were found to contain relevant articles. Only one journal, the BMJ, contained more than 10% (11 articles) of the relevant references.

Comment

This study confirms that searching electronic databases may uncover only half of all relevant studies1 and shows the importance of contacting other experts when compiling a systematic review. Over 300 of the references in this review were identified by experts working in the field, of which 40 were found subsequently to be eligible. Twenty four references (24% of the total number of eligible references) would have been missed entirely without the input of people working in the field.

The lack of sensitivity of electronic databases may be due to problems with indexing5: articles from journals that lie outside of the mainstream (which currently includes primary care) take some years to be electronically indexed; furthermore, near patient testing is an expanding field without unique medical subject headings (MeSH terms). The continuing work of the Cochrane Collaboration is improving this situation,5 but at the time of this study, making up the shortfall in articles by hand searching would not have been justifiable in terms of either time or money.

This study has quantified the contribution of experts to a systematic review and has found them to be an essential source for identifying literature. We suggest that appropriate experts should be consulted when performing a systematic review in a developing field that does not have a clearly defined specialist literature.