Article Text

Download PDFPDF

Why do peer reviewers decline to review? A survey
  1. Leanne Tite,
  2. Sara Schroter
  1. Correspondence to:
 Dr S Schroter
 BMJ Editorial Office, BMA House, Tavistock Square, London WC1H 9JR, UK; sschroter{at}bmj.com

Abstract

Background: Peer reviewers are usually unpaid and their efforts not formally acknowledged. Some journals have difficulty finding appropriate reviewers able to complete timely reviews, resulting in publication delay.

Objectives and methods: A survey of peer reviewers from five biomedical journals was conducted to determine why reviewers decline to review and their opinions on reviewer incentives. Items were scored on 5-point Likert scales, with low scores indicating low importance or low agreement.

Results: 551/890 (62%) reviewers responded. Factors rated most highly in importance for the decision to accept to review a paper included contribution of the paper to subject area (mean 3.67 (standard deviation (SD) 86)), relevance of topic to own work (mean 3.46 (SD 0.99)) and opportunity to learn something new (mean 3.41 (SD 0.96)). The most highly rated factor important in the decision to decline to review was conflict with other workload (mean 4.06 (SD 1.31)). Most respondents agreed that financial incentives would not be effective when time constraints are prohibitive (mean 3.59 (SD 1.01)). However, reviewers agreed that non-financial incentives might encourage reviewers to accept requests to review: free subscription to journal content (mean 3.72 (SD 1.04)), annual acknowledgement on the journal’s website (mean 3.64 (SD 0.90)), more feedback about the outcome of the submission (mean 3.62 (SD 0.88)) and quality of the review (mean 3.60 (SD 0.89), and appointment of reviewers to the journal’s editorial board (mean 3.57 (SD 0.99)).

Conclusion: Reviewers are more likely to accept to review a manuscript when it is relevant to their area of interest. Lack of time is the principal factor in the decision to decline. Reviewing should be formally recognised by academic institutions and journals should acknowledge reviewers’ work.

View Full Text

Statistics from Altmetric.com

Peer review is an important part of scientific publishing, but has been criticised for being ineffective, slow and expensive.1–3 Reviewers are usually unpaid and their efforts not formally acknowledged, but journal editors depend on them to inform publication decisions and to improve the quality of manuscripts. Editors of some journals have frequent difficulty finding appropriate reviewers who are able to complete timely reviews. This can result in publication delay.

Only a few empirical studies to date have attempted to find ways of improving quality of peer reviews.4–6 Tested interventions include the provision of written feedback,4 attendance at training workshops5,6 and the use of self-taught training materials,6 but none resulted in improvements in review quality. Other studies have evaluated methods of improving the timeliness of reviews.7,8 Pitkin and Burmeister7 conducted a trial and found that requesting permission from reviewers before sending manuscripts for review led to a higher number of refusals than when manuscripts were sent before seeking permission. However, those who agreed to review after a request completed their reviews more quickly than those who had not been initially asked. In another study they found that contacting late reviewers resulted in a review within 7 days in about two thirds of cases, whether contact was by telephone, fax or email.8

We conducted a survey of reviewers for five biomedical journals to determine the reasons why they decline to review, and their opinions on reviewer incentives.

METHODS

Questionnaire development

The questionnaire content was developed from two sources.

Interviews with reviewers

We conducted 28 semistructured telephone interviews with reviewers from four biomedical journals owned by the BMJ Publishing Group to determine the range of reasons why reviewers decline requests to review and what rewards might encourage reviewers. We used a random sample of reviewers for the journals Archives of Disease in Childhood, BMJ, British Journal of Ophthalmology and Sexually Transmitted Infections, who had declined a request to review more than once during the previous 12 months. A series of open-ended questions was asked about the most recent occasion respondents accepted or declined to review for the journal during the study period. The questions covered factors important in accepting and declining to review, and opinions and preferences for reviewer incentives. Responses were recorded by the interviewer at the time of interview and later tabulated.

Reviewer feedback

The BMJ Publishing Group’s online reviewing system asks reviewers to state a reason each time they decline a request to review. On the basis of recurrent responses, we developed a set of key reasons.

Questionnaire construction

Items from both sources were grouped into four broad questionnaire themes: reasons why reviewers decline to review, reasons why reviewers accept to review, opinions on financial incentives and opinions on non-financial incentives. All items were constructed as 5-point Likert scales, with low scores indicating low importance or low agreement. The questionnaire was piloted and refined before use in the survey.

Survey

After receiving approval for the study from the BMJ Ethics Committee, we generated a list of all invited reviewers for the journals Archives of Disease in Childhood, BMJ, Emergency Medicine Journal, Gut and Journal of Epidemiology & Community Health for a 9-month period. We divided reviewers for each of the journals into strata on the basis of the number of times they had declined or accepted a request to review during this study period. We then randomly selected 200 reviewers from across the strata from the samples of each journal using the random number generator in MS Excel, giving a total sample of 1000. Invitations to take part in the survey were sent by email, including a link to access the online survey. Reminders were sent by email to non-responders at weeks 2 and 5 after the initial mail.

RESULTS

Response rates

We received 110 automated responses from invalid email addresses. We received a response from 606 of the 890 (68%) remaining valid email addresses. Fifty five respondents completed only the opening questions of the survey and were excluded from the analysis. We analysed results for the five journals individually (Archives of Disease in Childhood, n = 115; BMJ; n = 133; Emergency Medicine Journal, n = 101; Gut, n = 90; Journal of Epidemiology & Community Health, n = 112). However, results across journals were broadly similar, so we present only the overall findings from the five journals combined (n = 551).

Respondent characteristics

Of the 551 respondents, 30% (n = 386) of the reviewers were women, 71% (n = 389) were aged <55 years, 66% (n = 362) were academics and 88% (n = 486) were active writers of research articles.

Reasons for accepting to review

Data are presented as the mean (standard deviation (SD)) of responses on a 5-point Likert scale.

In all, 429 respondents reported they had accepted a request to review during the study period. Factors rated most highly in importance to the decision to accept a paper to review included contribution of the paper to the subject area (3.67 (0.86)), relevance of the topic to own work/interests (3.46 (0.99)) and opportunity to learn something new from the paper (3.41 (0.96); table 1). A sense of professional duty and the reputation of the journal were also considered important factors. Factors rated as least important were monetary payment (BMJ reviewers only, 1.64 (0.76)), the reputation of the authors of the paper (1.78 (0.95)) and academic reward such as career enhancement (1.81 (0.96)).

Table 1

 Reasons for accepting to review (all journals, n = 429)

Reasons for declining to review

During the study period, 258 reviewers reported that they had declined a request to review. The most highly rated factor important in the decision to decline to review was conflict with other workload (4.06 (1.31), table 2). Having too many reviews for other journals (2.54 (1.45)) and a tight deadline for completing the review (2.70 (1.32)) were also important in the decision to decline. Factors rated as least important were having previously reviewed many papers on the same subject (1.26 (0.59)), length of the manuscript (1.29 (0.63)), comments not taken into account in the past reviewing experience (1.34 (0.81)), lack of formal recognition of reviewer contribution (1.38 (0.77)) and having to use the online review system (1.39 (0.92)).

Table 2

 Reasons for declining to review (all journals, n = 258)

Opinions on financial incentives

The survey yielded more agreement than disagreement that financial incentives will not be effective when time constraints are prohibitive (3.59 (1.01)) and that small financial incentives would not encourage reviewers to accept reviews (3.53 (0.95)), but many reviewers agreed that financial incentives encourage reviewers to accept requests to review (3.39 (0.86), table 3). There was little agreement that financial incentives could improve the quality of reviews (2.42 (0.98)), but responses to the item “financial incentives could compromise the quality of review” were mixed (3.08 (1.00)).

Table 3

 Financial incentives to review (all journals, n = 551)

Opinions on non-financial incentives

We found highest agreement that the following incentives would encourage reviewers to accept requests to review: free access or subscription to journal content (3.72 (1.04)); annual acknowledgement on the journal’s website (3.64 (0.90)); more feedback about the outcome of the submission (3.62 (0.88)) and quality of the review (3.60 (0.89)); and appointment of reviewers to the journal’s editorial board (3.57 (0.99), table 4). There was less agreement that publication of the review with the article (2.62 (1.09)), the option of adding the reviewer’s name at the end of the published article (2.77 (1.10)) and gifts would act as incentives for reviewers.

Table 4

 Incentives to review (all journals, n = 551)

DISCUSSION

Reviewers were more likely to accept a request to review a manuscript when the paper was relevant to their area of interest. Lack of time was the principal factor in the decision to decline. Participants regarded reviewing as an opportunity to learn something new in their area of interest and as part of their professional duty. They were not of the opinion that small financial incentives would be effective, but were favourable to the idea of some non-financial incentives. The most motivating incentives included free access or subscriptions to the journal, annual acknowledgement of reviewers on the journal’s website, feedback about the outcome of the manuscript submission and the quality of their reviews, and the appointment of the best reviewers to the journal’s editorial board.

Study implications

Peer review is an important and accepted part of scientific publishing and has become an indicator of perceived journal quality. Many reviewers regard journal peer review as part of their professional responsibility. However, academic institutions do little to train reviewers, and expect staff to review in addition to their formal workload. Although journal editors rely on reviewers to improve the quality of manuscripts and to inform decision making regarding the publication, reviewers are usually unpaid and their efforts not formally acknowledged by the journal. Interestingly, 80% of participants said that lack of formal recognition was not an important factor in their decision to decline. Some journals now offer reviewers continuing medical education or continuing professional development credits, but these are relevant only to those in medical posts.

Reviewers in our study indicated that they were not motivated by financial incentives, possibly because they saw reviewing as part of their academic role and were not usually paid for reviewing. Most journals cannot afford to pay reviewers consultancy rates for their time, but there are some inexpensive options that may act as acknowledgement or incentive for reviewers. For example, more journals could post annual acknowledgement lists of their reviewers on their websites, send letters of thanks to their most valued and regular reviewers, appoint their best reviewers to their editorial boards and give more feedback about the editorial outcome of the papers reviewed. However, although reviewers were favourable to the idea of feedback about their reviews, this would be resource intensive for journals.

Peer review has traditionally been a closed “black box” process. Some journals are considering a move to open peer review where the reviewers of published manuscripts are acknowledged in the publication and have the option of their review being made openly available. Reviewers in this study were divided over the idea of a published acknowledgment of the reviewer’s contribution and less keen on the publication of the review in the journal. Further research is necessary to gauge reviewers’ perceptions and concerns before switching to an open-review process.

What is already known on this topic

  • Peer review is central to the publishing process, yet reviewers’ efforts are not formally acknowledged.

  • Some journals have difficulty finding appropriate reviewers who are able to complete timely reviews, thus resulting in publication delay.

  • Little is known about why reviewers decline or accept invitations to review and which incentives are likely to motivate them to review.

What this study adds

  • Respondents regarded reviewing as an opportunity to learn something new in their area of interest and as part of their professional duty. Lack of time was the principal factor in the decision to decline.

  • Reviewers were not of the opinion that small financial incentives would be effective, but were favourable to the idea of some non-financial incentives.

Policy implications

  • Reviewing should be formally recognised by academic institutions, and journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.

With an increasing number of submissions to journals and demand on reviewers’ time, it is becoming increasingly important to solicit reviewers appropriately. Journals should remind reviewers to update their personal profiles, detailing their areas of expertise to ensure that they receive appropriate review requests and are not sent more than the agreed quota.

Study limitations

Our study was limited to a sample of reviewers with internet and email access as our survey was administered online. It is possible that reviewers who use electronic review systems have views different from those who use traditional paper-based reviewing. However, all journals published by the BMJ Publishing Group now require reviewers to use the journals’ online reviewing system, so respondents were representative of reviewers for these journals. Many other journals now use electronic tracking systems. It is also possible that reviewers working in specialties not covered by the journals in this study may have additional considerations when choosing whether or not to review for a journal. We deliberately selected journals with international reviewers and from different specialties, as well as a general medical journal, to increase the generalisability of findings across biomedical reviewers. Unfortunately personal characteristics of reviewers such as age, sex and reviewing experience are not recorded routinely on the manuscript tracking databases, so we were unable to compare responders with non-responders in terms of these factors.

Conclusion

Reviewing should be formally recognised by academic institutions, and journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.

Acknowledgments

We thank all the reviewers for taking part in the survey.

REFERENCES

View Abstract

Footnotes

  • Funding: This research was funded by the BMJ Publishing Group’s research budget. The study sponsor did not play a role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

  • Competing interests: LT and SS are employed by the BMJ Publishing Group.

  • Ethical approval: The BMJ Ethics Committee approved the study.

    Contributors: LT and SS designed the study and developed the questionnaire. LT coordinated data collection. Both LT and SS analysed the data, wrote the paper and approved the final version. SS will act as guarantor.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • In this issue
    Carlos Alvarez-Dardet John R Ashton