Abstract
The Journal of the American Society of Nephrology (JASN) gives authors submitting original research the option of suggesting qualified reviewers or those they wish to exclude. This historical habit often leaves us wondering whether author preferences correlate with reviewer recommendations and whether differences related to reviewer selection affect decisions by editors. In a self-study presented here, we found that author-suggested reviewers, as a group, make more positive recommendations than editor-suggested reviewers (P = 0.01), although the difference disappears when recommendations are compared with those of editor-suggested reviewers of the same manuscript (P = 0.081). The distribution of recommendations by author-excluded reviewers, as a group, did not differ from those by editor-suggested reviewers; however, author-excluded reviewers impart significantly more negative recommendations than other reviewers of the same manuscript (P = 0.029). We further explored whether such differences result from individual reviewer tendencies to give generally more positive or more negative recommendations than editor-suggested reviewers and found no such tendency. Finally, editorial decisions on manuscripts reviewed by author-suggested or author-excluded reviewers do not differ from those decisions on manuscripts assigned but not reviewed by them. JASN's policy of editors making decisions independent from individual reviewer recommendations minimizes the effect of selection bias on publication decisions.
The editors of JASN strive to publish only the very best research that significantly advances the field of nephrology. To maintain this standard, decisions on whether to publish a manuscript must be based on merit and novelty, uncolored as much as possible by reviewer bias. Although our editors make their decisions independently rather than relying on a calculated vote derived from a set of reviews, we often wonder if selection bias for reviewers affects publication decisions. One potential source of bias is reviewers suggested or excluded by authors. Indeed, studies performed at other journals show that author-suggested reviewers are more likely to recommend publication than editor-suggested reviewers;1–6 whether recommendations from author-excluded reviewers differ from other assigned reviewers remains unknown.
To explore whether author-suggested or -excluded reviewers influence editorial decisions at JASN, we asked whether such recommendations differ from those by reviewers selected independently by editors. We further asked whether receiving recommendations from author-suggested or -excluded reviewers leads to differences in the publication decision of the editors.
SELECTION OF REVIEWERS
This study is a retrospective analysis of submitted original research manuscripts that survived initial triage (approximately 35% of submitted manuscripts). Data were collected from the online submission system (ScholarOne). This system allows authors to suggest or exclude reviewers by entering their names and e-mail addresses and then designating them as “preferred” (suggested) or “nonpreferred” (excluded); the number of reviewer suggestions or exclusions is not limited. Titles, publication types, submitting author names, and editor decisions for all manuscripts describing original research submitted October 2008 through March 2009 were exported from ScholarOne as a CSV file. Information on the manuscripts for which authors made suggestions to exclude reviewers was collected as above for the period April 2007 through September 2009. Reviewer names (those suggested or excluded by authors and assigned by editors) and recommendations were then gathered by searching with each manuscript ID. The editors listed on the masthead of the September 2009 issue provided the editorial decisions used in this analysis. Recommendations and decisions were assigned ordinal categories as follows: Reject, category 1; reject unless substantially revised, category 2; accept if significantly revised, category 3; accept with revision, category 4; and accept as submitted, category 5. Nonparametric tests of significance (SPSS statistical program, version 18) were used because the data were not distributed normally.
FINDINGS OF OUR SURVEY
Authors Are More Likely To Suggest Reviewers than To Exclude Them; Editors Are More Likely To Assign Suggested than Excluded Reviewers
We originally collected all data during the period October 2008 through March 2009. Because so few manuscripts in this period were assigned author-excluded reviewers, we also collected data on author-excluded reviewers over the largest possible period, April 2007, when the current recommendation scores were implemented, through September 2009, when this survey was conducted.
To understand the relative importance of recommendations by author-suggested and author-excluded reviewers, we determined the proportion of manuscripts for which authors suggested or excluded reviewers, for which editors assigned reviewers authors had suggested or excluded, and on which such reviewers made recommendations (Figure 1). Authors were much more likely to suggest reviewers (73%) than to exclude them (21%, P < 0.0001). Similarly, editors were more likely to assign author-suggested reviewers (70% of the manuscripts for which authors made suggestions) than author-excluded reviewers (25% of the manuscripts for which authors made suggestions, P < 0.0001). Author-suggested and author-excluded reviewers did not differ in likelihood to complete reviews (77% versus 62%, P = NS). Thus, author-suggested reviewer recommendations, if they influence editorial decisions, might affect more manuscripts than recommendations by author-excluded reviewers.
Authors are more likely to suggest reviewers than exclude them and editors are more likely to assign suggested than excluded reviewers. (A) Proportions of the October 2008 through March 2009 sample of reviewed manuscripts for which authors suggested reviewers (green, blue, and purple), editors assigned author-suggested reviewers (blue and purple), and author-suggested reviewers made recommendations (purple). (B) Proportions of the April 2007 through September 2009 sample of reviewed manuscripts for which authors excluded reviewers (green, blue, and purple), editors assigned author-excluded reviewers (blue and purple), and author-excluded reviewers made recommendations (purple).
Author-Suggested Reviewers Are More Likely To Recommend Publication than Editor-Suggested Reviewers
We first compared recommendations by author-suggested reviewers to those of editor-suggested reviewers on all manuscripts assigned an author-suggested reviewer (Figure 2A). The median recommendation by author-suggested reviewers (category 3, accept if significantly revised), as a group, was the same as that for editor-suggested reviewers. However, author-suggested reviewers were much more likely to give a recommendation of category 4 (accept with revision, 41% versus 22%) and much less likely to give a recommendation of category 1 (reject, 11% versus 22%) than editor-suggested reviewers. Most importantly, the distributions of recommendations, as a group, differ significantly (P = 0.01).
Author-suggested reviewers are more likely to recommend publication than editor-suggested reviewers. (A) Distributions of recommendations by all editor-suggested and author-suggested reviewers of manuscripts in the October 2008 through March 2009 sample; P = 0.01, χ2 test. (B) Distribution of differences between author-suggested reviewers' recommendation and the median recommendation by other reviewers considering each manuscript in the same sample (n = 76); P = 0.081, Wilcoxon signed ranks.
To isolate differences between reviewers, we compared the recommendation given by each author-suggested reviewer to the median recommendation given by other reviewers considering the same manuscript. Author-suggested reviewer recommendations were more positive than other reviewers more often than they were more negative (50% versus 28%), and the median difference was positive (0.25, Figure 2B). However, the difference in recommendation between author-suggested and editor-suggested reviewers within the same manuscript was not significant, P = 0.081. Although author-suggested reviewers, as a group, are more likely than editor-suggested reviewers to recommend acceptance, their recommendations are not consistently more positive than those of other reviewers considering the same manuscript.
Author-Excluded Reviewers Give More Negative Recommendations than Other Reviewers Considering the Same Manuscript
We next compared recommendations given by author-excluded reviewers to those from editor-suggested reviewers on all manuscripts to which author-excluded reviewers were assigned in the period April 2007 through September 2009 (Figure 3A). Although the median recommendation by author-excluded reviewers (category 2; reject unless substantially revised), as a group, was lower than the median recommendation by editor-suggested reviewers (category 3, accept if significantly revised), the distributions of recommendations did not differ significantly.
Author-excluded reviewers give more negative recommendations than other reviewers considering the same manuscript. (A) Distributions of recommendations by all editor-suggested and author-excluded reviewers of manuscripts in the April 2007 through September 2009 sample; P = 0.166, χ2 test. (B) Distribution of differences between author-excluded reviewers' recommendation and the median recommendation by other reviewers considering each manuscript in the same sample (n = 41); P = 0.029, Wilcoxon signed ranks.
We compared recommendations by reviewers considering the same manuscript to control for differences among manuscripts (Figure 3B). Although the median difference, as a group, between author-excluded reviewer recommendations and those of others reviewing the same manuscript was zero, more of these differences (42.5%) were negative (author-excluded recommendations were more negative than the others) than were positive (25%). Furthermore, the difference in recommendation between author-excluded and other reviewers considering the same manuscript was significant (P = 0.029). We conclude that, in general, author-excluded reviewers are no less likely to recommend acceptance than editor-suggested reviewers, but tend to give more negative recommendations than other reviewers considering the same manuscript.
Author-Suggested and Author-Excluded Reviewer Recommendations Do Not Reflect a General Tendency of Each Reviewer To Give More Positive or Negative Recommendations than Other Reviewers
Author-suggested or excluded reviewers might make different recommendations than other reviewers because they generally view all manuscripts more positively or more negatively than others. To test this possibility, we compared the difference between each reviewer's recommendation and that of others considering the same manuscript for those manuscripts in which authors suggested or excluded them as reviewers to the median difference from other reviewers on all manuscripts each had reviewed since April 2007 (Figure 4). The distribution of the differences from other reviewer recommendations for manuscripts for which authors suggested these reviewers differed significantly from the distribution of median differences for all manuscripts the same reviewers had reviewed (P = 0.05). These reviewers were more likely to give more positive recommendations than others considering the same manuscript when authors suggested them as reviewers.
Author-suggested and author-excluded reviewers' recommendations do not reflect a general tendency of each reviewer to give more positive or negative recommendations than other reviewers. Distributions of two sets of differences between author-suggested (or author-excluded) reviewer recommendations and those of other reviewers considering the same manuscript: the specific instance during the studied period when the reviewer was (A) suggested or (B) excluded, and the median of such differences for all recommendations those reviewers made since April 2007. Significance of differences between distributions were determined by χ2 test; P = 0.05 for author-suggested and P < 0.0001 for author-excluded reviewers.
Similarly, the distribution of differences from other reviewer recommendations for manuscripts for which authors excluded these reviewers differed significantly from the distribution of median differences for all manuscripts the same reviewers had reviewed (P < 0.0001). These reviewers were more likely to give more negative recommendations than others considering the same manuscript when authors suggested them as reviewers. These results imply the reason author-suggested reviewers and author-excluded reviewers give different recommendations than editor-suggested reviewers is more likely related to the manuscript for which they are asked to review.
Author-Suggested and Author-Excluded Reviewer Recommendations Do Not Influence Editor Decisions
To determine whether receiving a recommendation from an author-suggested or author-excluded reviewer affected the final decisions of the editor, we compared editorial decisions on manuscripts reviewed by author-suggested or author-excluded reviewers to those on manuscripts assigned but not reviewed by the same type of reviewer (Figure 5). For both types of reviewers, the distributions of editor decisions did not differ significantly depending on whether manuscripts were reviewed by author-suggested or author-excluded reviewers. These findings suggest the differences between recommendations by these types of reviewers and those of editor-suggested reviewers do not affect the final decision.
Author-suggested and author-excluded reviewer recommendations do not influence editor decisions. Distribution of editor decisions on (A) manuscripts in the October 2008 through March 2009 sample reviewed by author-suggested reviewers of the manuscripts for which authors made suggestions and those reviewed only by editor-suggested and author-excluded reviewers (P = 0.365); (B) manuscripts in the April 2007 through September 2009 sample reviewed by author-excluded reviewers versus those reviewed only by editor-suggested and author-suggested reviewers (P = 0.164); χ2 test.
CONCLUSIONS AND COMMENT
Our analysis finds that both author-suggested and author-excluded reviewer recommendations at JASN sometimes differ from those of editor-suggested reviewers. Author-suggested reviewers, as a group, are more likely to recommend publication than editor-suggested reviewers, as at other journals,1–6 although this difference largely disappears when comparisons are made within each manuscript. Author-excluded reviewers give more negative recommendations than others considering the same manuscript. To our knowledge, the observation that author-excluded reviewers assess manuscripts more critically than others has no comparison in the literature. That this information was heretofore missing likely results from the low rate at which authors exclude reviewers and editors assign them (Figure 1B). Nonetheless, this finding is not surprising because we would expect authors to exclude reviewers they know to be overly critical of their work. These differences, however, do not reflect general tendencies by author-suggested or excluded reviewers to make more or less positive recommendations than other reviewers. Most importantly, these differences do not seem to affect the likelihood that an editor will decide to publish a manuscript. It is our culture to use reviewer recommendations as advisory to an editorial decision, though they are critical to the process of manuscript improvement.
Although we find differences in recommendations between groups of reviewers, the distributions of the differences within individual manuscripts (Figures 2B and 3B) reveal that author-suggested reviewers do not always give more positive recommendations than others, nor do author-excluded reviewers always give more negative recommendations. Although authors often correctly predict how particular reviewers will assess their manuscript, reviewers sometimes behave differently than authors expect. However, author-excluded reviewers' recommendations differ significantly from others considering the same manuscript, whereas those of author-suggested reviewers do not. This nuance suggests that authors are better able to predict which reviewers will be overly critical than which reviewers will be overly supportive. Alternatively, this difference might indicate that authors consider different criteria when suggesting reviewers than when excluding reviewers. Different criteria make sense given that a greater proportion of authors suggest reviewers than exclude them (Figure 1).
Our conclusions are somewhat limited by the retrospective approach taken here. Because we did not enroll participants, we could not influence reviewer assignment to match manuscript quality or the number of each type of reviewer assigned between groups. However, our triage process (which eliminates approximately 65% of submitted manuscripts) has the effect of normalizing for general quality, as approximately 44% of manuscripts surviving triage will ultimately be published (approximately 15% of submitted manuscripts), and the approach we used to evaluate the current data set has the advantage of avoiding a Hawthorne effect, in which study participants behave differently because they are aware they are being watched.2 Given the large number of papers evaluated in this self-study, the behavior of editors and reviewers regarding the manuscripts we studied here is likely a synthetic representation of our peer review process.
The differences we observed between recommendations by editor-suggested or author-suggested and excluded reviewers may result from a variety of factors. Because these reviewers do not generally tend to give more positive or more negative recommendations than others (Figure 4), observed differences likely result from something specific to the manuscript for which they were suggested or excluded.
Authors may consider their own self-interest when suggesting and excluding reviewers, basing these choices on their prediction of who would be most or least likely to recommend publication. Such predictions could be based on relationships between authors and potential reviewers or relationships between their areas of research. Suggestions and exclusions based on personal or professional relationships seem unlikely to contribute much to our findings, as editors are unlikely to assign reviewers they know to be associated with an author, and ethical conduct would require invited reviewers to decline if their review might not be objective.7 Therefore, author-suggested or -excluded reviewers most often assigned to manuscripts would likely be those whose selection reflects their knowledge of the research done by these potential reviewers.
Such suggestions might not necessarily be made with the intent to increase the chances that reviewers will recommend publication; authors may simply suggest those whose research is consistent with their own because they are well equipped to evaluate the manuscript. Nonetheless, reviewers suggested for this reason may make more positive recommendations than reviewers editors select because they view research that agrees with their own more positively than would researchers whose work does not support the manuscript's conclusions.
Such a tendency toward confirmation bias among journal reviewers has been demonstrated in other studies.8,9 One such study found that psychologists known to support a particular behavioral therapy tested in different versions of a fictitious manuscript were more likely to recommend acceptance of the one with positive results than the one that suggested the therapy did not work.8 The other study found that neurologists whose research agreed with the findings of another fictitious manuscript were more likely to recommend acceptance than those whose research disagreed.9 Such confirmation bias seems likely among author-suggested reviewers, but it could also affect recommendations from editor-suggested reviewers. Future prospective studies could compare the influence of confirmation bias among author-suggested and editor-suggested reviewers by asking how well their research agrees with that in each manuscript they are assigned to review.
We did not see an effect of recommendations by author-suggested or author-excluded reviewers on editorial decisions, which suggests that JASN's practice of encouraging editors to reach an independent decision protects against this type of reviewer bias. However, at journals whose editors rely heavily on a composite score of reviewer recommendations to reach decisions, the proportion of author-suggested and author-excluded reviewers that editors assign might affect publication decisions.
The opportunity for authors to suggest or exclude reviewers is allowed by many journals in the spirit of collegial deference to a stressful process. Our findings here support leaving this option in place at JASN.
DISCLOSURES
None.
Acknowledgments
We thank Dr. Mary Dietrich, of the Departments of Nursing, Psychiatry, and Biostatistics, for assistance with analyses. This work was originally presented at the Annual Meeting of the Council of Science Editors in May 2010.
Footnotes
Published online ahead of print. Publication date available at www.jasn.org.
- Copyright © 2011 by the American Society of Nephrology
In this issue
