Journal Peer Review University: A Seminar for Budding Reviewers

By: Michael Ernst, MD; Christopher Jaeger, MD; Caleb Nelson, MD, MPH; Christina Ching, MD | Posted on: 01 Sep 2022

The peer review process is a cornerstone of scientific publication and key aspect of many academic careers. This process is used to improve the quality of scientific publications through critical review and aid editors in decision making. However, there is a concern that this process is overly reliant on the volunteer work of a few dedicated reviewers. Across scientific fields there has been a widespread need to increase the number of quality reviewers.1 Scientific journals have tried many initiatives to increase their pool of reviewers, including sending reviews to authors who have previously submitted to a journal, searching cited authors on a paper, accepting suggested reviewers from submitted authors, and recruiting young faculty members and trainees.2 This is in line with recommendations from the World Academy of Young Scientists “that junior scientists should be involved in peer review of scientific literature and that it is a critical activity in their educational process.”3

Journals will not just benefit from the increased quantity of available reviewers by engaging trainees and young faculty members, but will also increase the diversity of their reviewers by recruiting from this pool of candidates. This is especially true in urology, where trainees represent a more diverse population than that of currently practicing urologists. In 2021 women made up 34% of all matched applicants in urology, 16% of fulltime faculty, and 10.3% of practicing urologists.4,5 Similar trends are seen for racial/ethnic minorities in urology where, based on the most recent AUA census data, 2.1% of practicing urologists were Black and 3.8% Hispanic, while for residents 3.1% were Black and 5.7% Hispanic.4 Intentional outreach, engagement, and support of young faculty and trainees to become reviewers can help to increase the diversity of voices contributing to the publication of scientific research.

While not robust, there has been some scientific literature describing the inclusion of trainees and young faculty in peer review. Generally, these have consisted of a mentorship or training program. American Journal of Neuroradiology (AJNR) developed a program to support 36 trainees as reviewers.2 These trainees were initially paired with 2 to 3 more senior reviewers. At first, they were sent simpler articles, such as case reports, but then were gradually given more complex and technically specific articles to review. After 2 years, the AJNR found young reviewers completed their tasks more quickly than senior reviewers with no significant discordance in disposition decisions between senior and junior reviewers, and that 25/36 trainees submitted articles to AJNR after becoming reviewers with a significantly higher acceptance rate, suggesting possible improved scientific writing and study design. Perhaps most encouragingly, all eventually migrated into their general pool of senior reviewers.

Several scientific publishers have guides available to assist young reviewers. These include written feedback from editors to reviewers,6 face-to-face or self-taught training,7 checklists,8 and structured mentorship programs.9 None of these, however, has demonstrated an improvement in the quality of reviews. In a study looking at the relationship of previous training and experience of journal peer reviewers to subsequent review quality, they found that academic rank, formal training in critical appraisal and statistics, and status as a principal investigator failed to predict performance of high-quality reviews. The only significant factors in predicting high-quality reviews were working at a university operative hospital vs other teaching environment, and relative youth (<10 years from finishing training).10 It is possible that these resources are most useful to increase recruitment of new reviewers and introduce them to the process of performing scientific review, as opposed to increasing quality of review. Ultimately, the ability to write a quality review varies by individual more than by training.

“Journals will not just benefit from the increased quantity of available reviewers by engaging trainees and young faculty members, but will also increase the diversity of their reviewers by recruiting from this pool of candidates.”

In light of these findings and in an attempt to increase peer review participation in urology, The Journal of Urology® and Journal of Pediatric Urology are joining to sponsor their first reviewer mentorship seminar at the upcoming Societies for Pediatric Urology annual meeting in October 2022 in Las Vegas, Nevada. In informal conversations with current urology trainees, it is clear they are interested but desire tools to participate in the review process. This ranges from “know how” of being involved to demystifying how to actually perform a review. Certainly comfort and feeling of preparedness would impact interest and willingness to participate in peer review and contribute to recruitment of junior reviewers.

This seminar, titled “Journal Peer Review University: A Seminar for Budding Reviewers,” will feature a panel discussion of senior reviewers and editorial board members from both journals on the review process, followed by smaller breakout group sessions composed of a senior reviewer mentor to a few trainees for a practice review. An article for review will be preassigned and the small groups will have the opportunity to go through the review together, with direct feedback from the mentor. Mentors will be identified prior to the session, and trainees will sign up in advance. Formal announcements and recruitment for participation of trainees will be announced closer to the meeting. It is the hope of The Journal of Urology® and Journal of Pediatric Urology to increase accessibility of the peer review process to those who might not be familiar with the process or who currently feel uncomfortable with participation, and thus increase the reviewer pool pipeline. In doing so, the goal is to better equip those being trained in many capacities as a urologist to be impactful participants in the peer review process. Lastly, the journals do plan to assess the response to such a seminar, with the hopes of implementing future such seminars across other subspecialties in urology, but with the pediatric team leading the way.

Please stay tuned for ways to participate. If you have any questions, however, please reach out to christina. ching@nationwidechildrens. org and caleb.nelson@childrens. harvard.edu.

  1. Kovanis M, Porcher R, Ravaud P, et al. The global burden of journal peer review in the biomedical literature: strong imbalance in the collective enterprise. Plos One. 2016;11(11):e0166387.
  2. Castillo M. Trainees in peer review: our experience. Am J Neuroradiol. 2014;35(2):211–213.
  3. Mainguy G, Motamedi MR, Mietchen D. Peer review—the newcomers’ perspective. PLOS Biol. 2005;3(9):e326.
  4. American Urological Association: 2020: The state of the urology workforce and practice in the United States. Available from: https://www. auanet.org/research/research-resources/aua-census/census-results.
  5. Findlay BL, Manka MG, Bole R, et al. Defining the current landscape of women in urology: an analysis of female applicants, residents, and faculty at AUA-accredited residency programs. Urology 2021;148:59–63.
  6. Callaham ML, Knopp RK, Gallagher EJ. Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA 2002; 287(21):2781.
  7. Schroter S, Black N, Evans S, et al. Effects of training on quality of peer review: randomized controlled trial. BMJ 2004;328(7441):673.
  8. Bruce R, Chauvin A, Trinquart L, et al. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med. 2016;14(1):85.
  9. Houry D, Green S, Callaham M. Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Med Educ. 2012;12(1):83.
  10. Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLOS Med. 2007;4(1):e40.
Top 300x250:
Bottom 300x250: