Viewpoints & Discussion:
Doing Peer Review: Reflections From an International Group of Postdoctoral Fellows
There is very little written regarding developing the skills of doing peer reviews. In this piece we use our own experience as postdoctoral fellows to offer our reflections on how to get the most out of doing peer reviews as a trainee researcher. We touch upon the variety and complexity of peer reviews, the debates concerning the nature and validity of peer reviews, the issue of conflict of interest, the menace of predatory journals, but also the potential gain from doing peer reviews. In sharing our reflections, we hope that future graduate students and postdoctoral fellows may be better prepared to do peer reviews and benefit from the experience.
Index Terms: research training; peer review; researcher development; professional development; research skills
Suggested Citation: Merry, L., Jarvis, K., Kupoluyi, J., Jomama Lual, J. O. (2017). Doing peer review: Reflections from an international group of postdoctoral fellows. Journal of Research Practice, 13(2), Article V2. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/579/475
Peer review is the process by which manuscripts are accepted by journals, abstracts are selected for conferences, and grants are approved for funding. It is critical to the research process as it determines what knowledge is generated and disseminated (Lee, Sugimoto, Zhang, & Cronin, 2013; Mayden, 2012). All academics have a responsibility to participate in peer review although many educational institutions do not provide any formal training to graduate students or postdoctoral fellows about how to be an effective peer reviewer (Provenzale & Stanley, 2006). We are four postdoctoral fellows from Canada, Nigeria, and South Sudan. Our backgrounds are diverse, including nursing, medicine, demography and social statistics, and epidemiology, and we are all working in global health. In this reflection we share our insight and provide suggestions on how novice reviewers, like ourselves, may get the most out of the experience of doing peer review. We emphasize in particular, the role of the mentor/supervisor in enhancing this experience.
The process of peer review usually begins with an invitation from the journal, funding agency, or conference organizers. It is acceptable to say “No” but it is always best to respond promptly so that the journal/granting agency/conference organizers can begin the search for someone else. This ensures that the review process is not extended and acceptance or rejection of the work is not delayed.
Generally, the role of a reviewer is to provide a constructive critique, including the strengths, limitations, and areas for improvement (Mayden, 2012; Provenzale & Stanley, 2006). The critique is typically based on the journal/granting agency/conference’s aim and scope, and the scientific soundness of the work. Feedback is usually provided in a written format, although for grants there may be a forum where reviewers have an opportunity to discuss their reviews verbally. Reviewers are given a form with general and broad items to respond to, such as the significance, ethical issues, and methodological quality of the research, leaving considerable discretion to the reviewer on how to complete the review. The extent of a reviewer’s comments can vary—some give detailed feedback, including comments on suggested edits for manuscripts and improvement of grant proposals, while others provide only brief remarks. Important to note, is that concerns or other feedback that are not to be shared with the authors/researchers, may be sent separately to the editors/funders/conference organizers, usually in a designated section. For all types of work reviewed (abstract, manuscript, or grant proposal), the reviewer is asked to advise by score, rank, and/or direct decision, whether the work should be accepted or rejected.
Completing a proper, thorough review can take several hours (Black, Van Rooyen, Godlee, Smith, & Evans, 1998); novice reviewers should anticipate that it could take much longer. Our experience is that it can take a full day to review, write up recommendations, and complete the required journal/granting agency/conference documentation (usually, online forms). For journal articles, a reviewer may also be asked to review a revised submission if a revision was recommended in the previous round. For grants the expectation may be to sit on a committee and to review multiple applications (up to 10) and attend a face-to-face meeting, which may be one to two days in length. Additional time beyond the initial review itself is therefore needed in these circumstances.
To have a sense of the length and depth of a review (for manuscripts), and process time, open-access journals are a good resource as there are many that publish reviewers’ comments and the timeline from submission to publication (e.g., several journals published by BioMed Central). Some journals may provide feedback on peer reviews or offer training and/or may share the reviews of others who reviewed the same manuscript. Similarly, if you sit on a grant review committee, you may have the opportunity to hear and read others’ reviews. Some funding agencies also offer observer programs to help researchers gain a better understanding of the peer-review process followed for grant proposals (Canadian Institutes of Health Research, 2017). As an observer, a trainee or young researcher may review applications and sit-in on the discussions of the review panel.
Asking supervisors or mentors for guidance is also a good method of learning to do peer reviews. This may include asking supervisors or mentors to show reviews they have done and/or having them act as co-reviewers. Reviewing with a peer, who is also in the learning process, is another effective way to learn. Any reviews that are done with a mentor or peer, however, must be reported to the journal/funding agency/conference organizers. Journals often have a place to provide this information or they may be informed in the “Comments to the Editor” section.
Trainees should be aware that there is debate about the role of reviewers, on what they should review exactly (e.g., science, ethics, writing-style, originality, and importance of the contribution), how they should comment (tone, depth), and whether or not they should even be in a position to accept or reject the work (Benos et al., 2007; Hojat, Gonnella, & Caelleigh, 2003). The utility and validity of the peer-review process itself is highly debated. There is a lot of subjectivity to the peer-review process and therefore one needs to think carefully about the role of peer reviewers as “gate-keepers” of knowledge (Hojat, Gonnella, & Caelleigh, 2003). If the science is sound and there are no ethical concerns, it is not always clear whether the work should be accepted or rejected. Many biases exist in the peer-review process, in particular it advantages research that is in English and that originates from high-income countries (Benos et al., 2007; Hojat, Gonnella, & Caelleigh, 2003). Undeniably, funding for grants is limited, so priorities need to be set which results in many studies not being funded. Similarly, conferences are restricted by time and space for presentations. In the case of manuscripts, however, there is a debate on whether there should be limitations on what is published especially with the advent of open-access journals and repositories (Ralph, 2016).
Reviewers are invited based on their expertise in specific content areas as well as their methodological knowledge. Before agreeing to review, one should ask whether one has sufficient knowledge to provide constructive feedback on the work. It is possible however to review even if one is not an expert in both the content area and the methodology, however, the review comments should be limited to what the reviewer thinks they are able to comment on.
Reviewers must also consider if they have any conflict of interest that might hamper the objectivity of their critique. Reviewers have to decline to review manuscripts when the circumstances of the review prevent balanced judgment, or benefit the reviewer personally or financially. This needs to be assessed carefully. For example, being situated within the same academic institution as the author of the item to be reviewed may be considered as a conflict of interest in some instances.
Trainee researchers may want to prioritize reviewing work that is of interest to them. For journal manuscripts, the prestige or impact factor of the journal may be considered. This has its limitations however, because articles in some research fields and articles using some methodological approaches are published in journals with lower impact factors (Amin & Mabe, 2000). Journals from low- and middle-income countries and non-English journals may also not have impact factors to report. It is ideal to get a range of experiences—different journals/funding agencies/conferences as well as types of review, grants/manuscripts/abstracts. If one needs to choose, reviewing manuscripts and grants may be more beneficial than reviewing conference abstracts, in terms of learning and gaining experience in how to do peer reviews.
One may also select to review for journals where one has published, or for granting agencies which have funded one’s research in the past. It is not uncommon to be asked by a journal to review shortly after one has published in it and, likewise, not uncommon to be asked by a funding agency to review shortly after one’s research has been funded. Similarly, one may choose to review for conferences organized by scholarly/professional associations where one is a member.
Lastly, it is important to note that some journals publish peer reviews and peer commentaries together with the manuscript, and if one does not feel comfortable having their comments made public, they should make it known to the journals in question. One also needs to be cognizant of “predatory journals.” The number of questionable journals is increasing worldwide. One should not accept to review for any of these journals as doing so may undermine one’s credibility and reputation (Kearney, 2015).
There is no set number regarding how many peer reviews a graduate student or postdoctoral fellow should do during their research training. There are a number of factors to consider. For example, it will depend on the type and length of the graduate or training program. More rigid and shorter graduate/training programs provide less opportunity for doing extracurricular research activities such as peer reviewing. It may also depend on the field and discipline. There may be greater expectations to participate in this type of activity in certain research areas and disciplines compared to others. The position one plans to apply for following one’s research training should be kept in mind as well. Peer review is a required skill in academia but may be less obligatory for an administrative position, for example. The skill however is valuable and transferable across most positions.
A mentor or supervisor is in a good position to advise on the number of reviews a trainee researcher should do. It will be based on workload and other commitments, level of research experience, writing abilities, and availability of mentoring capacity (if considerable support is needed). If one has more experience, one may accept to do more reviews. However, if one is being asked frequently by journals or funding agencies, it may not be wise to say “Yes” to every opportunity. While it is advantageous to hone one’s peer-review skills, it should not be at the expense of engaging in other activities that may be more advantageous for one’s research career (e.g., applying for a grant). The number of reviews each of us did in the last year ranged from 0 to 10.
Where to begin if one has never been asked to do a peer review? There are different ways that a graduate student or postdoctoral fellow can seek out peer-review opportunities. The first is to ask a supervisor. It is likely that a supervisor has many requests to review manuscripts or research proposals and could include their student/fellow in one of these activities. It is best to let the supervisor know of one’s interest when they first meet to plan their research training and objectives so that the supervisor may then identify reviews that are most suitable based on the student/fellow’s expertise and select those that fit with other planned activities.
A second way to be asked to do reviews is through publishing and receiving grants. When one submits an article to a journal or receives funding they may be asked to submit profile data that describe one’s areas of interest and expertise. The journal/funding agency will ask if they are interested in being a reviewer and if one agrees, their name is then added to the reviewer pool. Once one publishes articles and presents one’s work at conferences, one will also begin to become known in the field and one’s name may be put forward by other researchers to review their manuscripts or proposals.
A third way to find peer-review opportunities is through networking. If a supervisor or other colleagues know of the student/fellow’s interest in doing reviews, they may then suggest the student/fellow’s name to journals or granting agencies. Participating as a reviewer of abstracts for conferences may be a first step in doing peer reviews to learn the skill and put one’s name out there for more involved reviews (for journals and granting agencies). It is also a good idea to join listservs (funding agencies, research groups, and scholarly/professional associations) that may put calls out for reviewers for grant competitions, journal manuscripts, or conference abstracts.
A fourth possibility for being a reviewer is to contact journals and granting agencies directly and offering one’s services.
Reviewing papers, grants and abstracts is beneficial to a successful career in academia. Peer review provides an opportunity for the research trainee to update their knowledge and expose them to what other scholars are doing in their field of study. It is helpful for informing their work and to know how one can build on what others are doing. It might be a way to identify opportunities for future collaborations. It is also a means to improve one’s writing skills, to see different types of articles/grants and writing styles, and to learn more about publication and funding processes. Giving constructive feedback is an important and required skill for academics; it is needed for providing feedback to peers and also in the context of teaching/mentoring students and trainees. Doing peer reviews is also a welcome addition to one’s curriculum vitae, especially if one is planning on applying for an academic position.
Graduate and postgraduate training is a time to develop skills for a career in academia or for leadership positions. Peer review is an essential skill to develop because it fosters analytical and critical thinking, effective written and oral communication, and interpersonal skills. Supervisors and mentors can provide opportunities and assist trainees in a number of ways to develop peer-review skills. In sharing our reflections, we hope that future graduate students and postdoctoral fellows may be better prepared to do peer reviews and benefit from the experience.
Amin, M., & Mabe, M. (2000, October). Impact factors: Use and abuse. Perspectives in Publishing, 1, 1-6. Reprinted March 2004 in International Journal of Environmental Science and Technology, 1(1), 1-6.
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., . . . Polter, A. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145-152.
Black, N., Van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? Journal of the American Medical Association, 280(3), 231-233. Retrieved December 6, 2017, from https://jamanetwork.com/journals/jama/fullarticle/187762
Canadian Institutes of Health Research. (2017, August 22). Opportunity for early career investigators: Call for expressions of interest. Retrieved November 10, 2017, from http://www.cihr-irsc.gc.ca/e/50206.html
Hojat, M., Gonnella, J. S., & Caelleigh, A. S. (2003). Impartial judgment by the ‘gatekeepers’ of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1), 75-96.
Kearney, M. H. (2015). Predatory publishing: What authors need to know. Research in Nursing & Health, 38(1), 1-3.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the Association for Information Science and Technology, 64(1), 2-17.
Mayden, K. D. (2012). Peer review: Publication’s gold standard. Journal of the Advanced Practitioner in Oncology, 3(2), 117.
Provenzale, J. M., & Stanley, R. J. (2006). A systematic guide to reviewing a manuscript. Journal of Nuclear Medicine Technology, 34(2), 92-99.
Ralph, P. (2016). Practical suggestions for improving scholarly peer review quality and reducing cycle times. Communications of the Association for Information Systems, 38, Article 13.
Received 6 December 2017 | Accepted 7 December 2017 | Published 8 December 2017
Copyright © 2017 Journal of Research Practice and the authors