Recommendation 5. Foster Equity in Peer Review

Ensuring that a publication provides consistently fair and equitable evaluation of submitted papers through the peer review process is no simple exercise. Authors, editors, and reviewers come from different backgrounds (geography, education, field, race, gender, language, etc.) that inform their expectations and that consciously and unconsciously influence their perceptions of others and others’ perceptions of them.

This section provides several suggestions to help your publication create a more welcoming environment for all stakeholders: ongoing education for editors, editorial staff, and reviewers on bias awareness; accessible information and training on language inclusivity; alternative peer review models; and bias identification, assessment, and remediation in the peer review process.

Education and Training

Provide bias awareness or diversity training for editors and reviewers. We all have biases. Our challenge is to recognize and account for them. Implicit, or unconscious, bias is referred to as such because it happens automatically without intentional thought (Greenwald & Lai, 2020). Raising people’s awareness of their biases is a critical step to understanding when these biases may be influencing decisions. Recent work in this area suggests that providing counterexamples to these biases can also help change stereotypes, serving as a tool to disrupt biased judgments (see, e.g., Finnegan et al., 2015).

Bias awareness training can range from learning how to recognize and address unconscious biases to cultivating cultural awareness and humility. While live or in-person training programs can be valuable, they may also be time-consuming, difficult to organize, and expensive to run. Below is a selection of bias awareness training resources available online to help journal editors and publishers develop a training program that makes sense for their community.

Written toolkits and standards serve as key reference guides for editors, helping to ensure that manuscripts meet standards for equity, diversity, and inclusion with respect to methodology, language, conclusions, and other aspects. Such guidelines include:

• APA Journals’ Equity, Diversity, and Inclusion Toolkit for Journal Editors

• C4DISC’s Antiracism Toolkit for Organizations

On-demand interactive training or webinars can also be useful. These training programs provide a comprehensive overview of bias and its impact and serve as a foundation for other types of training. Examples include:

• Racial Equity Tools

• ReadySet Ally Workshop

• RacialEquity Institution

To learn more about other diversity training options for editors, see the following C4DISC webinar, which discusses ways to develop anti-bias training for editors.

Educate editors on resources that promote diverse research methods and transparent reporting standards. 

There are several reasons why editors need to understand the best practices for a diverse range of research methods and transparent reporting standards.

Researchers from diverse backgrounds and with different perspectives may use different research methods, which can lead to a more comprehensive understanding of a research question. For example, it has been shown that underrepresented researchers tend to produce more novel and innovative research (Johnson et al., 2016) while indigenous researchers often approach a problem from a holistic point of view compared with traditional Western methods (Hofstra et al., 2015). Embracing all of these diverse perspectives can enrich the research that is being published in your journal.

Promoting diverse research methods, along with transparent reporting standards, can help reduce bias in research. Additionally, researchers may have implicit biases that can influence their research design, data collection, and data interpretation. Encouraging a diverse range of research methods can help mitigate these biases by bringing in perspectives from underrepresented groups, while transparent reporting standards can help identify and address biases.

For more information about diverse research methods and transparent reporting standards, see APA’s Journal Article Reporting Standards for Quantitative, Qualitative, and Mixed Methods research, as well as the recent additional standards for reporting on Race, Ethnicity, and Culture. The EQUATOR network provides other reporting standards that may be suitable for your discipline.

Promote and encourage inclusive citations with authors and reviewers.

Not only are individuals from persistently marginalized groups often less likely to be published, promoted, or featured in leadership positions in journals, but they are often significantly less cited, pointing to issues of citational justice (Lloreda, 2022):

As an editor or publisher, you can contribute to citational justice by highlighting to your board and research community the importance of inclusive citations within the advancement of scholarship and the impact they can have on a researcher’s career. Encourage your authors and reviewers to actively keep abreast of new research and scholarship conducted by authors in various labs, consortia, and countries and to include and look for sources from diverse authors with whom your colleagues may have previously been less familiar and less inclined to cite. Practical tactics include the following:

• A citational justice statement in an editorial or journal diversity statement can encourage authors and reviewers to use tools that assess gender balance in citations in reference to the current literature. For example, Google citation transparency software adds inferred and probabilistic gender information to PubMed and Google Scholar searches based on author names. Other open-source tools probabilistically infer gender of first/last authors in bibliography entries and output a customizable diversity statement authors can submit with their manuscript (see attributes of the Gender Balance Assessment Tool; Sumner, 2018). While not without limitations (see limitations with name inference tools as it relates to demographic data collection), we hope that additional tools will enable similar checking for balance in citations to published work by authors of diverse race and ethnicity. (See example tools in the Resources section under Name Inference Tools.)

• Publishers can also support and facilitate the inclusion of an article-level citation diversity statement as part of the manuscript submission, to be published alongside the article. This allows editors and reviewers to take heed of the citation balance and potentially suggest relevant literature that may be missing or may be skewing a manuscript’s reference list. Examples include:

  • American Chemical Society (ACS): diversity and inclusion statement

  • Biomedical Engineering Society (BMES): citation diversity statement

  • Cell Press: inclusion and diversity statement

  • Nature Reviews Bioengineering: citation diversity statement

• Highlighting previously underrecognized work by people of color and authors from other persistently marginalized groups can ameliorate disparities in citation practices. As an example, American Psychologist recently published a special issue (guest edited by Fanita Tyrell et al., 2023) on foundational yet underrecognized contributions of Black scholars in psychology. These approaches can be extended to other persistently marginalized groups as important ways that publishers and editors can influence citational justice in their fields. Additionally, groups such as Cite Black Women aim to not only help draw attention to the research and scholarship of Black women but also to center their stories and ideas in knowledge production and education.

Support Inclusive Resources

Using inclusive language will help make all your authors, reviewers, and readers feel more welcomed and respected while avoiding unintentionally alienating any groups or individuals. This means using inclusive language in your own communications and encouraging your authors, reviewers, and editors to do so, as well as providing them with resources that support the publication journey for multilingual authors, such as translation services.

Promote style guides on avoiding bias in language. Several organizations have developed excellent inclusive guidelines to help guide researchers across a wide array of disciplines, such as C4DISC’s Guidelines on Inclusive Language and Images in Scholarly Communication. A recent article from Learned Publishing (Ashwell et al., 2023) provides an overview of three inclusive language guides across several STEM (science, technology, engineering, and mathematics) disciplines, and the American Psychological Association has developed Inclusive Language Guidelines to support the use of culturally sensitive terms.

Provide resources for non-native language authors and reviewers. As the research community is global, giving researchers access to resources in a variety of languages can help reduce barriers to publication.

ACS’s free Reviewer Lab course is available in both English and Chinese to support their global community of reviewers, and there is also a module within the ACS Author Lab for multilingual authors with tips to improve their writing.

When possible, translating instructions for authors makes them more accessible to everyone publishing in your journal. If translation is not possible, ensure your instructions follow the principles of plain language.

Ask reviewers to be on the lookout for noninclusive language. While reviewing papers, reviewers can help to identify non-inclusive language in the manuscript. They can also provide suggestions to authors on how to make the language used more inclusive, for example, in their research methods and descriptions of their sample populations.

Peer Review Models and Workflows

Explore alternative peer review models. The peer review model and workflow implemented by journals are often determined by the expectations within individual fields of study. However, journals and publishers may want to consider pursuing alternative peer review models, as they seek to address concerns about equity and bias in publishing.

The two alternative models that are most commonly cited in this context involve either more anonymity (double- and triple-anonymous review) or less—in some cases, none at all (open/transparent review).

• In double-anonymous peer review, reviewer and author names are not revealed to each other during any part of the process.

• In a triple-anonymous workflow, no one, including the editors, has access to the authors’ identities.

• In the various iterations of open or transparent review across publishing, reviewers and authors may be identified by name, the reviewers may collaborate with the editor before a decision is made, and the peer review history, including the review reports (anonymous or not), may be published alongside the final article as a PDF artifact.

Explore the Standard Terminology for Peer Review published by the National Information Standards Organization (NISO) for more information.

These alternative models aim to lessen bias in the review process, however, some have expressed caution (Haffar et al., 2019). We hope that by experimenting with various models in different publishing settings, publishers and editors will gain greater insight into which models are most equitable and under what circumstances.

Encourage reviewers to disclose whether anyone has assisted them with their review and offer that colleague credit. As noted in Recommendation 4, co-reviewing and mentorship programs for reviewers can provide early career researchers with an opportunity to gain experience with the peer review process. However, it is important to set up mechanisms to ensure that senior reviewers who submit their more junior colleagues’ feedback give and are given proper acknowledgment. If your journal permits co-reviewing, COPE recommends disclosing the names of all individuals who assisted in writing a review, so that “they are associated with the manuscript in the journal’s records and can also receive due recognition for their efforts” (COPE Council, 2017).

Consider diverse, multistep editor decisions. Besides exploring the pros and cons of alternative models and ensuring that all reviewers receive credit for their work, you should also consider your journal’s overall manuscript workflow. How many editors with content expertise evaluate a manuscript as it moves toward a final decision? Will decisions be approved by just one editor or more? Manuscripts handled by more than one editor may receive a more robust evaluation, and involving more editors may also minimize bias.

It is helpful to establish clear criteria for what is published within the journal’s scope and to include these criteria on the journal website and in decision letters. It is also important to ensure that you provide sufficient rationale for rejection to your authors. By using different decision letters for rejection types, you can systematically track rejection rationale to improve initial communication with authors and track emerging trends, including identifying potential bias.

Bias Assessment

Develop annual reporting to assess potential bias throughout the peer review process. As noted in Recommendation 2, gathering and ultimately reporting on demographic data of journal contributors is critical to developing a data-informed strategy, gauging areas for improvement with your journal, and keeping all stakeholders accountable. Journals typically report on submission, peer review, and publication trends to their board and/or publications committee on an annual or biannual basis. Submissions and publications are usually broken down by corresponding author country; this reporting can be enhanced further by analyzing the same trends, as well as peer review and manuscript outcomes, in the context of disaggregated, anonymized author, editor, and reviewer demographic data. (See Recommendation 2 for recommended practices for gathering these data).

• Manuscript milestones and outcomes to consider:

  • Submissions: Does the journal do a better job of attracting and recruiting papers from authors of particular geographical location, gender identity, ethnic origin, age, sexual identity, etc.?

  • Decisions, including immediate rejections and acceptances: Does the data show that editor decision-making exhibits potential bias?

    • Ensure that your submission data can be automatically broken out by rejections vs. acceptances before peer review. This may require upstream changes to system configurations for decision type or letter.

  • Peer review process

    • Invitation: Who is being invited to evaluate manuscripts?

    • Reviewer invitation response: How are they responding?

    • Agreed reviewer recommendation: Are there trends in how reviewers evaluate manuscripts by authors of particular geographical location, gender identity, ethnic origin, sexual identity, etc.?

General tips on creating reporting to assess potential bias can be found in the Using Measurement and Metrics section of the Antiracism Toolkit for Organizations (2021).