Identifying Patterns and Motivations of ‘Mega’ Peer-Reviewers

doi:10.21203/rs.3.rs-480702/v1

Download PDF

Research

Identifying Patterns and Motivations of ‘Mega’ Peer-Reviewers

https://doi.org/10.21203/rs.3.rs-480702/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 20 Feb, 2022

Read the published version in Research Integrity and Peer Review →

You are reading this latest preprint version

Background. The demand for peer reviewers is disproportionate to the supply and availability of reviewers. Identifying the factors associated with peer review behaviour can allow for the development of solutions to manage the growing demand for peer reviewers. The objective of this research was to identify factors associated with completing a large number of peer reviews in a given year.

Methods. A case-control study design was used to examine factors associated with individuals completing at least 100 peer reviews (‘mega peer reviewers’) from January 2018 to December 2018 as compared to a control group of peer reviewers completing between 1 and 18 peer reviews over the same time period. Data was provided by Publons, which offers a repository of peer reviewer activities in addition to tracking peer reviewer publications and research metrics. A series of independent sample t-tests and chi-square tests were conducted comparing characteristics (e.g., number of publications, number of citations, word count of peer review) of mega peer reviewers to the control group of reviewers.

Results. A total of 1596 peer reviewers had data provided by Publons. A total of 396 mega peer reviewers and a random sample of 1200 control group reviewers were included. Both groups were comprised of a greater number of males than females (mega peer reviews = 92.4% male, control reviewers = 70.0% male). Mega peer reviewers demonstrated a significantly greater average number of total publications, citations, receipt of Publons awards, and a higher average h index as compared to the control group of reviewers (all p < .001). Alternatively, the control group had a significantly greater average number of words in their peer reviews (mean = 332.48, standard deviation [SD] = 346.30) as compared to mega peer-reviewers (mean = 272.50, SD = 219.99).

Conclusions. There is a sub-set of highly active peer reviewers that complete a large proportion of peer review activities. These individuals demonstrate significantly different characteristics than reviewers completing a more typical number of peer reviews over a one year period. Additional research that considers motivations associated with peer review behaviour should be conducted to help inform peer reviewing activity.

Registration. https://osf.io/vxdhf/?view_only=313fd05399664b94bc7a9042aa225be3

Psychology

Peer review

characteristics of reviewers

Publons

Peer review involves a manuscript undergoing evaluation and scrutiny by experts in the same field of research.¹ The peer review system has been integrated into the scientific community for hundreds of years with the goal of validating academic work and improving the quality of published research.² When peer reviewers are aware and in agreement with the expectations and responsibilities of reviewing an article, and editors incorporate feedback in a timely manner, peer review has the potential to result in valuable feedback for the authors and improve the quality and usability of research findings.³

The sustainability of peer review relies on the availability and expertise of peer reviewers. Obtaining peer reviews that are high quality (i.e., useful for authors) is difficult for many journal editors. An inaugural Global State of Peer Review report, developed by Publons in collaboration with the Web of Science group (both owned by Clarivate Analytics), reported on (1) characteristics of peer reviewers, (2) efficiencies of the peer-review process, (3) quality of peer review, and (4) future considerations for peer review. Importantly, this report found an increasing demand of peer reviewers, disproportionate to the supply.³ Certain characteristics were found to be associated with the completion of peer review activities. For example, variability between regions for peer reviewer activity exists, with individuals from the USA and China contributing the greatest number of peer review.³ Regional variability in the incentives structure has been suggested as one factor that may partially account for these differences. The Publons’ 2018 Global Review Survey included over 11,800 researchers and found overwhelming agreement (84.8% of participants) that greater recognition and formalized incentives for peer review would increase willingness to serve as a peer reviewer and would positively impact the efficiency of the peer-review process.³ Traditional rewards exist, such as journal subscriptions, discounts for open accessing publishing, and acknowledgements through public “thank you” lists. However, these rewards are inconsistently applied among journals and do not meet the preferred rewards and incentives (e.g., waiver of publication fees) and recognition of peer (e.g., incorporated as part of the evaluation criteria for funding applications) being sought by researchers.^3-6

Findings from the Global State of Peer Review report includes a summary of trends found among a large sample of peer reviewers and the current strain on the peer review system. There remain substantial gaps in the understanding of characteristics of individuals that agree to serve as peer reviewers and the quality of peer reviews.³

Publons offers a repository where peer reviewers can document their peer review activities in addition to tracking publications and research metrics.⁷ Anecdotally, we have observed some researchers that are highly active in peer reviewing (i.e., individuals completing at least 100 peer reviews, annually – we refer to these individuals as ‘mega peer reviewers’) on the Publons website. A recent study also highlighted the unequal distribution of peer-reviewing tasks among small groups of researchers.⁸ Understanding the factors that relate to mega peer reviewing behaviour and the quality of these peer reviews can inform the design of strategies and interventions to facilitate system- and individual- level changes. Identifying the factors associated with peer review behaviour can provide a basis for keeping pace with the growing demand for peer reviewers. As such, the objective of this research was to identify the factors associated with individuals who were highly active peer reviewers in a given year as compared to a control group of peer-reviewers. Given the lack of research on mega peer reviewers, this was an exploratory project, and we did not form hypotheses.

The protocol for this study was registered within the Open Science Framework database

(https://osf.io/vxdhf/?view_only=313fd05399664b94bc7a9042aa225be3) before data collection began. This was a case-control study to retrospectively examine factors associated with mega peer reviewers as compared to a control group of peer reviewers. Mega reviewers were defined as individuals that completed peer reviews for 100 or more unique articles from January 2018 to December 2018. All aspects of this study were reported in accordance with the Strengthening (STROBE) reporting guidelines to facilitate the complete and transparent reporting of this work.⁹

Case Control Data

Participants

We gathered information from the Publons database. Publons tracks and publicizes peer-reviewer activities for individuals that create an account and connect their research activities to their profile. Individuals can download their peer-review, author, and editor metrics and this information can also be made public. Using the Publons database, two groups of individuals were of interest for this study, including (1) mega peer reviewers: all individuals that completed peer reviews for 100 or more unique articles from January 2018 to December 2018, inclusive (case group) (i.e., individuals completing approximately two peer reviews every week) and (2) a control group of individuals completing at least one peer review and less than 18 peer reviews over the same time period (i.e., individuals completing up to 1 peer review every 3 weeks). A random sample of controls were selected from Publons database.

Data Collection

A data scientist from Publons exported the following variables into a csv Microsoft excel file: peer reviewer characteristics [i.e., name, publons ID, institution, country of institution, number of publications based on Publons data, publications in Web of Science, publications in 2018, citations in 2018, total number of citations, h-index, presence of a Publons reviewer award (top 1% of reviewers in 22 research areas, top quality reviews based on editor rated evaluations)], review characteristics based on Publons data (i.e., number of unique manuscripts peer-reviewed in 2018, number of unique manuscripts reviewed each month, average number of words per review, average number of words per review at reviewer’s institution). Sex was not available on Publons. As such, sex was estimated by using the Genderize data base, which uses data collected from countries to assess the probability of the sex being associated with a given name (https://genderize.io/). For any sex that could not be estimated with more than 80% certainty, this was marked as missing data.

Sample Size Calculation

The mega peer-reviewers sample size was based on the number of peer-reviewers on the Publons website that met our inclusion criteria (i.e., greater than 100 peer-reviews 2018). For the control group, a sample size calculation based on the total number of reviewers that met the control group requirements (i.e., completing at least one review and less than 18 peer-reviews in 2018) was conducted using the standard deviation of the average word count which was estimated using preliminary data from Publons. A sample size calculation was conducted in R package (pwr) for a two sample t-test comparing mega peer reviewers and the control group. The pooled standard deviation was calculated, and a minimum sample size of 1167 was estimated (see Appendix 1). To determine the number of peer reviewers needed for the control group, a 1:1 random sample was selected and the standard deviation of the average word count of peer review quality was determined.

Data Analysis

Primary data analysis calculated descriptive characteristics of both samples of reviewers. The secondary data analysis involved conducting a logistic regression to compare the mega peer reviewer characteristics to the control group, treating mega-reviewing as a binary outcome. Given the exploratory nature of this study, the association between peer reviewer characteristics (i.e., sex, country of institution, number of publications in Web of Science, publications in 2018, citations in 2018, total number of citations, h-index, presence of Publons reviewer award) and review characteristics (i.e., average number of words per review, average number of words per review at reviewer’s institution) (independent variables) were compared in the two groups of reviewers (mega-reviewers and control group of reviews) within the regression model. Prior to conducting regression analyses, preliminary tests were performed to determine the appropriateness of analyses based on any violations of regression assumptions. Tests of multicollinearity were conducted, and independent variable tolerance values were reviewed. Four variables did not reach the recommended cut-off values for collinearity statistics,¹⁰ suggesting that variables were highly correlated with one another [tolerance values less than 0.1 for publications based on Publons (0.04), publications based on Web of Science (0.04), citations in 2018. (0.06), total citations (0.06). As such, independent samples t-tests for continuous data and chi-square analyses for categorical data were used. Analyses were conducted using SPSS version 27.0, and statistical tests were two sided with a significance value of P < .05

A total of 396 mega reviewers that completed >100 peer reviews during 2018 were included. A random sample of 1200 control group reviewers completing 18 peer reviews over the same time period. For reviewers that could have their sex estimated with at least 80% certainty (n= 1,315), 92.4% of mega peer reviewers were male, as compared to 70.0% of reviewers that were male in the control group. Mega peer reviewers completed reviews for an average of 138.8 (standard deviation [SD] = 82.3) articles as compared to the control group which peer reviewed an average of 4.1 (SD = 3.0) articles within the control group. Mega peer reviewers completed an average of 14.0 (SD = 2.0) peer reviews per month with the highest average number of reviews being completed in August (mean = 16.6, SD = 14.7) and October (mean = 16.6, SD = 15.3). The control group completed an average of 0.4 (SD = 0.1) peer reviews per month with the highest average number of reviews being completed in October (mean = 0.4, SD = 0.7) and November (mean = 0.5, SD = 0.7). Mega peer reviewers (n=396) completed a total of 54,953 peer reviews within 2018 as compared to the control group (n=1200) that completed a total of 4862 peer reviews in 2018. Characteristics of mega peer reviewers and the control group can be found in Table 1.

Independent Samples T-Tests and Chi-Square

A series of independent sample t-tests were conducted comparing mega peer reviewers to the control group of reviewers (see Table 2). Mega peer reviewers had a significantly greater average number of publications themselves (total), publications in 2018, citations (total), citations in 2018, and a significantly higher average h index as compared to the control group of reviewers (all p<.05). The control group had a significantly greater average number of words in their peer reviews and as did the average word counts of peer reviews conducted at the control groups academic institutions as compared to mega peer reviewers (all p<.05) (see Table 2).

The continent of reviewers significantly differed. The majority of mega peer reviewers were from Asia (33.0%), Europe (36.8%), and North America (18.9%). A similar pattern was found among the control group of peer reviewers with 41.1% from Europe, 25.8% from North America, and 21.3% from Asia. The remaining reviewers were from Australia (mega peer reviewers = 4.3%; control peer reviewers = 6.0%), South America (mega peer reviewers = 0.8%; control peer reviewers = 3.1%), and Africa (mega peer reviewers = 6.2%; control peer reviewers = 2.7%). Publons awards were significantly more present among mega peer reviewers with 88.1% of mega peer reviewers having received an award from Publons as compared to less than one percent of the control group reviewers (see table 3).

In the Publons database, 396 individuals peer reviewed at least 100 papers within a 12 month time-frame. This represents a substantial time commitment among mega peer reviewers in completing a task that is often perceived as burdensome.⁸ In addition to active peer reviewing activities, mega peer reviewers demonstrated a significantly greater average number of publications overall and within one year time frame reviewed. Mega peer reviewers also had significantly more citations overall and within the one-year time frame reviewed, greater receipt of Publon awards, and a higher average h index as compared to the control group of reviewers. Alternatively, the control group had a significantly greater average number of words in their peer reviews as compared to mega peer reviewers. Notably, mega peer reviewers were overwhelmingly male. This may align with previous research findings that demonstrate a gender bias among editors whereby editors invite fewer female peer reviewers as compared to males.¹¹ It may also reflect female academics managing multiple responsibilities at work and at home, resulting in little extra bandwidth to peer review articles.¹²

Mega peer reviewer’s altruism and dedication to peer reviewing should be acknowledged. These peer reviewing activities substantially impact research as the total number of articles reviewed in 2018 by mega peer reviewers was over 54,000. These articles were peer reviewed by 396 individuals, and this represents 11 times more peer reviews than the 1200 individuals in the control group completed. When considering the number of peer reviews completed by mega reviewers, it is possible that the level of detail provided to authors is less comprehensive compared to control peer reviewers. The word counts of submitted reviews by mega peer reviewers was significantly less than the word count of the control group of reviewers and the average word count of colleagues at the same institution. On the other hand, it is possible that mega peer reviewers are more direct in their comments and are able to provide a review that is equally as helpful to authors as compared to the control group of reviewers. Assessing the comments qualitatively and asking author feedback about the usefulness of reviewer comments will be important to better understand the impact of completing a substantial number of peer reviewing activities.

Both categories of peer reviewers provide approximately two thirds of a page of peer reviewer feedback per article with mega peer reviewers using fewer words. The average word count of other reviewers from the same institutions as the control reviewers was approximately to one and a half pages of text. Both groups of reviewers in this study provide less than a page of text for a review which may be inadequate for providing constructive feedback for an entire manuscript. Assuming a brief opening paragraph to precis the research paper under peer review (i.e., providing the authors with a measure of face validity about the peer reviewers understanding of the research report), followed by optimal reviewing with the help of a reporting guideline,¹³ along with any specific journal reviewing guidance, it is not clear that all of this information can be conveyed in so few words. Additionally, peer reviews are most helpful to authors when they are evidence based,¹⁴ often necessitating citing making the word length even longer.

Publons Academy modules provide relevant knowledge to all peer reviewers who are trainees, which could influence the quality and completeness of peer reviews. The need to prioritize depth over the number of completed peer reviews may need to be further emphasized in these modules when training reviewers. This emphasis on depth may also be relevant for incentive structures. Peer reviewing is often absent from incentive structures within academia,⁸ however, certain institutions have started to incorporate peer reviewing activities into career advancement.⁶ It is currently unknown if mega peer reviewers are rewarded at their institutions for peer reviewing and if there are other incentives contributing to mega peer reviewers behaviour. Future research that identifies qualitative and quantitative barriers and enablers associated with peer-reviewing behavior can provide a basis for keeping pace with the growing demand for peer reviewers. It can also identify facilitators and barriers to producing high-quality peer reviews. Conducting a survey with mega reviewers and control reviewers to better understand the current study findings was a planned part of this work but has not yet been completed. To inform change, also surveying editors and associated editors would provide a more thorough understanding of how mega peer reviewers receive ongoing journal request for peer reviews and understanding why editors may frequently invite specific reviewers.

There are limitations that should be considered when interpreting our results. First, the data used was collected from researchers that have an account through Publons database which may result in a selection bias. Second, the data collected for this work was not collected for the purpose of this study, limiting the variables that were available. For example, career stage of peer reviewers is not collected within the Publons database, although h-index may provide a proxy for this variable. Third, assessing the quality of peer-reviews was also not possible based on available data. Finally, considering geographical variability in peer reviewing activities was limited, due to few reviewers in either group being located in Australia, South America, or Africa.

The demand for peer reviewers continues to increase. Peer review can result in valuable feedback for the authors and improve the quality and usability of research findings. However, obtaining a high quality peer review is difficult for many journals. The current research found that mega peer reviewers complete a substantive number of peer review activities each year and demonstrate significantly different characteristics than a group of control peer reviewers. These characteristics are important to understand in order to increase the availability and quality of peer reviewers. Future research that identifies the factors associated with peer-review behaviour should be conducted to help inform designing strategies and interventions to facilitate system- and individual- level changes in peer reviewing activity.

STROBE: The Strengthening the Reporting of Observational Studies in Epidemiology

SD: Standard deviation

Ethics approval and consent to participate
Ethical approval was not required as this study did not involve human participants.
Consent for Publication
N/A
Availability of data and materials
The dataset supporting the conclusions of this article is available at https://osf.io/6awzr/?view_only=eeb8ccc0c3a7468095f0b7ef67b71508
Competing interests
David Moher is a Publons Academy advisor. All other authors declare no conflicts of interest.
Funding
No specific funding was received for this project. DM is supported by a University Research Chair (University of Ottawa). DBR is supported by a Canadian Institutes of Health Research Vanier Graduate Scholarship. ACT is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. All authors confirm their work is independent from the funders.
Authors’ contributions
(CReDiT: https://www.casrai.org/credit.html): Conceptualization: DM. Methodology: DBR, BP, JP, ACT, DM. Project administration: DBR. Investigation (data collection): DBR. Writing – Original Draft: DBR, DM. Writing – Review & Editing: All authors read and approved the final manuscript. Supervision: DM.
Acknowledgements
We would like to thank Julia Vilstrup Mouatt for providing input on this work. We would also like to acknowledge the data analyst staff at Publons Academy for assisting in compiling the data.

Elsevier. What is peer review? 2020. Available from: https://www.elsevier.com/reviewers/what-is-peer-review. Accessed March 1, 2020.
Spier R. The history of the peer-review process. TRENDS in Biotechnology. 2002;20(8):357-58.
Publons. 2018 Global State of Peer Review. 2018. Available from: https://publons.com/community/gspr#open-elq-form-slider-DLGSPR. Accessed December 20, 2020.
Tennant JP, Dugan JM, Graziotin D, et al. A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research. 2017;6:1151.
Nicholson J, Alperin JP. A brief survey on peer review in scholarly communication. The Winnower. 2016:1-8.
Moher D, Bouter L, Kleinert S, et al. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology. 2020;18(7):e3000737.
Publons. 2020. Available from: https://publons.com/about/mission. Accessed March 7, 2020.
Severin A, Chataway J. Overburdening of peer reviewers. A multi-disciplinary and multi-stakeholder perspective on causes, effects and potential policy implications. bioRxiv. 2021:1-19. https://doi.org/10.1101/2021.01.14.426539.
Von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Journal of Clinical Epidemiology. 2008;61(4):344-49.
Tabachnick B, Fidell LS. Using multivariate statistics: International edition. Pearson 2013.
Ross E. Gender bias distorts peer review across fields. Nature news. 2017. doi:10.1038/nature.2017.21685
Aldossari M, Chaudhry S. Women and burnout in the context of a pandemic. Gender, Work & Organization. 2021;28:826-834.
EQUATOR Network . Enhancing the QUAlity and Transparency Of health Research. 2021 Available from: https://www.equator-network.org/. Accessed February 10, 2021.
Moher D, Jadad AR. How to peer review a manuscript. Peer review in health sciences BMJ Books, London 2003:183-90.

Table 1. Descriptive Statistics
	Control (N)	Control Mean	Mega Reviewers (N)	Mega Reviewers Mean
Sex (% Male)*	986 (52.5%)	N/A	329 (25.0%)	N/A
Publications^a	1200	23.77	396	118.53
Publications^b	1200	20.57	396	99.32
Citation Count 2018	1200	1.46	396	10.21
Publications 2018	569	176.54	341	480.13
H Index	569	12.82	342	22.97
Citation Count Total	569	1517.26	341	3088.80
Articles Reviewed 2018	1200	4.05	396	138.77
Average Word Count	1200	332.48	396	272.50
Average Word Count for Institution	817	422.45	377	363.74

Table 2. T-Test Results Comparing Characteristics of Mega Reviewers and Control Reviewers
	Control (N)			Control Mean (SD)	Mega Reviewers (N)	Mega Reviewers Mean (SD)	P-Value
Publications^a		1200	23.77 (49.55)		396	118.53 (146.05)	.001
Publications^b		1200	20.57 (45.15)		396	99.32 (137.31)	.001
Publications 2018		1200	1.46 (3.58)		396	10.21 (16.58)	.001
Citation Count 2018		569	176.54 (1005.48)		341	480.13 (759.12)	.001
H Index		569	12.82 (12.73)		342	22.97 (18.29)	.001
Citation Count Total		569	1517.26 (8772.20)		341	3088.80 (5120.50)	.003
Articles Reviewed 2018		1200	4.05 (2.99)		396	138.77 (82.32)	.001
Average Word Count		1200	332.48 (346.30)		396	272.50 (219.99)	.001
Average Word Count of Institution		817	422.45 (190.39)		377	363.74 (237.79)	.001

Table 3. Chi-Square Analyses Results Comparing Characteristics of Mega Reviewers and Control Reviewers
	Control (% of total)	Mega Reviewers (% of total)	P-Value
Continent			.001
North America	269 (25.8)	70 (18.9)
Australia	63 (6.0)	16 (4.3)
Europe	428 (41.1)	136 (36.8)
Asia	222 (21.3)	122 (33.0)
South America Africa	32 (3.1) 28 (2.7)	3 (0.8) 23 (6.2)
Sex
Male	690 (70.0%)	304 (92.4%)	.001
Female	296 (30.0%)	25 (7.6%)
Publons Awards			.001
No	1193 (99.4)	47 (11.9)
Yes	7 (0.6)	349 (88.1)

Appendix1.docx

Download PDF

Journal Publication

published 20 Feb, 2022

Read the published version in Research Integrity and Peer Review →

Editorial decision: Revise before peer review
25 May, 2021
Reviewers invited by journal
21 May, 2021
Editor assigned by journal
04 May, 2021
Submission checks completed at journal
04 May, 2021
Editor invited by journal
04 May, 2021
First submitted to journal
29 Apr, 2021

You are reading this latest preprint version

Identifying Patterns and Motivations of ‘Mega’ Peer-Reviewers

Status:

Journal Publication

Version 1

Abstract

Introduction

Methods

Results

Discussion

Conclusions

List of Abbreviations

Declarations

References

Tables

Supplementary Files

Status:

Journal Publication

Version 1