Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Large-scale language analysis of peer review reports

The Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) Individual Fellowships (IF) funding programme is one of the largest and most competitive EU research programme (around 15% success rate). It would be expected that the applications of funded proposals have detected ethical issues well. The aim of this study was to describe the ethics evaluation process in the MSCA IF funding scheme . We first assessed the ethical issues identified by the applicants when preparing their proposals and compared these to the ethics issues identified by ethics experts during the ethics evaluation performed for those proposals retained for funding. In total, we analyzed 3074 proposals from three MSCA IF calls ( 2016, 2017 and 2018). Applicants identified almost two times fewer ethical issues compared to the experts performing the ethics review. Most of the ethics issues were identified in proposals from the Life Sciences and Social Sciences and Humanities research panels, which represented more than 60% of all proposals. Out of all 3074 proposals which were main listed and accepted for funding, 2225 (72.4%) received conditional ethics clearance, pending satisfaction of requirements defined by the ethics experts. The most prevalent requirements were within the Protection of personal data, Environment, Health and Safety, Humans, and non-EU countries categories. Our results emphasize that successful applicants to the MSCA IF programme, consisting mostly of postdoctoral researchers, are not adequately trained or knowledgeable about the ethics issues related to their proposed research, and that problem persists over time. Training in research ethics and proper guidance by the host institutions are paramount for narrowing the gap between the applicants’ ethics self-assessments and the results of the ethics review procedure. 


Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

<br>



**Chair:** Peter Van Den Besselaar, Vrije Universiteit Amsterdam


3. Peering into peer review

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Ivan Buljan

3

SHORT BIO

Presentations

Large-scale language analysis of peer review reports

Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES