Computational Studies Of Peer Review

In peer review, reviewers often have formal and informal occasions to interact with other reviewers of the same grant proposal or submission. The former happens, for example, when grant proposals are evaluated by sitting review panels who are tasked with collectively deciding the fate of a particular proposal. Despite the scholarly consensus that interactions (and thus social influence) can alter the outcome of collective decision making, to date social influence has rarely been a theoretical ingredient in simulation models of peer review. In our contribution we use a simple agent-based model of peer review to show that social influence processes can indeed change the dynamics and outcome of peer review. With our simulation experiment we showcase the incorporation of social influence processes in peer review models as a promising direction for future research.

Does social influence matter in peer review?

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers' opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers' opinions are largely adopted by agencies.

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

<br>



**Chair:** Simone Righi (University College London)


5. Computational studies of peer review

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Thomas Feliciani

3

4

SHORT BIO

Presentations

Computational Studies Of Peer Review

Does social influence matter in peer review?

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES