Background: Peer review is regarded as the gold standard decision-making approach for research funding. Around the globe, health research funders use peer review to inform fair and objective decisions to advance science. However, peer review is not without challenges. A large body of evidence ranging from journal articles, opinion letters, editorials, conference abstracts, thesis and blogs, report problems associated with peer review. These problems can be anything from high administrative costs, low reliability, potential biases (e.g. gender, career status), lack of equity, unreliable scoring and ranking, which can then contribute to a high burden to stakeholders involved in the application and decision-making system (1-5). Strategic priorities of the National Institute for Health Research (NIHR), one of the largest UK funders of health research, have endorsed and highlighted the need for further action on improving their peer review during the decision-making process for the allocation of research funding (6, 7). 

Aim: The aim of this study is to identify which elements of the peer review process and/or alternative approaches, support research funding allocation decision-making, for whom, in what contexts, and how. By conducting a realist synthesis, we will be better placed to understand which approaches and/or elements are likely to enhance the decision-making process for the allocation of research funding. 

Methods: A realist synthesis will provide an explanatory analysis of peer-review or alternative approaches to funding allocation that work in specific contexts, such as for different funding organisations, charities, Higher Education Institutes (HEIs), and research disciplines. The search strategy is reported elsewhere (6). Database searches were originally conducted during May-June 2016 and updated to December 2018. Relevant sources were also searched through Google Scholar and included a citation analysis for included studies. Searches of grey literature as well as databases outside health (e.g. IEEE, arXiv.org) were conducted to ensure materials outside of mainstream academic journals were captured.

Results: Electronic searches returned 1824 publications (after removing duplicates) and manual searches returned 36, resulting in 1860 publications. Five researchers independently screened 50 publications each (abstract and/or full text) and inter-rater reliability of paired researchers for agreement to include or exclude reached 80% agreement based on initial eligibility criteria (i.e. studies of any design exploring interventions or approaches for allocating research funds). This agreement validated the first-stage inclusion/exclusion criteria and the remaining publications (n=1610) were screened according to these criteria. This resulted in 215 publications that were subjected iteratively in relation to programme theory statements (A. if there’s peer review (as a process) then there is bias & burden to all stakeholders B. if variations to the typical peer review process are introduced then bias/burden/costs could be reduced, C. Studies have elements of A&B). Two researchers then discussed the assessment of the 215 included publications and made final decisions on inclusion. A total of 111 publications were selected to inform the research questions. All researchers in the team were involved in data extraction. The following section provides a brief description of the initial findings of the synthesis as per the realist methodology. 

Evidence map
Overall, publications were found to fall into two categories: 
1. Studies that generated evidence on known issues with peer review (bias towards early career researchers, innovation, gender; ability of peer review to predict future value of research; burden for funders and researchers to manage and write applications) 
2. Studies elaborating the need to find approaches different from peer review that can be used to make funding decisions and/or that propose novel approaches. 
The extraction process is ongoing, and the current emerging findings are taken from an in-depth analysis of 30 included publications (out of the 111). What we know so far is that only a handful of interventions have measured in some form the efficiency of peer review in terms of time or cost efficiencies or which have implemented slight or radical alternatives to the current peer review process. 

Context mechanisms outcomes
From the 111 included publications, we developed a map of the Context, Mechanism and Outcomes (CMO), which is central in the realist methodology to answer the questions of for whom, in what context and how an intervention works. Our CMO map shows that the contexts in which the studies have taken place are funders and higher education institutions focusing on developing interventions to increase efficiencies. Other studies are situated in academic contexts where research communities have explored peer review processes from an experimental point of view to enhance the discipline of Research on Research. From the funder perspective, interventions that have been developed to enhance peer review fall into three categories: 
1. Incremental changes to the funding process (modifying the number of stages in applications, modifying reviewers’ numbers and changing the size of the application) 
2. Organisation-level changes creating funding streams to address bias or systems that aid the identification and management of peer reviewers 
3. Alternative approaches with the most salient approaches (sandpits, random allocation, golden tickets, Delphi exercises, and community-based grants). 
A brief review of these categories using the existing/emerging evidence is provided below. 



Incremental changes to peer review 
The National Institutes of Health (NIH) recommendations to streamline peer review include shorter applications with focus on impact and innovation, and less on methods and preliminary data, (8) as well as establishing a numerical quota for the number of awards made to new investigators (Kaiser, 2008b), which aimed to match the application success rates of new investigators to those of established investigators. The Australian National Health and Medical Research Council proposed having a more focused application to reduce preparation costs and improving as a result the quality of peer review by reducing workloads (9, 10). However, some changes can have unintended negative effects, demonstrated in two cross-sectional surveys studies that by shortening the application increased the applicants time spent on preparing applications (11). This is an indication that changes in one context may shift the burden onto another, and this needs to be considered when proposing changes to peer review processes. Other areas under consideration include improvements to fairness, effectiveness and robustness of peer review through masking applications (12) increasing transparency (13) or developing peer review criteria to meet the goals of a specific research programme or discipline (14).

Alternative approaches to decision-making 
Novel approaches to funding decision-making that substitute (at least partially) peer review have also been discussed or implemented. The common approaches in the literature include sandpits, partial random allocation of research funds and Delphi’s. Sandpits are interactive workshops involving 20 to 30 participants, a director, a team of experts and independent stakeholders. The group comes together for a multi-day residential workshop to discuss ideas and participate in interactive sessions with the purpose of establishing research collaborations and receive funds. The proposals in the workshop are evaluated in a similar way to funding panels (15). The basic premise of the allocation of funds by chance retain an aspect of peer review where all proposals submitted to a call are assessed against the research programme requirements, and those potentially fundable are randomly allocated funding (16). A modified Delphi’s (or consensus surveys) proved to be an efficient, transparent, and equitable method of reviewing novel grant applications in a specialized field of research, where no local expertise was available. This process may also be useful for other peer review processes, particularly where there is limited access to local experts (17).
Preliminary conclusion: The approaches and interventions identified ebb and flow within the literature, perhaps influenced by multiple external factors including funding availability and the overall research landscape. Therefore, research that generates empirical evidence has an important role in enabling funders to make informed decisions about whether and how peer review processes work best, and the contexts in which alternatives may offer viable alternatives that support decision-making processes. Findings from this synthesis would encourage long-term implementation of successful mechanisms to the allocation of research funding.

Keywords: peer review, science of science, realist synthesis
Acknowledgements: We would like to thank Geoff Frampton, Karen Pickett, Jo Picot and Jonathan Shepherd at the Southampton Health Technology Assessments Centre for their collaboration in this study. The search strategy for this realist synthesis extends the original search used in their systematic map and review. Discussions with the team at the start of the study strengthened the overall design of the review and supported the Adding Value in Research NIHR initiative. We would also like to thank Justin Jagosh, Director of the Centre for Advancement in Realist Evaluation and Synthesis at the University of Liverpool for an insightful Realist Evaluation and Synthesis workshop and for sharing his expertise with us during the design of our study.
References
1. Hodgson C. How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol. 1997;50(11):1189-95.
2. Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS ONE. 2012;7(4):e35247.
3. Bollen J, Crandall D, Junk D, Ding Y, Borner K. From funding agencies to scientific agency: Collective allocation of science funding as an alternative to peer review. EMBO Rep. 2014;15(2):131-3.
4. Bishop D. Nature INDEX. 2018. [cited 2019]. Available from: https://www.natureindex.com/news-blog/luck-of-the-draw.
5. Bollen J, Carpenter S, Lubchenco J, Scheffer M. Rethinking Resource Allocation in Science. arXiv preprint arXiv:190407645. 2019.
6. Shepherd J, Frampton GK, Pickett K, Wyatt JC. Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PLoS ONE. 2018;13(5):e0196914.
7. Turner S, Bull A, Chinnery F, Hinks J, Mcardle N, Moran R, et al. Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study. BMJ Open. 2018;8(12):e022548.
8. Kaiser J. National Institutes of Health. Changes in peer review target young scientists, heavyweights. Science. 2008;320(5882):1404.
9. Herbert DL, Barnett AG, Graves N. Funding: Australia's grant system wastes time. Nature. 2013;495(7441):314.
10. Herbert DL, Graves N, Clarke P, Barnett AG. Using simplified peer review processes to fund research: a prospective study. BMJ Open. 2015;5(7):e008380.
11. Barnett AG, Graves N, Clarke P, Herbert D. The impact of a streamlined funding application process on application time: two cross-sectional surveys of Australian researchers. BMJ Open. 2015;5(1):e006912.
12. Ismail S, Farrands A, Wooding S. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009.
13. Gurwitz D, Milanesi E, Koenig T. Grant application review: the case of transparency. Plos Biology. 2014;12(12):e1002010.
14. Hawe P, Samis S, Di Ruggiero E, Shoveller JA. Population Health Intervention Research Initiative for Canada: progress and prospects. N S W Public Health Bull. 2011;22(1-2):27-32.
15. Guthrie S, Guérin B, Wu H, Ismail S, Wooding S. Alternatives to peer review in research project funding. Cambridge, UK; 2013.
16. Fang FC, Casadevall A. Research Funding: the Case for a Modified Lottery. mBio. 2016;7(2):e00422-16.
17. Holliday C, Robotin M. The Delphi process: a solution for reviewing novel grant applications. Int J Gen Med. 2010;3:225-30.



Decision-making approaches to grant funding allocation: insights from a realist synthesis

Background: Every year, thousands of research proposals are submitted to funding organisations in the hope of securing funding for research. These applications go through a number of assessment stages in which decisions are made about whether to decline the application, progress the application or fund the application (1–3). Funding organisations often have limited funds, large numbers of application submissions, and a need to demonstrate impactful spending, and so many of these applications go through multiple iterations and not all will be funded. 
Feedback to the applicants about these decisions are a vital part of the scientific process. Applicants rely on feedback to improve the quality of their application for securing successful funding and/or as a learning opportunity to facilitate the chance of success in future applications. It is therefore important that applicants receive constructive feedback in order to improve the quality of their application, change the content of their application or to understand why their application was or was not successful (was it competitive but there were stronger applications, or were significant improvements required?). A lack of constructive feedback could not only impact on professional development for researchers but could introduce unnecessary burden and negative experiences of researchers as applicants, in what is already a labour intensive research activity (1,4–7). 
For the National Institute for Health Research (NIHR) Evaluation, Trials and Studies Coordinating Centre (NETSCC), applicants receive written feedback by the research programme funding committees at both stage 1 and stage 2 of the application process. Stage 1 feedback is provided for applications that are progressed to stage 2, as well as for those that were unsuccessful. For applications that successfully progress to stage 2, feedback is again provided for both successful and unsuccessful applications at this second stage. Committees are encouraged to consider providing targeted feedback for each application rather than using generic phrases. Feedback points are drafted in the funding committee meetings and are provided to the applicants in written bullet point format. NIHR applicants also have access to guidance on what to include in their applications and the assessment criteria against which the application will be assessed. 

In order to support applicants throughout the funding process, it is vital that the guidance and feedback provided is of high quality, constructive and consistent. It is therefore important to understand the key components of the feedback provided to applicants at Stage 1 and 2 and how these are perceived by applicants. This will lead to increased knowledge on the quality and value of the feedback provided to applicants, whether improvements are needed regarding the type of feedback given and whether this is consistent across the NIHR research programmes managed by NETSCC, promoting the adding value in research approach and transparency agenda. 

Objectives: The aim is to identify and compare the key components of feedback that are provided to applicants by NIHR research programmes, whether these components are consistent across different sources of feedback (external peer review, funding committees at stage 1, funding committees at stage 2, guidance documents) and how the feedback is perceived by applicants.

Methods: To gain a NIHR funder perspective of feedback provided, qualitative content analysis of feedback and guidance documents from the Health Technology Assessment (HTA), Public Health Research (PHR), Efficacy and Mechanism Evaluation (EME), and Health Services and Delivery Research (HSDR) funding programmes was conducted. Following ethical approval, purposeful sampling was used to select commissioned-led and researcher-led, and successful and unsuccessful research applications from March 2018 and March 2019. To date, a total of 84 applicants across the four funding programmes have been contacted to gain approvals to access the feedback data. Using a phased approach, (1) a coding framework was created using inductive content analysis of external peer reviews, stage 1 and stage 2 funding committee feedback from the HTA programme, and (2) then applied to the stage 2 funding committee feedback from the remaining funding programmes using deductive content analysis. (3) The final phase will compare the key components across funding programmes and to the guidance criteria provided for each programme. 
To gain an NIHR applicant perspective about feedback received from funding committees, an online survey was conducted in October 2019. Following ethical approval, the survey used purposeful sampling and targeted chief investigators of applications submitted to HTA, EME, PHR and HSDR research programmes during March 2018 and March 2019 to ensure the sample reached commissioned-led, researcher-led and successful and unsuccessful applicants. The survey was distributed via email to 770 applicants and was also published on Twitter. The survey used both closed and open questions to explore how useful the feedback was and considerations for improvements for both stages 1 and 2. A mixed methods analysis using descriptive statistics and inductive thematic analysis was conducted on closed and open question responses.
Results: From the NETSCC perspective, qualitative content analysis of feedback and guidance documents is on-going. To date, feedback documents relating to 25 HTA applications have been analysed. From this, a coding framework has been created. Seven key components (with sub-components) were identified from the feedback given to HTA applicants:
(1) Acceptability to patients and professionals: feedback on patient and public involvement in the study design.
(2) Team and infrastructure: feedback on justification of the roles and time of the team as well as potential conflicts of interest for the team.
(3) Justification of project: feedback on the scientific rationale, relevance and need of the project.
(4) Study design: feedback was primarily around clearly defining and justifying study parameters, outcomes and sample size.
(5) Risks and contingencies: feedback highlighted the need for consideration to be given to recruitment, retention and attrition, and feasibility of the project.
(6) Value for money and costings: feedback around cost elements of the application.
(7) Pathway to impact and implementation: feedback around implementation plans and whether project will have clinical impact.
The most frequently reported components were study design; risks and contingencies; and team and infrastructure. The majority of the components were consistently reported across the funding committee and external peer reviewer comments, except for the justification of project component, which was only provided in feedback to applicants at stage 1. Recruitment for documents in the remaining research programmes commenced in October 2019. To date, we have recruited 23 applicants across PHR (8), EME (6) and HSDR (9), and the coding framework is being applied to stage 2 funding committee feedback and guidance documents. 
From the applicant perspective, 147 applicants completed the online survey (response rate of 19%). After removing data from applications under review, non-NETSCC research programmes and incomplete datasets, a final sample of 115 applicant data was analysed. Overall, more than half of applicants agreed that the feedback provided by the funding committee was useful (52%) and of benefit (53%). However, more than half of applicants also disagreed that the feedback was of good quality (54%) or that it was helpful for developing future applications or work (57%). Responses differed depending on whether the applicant had been successful in securing funding, with at least 77% of funded applicants responding positively about the feedback and less than 37% of unsuccessful applicants responding positively. Moreover, feedback at each stage was more likely to be seen positively if the application was moved on to the next stage/funded whilst feedback at the point of rejection was more likely to be considered negatively. For example, unsuccessful applicants at stage 2 regarded stage 1 feedback positively, but regarded stage 2 feedback negatively, finding it unhelpful and unclear on areas for development. The open-ended responses also suggested that feedback could be more detailed and directive to facilitate changes and that it would be helpful to get some indication of how important each comment was (is it mandatory and which to focus more attention on). In contrast to the findings from the NIHR funder perspective, feedback from the applicant perspective showed that feedback is not always seen as being consistent across stages, with guidance, briefs or within the same feedback document. 
Discussion: The findings will lead to increased knowledge on the quality, transparency and consistency of feedback provided to applicants, whether this is consistent across research programmes and how useful the feedback is to applicants. The preliminary findings show that key components provided in feedback by the different NIHR research programmes were consistently reported across both the funding committee and external peer reviewer comments. However, applicants reported that the content of the feedback was not always consistent across both stages and within stages, and that although applicants found this feedback useful overall, there are a number of areas in which feedback and guidance could be improved. Once the full analysis has been completed, recommendations can then be made for NIHR research programme guidance and feedback to applicants that supports the chance of a successful application, promoting the adding value in research approach and transparency agenda. 
This work is part of a wider series of studies being undertaken by the NIHR Research on Research team on peer review and decision-making in funding allocation. This wider work includes the studies reported here, a realist synthesis to identify contexts in which different decision-making processes work, a survey with international funders to understand current funding practices, qualitative analysis to understand stakeholders’ expectations and an observational study of funding committees and will provide much needed evidence to help inform funders in peer review and decision-making activities.

References:
1. Guthrie S, Ghiga I, Wooding S. What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies. Santa Monica, CA; 2018. 
2. Ismail S, Farrands A, Wooding S, Europe R. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009. 
3. Publons part of the Web of Science Group. Grant Review in Focus. Global State of Peer Review Series. 2019. 
4. Gluckman P. Which science to fund: time to review peer review? 2012;(December):11. 
5. Graves N, Barnett AG, Clarke P. Cutting random funding decisions. Nature. 2011;469(7330):299. 
6. Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers. BMJ Open. 2014;4(3):e004462. 
7. Herbert DL, Barnett AG, Clarke P, Graves N. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open. 2013;3(5):1–6. 



Identification and comparison of key criteria of funding decision feedback to applicants: A funder and applicant perspective

<br>


**Chair:** Jadranka Stojanovski, University of Zadar


Lightning session 2

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


PEERE

The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Alejandra Recio-Saucedo

2

2

SHORT BIO

Presentations

Decision-making approaches to grant funding allocation: insights from a realist synthesis

Identification and comparison of key criteria of funding decision feedback to applicants: A funder and applicant perspective

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES