Peering at peer review revealed high degree of chance associated with funding of grant applications
Accepted 12 December 2005. published online 27 March 2006.
Abstract
Background and Objectives
There is a persistent degree of uncertainty and dissatisfaction with the peer review process underlining the need to validate the current grant awarding procedures. This study compared the CLassic Structured Scientific In-depth two reviewer critique (CLASSIC) with an all panel members' independent ranking method (RANKING). Eleven reviewers, reviewed 32 applications for a pilot project competition at a major university medical center.
Results
The degree of agreement between the two methods was poor (kappa = 0.36). The top rated project in each stream would have failed the funding cutoff with a frequency of 9 and 35%, depending on which pair of reviewers had been selected. Four of the top 10 projects identified by RANKING had a greater than 50% of not being funded by the CLASSIC ranking. Ten reviewers provided optimal consistency for the RANKING method.
Conclusions
This study found that there is a considerable amount of chance associated with funding decisions under the traditional method of assigning the grant to two main reviewers. We recommend using the all reviewer ranking procedure to arrive at decisions about grant applications as this removes the impact of extreme reviews.
aDivision of Clinical Epidemiology R4.29, McGill University Health Center, RVH Site, 687 Pine Avenue West, Montreal, H3A 1A1, Canada
bHealth Technology Assessment Unit, R4.14, McGill University Health Center, RVH Site, 687 Pine Avenue West, Montreal, H3A 1A1, Canada
cDivisions of Infectious Diseases/Immunodeficiency, Royal Victoria Hospital, McGill University Health Centre, 687 Pine Avenue West, Montreal, H3A 1A1, Canada
dDepartment of Psychology, Science Pavilion, Concordia University, 7141 Sherbrooke Street West, Montreal, Quebec H4B 1R6, Canada
eDepartment of Pediatrics and of Epidemiology and Biostatistics, Montreal Children's Hospital Research Institute, 4060 Ste Catherine Street West, #205, Westmount, QC H3Z 2Z3 Canada
fMcGill University Health Centre, Room D6-156, MGH site, 1650 Cedar Avenue, Montreal Quebec H3G 1A4 Canada
ABSTRACT