Development of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts

Abstract 

Research on the value of peer review is limited by the lack of a validated instrument to measure the quality of reviews. The aim of this study was to develop a simple, reliable, and valid scale that could be used in studies of peer review. A Review Quality Instrument (RQI) that assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and on two aspects of the review (constructiveness and substantiation of comments) was devised and tested. Its internal consistency was high (Cronbach’s alpha 0.84). The mean total score (based on the seven items each scored on a 5-point Likert scale from 1 to 5) had good test-retest (K_w = 1.00) and inter-rater (K_w = 0.83) reliability. There was no evidence of floor or ceiling effects, construct validity was evident, and the respondent burden was acceptable (2–10 minutes). Although improvements to the RQI should be pursued, the instrument can be recommended for use in the study of peer review.

Keywords:  Peer review, reliability, validity, review quality, internal consistency, respondent burden