Quality versus Risk-of-Bias assessment in clinical research

Bias negatively impacts clinical epidemiologic research, and to prevent this requires the placement of methodological safeguards within research studies. Such safeguards may be within the design, conduct, and analysis of a study. Before a body of research can get its results embedded into evidence-based practice, there should be a quality assessment for the presence of such safeguards against bias in the various included studies on the topic to ensure the credibility of research results, and this is formally undertaken using a quality assessment. More recently, this assessment has also been called the risk of bias assessment. The distinction between them is that:

• a)
  quality assessment is the assessment of the inclusion of methodological safeguards within a study, and
• b)
  risk of bias assessment concerns the implication of the inclusion of such safeguards for study results.

This distinction is not just semantic and reflects differences related to how the tool is intended to be used. For example, the quality assessment also deals with the number or type of safeguards present and does not include signaling questions and criteria for judgments, both of which are required for risk of bias assessment as the latter requires a subjective degree of risk as the outcome of the assessment. The Cochrane collaboration had previously suggested a similar distinction suggesting that quality assessment should refer to the extent to which a study was designed, conducted, analyzed, interpreted, and reported to avoid systematic errors, while risk of bias assessment should refer to what flaws in the design, conduct, and analysis affect the study results []. It is important to note that the terms “quality” and “risk of bias” have been used interchangeably in the epidemiologic literature to describe the methodological conditions associated with the validity of study results and at times, purely reporting items have crept into quality assessments that have no bearing on systematic error []. Some researchers have taken this to suggest that quality assessment should refer to reporting such items [] irrespective of whether these are safeguards against bias or not, and this is incorrect. We should henceforth use the term quality assessment when we refer to the measurement of the extent that methodological safeguards against bias have been implemented and risk of bias assessment when we refer to bias judgments based on such quality assessment. Judgments are usually made in terms of low/high risk of bias or high/low quality of a study, which are terms that have also been used interchangeably. Reporting checklists (and there are many) must be distinguished from a quality assessment tool, checklist or scale.

All evidence syntheses require that such assessment be formally done of the included studies so that users of such research are cognizant of how much the results can be trusted. When a meta-analysis is included, the quantitative results should also be interpreted in the light of either the quality assessment or risk of bias assessment of included studies. This can be taken a step further, and these assessments (particularly the quality assessment) can be used for bias adjustment of meta-analysis results [].

These assessments require tools that include a list of safeguard items likely to influence internal validity (quality assessment) or estimated intervention effects (risk of bias assessment) that can be looked for in the published research. By and large, these safeguard items are the same, and the only reason why they may be labeled by either term is the intent of the particular assessment. Of note, a study that implemented all safeguards listed in a tool may not necessarily be unbiased, and one that applied none is not necessarily biased, and this applies to both assessment types.

The final point is that the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) group [] uses the term “quality of evidence” to reflect the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation. Therefore, in addition to the risk of bias assessment of individual studies making up the body of evidence, four other issues are rated under the “quality of evidence” umbrella: inconsistency, indirectness, imprecision, and publication bias. The latter is not actually a quality assessment in terms of what is meant by this term for individual study assessment of internal validity and uses "quality" in a more generic sense making this a distinct concept from what we mean when we undertake a quality assessment of individual studies. Of note, GRADE incorporates a risk of bias assessment within this generic quality construct.

The use of either type of tool, despite similar content, differs slightly. The best use of a quality assessment tool is to enumerate the safeguards implemented in a study as a count. We are avoiding the use of the word score here because it has received bad publicity [] although not for the right reasons []. Once the safeguards are counted, stratification of studies for subgroup analysis by such counts is not recommended []. There remains no quantitative utility for risk of bias judgments (apart from their descriptive use) when we take away stratification, and all bias adjustment methods [] require a quality assessment as input. Such input requires conversion of the counts of safeguards from a quality assessment into a relative ranking of studies that can then be examined analytically []. Such ranking is achieved by dividing all study counts by the highest count in the group of studies; thus, study ranks vary from 1 downwards to zero. Again, a study of rank 1 is not necessarily devoid of bias, and quality assessment can only help with the relative assessment of studies. Bias quantification is not possible from a quality assessment. An example of a study ranking is given in Table 1.

It should be noted that the (seemingly “arbitrary”) application of rules to quality assessment tools turns them into the risk of bias assessment tools. The underlying safeguards are the same, but the risk of bias assessment tools needs an empirical construct for making the hierarchal risk of bias judgments (high/low risk which is equivalent to low/high quality). These constructs can vary, such as missing important items to the number of missing items or domains with missing items depending on the tool authors’ judgments regarding possible influence on estimates of effect. One disadvantage of this process is that assessments are tool dependent and consistency (as opposed to agreement) is not an expectation as we move from tool to tool. An example of a risk of bias judgment is also given in Table 1. Of note, researchers can use a risk of bias assessment tool for quality assessment but not vice versa unless the empirical construct to be used for such judgments is first defined.

A study reported in the BMJ [] illustrates these points. The authors used the Jadad quality scale to assess the quality of five randomized controlled trials of paracervical anesthesia for pain control and considered six safeguards through the tool. They then checked the studies for the presence or absence of safeguards denoted 1 and zero respectively and counted how many were present (Table 1). Finally, they also created their own rule for risk of bias assessment (>3 safeguards = low risk or high quality). We added the column on quality rank for illustration. The latter can be used for quantitative bias adjustment [], and this is illustrated in Fig. 1. Of note, instead of the Jadad quality scale, researchers in such a study may now use the RoB 2 risk of bias tool from Cochrane, which has explicit albeit equally “arbitrary” rules for arriving at the judgment that obviates the creation of one by the authors as was done with this example.

Luis Furuya-Kanamori: Writing - review & editing. Chang Xu: Writing - review & editing. Syed Shahzad Hasan: Writing - review & editing. Suhail A. Doi: Conceptualization, Methodology, Writing - original draft, Funding acquisition.