Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy

Abstract 

Objective

This study assessed the impact of systematic review and data extraction experience on the accuracy and efficiency of data extraction in systematic reviews.

Study Design and Setting

We conducted a prospective cross-sectional study from October to December 2006. Participants were classified as having minimal, moderate, or substantial experience in systematic reviews and data extraction. Three studies on insomnia treatment were extracted. Our primary outcome was the accuracy of data extraction. Data sets of each experience level were analyzed for errors in data extraction and results of meta-analyses. Additionally, the time required for completion of data extraction was compared.

Results

Error rates were similar across the various levels of experience and ranged from 28.3% to 31.2%. Mean rates for errors of omission (11.3–16.4%) were generally lower than those for errors of inaccuracy (13.9–17.9%). There were no significant differences in error rates or accuracy of meta-analysis results between groups. Time required approached significance, with minimally experienced participants requiring the most time.

Conclusion

Overall, there were high error rates by participants at all experience levels; however, time required for extraction tended to decrease with experience. These results illustrate the need to develop strategies aimed at mastery of data extraction, rather than reliance on previous data extraction experience alone.

Keywords: Systematic review, Data interpretation, Statistical analysis, Meta-analysis, Data extraction, Data-handling error