Systematic Review Rigor: New Research
Systematic reviews, crucial for shaping policy, are under scrutiny.New research reveals that some reviews incorporate retracted articles, raising serious research integrity concerns. A recent JAMA Network Open study highlights that low-quality studies from paper mills are making their way into the process. this is problematic, as it could wrongly influence decisions.The reliance on static reviews is addressed by innovators aiming at “living” synthesis models. AI tools like otto-SR, designed to automate systematic reviews, are emerging as promising solutions, potentially speeding up the process substantially.However, experts like Christian Cao, stress that the need for human oversight should remain. News Directory 3 is following the ongoing developments! Discover what’s next for systematic reviews as AI evolves and new gold standards are explored.
Retracted Studies Undermine Systematic Reviews, Raising Research Integrity Concerns
Systematic reviews, designed to strengthen research by pooling data from multiple studies, are gaining influence in shaping policy. However, a recent study highlights that not all systematic reviews are created equal, raising concerns about research integrity.
The study, published in JAMA Network open, found that a small percentage of systematic reviews included retracted articles originating from paper mills.These journals produce low-quality research, frequently enough to help researchers advance their careers.
Jack Wilkinson, a biostatistician at the University of Manchester, noted the study reinforces concerns that systematic reviews can inadvertently amplify the influence of problematic studies. he has previously analyzed the impact of untrustworthy studies on Cochrane reviews.
Lisa Bero, an evidence synthesis expert, said the term “systematic review” has been co-opted, with some studies labeled as such failing to meet established criteria for data synthesis.
Genyang Tang, a research integrity scientist at the University of Calgary and author of the new study, examined 200,000 systematic reviews and found 299 cited retracted articles. He cautioned that his reliance on the Web of Science may have skewed the sample toward more rigorous reviews.
Maryam Zaringhalam, senior director of policy at the Center for Open Science, emphasized the importance of determining whether the inclusion of retracted articles significantly alters the conclusions of systematic reviews.
The study also revealed that in many cases, articles were retracted after the systematic review’s publication, highlighting the static nature of many reviews. “Living evidence syntheses,” which are regularly updated, have emerged as a potential solution.
Christian Cao, a medical student at the University of Toronto, experienced the time-intensive nature of systematic reviews while working on SeroTracker, a living systematic review estimating COVID-19 infection rates. His team screened over 100,000 articles.
Cao and his colleagues are developing an artificial intelligence tool, otto-SR, to automate the review process. Testing showed the tool could reproduce a Cochrane review issue in two days, compared to the years required for manual completion. The results are available in a preprint.
Experts caution that AI is not a panacea and could facilitate the production of poor-quality reviews. Cao hopes his tool will spark discussion about establishing a new gold standard for systematic reviews.
“We aren’t saying that we have 100% or 99% data extraction accuracy. We’re saying we have 93.1% and we’d still like to test on more reviews… but humans also really suck at systematic reviews,” said Cao.
What’s next
The ongoing growth and refinement of AI tools like otto-SR may offer a path toward more efficient and accurate systematic reviews, but careful oversight and validation will be crucial to ensure research integrity and maintain the reliability of evidence-based policy decisions.