AI vs Doctors: Medical Case Accuracy

MAI-DxO also achieved correct diagnoses more cost effectively than physicians, the company said in a Monday blog post.

“For AI to make a difference, clinicians and patients alike must be able to trust its performance,” the post said. “That’s where our new benchmarks and AI orchestrator come in.”

Earlier benchmarks used to evaluate AI systems in medicine were based on the United States Medical Licensing Examination (USMLE), which is based on multiple-choice questions, favors memorization and therefore overstates the apparent competence of AI systems, according to the post.

To overcome the limitations of that test, Microsoft AI developed a new one that requires sequential diagnosis and uses 304 recent cases published by NEJM, the post said. This test requires AI models and human physicians to ask questions, order tests and work toward a final diagnosis.

Microsoft AI’s test also includes a virtual cost that reflects real-world healthcare expenditures, per the post.

The MAI-DxO, being an orchestrator, accesses multiple language models and integrates diverse data sources, according to the report. It is also configurable so that it can be told to operate within defined cost constraints.

“Together with our partners, we strongly believe that the future of healthcare will be shaped by augmenting human expertise and empathy with the power of machine intelligence,” the post said. “We are excited to take the next steps in making that vision a reality.”

The PYMNTS Intelligence and AI-ID collaboration “Generative AI Tracker®: Generative AI Can Elevate Health and Revolutionize Healthcare” found that while Americans are enthusiastic about the potential benefits of You have in Healthcarethey still feel uncomfortable with the idea of healthcare providers relying on AI or replacing their medical professionals with this technology.

Sixty percent of Americans said they are uncomfortable with a provider relying on AI in their healthcare, while 57% believe using AI to diagnose diseases and suggest treatments would harm the patient-provider relationship, according to the report.