AI Enhances Detection of Social Determinants of Health: New Research Insights
- SEATTLE, November 21, 2024—Researchers from the Institute for Systems Biology (ISB) studied how artificial intelligence (AI) can reveal social factors affecting health.
- The ISB team worked with Providence, using large language models (LLM) based on generative pre-trained transformers (GPT) to analyze over 25,000 clinical notes from 795 pregnant women.
- First, it assessed AI's ability to detect current versus past housing instability and to provide direct quotes from clinical notes.
New Research on AI and Social Determinants of Health
SEATTLE, November 21, 2024—Researchers from the Institute for Systems Biology (ISB) studied how artificial intelligence (AI) can reveal social factors affecting health. Their peer-reviewed findings are available online.
The ISB team worked with Providence, using large language models (LLM) based on generative pre-trained transformers (GPT) to analyze over 25,000 clinical notes from 795 pregnant women. They aimed to identify housing instability using GPT-4, GPT-3.5, and a human review process.
This study improved on previous research in two main ways. First, it assessed AI’s ability to detect current versus past housing instability and to provide direct quotes from clinical notes. Second, it evaluated how de-identifying notes impacted AI performance.
Among the technologies tested, GPT-4 was the most effective in identifying housing instability. It outperformed humans in recall but lagged in precision, meaning humans were better at confirming instances of no housing instability. Humans excelled at quoting the correct evidence from notes.
ISB Professor Jennifer Hadlock stated that LLMs offer an economical way to identify patients needing outreach. GPT-4 provided text similar to human selections. Notably, it did not generate incorrect information, likely due to clear instructions requiring quotes from notes.
Despite these benefits, the AI sometimes misinterpreted notes, which could lead to confusion. This is concerning since housing issues are linked to other dangers, such as domestic violence. Hadlock emphasized the importance of human review, stating that professionals must carefully evaluate patient situations for safety.
The researchers discovered that AI’s recall was worse with de-identified notes, which used an automated technique called “hide in plain sight.” This method substitutes sensitive information with realistic alternatives, sometimes changing crucial details and compromising the accuracy of housing instability identification.
Alexandra Ralevski, the study’s lead author, highlighted the need for improved de-identification methods. Protecting privacy is crucial, but it’s also essential not to lose vital details regarding social determinants of health.
About ISB
The Institute for Systems Biology (ISB) is a non-profit biomedical research organization in Seattle. ISB addresses significant human health issues, including cancer, aging, and chronic illness. The organization collaborates with Providence, a major healthcare system in the U.S.
Explore more about ISB at isbscience.org or follow them on social media platforms like YouTube, Facebook, and Twitter.
For the original press release, visit Business Wire.
Media Contact
Joe Myxter
Director of Communications, ISB
jmyxter@isbscience.org