Trustworthy AI: Keeping Humans in the Loop
- Here's a breakdown of the key concerns and points raised in the provided text regarding AI in healthcare, specifically focusing on Large Language Models (LLMs):
- * False Positives: UK radiologists report that AI-assisted imaging can increase false positive results compared to unassisted screening.
- * Rapid,Unchecked Growth: The launch of ChatGPT and similar llms has led to a surge in chatbot use in healthcare,but this growth is largely unregulated.
Here’s a breakdown of the key concerns and points raised in the provided text regarding AI in healthcare, specifically focusing on Large Language Models (LLMs):
1. AI-Assisted Imaging Concerns:
* False Positives: UK radiologists report that AI-assisted imaging can increase false positive results compared to unassisted screening. This leads to unnecessary tests and patient anxiety.
* Hopeful Signs: Despite this, prospective studies and careful implementation show promise for AI in imaging.
2. The Rise of Unregulated Generative AI (LLMs):
* Rapid,Unchecked Growth: The launch of ChatGPT and similar llms has led to a surge in chatbot use in healthcare,but this growth is largely unregulated.
* distinction from Regulated AI: LLMs differ from “foundation models” or “Software as a Medical Device” which are subject to regulation.
3.Real-World LLM Adoption (Examples):
* China (DeepSeek): A cost-effective, open-source LLM (DeepSeek) was quickly adopted by 750 Chinese hospitals for administrative and clinical support, operating in a “regulatory gray area.”
* united States (Open Evidence): The Open Evidence platform is used by 40% of US physicians to answer questions about treatment and labs, backed by medical journal content. It’s largely unregulated beyond data privacy compliance.
* AI Scribes: Commercial AI scribes are recording doctor visits.
4. Regulatory Shortcomings:
* Insufficient Frameworks: Existing regulations (treating AI as Software as a Medical Device) are not adequate for the rapid evolution of LLMs.
* Declining Disclaimers: A study shows a significant decrease in disclaimers accompanying medical advice from LLMs (from 26% in 2022 to 1% in 2025). This is a major concern.
* Unlicensed Therapy Concerns: There’s growing public worry that LLMs are being used as unlicensed therapy chatbots.
* AI-Mediated Detriment: In certain specific cases, interactions with chatbots have led to negative outcomes.
In essence, the article highlights a critical gap between the rapid advancement and adoption of llms in healthcare and the lack of appropriate regulation and oversight. This poses risks to patient safety, trust, and possibly, the doctor-patient relationship.