AI’s Hallucinations in the Courtroom & OpenAI’s Risky Health Advice

Artificial intelligence is rapidly infiltrating nearly every aspect of modern life, and the legal system is no exception. But recent events are raising serious questions about the readiness – and the wisdom – of handing over critical tasks to algorithms prone to error. Simultaneously, a shift in OpenAI’s messaging around its GPT-5 model is sparking concern about the potential for AI-driven health misinformation. Let’s break down what’s happening,and why it matters.

AI’s Early struggles with Legal Accuracy

The promise of AI in law is compelling: faster research, streamlined case summaries, and automated drafting of routine orders could alleviate the immense backlog plaguing courts across the US.But the reality, as recent cases demonstrate, is far more fraught with peril.

We’ve seen a disturbing trend of AI systems confidently presenting false information as fact. Lawyers have submitted briefs citing non-existent cases,and even AI experts have fallen victim to “hallucinations” – instances where the AI fabricates information – in sworn testimony. A Stanford professor specializing in AI and misinformation himself recently submitted flawed evidence in a deepfake case, highlighting just how easily these errors can slip through, even with expert oversight.

This isn’t simply a matter of inconvenience. In the legal realm, accuracy is paramount. Incorrect information can lead to unjust outcomes, erode trust in the system, and possibly jeopardize individual liberties. The early experiments with AI in courts are serving as a stark warning: proceed with extreme caution. While the technology could be beneficial with robust safeguards, the current risk of relying on demonstrably unreliable information is too high.

GPT-5: Underwhelming Performance & a Concerning New Direction

OpenAI’s GPT-5 was hyped as a potential leap towards “artificial general intelligence” (AGI) – the holy grail of AI development. The expectation was a transformative model capable of human-level reasoning and problem-solving. What we’ve gotten, however, is…underwhelming.

While improvements have been made, GPT-5 hasn’t delivered the revolutionary capabilities many anticipated. But perhaps more concerning than its performance is a shift in OpenAI’s messaging. The company is now explicitly suggesting users employ its models for health advice.

This is a perilous game. while AI can potentially assist in healthcare – analyzing data, identifying patterns – it is absolutely not equipped to provide reliable medical guidance. The potential for misdiagnosis, inappropriate treatment recommendations, and the spread of harmful misinformation is enormous. OpenAI’s decision to actively promote this use case feels less like innovation and more like a reckless disregard for public safety.

What Does This Mean for the Future?

These two developments – the courtroom errors and OpenAI’s health advice push – are connected by a common thread: the limitations of current AI technology and the need for responsible development and deployment.

We’re witnessing a crucial moment.The initial enthusiasm for AI is colliding with the harsh realities of its imperfections. It’s a wake-up call for the legal profession, the tech industry, and the public.

Here’s what needs to happen:

Rigorous Testing & Validation: Before AI is integrated into critical systems like the legal system, it must undergo extensive testing and validation to minimize the risk of errors.
Human Oversight: AI should be used as a tool to assist humans, not replace them. Critical decisions should always be made by qualified professionals.
Transparency & Accountability: The algorithms used in these systems must be obvious, and there must be clear lines of accountability when errors occur. Responsible Innovation: Tech companies like OpenAI need to prioritize safety and ethical considerations over rapid deployment and profit.Promoting potentially harmful applications, like AI-driven health advice, is simply unacceptable.

The future of AI is not predetermined. It’s up to us to ensure that this powerful technology is developed and used responsibly, ethically, and in a way that benefits humanity – not puts it at risk.

Stay informed: [Read the full story on judges using AI](https://www.technologyreview.com/2025/08/11/1121460/meet-the-early-adopter-judges-using-ai/?utmsource=thedownload&utmmedium=email&utm