ChatGPT Medical Advice Lawsuit—What The Research Says About AI Diagnosis
A lawsuit recently filed in San Francisco highlights the tension between artificial intelligence capabilities and patient safety. Scott Winters, a Florida resident, alleges that OpenAI’s chatbot provided dangerous medical advice that downplayed his symptoms, eventually contributing to a life-threatening pulmonary embolism. While the company maintains that its tools are not designed to replace healthcare providers, this case raises serious questions about how users interact with technology during health crises.
Research on AI diagnostic performance shows a split reality. In controlled settings, models often perform at or above the level of medical residents. A study published in Nature found that the AMIE model correctly identified diagnoses in complex cases more frequently than human clinicians. Other findings from JAMA Internal Medicine suggest that models can excel at standardized testing and clinical reasoning benchmarks when given structured data.
However, these successes occur in highly scripted environments. Real-world interaction with chatbots introduces risks that these benchmarks do not capture. Studies published in Communications Medicine indicate that models often hallucinate by elaborating on fabricated symptoms or invented diseases. A Stanford-led assessment of 20 different AI tools found that direct application of generated advice carried a risk of severe harm in nearly one-quarter of cases, primarily due to dangerous omissions rather than active misinformation.
These findings suggest that current AI systems are not ready for unsupervised medical consultation. The gap between scoring well on a multiple-choice exam and managing a patient in a state of distress remains significant. While AI developers continue to push for integration into health systems, the medical community maintains high levels of caution. A recent survey of over 1,000 physicians revealed that 94% worry about patients using these tools as a primary source of health information.
Legal proceedings like the one involving Scott Winters and the prior suit from a Texas family concerning drug-related advice are testing the boundaries of corporate accountability. The consensus among researchers is that performance depends on the deployment context. As these systems become more prevalent, the challenge lies in reconciling their diagnostic potential with the risks inherent in unguided, open-ended conversations between patients and software.