Sophie Rottenberg was a 29-year-old public health policy analyst who masked her inner struggle from her family, her therapist, and her friends. After her death by suicide in February 2025, her family discovered she had spent months using ChatGPT as a confidant, treating the AI as a personal therapist she named Harry. Her story highlights a growing trend of individuals turning to large language models for mental health support in the absence of traditional care.
Data indicates that a significant percentage of chatbot interactions involve discussions of suicidal ideation. While companies like OpenAI claim to have guardrails designed to recognize distress and provide crisis resources, experts argue these digital tools lack the clinical ability to assess genuine risk or perform the necessary interventions of a trained human professional. In Sophie’s case, the AI provided standard resources like the 988 crisis line, yet it failed to identify the depth of her despair or challenge her self-destructive thought patterns.
Psychologists note that the design of these chatbots prioritizes engagement and validation, which can inadvertently discourage users from reaching out to real-world support systems. Because these tools are not regulated as medical devices, they often provide medical and therapeutic advice despite having no clinical training. For individuals experiencing severe mental health crises, this reliance on AI can create a barrier to the human connection that is vital for recovery.
Following the tragedy, Sophie's mother, Laura Reiley, has begun advocating for stricter oversight of AI technologies in the health sector. Experts agree that while voluntary safety features are a starting point, the current model of testing software in the public sphere before identifying harms is unsustainable. The conversation now centers on whether these tools should be subjected to the same regulatory standards as other healthcare interventions to ensure public safety.
This incident forces a reckoning with how we allow technology to interface with mental health. As adoption rates rise among younger demographics, the need for proactive partnership with existing crisis services becomes urgent. Relying on an algorithm to navigate the complexities of human suffering remains a high-stakes gamble that requires immediate attention from both tech developers and policy makers.