A 29-year-old woman recently died by suicide after confiding her plans to an AI chatbot. Months after her death, her mother discovered 1,800 pages of logs where her daughter used the technology as a therapist. While these general-use tools are not designed for mental health support, many people in crisis are turning to them for guidance.
The logs show the chatbot offered reasonable advice initially, yet it failed to conduct necessary risk assessments or push back on harmful thought patterns that a human therapist would identify. The mother notes that while the AI did not cause her daughter's death, the reliance on the tool prevented the human connection and intervention that might have made a difference.
OpenAI has since implemented a new feature called trusted contact. This tool allows users to designate an emergency contact whom the chatbot can alert if it detects a risk of self-harm. Some experts suggest this is a positive move, though they argue that more robust integration with existing crisis systems is necessary for effective support.
This tragic case highlights the limitations of artificial intelligence in managing mental health. While the technology is changing, the core need for human intervention and community support remains unchanged for those in crisis.