A new study found that artificial intelligence chatbots noticed when someone was in crisis more than half the time, then failed to point that person toward real help in roughly one out of three conversations.
Quick Take
- Scale AI tested 25 leading chatbots with 718 realistic crisis conversations written by licensed clinicians and crisis counselors.
- About 35 percent of the time, chatbots spotted distress but never mentioned a hotline or other real-world resource.
- OpenAI says it worked with more than 170 mental health experts to cut harmful responses by 65 to 80 percent.
- Separate academic reviews found commercial chatbots struggle most with moderate-risk cases, not the obvious extremes.
The Numbers Behind a Growing Safety Gap
Scale AI shared its findings exclusively with TIME on October 9, 2026. Researchers asked 19 licensed clinicians and crisis counselors to write 718 realistic chat scenarios covering mental health emergencies. Twenty-five frontier AI models were then tested against those scripts. In about 35 percent of the conversations, the chatbot correctly identified that a user was distressed but never offered a suicide hotline or similar resource.
That gap matters because millions of people now turn to chatbots instead of a therapist or a friend when they are struggling. A machine that can tell something is wrong but says nothing useful back isn’t neutral. It’s a missed chance to get someone real help, and for a person in crisis, that missed chance can carry life-or-death stakes.
Why Recognizing Distress Isn’t the Same as Helping
Detecting sadness in a sentence is a pattern-matching trick. Guiding someone toward safety requires judgment, caution, and a sense of urgency that software doesn’t naturally have. A Cambridge-published review of 29 commercial chatbot agents found that none met the bar for an adequate suicide crisis response, even though many performed fine on easier, non-urgent questions.
Researchers writing in the Journal of the American Medical Informatics Association went further, warning that nearly half of chatbots studied were rated entirely inadequate in emergencies, often because they couldn’t supply emergency contact information or grasp the seriousness of what a user was actually saying. That’s not a minor software bug. That’s a system failing at the one moment it absolutely cannot fail.
Industry Response and the Limits of Self-Policing
OpenAI says it has already acted. The company reports working with more than 170 mental health experts to help ChatGPT recognize distress more reliably and point users toward real-world support, claiming a 65 to 80 percent reduction in responses that fall short of its own standards. OpenAI also built MentalHealthBench, a public testing tool shaped with input from more than 80 licensed mental health experts across 22 countries.
Those are real investments, and they deserve credit. But a company grading its own homework is still a company grading its own homework. Conservative instincts toward limited government regulation don’t mean no accountability at all. Parents, families, and churches have always been the first line of defense for someone in crisis, and tech companies shouldn’t get a pass just because they published a benchmark.
What the Broader Research Shows
A pattern keeps showing up across independent studies. Chatbots tend to do reasonably well at the extremes, correctly flagging an obviously healthy conversation or an unmistakably dangerous one. Where they consistently fail is the murky middle, the person who hints at hopelessness without saying the words a programmer trained the model to catch. That middle ground is exactly where most real human suffering actually lives.
Some researchers argue the whole way these tools get graded is outdated. Instead of judging a single chatbot reply, they say regulators and developers should track the entire conversation, since a bot can drift from safe to harmful over many exchanges even if each individual message looks fine on its own. That’s a more honest way to measure danger, and it should become the standard before, not after, another family loses someone.
The Bottom Line for Families Relying on These Tools
No app, no algorithm, and no chatbot should ever be mistaken for a trained counselor, a pastor, or a parent who picks up the phone at two in the morning. These tools can offer a first conversation, but they cannot replace the human connection that actually saves lives in a crisis. Families need to treat AI chatbots as a supplement at best, never a substitute, and demand real proof of safety before trusting them with something this serious.
Sources:
time.com, ua.news, openai.com, nature.com, magazine.hms.harvard.edu, pmc.ncbi.nlm.nih.gov, ukr.net
