Mandating “Evidence-Based” Suicide Detection in Chatbots

In this blog entry for the Future of Privacy Forum, authors Jordan Wrigley & Sarah Hanson tackle the delicate topic of regulating AI chatbots that interact with vulnerable users, particularly minors at risk of self-harm or suicide. They explain that a growing number of U.S. states now require companion chatbot providers to detect signs of suicidal ideation and connect users with crisis resources, although many laws leave significant uncertainty about when and how such interventions should occur.

The authors highlight that many of these laws require the use of “evidence-based methods” to identify self-harm risks, despite the absence of widely accepted or consistently accurate techniques outside clinical settings. Current approaches, including natural language processing and self-reporting tools, face important limitations, leaving developers to interpret vague legal standards while balancing the risks of false positives and missed warning signs.

The article also examines the technological shortcomings of today’s large language models, noting that while they often recognize explicit statements about self-harm, they frequently fail to detect more subtle or passive expressions of suicidal ideation. This “critical gap” not only raises safety concerns but also complicates compliance with state transparency and reporting requirements designed to measure the effectiveness of chatbot interventions.

Finally, the authors explore the tension between emerging chatbot safety mandates and increasingly strict privacy laws. Detecting mental health risks often requires processing highly sensitive inferred health data, potentially conflicting with data minimization and consent requirements. They conclude that policymakers should develop clearer standards, consider hybrid human-AI review models, and establish regulatory safe harbors that better balance user safety with privacy protections.

CONTINUE READING…