Mental-health apps are filling in a critical need — but they have a dark side

How do you make money off people caught in a mental-health emergency?

Loris, a startup that helps companies improve their customer-service conversations, found a way. Founded in 2018, Loris used data generated from text conversations with people in distress to make "empathetic" customer-service software. The source for the data? Crisis Text Line, a nonprofit suicide-prevention hotline and the parent company of Loris.

Crisis Text Line, now in its 10th year of operations, uses artificial intelligence to respond to people experiencing emotional abuse, self-harm, and suicidal thoughts. A Politico investigation last year explained how the help line created Loris to develop and sell AI software that could guide customer-service agents through live chats with customers using proven "de-escalation techniques, emotional-intelligence strategies, and training experience."

At the heart of the arrangement was what Crisis Text Line called the "largest mental health data set in the world" — 219 million messages from more than 6.7 million conversations over text, Facebook Messenger, and WhatsApp.

Exploiting data for profit is par for the course in the modern technology business — search engines, social-media platforms, and streaming apps all gather and monetize the data they gain from users. But commercializing the data from a crisis line is vastly different from mining data on the binge-watching or online-shopping habits of customers. The data used for Loris came from people who were at the lowest point of their lives, when they may not have been able to truly understand and consent to how it would then be used.

After the Politico story sparked outrage, the companies claimed that they used only anonymized data — though research has found that anonymization is not fool-proof. Eventually, the furor became so great that Crisis Text Line severed all ties with Loris and told the company to delete any previously transferred information. But the Loris mess was just one of a growing number of mental-health-data fiascos.