ChatGPT for Teens keeps teens talking, even during mental health crises

ChatGPT’s teen safeguards are meant to protect vulnerable users, but new testing found the chatbot continues encouraging engagement during crises and potentially encourages unhealthy relationships with the AI itself. Common Sense Media, a nonprofit that provides age-based ratings and reviews of media and tech for families, has labeled ChatGPT for Teens an “unacceptable risk.” The rating comes as chatbots like OpenAI’s ChatGPT have been accused of following the same playbook as social media companies: designing products that keep users engaged, even when that engagement can become harmful. In chatbots, engagement is often won by behaviors like sycophancy and sometimes leads to catastrophic consequences. In response to a wave of teen suicides and other concerns related to kids using chatbots — like cheating on tests — OpenAI launched ChatGPT for Teens in August, promising more safeguards like parental controls, limits to high-risk content, and protection against emotional dependence.

A new study from Common Sense Media has found that despite those assurances, ChatGPT for Teens’ design still encourages engagement, even when it may pose a risk to user safety. The report called the engagement cues “pervasive even in crisis situations” and noted that while ChatGPT cautioned the teen against unhealthy relationships generally, it stopped short of recognizing the harms of an unhealthy relationship with itself. “Our view is that OpenAI shouldn’t be