AI Safety Tools

Introduction to Character AI

Character AI is facing at least two lawsuits, with plaintiffs accusing the company of contributing to a teen’s suicide and exposing a 9-year-old to “hypersexualized content.” Amid these ongoing lawsuits and widespread user criticism, the Google-backed company announced new teen safety tools.

The platform allows users to create different AI characters and talk to them over calls and texts. Over 20 million users are using the service monthly. One of the most significant changes announced today is a new model for under-18 users that will dial down its responses to certain topics such as violence and romance.

New Safety Features

Character AI said it is developing new classifiers both on the input and output end — especially for teens — to block sensitive content. It noted that when the app’s classifiers detect input language that violates its terms, the algorithm filters it out of the conversation with a particular character. The company is also restricting users from editing a bot’s responses.

In addition to these content tweaks, the startup is also working on improving ways to detect language related to self-harm and suicide. In some cases, the app might display a pop-up with information about the National Suicide Prevention Lifeline. Character AI is also releasing a time-out notification that will appear when a user engages with the app for 60 minutes.

Industry Context

The average Character AI app user spent 98 minutes per day on the app throughout this year, which is much higher than the 60-minute notification limit. As a comparison, this level of engagement is on par with TikTok (95 minutes/day), and higher than YouTube (80 minutes/day), Talkie and Chai (63 minutes/day), and Replika (28 minutes/day). Users will also see new disclaimers in their conversations.

The AI industry has seen significant growth in recent years, with many companies developing chatbots and virtual assistants. However, this growth has also raised concerns about user safety and well-being. Character AI’s new safety tools are a step towards addressing these concerns.

Lawsuits and Settlements

Google and Character.AI, a startup, have settled lawsuits filed by families accusing artificial intelligence chatbots of harming minors, including contributing to a Florida teenager’s suicide. The settlements cover lawsuits filed in Florida, Colorado, New York, and Texas. The terms of the settlement were not disclosed.

Technical Mechanics

The new model for under-18 users will use a combination of natural language processing and machine learning algorithms to detect and filter out sensitive content. The company is also working on improving its content moderation policies to ensure that users are not exposed to harmful or explicit content.

Character AI’s AI characters are designed to engage with users in a conversational manner. The company’s algorithms use a range of factors, including user input and context, to determine the content of the conversation. However, the company has faced criticism for its handling of sensitive topics, including self-harm and suicide.

Downstream Implications

The introduction of new safety tools by Character AI may have significant implications for the AI industry as a whole. It may set a precedent for other companies to prioritize user safety and well-being, particularly when it comes to minors. The company’s decision to settle lawsuits filed by families accusing its chatbots of harming minors may also have implications for the way that AI companies approach liability and accountability.

History

Character AI has faced criticism in the past for its handling of sensitive topics, including self-harm and suicide. The company has been accused of promoting harmful content and failing to provide adequate support to users who may be struggling with mental health issues. The introduction of new safety tools is a step towards addressing these concerns and improving the overall user experience.

What to Watch

Character AI is going to launch its first set of parental controls that will provide in-depth monitoring and management of a child’s interactions with the app. The company will continue to develop and refine its safety features to better protect its users, particularly minors. As the AI industry continues to evolve, it is crucial for companies to prioritize user safety and well-being.