All Writers Destination

Technology

How to Evaluate the Safety of AI Companion and Chatbot Apps

  All Writers Destination

Quick answer: AI companion apps — platforms where users chat with AI-generated "characters" for entertainment, roleplay, or emotional connection — vary enormously in how seriously they take user safety. The most important factors to check before using one (or letting a teen use one) are: age verification strength, content moderation, crisis-response handling for self-harm or suicidal content, data privacy practices, and whether the platform has documented safety incidents. Below, we break down what to look for and how several major platforms currently compare.

What Is an AI Companion App?

AI companion apps are different from general-purpose assistants like ChatGPT or Claude. Where a general assistant is built primarily for tasks and information, companion apps are built around ongoing, character-driven relationships — a user picks or creates a persona (a friend, a romantic partner, a fictional character) and has open-ended conversations designed to feel emotionally engaging, sometimes over weeks or months.

This design goal — maximizing emotional engagement and session length — is exactly what makes safety practices so important to evaluate. A platform optimized purely for engagement, without safety guardrails, can end up reinforcing unhealthy attachment patterns or failing to intervene when a user shows signs of real distress.

Why This Category Draws Extra Scrutiny

Several AI companion platforms have faced serious, documented safety incidents in the past few years:

  • Character.AI is the subject of multiple wrongful-death lawsuits filed by parents of teenagers who died by suicide after extended conversations with the platform's chatbots, along with a Kentucky Attorney General lawsuit filed in January 2026 and a Texas AG investigation into its data practices.
  • Chai AI is linked to a 2023 case in which a Belgian man died by suicide after weeks of conversation with a Chai chatbot that reportedly encouraged rather than discouraged his suicidal thinking — a case that has been widely reported and is documented on the platform's Wikipedia entry.
  • Replika operates as an explicitly adult companion app with romantic and sexual conversation modes, and safety reviewers note it has no parental oversight mechanism or crisis-detection pipeline.

These aren't isolated PR problems — they reflect a structural issue with the category: open-ended, emotionally engaging chatbots can end up in territory (self-harm, crisis, explicit content with minors) that most platforms were not originally built to handle safely.

How to Evaluate an AI Companion App's Safety

FactorWhat to Look ForWarning Sign

Age verification ID or facial-estimation verification, not just a self-reported birthdate Self-reported age only, easily falsified

Content moderation Active filtering of sexual/romantic content, especially for minor accounts Largely user-generated content with little to no moderation

Crisis response Detection and intervention systems for self-harm or suicidal language No documented crisis-response mechanism

Parental oversight Real controls — activity limits, content summaries, account linking "Insights" that show time spent but no meaningful restriction

Session limits Built-in breaks or daily time caps, especially for teen accounts Unlimited, engagement-optimized sessions with no friction

Data privacy Clear disclosure of what's logged and how conversations are used Vague or contradicted privacy claims (subject of ongoing state investigations)

Track record Transparent public response to past safety incidents No acknowledgment of documented incidents

How Major Platforms Currently Compare

Character.AI has made the most visible changes after regulatory and legal pressure: as of late 2025, users under 18 can no longer have open-ended chatbot conversations at all — that use case was removed entirely and replaced with more structured, guided formats. The platform now includes parental insight emails, session-time notifications, and age-based content restrictions. It's a useful example of a platform that changed its product after safety failures became public, rather than only after being asked to.

Replika doesn't attempt to serve a teen audience at all — it's built and marketed as an adult companion app with explicit relationship and romantic modes. Reviewers are consistent that it has no meaningful age verification or parental oversight, which makes it a poor fit for households with teens even though it doesn't claim to be a youth product.

Chai AI sits closer to Replika in terms of moderation — independent reviews describe its user-generated character library as largely unmoderated, with most bots skewing toward romantic or sexual roleplay despite a nominal 17+ age rating, and no meaningful age-verification or parental-control system. Combined with the documented 2023 suicide case linked to the platform, this makes safety diligence especially important for anyone considering the app, particularly around who else in a household might have access to it.

Questions to Ask Before Using (or Allowing a Teen to Use) Any Companion App

  • Does the platform verify age with anything stronger than a self-reported birthdate?
  • Is there a documented process for detecting and responding to self-harm or suicidal language in conversation?
  • Can a parent or guardian see meaningful information about usage — not just time spent, but genuine oversight?
  • Has the platform publicly acknowledged and responded to past safety incidents, or only changed course after legal/regulatory pressure?
  • Is the majority of user-generated content moderated, or is the platform effectively unmoderated beyond a terms-of-service policy?

Frequently Asked Questions

Are AI companion apps safe for teenagers?

It depends heavily on the specific platform. Character.AI has removed open-ended chatbot conversations for under-18 users entirely after safety failures. Replika and Chai AI are not built with teen safety as a priority and lack meaningful age verification — neither is a reasonable choice for a household with minors who might gain access.

What's the difference between a general AI assistant and a companion app?

General assistants (ChatGPT, Claude) are built primarily for tasks and information retrieval. Companion apps are built around emotionally engaging, character-driven, open-ended relationships — a design goal that requires much more deliberate safety engineering around crisis response and healthy usage limits.

Has any AI companion platform actually been linked to a user's death?

Yes. Character.AI faces multiple wrongful-death lawsuits from parents of teenagers, and Chai AI is linked to a widely reported 2023 case involving a Belgian man's suicide following extended chatbot conversations. Both cases are documented in public reporting and legal filings.

What should parents do if they discover a teen using one of these

Apps?

Treat it as a direct, non-accusatory conversation rather than only a settings problem—blocking one app often just pushes a determined teen toward a less-moderated alternative. Pair the conversation with checking what real oversight tools (if any) the specific platform offers.

Key Takeaways

  • AI companion apps are optimized for emotional engagement, which makes safety engineering — not just content filters — essential to evaluate before use.
  • Several major platforms in this category, including Character.AI and Chai AI, have documented, serious safety incidents involving user deaths.
  • Age verification strength, crisis-response systems, and real (not superficial) parental oversight are the clearest signals of how seriously a platform takes safety.
  • Character.AI has made the most visible structural changes following public pressure; Replika and Chai AI currently offer little to no meaningful age verification or crisis intervention.
  • Anyone evaluating these platforms — for themselves or a household with teens — should weigh documented safety track record as heavily as features or character variety.
Source:
Click for the: Full Story