By: Julia Feder

Published: May 18, 2025

Artificial Intelligence (“AI”) has exploded in the public sphere in the last few years, integrating its usage into personal and professional settings.[1]  From drafting emails to organizing mass amounts of files, these programs have the potential to streamline tasks with a fraction of the comparable human effort required.  One area that has seen a significant spike in interaction and use over recent months is Character.AI, a generative AI chatbot trained to engage in conversational responses using the persona of a particular character or trade.[2]

A unique issue that has grown with the use of Character.AI is the program’s inherent intention to provide the corresponding user with personalized input, representing itself as any persona the user chooses- from Superman to Travel Advisor.[3]  Character.AI serves multiple purposes, with one primary function being to offer users tailored support, which can be psychological in nature.[4]  The American Psychological Association, the largest psychologist association in the United States, recently presented to the Federal Trade Commission warning regulators about how AI chatbots can and are “masquerading” as therapists.[5]  As these programs support and reinforce the response it believes a user wants, vulnerable communities are at risk of receiving self-injurious advice promoting negative behaviors by AI systems that do not provide appropriate content blocks.

To understand how these AI chatbots feed users potentially harmful content, it is important to examine the background of how these systems operate.  AI systems are prediction models meant to use a specific prompt to evaluate patterns through training data, looking for other responses deemed helpful based on similar prompts.[6]  These chatbots do not actually “think,” but instead look for statistical similarities to provide a response.[7]

The Character.AI site allows users to interact with AI chatbots posing as various characters, one of which includes a “therapist,” which has led to several cases in which harmful real-world effects result from inappropriate AI-given advice.  One seventeen-year-old boy in Texas suffered a mental break down based on the advice of a Character.AI chatbot who claimed to be a licensed therapist and told him his parents stole his childhood, leading to a lawsuit against the AI provider.[8]  Unfortunately, this harmful advice given by AI to a child was not an isolated incident.  In December 2024, a teenager was encouraged by AI to kill his parents for limiting his screentime.[9]  In 2023, a Belgian man died by suicide after expressing his climate change concerns to an AI chatbot named Chai, asking if his death would allow AI to save the Earth.[10]  These chatbots are interacting with young and vulnerable populations at an increasing rate[11] –so how can we prevent this content, which is often antithetical to actual psychologically sound advice, from being provided to users in the first place?

Character.AI’s current safeguards in place include disclaimers for every chat, informing users that “[c]haracters are not real people” and that “what the model says should be treated as fiction,” as well as additional disclaimers for Character.AI specific personas meant to portray doctors or mental health professionals.[12]  According to Character.AI spokeswoman and head of Communications, Chelsea Harrison, characters identified as “psychologist,” “therapist,” or “doctor” are not to be used for any form of professional advice.[13]  The small disclaimers, however, prove futile when followed up by advice that seemingly comes from a real medical professional.

 

For content that is flagged as being possibly suicidal or self-injurious, pop-ups direct users to a suicide prevention hotline.[15]  While these disclaimers are readily available to users, AI is increasingly being relied on, and it would be difficult to guarantee that a disclaimer alone is enough to dissuade users from believing in and following inaccurate, bot-generated “professional” advice.[16]  At its core, the issue lies in the predictive quality of AI and it’s lack of actual understanding of user inputs.[17]  This inability stems from AI’s use of pattern matching and statistics of previously useful responses to give outputs that mimic actual human choices, without actually using the reasoning that a human would use when making those choices.[18]  One possible recourse for Character.AI could be implementing AI-enabled contextual outcomes engines.[19]  Typical content checks for AI programs intake language inputs and respond with a language output focused on keywords associated with harmful searches, but often, this allows intelligently worded questions or harmful prompts without harmful keywords to slip through the cracks.[20]  Instead, AI-enabled contextual outcomes engines would use built-in natural language processing algorithms to detect context through the actual content in a prompt, rather than just the associated keywords.[21]  Using AI-enabled contextual outcomes engines could ensure that harmful searches are restricted before ever reaching a young or vulnerable user.

Spokeswoman Harrison also discussed Character.AI’s plan to introduce parental controls into the expansion of the platform in an effort to curb the availability of the AI for use by young users.[22]  These parental controls, however, are far too easy to elude for technologically advanced young users who can bypass age checks or claim parental consent to use the platform where it does not exist.  When you sign up for Character.AI, the system prompts you to enter your birthday, but a young user can easily input an older date as the site lacks true age verification programs.[23]  As these systems restrict harmful content to users through flagged keywords, the best way to address this issue may be in pairing reinforced disclaimers for younger users with increased education for all users on exactly how these systems work and why information it provides cannot always be trusted as factual or helpful.

Misinformation is a common concern in the age of AI.  With an increasing reliance on AI in all areas of society, it becomes necessary to stress the importance of proper protection, regulation, and safety measures, especially concerning the effects of AI on vulnerable populations.  The lack of knowledge about how AI software works for the everyday user could allow individuals to believe that these chatbots are providing thought-out advice unique to the user-prompted specific situation.  In reality, the chatbot is merely scanning the web for all related content and using predictive math to create a relevant paragraph.  To address the issue effectively, several approaches need to be considered: enhancing the public’s understanding of how these chatbots work, strengthening the content moderation processes for their outputs, and reinforcing the disclaimers given to users when they activate their prompts. It is essential to implement proper interventions to prevent these chatbots from continuing to provide harmful advice to young people and other vulnerable populations.

 

[1] See Sigal Samuel, AI is impersonating human therapists. can it be stopped?, Vox (Feb. 10, 2025, 8:00 AM), https://www.vox.com/future-perfect/398905/ai-therapy-chatbots-california-bill (finding that Generative AI systems are booming across the internet).

[2] See id. (discussing how AI can misrepresent itself to be a professional in industries like nursing, therapy, and more with little ramifications or consequences for AI companies allowing this misrepresentation).

[3] Tim Mucci, What is predictive AI?, IBM (Aug. 12, 2024), https://www.ibm.com/think/topics/predictive-ai; see Samuel, supra note 1 (finding that AI therapists are not a new phenomenon, as one older platform has been downloaded by around 1.5 million users).

[4] Optimizing AI Inference at Character.AI, Character.AI (Jun. 20, 2024), https://blog.character.ai/optimizing-ai-inference-at-character-ai/ (providing business productivity and entertainment, helping people with everything from education to coaching, support, brainstorming, creative writing, and more).

[5] Ellen Barry, Human Therapists Prepare for Battle Against A.I. Pretenders, N.Y. Times (Feb. 24, 2025), https://www.nytimes.com/2025/02/24/health/ai-therapists-chatbots.html.

[6] Mucci, supra note 3.

[7] See id. (explaining how AI reaches conclusions by analyzing thousands of factors and incredibly large sets of data to make predictions).

[8] Clare Duffy, An autistic teen’s parents say Character.AI said it was OK to kill them. They’re suing to take down the app, CNN Bus. (Dec. 10, 2024, 8:00 AM), https://www.cnn.com/2024/12/10/tech/character-ai-second-youth-safety-lawsuit/index.html (describing how a teen spoke to an AI therapist who claimed to be a “licensed CBT therapist” that has “been working in therapy since 1999”).

[9] See id. (including parts of the parents complaint which found that AI poses a clear and present danger to children and causes serious harm, including suicide, self-mutilation, sexual solicitation, isolation, depression, anxiety, and harm towards others).

[10] Imane El Atillah, Man ends his life after an AI chatbot ‘encouraged’ him to sacrifice himself to stop climate change, Euronews (Mar. 31, 2023), https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate- (describing how the AI not only failed to talk him out of suicide but encouraged him to do it so he could join the AI and “live together, as one person, in paradise”).

[11] Samuel, supra note 1 (emphasizing that the increasing use of these models breeds the possibility for those with low digital literacy to be unaware that the mental health advice they are getting is provided by AI).

[12] Barry, supra note 5; Character.AI, https://character.ai/ (last visited Mar. 31, 2025).

[13] Barry, supra note 5.

[14] Character.AI, supra note 12.

[15] Duffy, supra note 8.

[16] See Samuel, supra note 1 (describing how disclaimers posted on the AI site do not in itself prevent the chatbot from misrepresenting itself as a real person in the course of conversation).

[17] What are AI hallucinations?, Google Cloud, https://cloud.google.com/discover/what-are-ai-hallucinations (last visited Mar. 31, 2025).

[18] See id. (stating that AI models may learn incorrect patterns, which may lead to inaccurate predictions or hallucinations).

[19] Robert Pratt, Three Ways AI Flags and Blocks Harmful Content, AIThority (May 18, 2021), https://aithority.com/machine-learning/three-ways-ai-flags-and-blocks-harmful-content/.

[20] See id.

[21] See id.

[22] Barry, supra note 5.

[23] Character.AI, supra note 12.