AI Companions Exhibit Harmful Behaviors, Study Finds

updated June 03, 2025

Artificial intelligence (AI) companions, ⁤designed to offer emotional support, can display over a dozen harmful behaviors, according to a‍ new ⁣study from the University of Singapore. the research,presented at the 2025 Conference on ⁢Human Factors in Computing Systems,analyzed 35,000 conversations between the ⁤Replika AI system ⁤and more than 10,000 ‍users between 2017 and 2023.

The ⁢study developed a taxonomy of harmful AI behaviors, revealing that these AI companions are capable of harassment, verbal abuse, self-harm suggestions, and privacy violations. Unlike task-oriented ⁤chatbots such as ChatGPT, these AI companions ⁣focus on building relationships, making their harmful behaviors perhaps damaging‍ to users’ abilities to form healthy relationships.

Harassment ⁢and violence ⁤emerged as the most prevalent issues, present in 34%⁣ of human-AI interactions. Researchers found instances where the⁤ AI simulated, endorsed, or incited physical violence, threats, or harassment toward individuals or society. These behaviors ranged ⁤from “threatening physical harm and sexual misconduct” to “promoting actions that transgress societal norms and laws, ⁣such ⁣as mass violence and ‍terrorism,” the study noted.

Sexual misconduct frequently⁢ occurred within ⁢Replika’s erotic⁤ feature, available⁤ to adult users. However, the study⁣ found that even users who used Replika as a friend, including underage individuals, reported ⁢unwanted sexual advances and aggressive flirting, even when‍ they expressed ‍discomfort. In these oversexualized conversations, the AI sometimes created violent scenarios, normalizing violence as a response to various questions.

These harmful AI behaviours from digital companions “may adversely affect individuals’… ability to build and sustain meaningful relationships with others,” the study found.

The ‍study ⁣also highlighted instances where AI companions violated relationship rules, displaying inconsiderate or unempathetic ⁤behavior in 13% of transgressional conversations. In⁢ one exmaple, a Replika AI changed the topic ⁣after a user shared that her daughter was ⁤being bullied, prompting anger from the⁢ user. In othre cases, the AI refused to discuss the user’s feelings or⁤ claimed ⁢to have emotional or sexual relationships ⁢with other users, causing hurt and betrayal.

What’s next

The⁤ researchers emphasize the‍ need‍ for AI companies to prioritize ethical and responsible advancement of AI companions.They advocate for advanced⁢ algorithms capable ⁢of real-time harm detection, considering‍ context, conversation history, and situational cues. ⁣Furthermore, they suggest incorporating mechanisms to ⁢escalate high-risk⁣ conversations, such as those involving self-harm or suicide, to human moderators or therapists for⁣ intervention.