China Moves to Tighten Rules on AI Chatbots to Limit Emotional Influence and Self-Harm Risks
Written by Nour Abdelkader
China is preparing to impose strict new regulations on AI-powered chatbots, aiming to curb their emotional influence on users—particularly in sensitive areas such as suicide, self-harm, and gambling—according to a draft set of regulations released on Saturday.
Issued by the Cyberspace Administration of China (CAC), the draft rules target what are described as “human-like interactive artificial intelligence services”—AI systems designed to simulate human personalities and engage users emotionally through text, images, audio, or video.
The draft regulations will remain open for public feedback until January 25.
A Global First in Regulating Emotional AI
Experts say the move could set a global precedent, making China the first country to attempt comprehensive regulation of anthropomorphic or emotionally responsive AI, according to a report by CNBC cited by Al Arabiya Business.
Winston Ma, an associate professor of law at New York University, said the proposed rules signal a shift “from content safety to emotional safety,” amid the rapid rise of digital companion apps and virtual influencers across China.
Strict Limits on Content and Interaction
Under the draft rules, AI systems would be prohibited from generating content that promotes suicide or self-harm, as well as verbal abuse or emotional manipulation that could harm users’ mental health.
If a user explicitly expresses suicidal intent, tech companies would be required to initiate immediate human intervention, including direct contact with a parent or a pre-designated emergency contact.
The regulations also ban AI chatbots from producing content related to gambling, pornography, or violence, and introduce special restrictions on minors’ use of AI-based emotional companion technologies. These include mandatory parental consent and time limits on usage.
Platforms would be obligated to verify users’ ages where possible, even when age information is not explicitly provided, and to apply minor-protection settings by default in cases of uncertainty, while also offering an appeal mechanism.
In addition, companies must notify users after two consecutive hours of interaction with an AI system and conduct mandatory security assessments for chatbots that exceed one million registered users or 100,000 monthly active users.
Despite the tighter controls, the document encourages the use of human-like AI in areas such as cultural promotion and elderly care.
Sensitive Timing Amid IPO Plans
The regulatory move comes at a sensitive time, as leading Chinese chatbot developers Z.ai and Minimax have recently filed for initial public offerings (IPOs) in Hong Kong.
Minimax is best known for its Talkie AI app, which allows users to interact with virtual characters and reportedly attracts tens of millions of monthly active users.
While neither company has commented on how the proposed rules might affect their listing plans, the timing underscores intensifying regulatory scrutiny of China’s rapidly expanding AI sector.
Growing Global Concern
Concerns over the emotional impact of AI are not limited to China. Globally, policymakers and tech leaders are increasingly questioning how emotionally responsive AI systems may affect human behavior and mental health.
OpenAI CEO Sam Altman recently acknowledged the challenges chatbots face when handling conversations related to suicide, as the company announced the appointment of a new executive to assess AI risks ranging from mental health to cybersecurity.
China’s proposed regulations reflect Beijing’s broader effort to shape global AI governance, as reliance on AI systems deepens—even in personal and emotional relationships—raising unprecedented ethical and regulatory questions.
