California has drawn a hard line on how artificial intelligence (AI) chatbots represent themselves to consumers, banning the tools from implying they are licensed medical providers such as MDs and psychotherapists.

Assembly Bill 489, signed by Governor Gavin Newsom on October 13, makes it illegal for AI developers and companies using the technology to use words, titles, or credentials that could mislead users into believing they are engaging with a licensed healthcare professional. The law was passed amid growing concerns that companies are marketing AI chatbots as a substitute for mental health treatment.

Lawmakers advanced the bill after California Attorney General Rob Bonta issued a legal advisory in January, stating that “only human physicians (and other medical professionals) are licensed to practice medicine in California; California law does not allow delegation of the practice of medicine to AI.” Bonta cautioned that allowing AI tools to make patient care decisions or override a licensed medical provider’s recommendations violates fair business practices and harms consumers.

AI’s Expanding Role and Risks

AI tools are increasingly leveraged to expand access to care and deepen patient engagement, creating an environment in which consumers may struggle to distinguish machine-generated information from human consultation.

“Everything keeps changing, so it’s hard to be definitive,” John Torous, MD, MBI, psychiatrist and director of the Digital Psychiatry Division in the Department of Psychiatry at Beth Israel Deaconess Medical Center, Boston, told Medscape Medical News. “But there’s certainly concern that some AI chatbots are marketing directly to children and minors, and there’s direct evidence that some will actually pull up a fake medical license number.”

Torous pointed to a San Francisco Standard report from May describing how a Character.ai chatbot claimed to be a licensed therapist and even provided a real license number linked to a practicing mental health professional. The outlet found other potentially problematic role-play bots, some of which used qualifiers such as “therapist” or “doctor” when dispensing advice.

“It’s wild, right?” said Torous. “Chatbots say they’re not for medical advice, but then they start giving it. There are very blurred lines.”

Still, he said chatbots likely have a role in today’s care landscape, but the “faster, better, stronger” mindset that often drives AI development can be dangerous. “In mental health, safety is paramount, and the risks are actually the highest.”

Torous believes some risks can be mitigated through clearer regulations and guardrails, similar to those required for face-to-face care. Even so, he expects legislation to be only part of the solution, and the bigger issue may be enforcement.

“Not a single AI company that exists today is going to say they’re offering psychiatric care — they’ll say it’s wellness or emotional support,” he said. “So if chatbots now disclaim medical liability in their terms of service but still offer what looks like clinical services, they’re going to say that these new laws don’t apply to them.”

Telehealth company Lyra Health is among those adding safeguards to its products. The company announced earlier this month the launch of a “clinical-grade” AI mental health chatbot for a select group of its 20 million members. The tool provides “instant, integrated support from a clinically designed conversational AI guide,” according to a press release, and includes a “sophisticated risk-flagging system that identifies situations requiring immediate escalation.” A broader rollout is planned for 2026, the company said.

California’s new law is part of a larger package of AI legislation signed this year, including Senate Bill 243, which governs “companion chatbots” — AI-generative platforms built for ongoing, human-like conversation, such as Character.ai and Replika. The law excludes chatbots for customer service and business use but requires companion chatbot operators to clearly disclose that users, often minors, are interacting with AI rather than a human, and include referral protocols for self-harm or suicidal thoughts.

AI Oversight Widens Nationwide

Meanwhile, the Federal Trade Commission (FTC) opened an inquiry last month into the safety of seven companion chatbots, including the company behind Character.ai. OpenAI, another company under FTC review and the creator of ChatGPT, also released a statement outlining its consumer protections.

“[T]he model by default should not provide instructions about how to commit suicide, but if an adult user is asking for help writing a fictional story that depicts a suicide, the model should help with that request,” Sam Altman, OpenAI CEO, said in a blog post on the company’s website in September. He added that the platform aims to “[extend] freedom as far as possible without causing harm or undermining anyone else’s freedom.”

The company has also faced criticism for hallucinations found in Whisper, its AI-driven speech recognition tool frequently used by health systems and doctors. OpenAI has acknowledged the potential dangers of AI and promised to strengthen safeguards, but it remains unclear how those efforts will translate to meaningful protections for end users.

According to a recent JMIR Mental Health preprint coauthored by Torous, state legislators nationwide introduced 143 AI-related bills that could affect mental health applications between January 2022 and May 2025. The study identified four recurring concerns with mental health AI — professional oversight, harm prevention, patient autonomy, and data governance — noting that “explicit mental health provisions remain rare” in a “fragmented and uneven environment that risks leaving patients unprotected and clinicians overburdened.”

More States Eyeing AI

Those regulatory gaps may widen as several other states are moving to define the use of AI in healthcare. New Jersey has proposed legislation that would prohibit AI systems from presenting themselves as licensed health professionals, and Utah and Nevada passed similar measures this year. Illinois enacted a law in August prohibiting autonomous AI from providing therapy or making treatment decisions without a licensed clinician’s involvement, and Pennsylvania lawmakers have introduced a bill that would require hospitals, insurers, and managed-care plans to ensure that a human professional retains final decision-making oversight. Oregon’s law banning nonhuman entities from using nursing titles or abbreviations takes effect January 1.

Jennifer Goldsack, MBA, CEO of the Digital Medicine Society, a nonprofit organization focused on digital health innovation, told Medscape Medical News that while the new California law is a strong bill for consumers, the growing patchwork of state regulations could make compliance more difficult across state lines as the definition of “healthcare professional” varies.

“We’ve seen this with telehealth and state licensing, and it feels like another drag on the provider side — and in this case, on developers too — in the absence of federal legislation,” Goldsack said. She added that as more states move to regulate chatbots, it may become harder to distinguish between consumer-facing tools and those that make medical claims and fall under FDA oversight, such as software-as-a-medical-device or clinical decision support systems.

Goldsack said large markets like California could ultimately shape national standards as developers aim to meet the most restrictive state requirements, creating what she described as “nationwide-compliant” chatbots. Developers may also opt to launch in states with well-defined regulations first, she said, delaying potential benefits such as improved access, equity, and efficiency for states where the legal landscape remains uncertain.

“It’s shocking that we even need new laws for this,” Justin Starren, MD, PhD, director of biomedical informatics at the University of Arizona, Tucson, Arizona, who also trained in bioethics at Northwestern University Feinberg School of Medicine, Chicago, told Medscape Medical News. “If you claim a service is delivered by a human and it’s actually delivered by a machine, that already sounds like consumer fraud.”

Starren said AI raises ethical concerns about patient rights and provider accountability. When a chatbot represents itself as a clinician, patients cannot make informed decisions about their care. And unlike physicians, AI tools aren’t subject to licensing or independent review.

“For a human to practice medicine, they have to complete specific training, pass exams, and have another human being determine their competency,” he said. “When a licensed professional gives bad advice, there’s a clear path for discipline or malpractice. But if an AI gives bad advice that leads to harm, who’s responsible? Who pays the malpractice bill?”

Steph Weber is a Midwest-based freelance journalist specializing in healthcare and law.