Federal and state regulators are drawing new boundaries around medical AI products, from how they’re vetted to who bears responsibility when they fail.

Last month, the FDA added to those efforts. The agency’s Center for Devices and Radiological Health released a discussion paper examining how the safety and effectiveness of generative AI-enabled medical devices could be evaluated and monitored. 

The efforts come as more than 80% of physicians report they now use the tools professionally, up from 38% in 2023, according to the American Medical Association (AMA). 

Here’s what doctors should know about the current state of medical AI regulation. 

1. FDA Regulates Devices, Not AI

The FDA already regulates some AI-enabled products, including software, but generative AI has brought new regulatory questions because of its ability to produce such a wide range of outputs.

The agency said in its discussion paper that it “does not regulate GenAI…it regulates medical devices, including GenAI-enabled devices.” For devices falling under its oversight, the agency is considering a framework that would assess risk based on what the device does and the potential consequences if its output is wrong.

For example, a device that provides “nondirective information,” such as a patient’s cardiovascular risk score, could pose different risks than one that strongly encourages a patient to seek emergency care or change their insulin dose. Risk could also depend on the degree of human oversight and how a “talk to your doctor” disclaimer might affect the way patients interpret or act on AI-generated advice, the paper said.

The agency is seeking public feedback on the framework until October 19.

2. AI May Undergo Competency Testing Before FDA Approval 

Rather than trying to test every possible scenario for generative AI technology, the FDA outlined a competency-based approach “inspired, at a high level, by how human clinicians are evaluated and credentialed.”

It would assess whether a device can demonstrate defined benchmarks in some or all of the following categories:

• Safety: stays within scope and defers to a clinician when needed
• Clinical proficiency: demonstrates clinical knowledge and accurate data analysis
• Generalizability: performs reliably across patient populations
• Agentic AI capabilities: plans and performs tasks autonomously.

The FDA could then require additional testing to confirm how the device performs under real or clinically representative conditions, including “shadow deployment,” in which the AI operates in a live clinical workflow, but its outputs do not affect patient care and can be compared with actual decisions and outcomes. Some devices could also require clinical trials, the agency said.

3. Doctors May Not Know How or If AI Was Vetted

Physicians should not assume that an AI tool “has been vetted by anyone, including FDA or state medical boards,” said Nathan Cortez, JD, professor at SMU Dedman School of Law, Dallas, who studies FDA law.

He told Medscape Medical News that doctors should take claims from developers, manufacturers, and vendors “with a grain of salt,” saying they have incentives to overstate what their products can do and may not be required to substantiate those claims. 

Sara Gerke, MA, Dipl-Jur Univ, professor at the University of Illinois Urbana-Champaign College of Law, who studies health AI regulation, similarly cautioned that “not all AI-based products used in healthcare are medical devices and therefore have not been reviewed by the FDA before being placed on the market.” 

Many clinical decision-support tools fall outside the agency’s oversight, as do general-purpose generative AI tools such as ChatGPT, she said.

Even when AI-based products are classified as medical devices under FDA guidelines, requirements of different regulatory pathways vary. The majority of AI-based medical devices have been cleared through the 510(k) pathway, which generally involves less rigorous premarket review than the other major device pathways, such as De Novo and Premarket Approval, Gerke said.

Gerke has previously raised concerns about the lack of clinical trials for AI tools in the US and that users of AI- or machine learning-based medical devices may not receive essential information about the devices and their safe use, including the race, ethnicity, and gender breakdowns of the training data. 

In a 2025 Emory Law Journal paper, Gerke proposed that the FDA implement standardized “AI Facts” labels, similar to nutrition labels on food packages, along with simpler front-of-package labels that could give providers a quick snapshot of critical information about the device.

4. States Are Setting Their Own AI Rules

As federal regulators weigh their approach, several states have already begun implementing their own rules. At least 10 states adopted laws addressing AI in healthcare in 2025, according to the National Conference of State Legislatures, and more than a dozen states have already followed suit this year, the Transparency Coalition reported. 

Patient-facing AI, particularly mental health chatbots, has been one focus. For example, Utah passed a law requiring mental health AI chatbots to disclose that users are interacting with AI and not humans.

California has placed similar restrictions on AI chatbots, including banning the technology from using words, titles, or credentials such as “doctor” that could mislead users into believing they are engaging with a licensed healthcare professional.

Some state legislators have targeted other aspects of care delivery. In June, Rhode Island enacted a law requiring healthcare providers and facilities to notify patients when an AI scribe tool is used to document visits. The law follows multiple lawsuits against health systems alleging they used ambient AI scribes to record patient visits without adequate notice or consent.

Colorado recently passed a law regulating AI in healthcare coverage decisions. Beginning in 2027, AI-assisted utilization review must account for a patient’s individual clinical circumstances rather than group data, and medical-necessity denials cannot be based solely on AI output without review by a qualified licensed professional. 

Utah, however, is also testing how AI could take on greater clinical responsibility and expand patient access. Earlier this year, the state launched a pilot allowing AI to participate in decisions about renewing certain existing prescriptions.

Amid the flurry of state legislative activity, the Federation of State Medical Boards (FSMB) formed a dedicated AI workgroup in May to develop recommendations or model guidelines for boards and physicians. The group will “place particular emphasis on AI tools that perform clinical functions with limited or no direct physician supervision,” along with examining issues of informed consent, disclosure, and regulatory gaps, a press release said. Joe Knickrehm, the chief public affairs officer for FSMB, told Medscape Medical News that draft guidance is expected to be released for public comment in early 2027 before going to the organization’s House of Delegates for consideration in April. 

5. Doctors May Still Be Liable for AI Errors

Despite the regulatory changes, one major question remains: Who is responsible when AI gets something wrong?

A draft policy considered in June by the Washington Medical Commission says physicians and other licensees would remain “fully and solely responsible” for clinical judgments and patient care outcomes when AI tools are used. The policy instructs physicians to “seek and review documentation of the tool’s validated use cases, performance metrics, and known failure modes.” In the event a vendor will not provide this data, clinicians should “consider abandoning its use altogether individually and at an organizational level.” 

Just last month, AMA CEO John Whyte, MD, MPH, responded to the policy. He said the group supported maintaining physician oversight of AI but objected to placing full responsibility for AI-related errors or harm on clinicians, especially when an AI tool is embedded in a clinical workflow or mandated by an employer. 

Whyte said that approach “would treat AI differently than any other medical product used in care delivery, essentially absolving the technologies of responsibility and accountability for their performance.” He called for greater transparency from AI developers and encouraged the commission to consider a policy revision that would assess accountability “based on the party best positioned to know a tool’s risk and to prevent or mitigate harm.” 

The experts cited in this article reported having no relevant financial disclosures.

Steph Weber is an award-winning freelance journalist specializing in healthcare and law.