Healthcare organizations spent the first phase of the artificial intelligence (AI) boom asking whether the technology worked. The next phase will be defined by a harder question: Who is responsible when it does not?
AI is moving deeper into clinical decisions, patient communications, claims administration and health-data exchange as the regulatory environment remains fragmented. Federal agencies are reconsidering parts of their oversight, states are pursuing separate approaches and existing healthcare, privacy and consumer-protection laws are being applied to systems they were never designed to govern.
That does not create a regulatory vacuum. It creates a more complicated form of exposure.
“Self-governance matters because defensibility matters. We have already seen that risk is manifesting with respect to the use of AI technology in the healthcare sector,” Alaap Shah, member of the firm at Epstein Becker Green, told Competition Policy International (CPI), a PYMNTS company, in an interview.
As a result, AI-native governance capabilities are becoming both legal protection and commercial infrastructure across healthcare.
Existing Law Still Applies to Healthcare AI
One of the most dangerous assumptions in enterprise AI is that organizations can wait for lawmakers to produce a clear rulebook. Healthcare companies do not have that luxury. Laws governing privacy, discrimination, consumer protection, contracts and professional duties already apply to AI-enabled activity, even when they do not mention artificial intelligence directly.
“There are existing bodies of law that, while not passed or promulgated for the reason of AI, are still applicable to AI solutions,” Shah said, noting that the question for healthcare firms is not simply whether an AI-specific law applies but whether the system creates risks covered by older legal obligations.
A patient-facing model that produces different recommendations across demographic groups may generate discrimination exposure. A clinical tool that influences a physician’s decision may become relevant in a malpractice case. AI is not replacing healthcare’s existing liability structure. It is entering it.
The Food and Drug Administration (FDA) continues to regulate certain software as a medical device, but its approach leaves room for interpretation. Shah called the boundary “still a little murky.”
“There are still going to be situations where a company may push right up against that line and FDA may come back and say, ‘Actually, this is something we’d like to regulate,’” he said, noting that state medical boards add another layer.
A developer may describe a product as decision support. A medical board may see a system behaving like an unlicensed clinical operator.
“I think there’s going to be fragmentation for quite a while,” Shah said.
Healthcare’s AI Governance Must Become a Defensible Record
There’s another kink in the AI machine for healthcare. Healthcare organizations often do not control the models they deploy. Vendors may own the system logs, training process and performance data, but when a tool fails inside a hospital, the hospital may still be expected to explain what happened. That makes contracts central to AI governance.
Hospitals need more than standard promises about security and compliance. They need logging requirements, preservation obligations, audit rights and clear rules governing access to the evidence a model produces.
“To the extent that any events could be logged in the AI processing, that is something that needs to be happening so we can understand how that AI operated and why the input led to the output,” Shah said.
A clinician cannot defend a decision influenced by a system whose operation cannot be reconstructed. Nor can a hospital investigate an adverse event if the vendor controls the relevant records. The black box is becoming a contractual problem.
AI also challenges one of healthcare’s most familiar privacy safeguards: de-identification.
Removing direct identifiers can reduce risk, but AI systems can combine datasets, infer attributes and reconnect information that appeared anonymous in isolation.
“It’s a real possibility that AI algorithms could re-identify individuals if sufficient data gets put in, even if de-identified in the first instance,” Shah said, adding that bias presents a similar problem.
AI can influence treatment, prior authorization, insurance coverage and patient communications. When those systems produce different outcomes for protected groups, organizations may face litigation, regulatory scrutiny or public backlash regardless of federal enforcement priorities.
That makes internal governance more important. Organizations need to inventory their AI systems, classify them by risk, assign accountable owners, train employees, document controls and allocate responsibility across vendors. The goal is not to prove that failure was impossible. It is to prove that the organization was not careless.
Watch the full PYMNTS TV interview with Alaap Shah to hear more about:
- Why healthcare AI governance is becoming a legal defense strategy, not just a compliance exercise. Shah said that as AI-specific regulation remains fragmented, organizations must build documented oversight, policies and risk controls that demonstrate they acted responsibly when litigation or regulatory scrutiny inevitably arrives.
- How AI is turning vendor contracts into critical risk infrastructure. The discussion explores why hospitals need audit rights, model logging, record preservation and clear allocations of responsibility, since developers—not providers—often control the evidence needed to explain how AI reached a clinical recommendation.
- Why healthcare AI’s greatest risks extend beyond regulation to trust. Shah said re-identification, algorithmic bias and AI-generated data errors can propagate across interconnected healthcare systems, making transparency, data provenance and governance essential for protecting both patients and organizational reputation.
