As we move into 2026, the healthcare industry stands at a pivotal moment with artificial intelligence. The adoption of AI in healthcare is increasing, but so are the associated risks. Currently, the most common form of artificial intelligence is large language models (LLMs).

Large language models like ChatGPT, Gemini, and Claude are no longer just experimental “toys” for early adopters; they now form the basis for clinical documentation, surgical planning, and mental health triage. However, for HIPAA compliance managers, healthcare providers, IT professionals, and attorneys, this early phase has transformed into a complex landscape filled with regulatory and clinical liabilities.

LLMs have quickly advanced recently, offering many options. Most are publicly accessible or open source. This transparency, which allows LLMs to gather and learn from data across the internet, also introduces risks. There are privacy and security concerns for anyone using them.

All HIPAA-regulated entities must understand the risks AI presents in healthcare: covered entities such as hospitals, clinics, health plans and insurers, along with physicians, nurses, therapists, dentists, pharmacists, and counselors, as well as business associates like billing vendors, collection firms, and IT support, should all address these risks by establishing AI policies and training staff to ensure HIPAA privacy and security.

The challenge is threefold:

1. Organizations must keep Protected Health Information (PHI) private amid data-hungry, data-driven models.
2. Organizations must ensure the integrity of PHI—protecting it from the “hallucinations” and bias that can turn a helpful assistant into a source of medical liability.
3. Organizations must comply with a range of recent state laws that aim to regulate AI in healthcare.

In 2026, the Office for Civil Rights (OCR) will shift from its previous flexible enforcement approach. Once the HIPAA Security Rule updates are finalized, the “addressable” standards of the past will become “required,” and AI systems will undergo increased scrutiny.

The Top Three HIPAA Privacy Concerns for AI in Healthcare

I. Impermissible Disclosures

The “Shadow AI” Epidemic

The most persistent privacy threat is “Shadow AI“—the unauthorized use of consumer-grade AI tools by well-intentioned staff in violation of policy.

Despite the availability of private or enterprise solutions, many clinicians still rely on public models like ChatGPT, Claude, or Gemini to summarize complex patient histories or draft appeal letters to insurers.

• The Problem: Public models often lack a Business Associate Agreement (BAA). Every prompt entered is included in the vendor’s training data or telemetry logs.
• The 2026 Reality: “Prompt leaking” is an established breach category. If a nurse enters a patient’s unique history into a non-BAA tool as a prompt, that data is outside the “covered entity’s” control and constitutes an unauthorized disclosure.

Re-identification and the “Mosaic Effect”

Traditional de-identification, known as the “Safe Harbor” method, is failing. In 2026, AI’s pattern recognition skills are so advanced that “anonymized” data often isn’t truly anonymous.

• The “Mosaic Effect”: This occurs when an AI cross-references a de-identified medical record with public datasets—such as voter registrations, property records, or social media. By combining these pieces, the AI can re-identify a patient with surprising accuracy.
• Compliance Risk: Lawyers now argue that “Safe Harbor” is no longer a sufficient defense if the entity knew or should have known that an AI could re-identify the data.

Your Inputs are Training the LLM

Under HIPAA, PHI can be used for “healthcare operations’ without authorization. However, a growing legal divide exists: Does training a commercial AI model for a vendor count as healthcare operations?

A larger concern is that this could qualify as a “sale of PHI,” which would require authorization if the hospital directly or indirectly receives remuneration from, or on behalf of, the recipient of the PHI.

The issue arises when an AI vendor offers to provide the AI product under development to the hospital free of charge in exchange for the hospital’s disclosure of PHI.

• The Conflict: If a hospital shares data with a vendor to enhance a diagnostic tool that the vendor then sells to other hospitals, it appears less like “operations” and more like “product development,” and may even constitute a sale of PHI that requires a patient’s authorization.
• 2026 Standard: Regulators are increasingly requiring express patient authorization for any use of PHI that contributes to a commercial AI product, even if the primary goal is “quality improvement.”

II. Clinical and Legal Risks: Hallucinations and Bias

Hallucinations

Privacy involves safeguarding data from exposure; integrity emphasizes keeping data accurate. In clinical settings, AI “hallucinations”—where the model confidently produces incorrect information—are more than just bugs; they pose safety risks that can lead to significant HIPAA and malpractice liabilities.

A recent episode of the popular TV show, The Pitt, highlights the controversy around using AI in the emergency room. (Season 2, Episode 2). One doctor introduces a speech-to-text and medical scribing tool to record doctor-patient conversations, which helps reduce charting time. However, as the AI tool enters notes into the chart, one of the attending physicians notices it has suggested the wrong medication for the patient.

Bias

When sources lack diversity or representativeness, LLM output can be biased. Human scientists and researchers are trained to prevent bias, and the peer review process provides safeguards to identify and correct bias that could distort conclusions. So far, LLMs lack built-in safeguards.

A research paper from ScienceDirect on LLMs in healthcare explains:

“Of major concern is the potential for inherent bias within the training data sets to exacerbate cognitive biases and discriminatory practices already prevalent throughout society. Biases that carry over from the training data to LLMs used in healthcare settings could negatively impact people of a specific race, gender, age, sexual orientation, or political ideology.”

Behavioral Health and Counseling: Distorting the Narrative

The foundation of therapeutic counseling is the human-to-human relationship between the client and clinician, built on empathy, understanding, and trust. While it may offer advice or guidance, an artificial intelligence platform cannot provide genuine human empathy.

In mental health, AI’s tendency to be “agreeable” can be harmful. The agreeable response might miss clues and favor one interpretation over others.

• The Risk: An AI scribe might summarize a session and “hallucinate” a detail about a patient’s trauma (e.g., suggesting a specific person was an aggressor when they weren’t). If the therapist relies on this summary, they risk losing trust with their patient, damaging the therapeutic alliance, and potentially leading to a wrongful diagnosis claim.

Bias in the therapeutic setting is harmful, especially for patients of a specific race, gender, age, sexual orientation, or political ideology not represented in the LLM output.

The state of Illinois passed a law last year that completely prohibits the use of AI in counseling and psychotherapy. (More on that below.)

Surgical Discharge: Creative Recovery Plans

Discharge notes are a patient’s survival manual.

• The Risk: Models often prioritize “statistically common” outcomes over the specific reality of a surgery. An AI might hallucinate that a patient’s vitals were “stable throughout” because “stable” is a common word in medical text, even if the actual intraoperative record showed a life-threatening spike in blood pressure.

Prescribing: The Decimal Point Disaster

• The Risk: AI models rely on probability, not precise math. A common mistake is confusing 0.5mg with 5.0mg. In the “probabilistic” world of an LLM, these numbers are close; in the human body, that 10x difference can be a lethal overdose.

III. States Are Regulating AI in Healthcare

In the absence of federal guidance on the use of AI in healthcare, states are passing laws to restrict its use or require transparency.

For example, Illinois passed the “Wellness and Oversight for Psychological Resources Act” (HB 1806), which took effect on August 1, 2025, prohibiting the use of artificial intelligence to provide direct therapy, psychotherapy, or clinical decision-making. The law ensures only licensed professionals perform these services, aiming to prevent harm from unregulated AI chatbots. 

In addition, California’s Transparency in Frontier Artificial Intelligence Act (the “California TFAIA” or SB 53) and Texas’s Responsible Artificial Intelligence Governance Act (the “Texas RAIGA” or HB 149) are two prominent examples of state AI laws effective as of January 1, 2026.

California and Texas now require healthcare providers to:

1. Conspicuously disclose the use of AI in diagnosis or treatment.
2. Provide a “human-centered” alternative for patients who opt out of AI-driven interactions.
3. Implement an AI Bill of Materials to track exactly what data was used to train the clinical models they deploy.

Many other states have enacted laws to regulate or restrict AI in healthcare, including Nevada, Utah, and New York.

Red Flag Guide: What to Look For

Compliance managers and providers must be trained to review AI drafts with a skeptical eye. In 2026, providers should look for these specific signs of model failure:

• The “Normalizing Bias”: If a patient describes a “sharp, localized pain in the left calf” (a DVT red flag), and the AI summarizes it as “Patient reports general lower-body discomfort,” it has softened a critical symptom into a generic phrase. This is a red flag for clinical negligence.
• Gender/Name Flips: If the AI switches pronouns (he/she) or incorrectly attributes a mother’s medical history to the patient, the model has “lost the thread.” This signifies that the entire note’s integrity is compromised.
• The “Phantom” Stat: Be cautious of very specific percentages like “Patient reported a 22% improvement in anxiety.” Patients rarely express themselves in percentages; the AI is probably making up a metric to seem more “professional.”

Recent Developments: Mitigating the Risks

The good news is that new tools are available to combat these ghosts in the machine. We have shifted from “blind trust” to “verified output.”

Human-in-the-Loop (HITL) 2.0

The gold standard for 2026 is the Mandatory Review Interface. AI-generated notes are no longer sent directly to the electronic health record (EHR) system.

Instead:

• Confidence Scoring: The AI highlights text in yellow when it is less than 85% confident in the fact.
• Click-to-Validate: A physician must manually click a “verify” button on key assertions (dosages, diagnoses, allergies) before the note can be signed.

Retrieval-Augmented Generation (RAG) and Source Anchoring

To prevent models from “making things up,” providers should now adopt RAG.

• How it works: Instead of letting the AI answer from its “memory,” the system requires the AI to consult a specific knowledge base (such as the hospital’s actual lab results or the FDA’s drug database).
• Source Anchoring: When a doctor hovers over a dosage in an AI draft, the system displays a “snippet” of the original source, such as the physician’s voice or the lab report. If no snippet appears, it’s a hallucination.

A Call to Action to Manage AI in Healthcare

In 2026, the guiding principle is clear: the AI suggests, the human decides. We must not allow the efficiency of Large Language Models to distract us from the core HIPAA requirements: the safety, privacy, and integrity of patient records.

For healthcare attorneys, the focus must shift from simply maintaining “Privacy Policies” to establishing strong “Data Integrity Protocols.” For compliance managers, the task is no longer just about preventing leaks; it’s about thoroughly auditing the black box of AI logic.

Next Steps for Your Organization:

• Establish AI Use Policies: Decide whether and how AI may be used, and train staff on your protocols.
• Audit for Shadow AI: Use Cloud Access Security Brokers (CASBs) to monitor if staff access public AI endpoints.
• Update your BAA: Ensure your AI vendors include “No-Retention” clauses and guarantees of “No-Training-on-User-Data.”
• Implement HITL: Make sure your AI tools always require active human verification for every clinical claim.