Using an LLM With PHI: What Makes a Setup HIPAA Compliant
Large language models (LLMs) can help healthcare teams summarize documents, answer questions, and organize clinical information. Using these tools with protected health information (PHI) requires careful planning.
A prompt may contain a patient's name, diagnosis, medical record number, or treatment details. Sending that information to an unsuitable service creates real privacy and security risk.
A HIPAA compliant LLM setup isn't defined by the model alone. It depends on the provider, the service configuration, the data flow, access controls, contracts, and the safeguards the organization applies around all of it.
What Counts as PHI in an LLM Workflow?
Protected health information includes identifiable health information created, received, maintained, or transmitted by a covered entity or business associate, subject to HIPAA's definitions and exceptions.
In an LLM workflow, PHI can show up in more places than people expect:
- Patient questions and prompts
- Clinical notes and discharge summaries
- Uploaded medical records
- Retrieved documents used to answer a question
- Model-generated responses that repeat identifiable information
- Application logs, error reports, and conversation histories
Removing a patient's name doesn't always make information de-identified. Dates, locations, rare conditions, and combinations of details can still identify someone. Before sending data to a model, identify what information enters the system, where it goes, and which services can access it.
What Actually Makes an LLM HIPAA Compliant
A model can't make an entire application HIPAA compliant on its own. The organization has to evaluate the complete service and how it's actually used.
1. Check the Business Associate Agreement
A Business Associate Agreement (BAA) is a contract that sets out permitted uses and disclosures of PHI and the business associate's responsibilities for protecting it.
When a cloud or AI provider creates, receives, maintains, or transmits PHI on behalf of a covered entity or business associate, a BAA may be required, depending on the arrangement and the applicable HIPAA rules. Before selecting a provider, confirm:
- Whether the specific service and intended use are actually covered by its BAA
- Which data the provider may process
- Whether subcontractors are involved and how they're covered
- How security incidents are handled
- What happens to data when the service ends
A BAA matters, but it doesn't automatically make the complete application compliant. The healthcare organization still has to apply its own administrative, physical, and technical safeguards.
2. Evaluate the Provider and Service Configuration
Don't assume every model from a provider supports the same healthcare use case. BAA availability, eligible services, data retention options, and security features can vary by product and configuration, even within the same provider.
Check the provider's current contractual terms and documentation, and confirm that the exact model and API service in use is covered, not just the provider's product line in general. Review whether the service supports the required data handling, access restrictions, encryption, retention settings, and audit capabilities. Don't send PHI to a service until those checks are complete.
Protect PHI Across the Data Flow
An LLM application usually includes a user interface, a backend, a model API, a database, a document store, a monitoring service, and a logging platform. PHI can pass through several of these components, not just the obvious ones. Map the complete data flow before deployment.
Control Logging and Data Retention
Prompts and model responses can contain PHI even when the main database is well protected. Logs, debugging tools, analytics platforms, and error reports can create additional, easy-to-overlook copies of sensitive data.
- Restrict access to logs
- Avoid putting unnecessary PHI into application logs in the first place
- Set and actually enforce retention periods
- Encrypt data in transit and at rest where appropriate
- Review monitoring and analytics services before sending them sensitive information
- Test how deletion requests and retention policies work across every connected service, not just the primary database
Don't assume a provider's default logging or retention settings meet your requirements. Review the actual service configuration and the applicable contractual commitments. For more on this specific problem, see Audit Logging for Healthcare Apps: Beyond the HIPAA Checkbox.
Apply Access Controls
Use role-based access controls and least-privilege permissions to limit who can view PHI. A user should only access records needed for their authorized role and task.
Service accounts and backend components need restricted permissions too. Protect API keys, rotate credentials when appropriate, and avoid placing secrets in source code or client-side applications. Access controls stay important even when a model provider offers strong security features of its own, the application is still responsible for controlling which users and services can reach its data.
De-Identification and Access Controls Serve Different Purposes
These two safeguards are related, but they aren't interchangeable.
De-identification aims to remove or manage identifying information so data no longer meets the applicable definition of PHI. HIPAA provides specific methods for de-identification, including Safe Harbor and Expert Determination. Access controls restrict who can view or use information, they don't remove identifiers from the information itself.
Before using real patient information in prompts, determine whether the data needs to be de-identified and whether the proposed method actually fits the use case. Simple name removal isn't proof that data is no longer identifiable. Where PHI needs to stay identifiable, apply the required safeguards and make sure the service is authorized for that use.
Where RAG Fits Into a HIPAA Compliant LLM Setup
Retrieval-augmented generation (RAG) lets an LLM application retrieve relevant information from approved documents or databases before generating an answer. In healthcare, RAG can help an application answer questions using internal policies, clinical documents, or authorized records. But retrieving information from a protected source doesn't automatically make the workflow secure.
A safer RAG design includes:
- Access checks before retrieving documents
- Permissions that match the user's role and authorized records
- Protection for document stores, embeddings, and retrieval indexes
- Controls that reduce the risk of exposing information from unrelated records
- Logging of relevant access and system events without unnecessarily storing PHI
- Testing for unauthorized retrieval, prompt injection, and data leakage
A user should never receive information from another patient's record just because the retrieval system found it relevant to a question, the system has to enforce permissions before documents ever reach the model. Embeddings and vector indexes need review too, they can represent information derived from sensitive records and shouldn't automatically be treated as harmless or anonymous. For a deeper look at this, see Building Secure and Compliant RAG for Healthcare.
A Practical Checklist for Using an LLM With PHI
This supports implementation planning, it isn't a substitute for a formal HIPAA risk analysis or legal advice.
| Area | What to Check |
|---|---|
| PHI identification | Identify sensitive data in prompts, documents, responses, and logs. |
| Vendor agreements | Confirm applicable BAAs cover the services handling PHI. |
| Data flow | Map where information travels and which services process it. |
| Data minimization | Send only the information required for each task. |
| De-identification | Follow a recognized HIPAA method when relying on de-identified data. |
| Access controls | Verify permissions before retrieving or sharing patient records. |
| Encryption | Protect information in transit and at rest. |
| Logging and retention | Minimize sensitive logs and establish deletion policies. |
| RAG security | Restrict retrieval and protect document stores and embeddings. |
| Testing | Check for unauthorized access and cross-patient data exposure. |
| Incident response | Establish procedures for investigating potential data exposure. |
For a broader implementation guide, see What It Actually Takes to Build a HIPAA-Compliant Healthcare Platform and HIPAA Compliance for SaaS: A Practical Checklist.
FAQs
Is There Such a Thing as a HIPAA Compliant LLM?
An LLM may be available through a service that supports HIPAA-regulated use, but compliance depends on the specific service, contractual terms, configuration, and safeguards across the complete application. A model name alone doesn't establish compliance.
Does a BAA Make an LLM Application HIPAA Compliant?
No. A BAA establishes contractual responsibilities between the relevant parties. The organization still has to apply appropriate safeguards, manage access, review data flows, and meet its own HIPAA obligations.
Can I Send PHI to an LLM API?
Only after confirming the specific service and use are appropriate for PHI, the required agreements are in place, and the application has suitable safeguards. Don't assume every API or account type supports this use.
Does RAG Prevent PHI Leakage?
No. RAG can help ground answers in approved sources, but it can also expose sensitive records if retrieval permissions or application controls are weak. Access checks and security testing are essential, not optional extras.
The Bottom Line
Using an LLM with PHI takes more than selecting a capable model. Healthcare teams need to understand their data flows, verify provider terms, apply access controls, manage logging and retention, and test the complete application, not just the model in isolation. A carefully designed setup can reduce risk while still supporting useful healthcare workflows, which starts with mapping where PHI actually travels and documenting the safeguards needed before anything goes live. For related work, see Healthcare AI Agents Powered by MCP: Transforming Clinical Workflows.
