HIPAA checklist for AI startups using LLMs on patient data

1. Know your role

• Are you a business associate? If you create, receive, maintain or transmit PHI on behalf of a covered entity (a provider, health plan or clearinghouse), you generally are, and HIPAA applies to you directly. See the definition in 45 CFR 160.103.
• Sign a BAA with each covered-entity customer before PHI flows. Required contract terms are in 45 CFR 164.504(e).

2. Map every place PHI goes

• Draw the data flow: ingestion (uploads, EHR/FHIR, email, fax), storage, queues, model calls, retrieval indexes, outputs, exports.
• List every vendor that touches PHI: cloud, LLM provider, OCR, vector database, email or SMS, support tools, error tracking, analytics, logging, backups.
• Mark where PHI shouldn't be but often is: logs, traces, prompt caches, analytics events, crash reports, browser storage, screenshots in tickets.

3. Close the BAA chain to your vendors

• Every subcontractor that handles PHI for you needs a BAA with you (business associates must obtain the same assurances from their subcontractors: 45 CFR 164.308(b)).
• Check the BAA's scope, not just its existence. Vendor BAAs usually cover specific products, endpoints or features. Consumer apps, beta features and some tools are often excluded. See the LLM BAA matrix.
• Use the provider surface the BAA actually covers (for example, a cloud-hosted model endpoint versus a vendor's consumer app), and configure the account as the BAA requires.
• Turn off training on your data, and enable zero data retention or the shortest available retention where the vendor offers it. Record what you enabled and when.

4. Build the technical safeguards

The Security Rule's technical safeguards are in 45 CFR 164.312: access control, audit controls, integrity, authentication and transmission security.

• Encryption in transit (TLS everywhere, including internal hops) and at rest (managed keys, restricted key access).
• Least-privilege access with SSO and MFA, separate production access, break-glass procedures, quarterly access reviews.
• Audit logging of who accessed which records and when, kept tamper-resistant and free of PHI content.
• PHI-safe logging: redact or hash identifiers in application logs, traces and LLM observability tools. Never log full prompts or completions that contain PHI unless that store is covered and controlled.
• Infrastructure as code (Terraform or similar), so controls are reviewable, repeatable and provable to auditors.
• No PHI in dev or test. Use synthetic data for development, demos and evaluations.

5. AI-specific guardrails

• Tenant isolation for retrieval: separate indexes, or strict per-tenant filters enforced server-side, so one customer's records can't enter another's prompt context.
• Minimum necessary: send the model only the fields a task needs, and strip direct identifiers where they add nothing.
• Output handling: treat model outputs that contain PHI exactly like the source records (storage, access, retention, deletion).
• Human review where outputs drive clinical, legal or benefits decisions. Keep the review trail.
• Evals on synthetic data before every model or prompt change, to catch quality regressions and PHI leakage.

6. Paperwork that security reviews ask for

• A written risk analysis and risk-management plan. These are required (45 CFR 164.308(a)(1)), and they're the first thing many reviewers ask for.
• Security policies, workforce training records, vendor list with BAA status, data-retention and deletion policy, incident-response plan.
• Breach notification: a business associate must notify the covered entity without unreasonable delay, and no later than 60 days after discovering a breach (45 CFR 164.410). Your contracts may require faster.
• SOC 2 Type I or II, plus a reusable security-questionnaire answer bank (many health systems ask for both).

Want this done for you?

The PHI-Safe AI Architecture Sprint covers sections 2–6 in about 30 days, with pull requests into your repos and an evidence pack at the end.