A Security Risk Assessment Guide for AI Medical Records Indexing Startups
If your startup uses AI to index medical records, a security risk assessment should cover more than the model itself. It should account for every place electronic protected health information (ePHI) is created, received, maintained, or transmitted—including data pipelines, people, cloud services, and vendors.
The HIPAA Security Rule requires covered entities and business associates to assess potential risks and vulnerabilities to the confidentiality, integrity, and availability of ePHI. The HHS guidance on risk analysis explains that the rule does not prescribe one assessment method; the approach should fit the organization and its environment.
How to inventory ePHI and map AI medical records data flows
Start by identifying the ePHI your startup handles and where it moves. Include data received from customers, created during indexing, stored temporarily for processing, returned as results, or retained in logs, backups, support tickets, and test environments. HHS recommends identifying all ePHI an organization creates, receives, maintains, or transmits, including information held in systems and applications.
Draw a data-flow map from intake through deletion. For each stage, record the system, data type, responsible team, access roles, storage location, transfer method, retention period, and any downstream service. Map both routine flows and exceptions, such as failed jobs that place records in a queue or staff who access an account to troubleshoot an indexing error.
- Inputs: Clinical documents, scanned images, metadata, identifiers, and customer-provided annotations.
- Processing: Optical character recognition, text extraction, classification, indexing, model inference, and human review.
- Outputs: Search indexes, extracted fields, confidence scores, summaries, logs, and exported records.
- Copies: Training and test datasets, cached files, snapshots, backups, and local downloads.
Include systems that may not look like clinical applications but can still expose ePHI, such as analytics tools, error monitoring, collaboration platforms, and customer support systems. HHS’s risk analysis guidance notes that an inventory may include systems and applications used to access or house ePHI, as well as external sources such as vendors and consultants.
Which threats and vulnerabilities should the risk analysis assess
Assess threats to confidentiality, integrity, and availability across the mapped data flows. Consider unauthorized access, accidental disclosure, compromised credentials, malicious insiders, ransomware, service outages, lost devices, misdirected transfers, and improper changes or deletion of records. The HHS risk analysis guidance calls for evaluating relevant threats and vulnerabilities and considering the likelihood and impact of potential harm.
Include AI-related risks alongside familiar cybersecurity threats. For example, assess whether malicious or corrupted documents could manipulate extraction results; whether a compromised model or software dependency could alter outputs; and whether an overly broad prompt, interface, or retrieval process could expose records to an unauthorized user. Treat these as scenarios to evaluate in your own system, not as assumptions that every AI tool has the same weaknesses.
For each threat, identify the specific vulnerability that could make it possible. Examples of vulnerabilities include unpatched software, weak authentication, excessive access permissions, exposed storage, insecure interfaces, missing audit logs, inadequate backups, or incomplete vendor controls. A useful assessment describes the path from threat to vulnerability to potential impact—for example, a stolen support account combined with broad production access could expose many customer records.
How to scope AI systems and third-party vendors
Define the assessment boundary to include the full service, not just the model. Identify model endpoints, APIs, orchestration layers, data stores, identity systems, development and production environments, human review tools, and customer integrations. Record whether each component is built in-house, licensed, hosted, or operated by a third party, and specify whether it can receive, retain, or access ePHI.
Determine your HIPAA role based on the actual services and data access. A startup that performs indexing on behalf of a covered entity and handles PHI may be a business associate; a vendor that only provides software without access to the customer’s PHI may not be. HHS explains these distinctions in its Business Associates FAQs.
Review business associate agreements (BAAs) and subcontractor arrangements before ePHI is shared. If a cloud service provider creates, receives, maintains, or transmits ePHI for a covered entity or business associate, HHS says the provider is a business associate even when it stores encrypted data without a decryption key. See HHS guidance on HIPAA and cloud computing. A BAA does not replace your own assessment of the provider’s configuration, responsibilities, and security controls.
For each vendor, review what data it can access, where it is processed, how long it is retained, whether it may be used for model training, how incidents are reported, and how data is returned or deleted when services end. Assess cloud security configuration as well: identity and access settings, network exposure, encryption, logging, backup arrangements, and separation between customer environments.
Which administrative physical and technical safeguards should be evaluated
Evaluate safeguards across administrative, physical, and technical areas, then determine whether they are reasonable and appropriate for your organization’s risks. HHS describes these safeguard categories in its Summary of the HIPAA Security Rule; the rule is flexible and technology-neutral, so assess how controls work in your actual environment.
- Administrative safeguards: Assigned security responsibility, workforce access approval and training, incident response, vendor oversight, contingency planning, periodic evaluation, and documented security policies.
- Physical safeguards: Facility access, workstation use, protection of devices that can access ePHI, and controls for receiving, moving, reusing, and disposing of hardware or media.
- Technical safeguards: Role-appropriate access, authentication, audit logging, integrity protections, and transmission security for systems that create, receive, maintain, or transmit ePHI.
Test whether controls address realistic startup workflows. For instance, check that engineering staff cannot access customer records by default, that support access is authorized and logged, and that test environments do not contain live ePHI without an approved need and safeguards. Review whether logs themselves could capture record contents or identifiers.
HHS identifies certain Security Rule implementation specifications as “addressable,” not automatically optional. An organization must determine whether each is reasonable and appropriate; if not, it must document the basis and adopt an appropriate alternative when required. See the HHS Security Rule summary for this distinction.
Ready to assess your HIPAA security risks?
Join thousands of organizations that use Accountable to identify and fix their security gaps.
Take the Free Risk AssessmentHow to protect AI training data and model integrity
Separate data used to operate the indexing service from data used to develop or improve models. Before using customer records for training, testing, evaluation, or fine-tuning, document the permitted purpose, authorization, access controls, retention, and deletion process. A BAA should define permitted uses and disclosures; do not assume that receiving ePHI for indexing automatically authorizes using it to train a model.
Protect AI training data integrity by controlling who can add, modify, approve, or remove examples. Track dataset sources and versions, validate data changes, preserve provenance records, and test for unexpected shifts in labels or content. Restrict production data access and use de-identified or synthetic data for development where appropriate, while checking that the chosen data still supports meaningful testing.
Model integrity also depends on securing code, dependencies, model files, configuration, deployment credentials, and update pipelines. Limit privileges for people and services that can change these components; review changes; and retain enough records to trace which model and data version produced an indexing result.
Assess model output data leakage: outputs may reveal patient information if the system returns record text to the wrong user, exposes data through logs or error messages, or reproduces information retained in model training. The NIST Generative AI Profile recommends examining privacy risks such as outputting training samples, model extraction, and membership inference. Use testing and access restrictions to check whether outputs disclose information beyond the intended user’s authorization.
How to document risk levels and corrective actions
For each risk scenario, document the affected ePHI, system or vendor, threat, vulnerability, existing safeguards, likelihood, potential impact, and resulting risk level. Explain the reasoning behind the rating so that another reviewer can understand why one scenario is more urgent than another. HHS advises documenting risk levels and corrective actions, and does not require a specific documentation format; see its risk analysis guidance.
Maintain a corrective-action register linked to the assessment. For every action, identify an owner, priority, target date, interim safeguard if needed, and evidence of completion. Distinguish risks that have been reduced from risks that remain open, and record the rationale for decisions about how to address them.
Keep the assessment, policies, and related records available to those responsible for implementing them. HHS’s Security Rule summary states that required documentation must generally be retained for six years from its creation or the date it was last in effect, whichever is later.
When to reassess security risks as systems and operations change
Reassess when a change could affect ePHI or the safeguards protecting it. Triggers can include launching a new model, changing a cloud service or configuration, adding a vendor, connecting a new customer system, changing data retention, expanding staff access, moving to a new facility, or responding to a security incident. HHS says risk analysis should be ongoing and performed as new technologies and business operations are planned.
The Security Rule does not set one universal schedule for a comprehensive risk analysis. Establish a review cadence that fits your environment, and supplement it with reassessments when meaningful changes or newly recognized risks arise. HHS discusses both periodic reviews and event-driven updates in its guidance on risk analysis.
For a startup, the practical goal is a living assessment: keep the ePHI inventory and data-flow map current, verify vendor and cloud controls, track AI training data integrity and model output data leakage risks, and assign owners to corrective actions. This connects the HIPAA Security Rule risk analysis to day-to-day decisions that protect the confidentiality, integrity, and availability of ePHI.
FAQs.
What ePHI should an AI medical records indexing startup include in its risk assessment?
Include all ePHI the startup creates, receives, maintains, or transmits, not only the original medical record. Consider source documents and images, identifiers, extracted fields, search indexes, model inputs and outputs, human-review copies, logs, support records, test data, caches, and backups. Map where each item is stored or processed and who or what can access it.
How should a startup assess risks introduced by AI vendors and cloud providers?
Inventory each provider’s access to ePHI, service role, processing locations, retention and deletion practices, security controls, incident reporting, and subcontractors. Review relevant contracts and BAAs where required, and assess the provider’s actual configuration and data flows rather than relying only on a general security statement. HHS’s cloud computing guidance explains that a cloud provider handling ePHI on behalf of a regulated entity may be a business associate.
How can AI training data and model outputs expose patient information?
Patient information could be exposed if training data contains ePHI and a model reproduces examples, if outputs or logs disclose record content, or if an interface returns information to someone without authorization. Assess permitted data uses, access controls, retention, output handling, and privacy testing. The NIST Generative AI Profile identifies training-data output, model extraction, and membership inference as privacy risks to evaluate.
When should an AI security risk assessment be updated?
Update it when changes to technology, vendors, data flows, staffing, or operations could affect ePHI security, and after incidents or newly identified risks. Maintain a periodic review cadence appropriate to your environment, but do not treat a particular annual interval as a universal HIPAA requirement. HHS describes risk analysis as an ongoing process and recommends considering changes as they are planned.
Table of Contents
- How to inventory ePHI and map AI medical records data flows
- Which threats and vulnerabilities should the risk analysis assess
- How to scope AI systems and third-party vendors
- Which administrative physical and technical safeguards should be evaluated
- How to protect AI training data and model integrity
- How to document risk levels and corrective actions
- When to reassess security risks as systems and operations change
- FAQs.
Ready to assess your HIPAA security risks?
Join thousands of organizations that use Accountable to identify and fix their security gaps.
Take the Free Risk Assessment