GxP AI Validation on a Lakehouse: The 21 CFR Part 11 Evidence

Quick answer: GxP AI validation on a lakehouse has to prove two things. The system must be fit for its intended use, and the regulated records it touches must stay protected. The evidence is approved requirements, controlled model versions, risk-based evaluations, traceable outputs, audit trails, retained records and documented human review. Electronic signature controls apply wherever the workflow signs a record.

FDA says CDER has drawn on its experience with over 500 submissions with AI components from 2016 to 2023, and in January 2025 the agency published draft guidance on AI credibility. On a Databricks lakehouse, reliable table time travel covers only the past 7 days by default. GxP AI validation gets hard in the space between those two facts.

The trouble is specific. An LLM words its answer differently on each run, so fixed expected-value tests stop working. A vector index rebuild changes the SOP context the model retrieves, with no code release to flag it. A hosted foundation model changes behind the same endpoint name. Routine table cleanup removes the history an inspector asks to see. Then someone asks how an answer was produced, and nobody can rebuild it.

I have spent 30+ years implementing enterprise systems inside IBM, Ernst & Young, Johnson & Johnson and McKesson. In regulated work, the validation question stays the same: show what the system did, who approved it, and which requirement it met. For mid-market life-sciences teams, GxP AI validation starts there.

What does 21 CFR Part 11 ask of an AI system’s records?

Start with three things: the regulated record, the predicate rule that requires it, and the boundary of the electronic system. Part 11 covers electronic records and signatures within its scope. It creates no separate approval category for AI models (see 21 CFR Part 11, particularly § 11.1).

FDA explains how it reads the rule in its September 2003 final guidance, Part 11, Electronic Records; Electronic Signatures: Scope and Application. There, FDA applies enforcement discretion to validation, audit trails, record retention and record copying.

Part 11 stays in effect, and predicate rules still apply. FDA keeps enforcing access limits, operational and authority checks, and electronic signature requirements. You need a documented applicability assessment, not a blanket claim that every AI interaction carries the same requirements.

For a closed system, § 11.10 covers validation, accurate copies, record protection, access limits, audit trails and authority checks. We give each requirement a testable control and a named evidence artifact.

Take an assistant that drafts a deviation summary for review in a quality management system. Define where the draft lives, which output becomes the retained record, and who approves it.

Then draw the boundary. The lakehouse, the application, the identity provider, the external model service and the quality system all belong in the assessment when they touch that record.

At Analytics AIML, we build the controls and the evidence. Your quality unit, regulatory affairs team and counsel decide applicability, accept residual risk and make the compliance call.

Why does AI strain traditional computer system validation?

Traditional computer system validation (CSV) leans on scripted tests with fixed expected results and screenshot-heavy evidence. FDA’s Computer Software Assurance for Production and Quality Management System Software (CSA) guidance, final and dated February 2026, applies to medical device production and quality system software under 21 CFR Part 820. It moves toward risk-based assurance, with unscripted and exploratory testing where the risk is lower. Drug and biologics quality units can adopt the same risk-based thinking alongside GAMP 5 Second Edition, whose Appendix D11 covers AI and machine learning. Neither framework removes the specific problems below.

What happens when the same input produces different answers?

A conventional test compares an output with a fixed expected value. Generative AI breaks that pattern, because wording and coverage shift between runs. A fluent summary still fails if it drops a material deviation or invents an explanation.

Set acceptance around the task: factual accuracy, material omissions, source support, refusal behavior and escalation. Keep individual results next to aggregate scores so reviewers see where failures cluster.

For variable outputs, specify repeated-run testing and justified acceptance criteria before you execute. A good average must never hide a dangerous failure category.

Which change actually changed the system?

Swapping the model is an obvious change. So is a revised system prompt, retrieval filter, document parser or approved procedure, because each one changes what reaches the model. Even untouched code produces different results when its reference corpus changes without a recorded release boundary.

Retrieval-augmented generation (RAG) adds quieter changes. Chunking logic, an embedding model refresh or an asynchronous vector index rebuild each alter the retrieved context without any software release. Version the chunking configuration, the embedding model and every index build, and record which index version served each answer.

Define each release as a bundle: model identity, prompt, code, retrieval settings, index version and reference-data versions. Tie every bundle to an approved change record.

For externally hosted models, decide how you detect and assess provider changes. A provider that updates weights or safety filters behind the same API name invalidates earlier test runs without telling you. Where reproducibility matters, serve a pinned model version on a governed model-serving endpoint. If the provider offers no reproducible version, document that limit and decide whether the intended use still holds up.

What happens when retrieval ignores record permissions?

A retrieval layer that indexes restricted GxP documents under one service identity hands passages to users who lack § 11.10(d) access to the underlying record. The answer looks harmless, but the access control has been bypassed.

Enforce the requesting user’s permissions at retrieval time, not only at ingestion. Test with users who hold partial access, and confirm that restricted passages never reach the prompt.

Where does the human decision disappear?

A human-in-the-loop checkbox says little about what the reviewer actually checked. A reviewer who sees only a polished answer has nothing to push back against. Show the reviewer the source material, uncertainty signals and a clear path to reject or escalate.

Retain the original AI output, later edits, reviewer identity and disposition as separate events linked to the record. Define what review means and how exceptions get handled.

Test rejected answers and unavailable reviewers, not just clean approvals. Otherwise the system’s most consequential behavior sits outside the evidence package.

That is why we start with workflow boundaries and decision rights. More documentation will not repair an undefined process.

What does risk-based validation look like for a model that changes?

Risk-based GxP AI validation starts with one specific intended use. “Summarize approved procedures for a trained reviewer” draws a far clearer boundary than “support quality decisions.” Name the users, input population, permitted actions, excluded uses and the consequence of a wrong output.

In January 2025, FDA published Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products. It proposes a risk-based credibility framework tied to a context of use. FDA labels it Draft Level 1 Guidance: not for implementation, with non-binding recommendations.

The draft weighs model risk by two factors: model influence and decision consequence. Its scope is AI that produces information supporting regulatory decisions on drug safety, effectiveness or quality. It is not a validation mandate for every internal AI tool.

Our validation package separates system behavior from model performance. System tests cover permissions, record handling, interfaces, failures and recovery. Model evaluations check outputs against approved task criteria across representative and difficult cases.

• Intended use: Name the decision supported, the responsible user, the record produced and the prohibited actions.
• Risk assessment: State the consequence of error and the controls that reduce exposure.
• Requirements: Turn the workflow into observable behavior and traceable acceptance criteria.
• Evaluation plan: Set test populations, reference answers, scoring methods and failure thresholds before testing starts.
• Release decision: Tie executed tests, deviations, reviewer conclusions and approval to the exact release bundle.

A 21 CFR Part 11 validation protocol template, organized around lifecycle deliverables such as specifications, a requirements traceability matrix (RTM) and, where your procedures use them, IQ, OQ and PQ records, gives you a starting structure for these artifacts. Fill it with the real workflow, dependencies and failure modes. A completed template without executed evidence proves only that someone completed a template.

How should a gold-layer evaluation dataset validate RAG and LLM outputs?

Build a versioned gold-layer evaluation dataset. Give it stable case IDs, controlled inputs, reference evidence, expected behavior and risk categories.

Keep evaluation cases apart from development examples. Document how subject-matter reviewers set and revise reference answers.

Databricks documents evaluation inputs and optional expectations in its Evaluation dataset reference. Record the dataset version, scoring configuration, individual results and run ID in every release evaluation.

Load the set with hard cases: contradictory documents, missing evidence, obsolete procedures, access-restricted content and instructions hidden inside retrieved text. A document assistant must refuse unsupported questions. For RAG, score retrieval separately from generation, so you know whether a wrong answer came from the wrong passage or the wrong reading of the right one. For extraction, inspect errors by document type and field consequence.

Automated scoring speeds up evaluation, but qualified reviewers still judge task-specific adequacy. If an AI judge scores outputs, version its configuration and measure its agreement with those reviewers. The evaluator is part of the evidence design.

Which lakehouse controls produce Part 11 evidence?

A governed lakehouse produces usable evidence when identity, versions, execution events and review records share consistent identifiers. The table maps selected provisions to a proposed implementation. It is an engineering map, not a claim that any platform delivers compliance.

The regulatory references are § 11.10, § 11.50 and § 11.70. Training, written procedures, documentation controls and other applicable provisions also belong in your assessment. If you also supply the EU, run the same mapping against EU GMP Annex 11 on computerised systems, and track the European Commission’s proposed Annex 22 on AI.

Which record is the original?

ALCOA+ asks for records that are attributable, legible, contemporaneous, original and accurate, plus complete, consistent, enduring and available. AI workflows blur the “original” part. MLflow traces capture prompt and response telemetry, which helps debugging, but trace data is not automatically your GxP record.

Decide which output is the original record. Write it to the retained store or the quality system at the time of the event, attributed to the user and the release bundle. Link the trace to it by correlation ID, and set the trace’s own retention by its role in the evidence.

How do model versions connect to the approved release?

Databricks documents registered model versions, access controls, aliases and lineage in Manage model lifecycle in Unity Catalog. An alias is a mutable pointer. Record the resolved model version used at execution, not just an alias like “Champion.”

Our release manifest also records the code revision, prompt version, index version, dependency configuration, evaluation dataset version and approval reference.

For an external model, capture provider identifiers and whatever version information exists. A local registry entry does not prove the external weights stayed the same.

Where do lineage and audit trails meet?

Databricks captures lineage automatically for supported operations, as described in Lineage in Unity Catalog. The same documentation lists limitations, so test coverage across your actual workflow. Source extraction, application actions and final quality records need explicit connections where automatic capture stops.

Carry one correlation ID from source ingestion through the bronze, silver and gold layers of the medallion architecture into the AI output and the review record. Keep source IDs, pipeline code versions, retrieval references and execution IDs together. Reconstruction then becomes a controlled query, not an investigation across disconnected screenshots.

Platform audit events add a second evidence stream. The Databricks Audit log system table reference documents identity, time, action and request fields. Test which events land there, and build application event capture for the record changes and review actions it misses.

When does an electronic signature enter the design?

Add signature controls wherever the workflow signs an electronic record. Section 11.50 requires the signer’s printed name, the date and time of signing, and the meaning of the signature. Section 11.70 requires the signature to be linked to its record. Ordinary application logging does not meet those requirements.

An MLflow model signature describes input and output schemas. It is a different concept entirely and never stands in for a person’s electronic signature. When signing happens in the quality system, test that the signed revision matches the exported lakehouse record and its supporting evidence.

How do you keep validation evidence current after go-live?

GxP AI validation holds up when approved operations emit evidence on their own. Evaluations write result records, deployments write manifests and review actions write dispositions. Scheduled evidence checks flag missing links before anyone needs the release package.

Build change assessment around model swaps, prompt edits, retrieval and index changes, revised source procedures, dependencies and permission changes. Document which tests each change triggers and who approves the scope. Submit these as proposed controls for quality approval; not every edit needs the same testing.

Monitor task-specific failure categories, reviewer overrides, missing source references and input drift. Set escalation thresholds and owners before production use. Monitoring spots emerging problems; controlled release testing decides whether a revision meets its acceptance criteria.

What happens to evidence after routine cleanup?

Databricks says it plainly in Work with table history: “Don’t use table history as a long-term backup solution for data archival.” The same page limits time travel to the past 7 days unless you set both data and log retention to larger values. Retention settings and archive design are separate decisions.

Define retention by evidence category: source records, evaluation datasets, model artifacts, prompts, index builds, execution records, review events and approvals. Link dependencies so deleting a reference dataset never strands a retained evaluation report. Protect archive administration separately from day-to-day operation.

Test restoration and export with historical records. Reconcile record counts, identifiers, attachments, review history and, where applicable, signature information. Assign an owner to resolve missing artifacts and document the result.

Our AI governance and compliance services connect this control design to data engineering and operating responsibilities. The goal is an evidence chain that survives routine deployments, staff changes and infrastructure retirement.

What changes when AI starts taking actions?

An assistant that drafts text is a different operating problem from an agent that updates records or calls production systems. Every permitted tool action adds an authorization boundary, a failure path and one more event to reconstruct. Evaluate the full action sequence, not just the generated answer.

For agentic workflows, use an explicit allowlist of actions, constrained service identities, approval gates and safe stop conditions. Preserve attempted actions as well as completed ones. Test denied access, duplicate requests, partial failures and recovery from interrupted runs.

Evidence automation grows with that scope. The pipeline flags any release missing evaluation results, unresolved deviations or required approvals. Automation organizes the evidence; your company still decides whether it supports the intended use.

What should you ask before approving a GxP AI workflow?

The difference between demo-grade and production-grade AI is the evidence behind it. Before your quality unit signs off, or before you hire a validation partner, ask five questions:

• Reconstruction: Can we rebuild one sample record from source data through index, model, prompt, output and review?
• Release control: Is the release bundle versioned and tied to an approved change record?
• Acceptance: Were acceptance criteria and failure thresholds set before testing started?
• Review: Does the workflow capture edits, rejections and escalations as separate events?
• Retention: Will the evidence survive table cleanup, staff turnover and platform changes?

If any answer is no, fix the process before you scale the model.

Contact us about life-sciences lakehouse implementation. Bring one defined workflow and a sample record to reconstruct. We will map the controls, the evidence and the responsibilities for your quality unit to review. Secure. Solve. Scale.

Frequently Asked Questions (FAQs)

Does Databricks make an AI workflow Part 11 compliant?

No. No platform delivers compliance on its own. Databricks supplies technical capabilities that still need configuration, testing, procedures and an applicability assessment. Your quality unit, regulatory affairs team and counsel make the compliance call.

Does every prompt change require full revalidation?

Not automatically. An approved change-assessment process sets the testing scope. Check whether the prompt change affects intended use, record content, retrieval behavior or risk controls, then retain the assessment, affected tests, results and release decision.

Is a human reviewer enough to validate an AI output?

No. Human review is one control inside the workflow. Define what reviewers inspect, give them the source evidence, and preserve edits, rejections and escalations. Then test the review process alongside the model and record-handling controls.

How does FDA Computer Software Assurance (CSA) apply to GxP AI and LLM validation?

FDA’s final CSA guidance, Computer Software Assurance for Production and Quality Management System Software (February 2026), is written for medical device production and quality system software, so it does not directly govern a drug-side AI workflow. Its risk-based approach still transfers: put scripted acceptance tests on the failure categories where a wrong output affects product quality or patient safety, and use documented exploratory testing where risk is lower. Your quality unit decides how far to adopt it, and any validation partner should show how its test plan scales with risk.

How do you maintain ALCOA+ data integrity for RAG and lakehouse AI models under 21 CFR Part 11?

Version every input that shapes an answer: source documents, chunking and embedding configuration, index builds, prompts and the resolved model version. Retain the original output as a contemporaneous record attributed to the user and reviewer. Restrict retrieval to what the requesting user can access, and test export and restoration so the record stays enduring and available.