Data Lineage for Compliance Audits: Prove Where Regulated Data Went

The Short Version

Data lineage is the record of where a piece of data came from, what touched it, and where it went next. For an audit, it has to be evidence, not a drawing. On a governed lakehouse, Unity Catalog captures table and column lineage automatically for work that runs through it. Your job is to route regulated data through that path and close the gaps around it.

• Cost: the lineage capture and its system tables are free to use; you pay for the compute that queries them.
• Effort: the hard part is routing work through governed tables, not switching lineage on.
• Risk: lineage stops at exports, spreadsheets, renames and tools outside the catalog.
• When it fits: any team that has to answer “who used this data, and where did it go” on demand.

Every audit I have sat through ends up at the same question. Not “do you have a data catalog?” but “show me where this customer’s record went after it left the source system.” Most teams answer with a Visio file somebody drew two years ago. That file is an opinion. An auditor wants a record.

The record already exists if the work runs through the right place. Databricks Unity Catalog captures lineage for queries, notebooks, jobs, Lakeflow pipelines and dashboards, and writes it to system tables that keep a rolling 365 days of history. One date matters for anyone with a long audit window: lineage captured before 1 September 2024 is not available at all.

So the real difficulty is not the tooling. It is data silos. Regulated records live in an ERP, a CRM, a file share and three analysts’ laptops, and no single system saw the whole trip. This guide shows how to build lineage an auditor accepts, where it breaks, and how to close those breaks.

What an Auditor Actually Asks For

Regulators rarely say the word lineage. They ask for outcomes that are impossible without it. Colorado’s replacement AI law, SB 26-189, signed on 14 May 2026 and effective 1 January 2027, requires developers and deployers of covered automated decision-making technology to “retain records necessary to demonstrate compliance with the act for at least 3 years.” In California, law-firm summaries of the CCPA regulations report that businesses using automated decision-making for significant decisions must respond to consumer access requests by 1 January 2027.

Both rules land on the same engineering question. Which inputs fed this decision, which tables held them, and which people and jobs touched them along the way? In practice an auditor or an internal reviewer asks for four things:

• Origin. Which source system produced the record, and when it landed.
• Path. Every table, view and pipeline the record passed through on the way to a report, a model or an export.
• Access. Who read it, and whether that person was entitled to.
• Consumers. Every dashboard, model, file or downstream system that received it.

Lineage answers origin, path and consumers. The audit log answers access. You need both, joined, and you need them kept for longer than the platform keeps them by default.

How Unity Catalog Captures Lineage

The documentation is precise about scope: Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, and aggregates it across every workspace attached to the metastore. If a notebook, a SQL query, a Lakeflow pipeline or a job reads one Unity Catalog table and writes another, the edge is recorded.

Two system tables hold the evidence. system.access.table_lineage carries a record for each read or write event on a Unity Catalog table or path. system.access.column_lineage does the same at column level, for events that have a source. Each row names the source and target, the entity that did the work (a notebook, job or pipeline, with its ID), the user or service principal, and the event time. Both tables are generally available.

Access to the graph follows the same permissions as the data. A reviewer needs at least the BROWSE privilege on the parent catalog to see lineage, and workspace objects such as notebooks and dashboards carry their own permissions. That matters in an audit: the lineage view itself is access-controlled, so you can hand an assessor a scoped role without opening the whole estate.

The audit log sits beside it in system.access.audit, which records audit events from workspaces in the region. The docs label that table Public Preview, so treat it as a strong evidence source that you verify, not a finished compliance control. Joining the lineage tables to the audit log on user and time turns “this table fed that model” into “this analyst queried this table on this date, and this job wrote the result to that model’s feature table.”

Building Lineage an Auditor Will Accept

This is the sequence we run on a governed lakehouse. The order matters more than any single step.

1. Inventory and tag the regulated data first

You cannot prove where regulated data went if nobody agreed which data is regulated. Start with a list of the source fields that count: identifiers, financial attributes, health or employment data, anything that feeds a consequential decision. Apply governed tags in Unity Catalog so the classification travels with the column instead of living in a spreadsheet.

2. Land every source through a governed path

Bring ERP, CRM and SaaS data into bronze tables through Lakeflow Connect managed connectors or Auto Loader. The very first hop is a registered Unity Catalog table, so lineage starts at the edge of your estate instead of halfway through it.

3. Transform by table name, never by path

Build silver and gold in Lakeflow Declarative Pipelines and reference tables by their three-part names. Column-level lineage is not captured when a source or target is referenced as a storage path, so a pipeline that reads from a bucket URL leaves a hole exactly where an auditor will look. Our medallion architecture guide covers the layer design this depends on.

4. Orchestrate every hop in Lakeflow Jobs

Lakeflow Jobs (formerly Workflows) gives each run a job ID and a task ID, and the lineage record carries the entity that performed the write. Ad hoc notebook runs have no stable owner and no run history to match against. Orchestration is what turns lineage edges into an explainable timeline.

5. Persist the evidence past the default window

System tables keep a rolling year of data. A three-year record-keeping duty outlives that by two years. Schedule a small job that appends the lineage and audit tables into your own Delta tables in a restricted catalog, partitioned by date, with Delta retention settings chosen deliberately rather than left at defaults.

6. Join lineage to access and test it

Build one gold view that answers the four auditor questions for a given record or table. Then pick a real decision from last quarter and walk it end to end. If the view cannot reconstruct the trip, you have found your gap before the auditor did.

Where the Trail Goes Cold

Automatic capture is not total capture. These are the breaks we find on almost every assessment, ordered from the most common to the rarest. Each one compounds the next, because a single missing edge makes every downstream answer unprovable.

1. Exports and spreadsheets. The moment someone downloads a result to a CSV and emails it, the trail ends inside your estate and restarts nowhere. The audit log can show that a query ran and who ran it; it cannot show what happened to the file afterwards. The fix is process: publish governed views and dashboards so the export has no reason to exist.
2. Tools outside the catalog. A BI tool, a reverse ETL product or a third-party scheduler that reads through its own connection is invisible to Unity Catalog unless you register it. Unity Catalog lets you register external systems as external metadata objects so they appear in the lineage graph, but somebody has to own that registration.
3. Path-based reads. A job that reads a bucket path instead of a table name gives you, at best, table-level lineage and no column detail.
4. Renames. The docs are blunt: lineage is not preserved for renamed catalogs, schemas, tables, views or columns. A tidy-up sprint that renames forty tables quietly severs the history of all forty.
5. Code the engine cannot see into. Column-level lineage is not captured through user-defined functions, RDD operations or global temp views, and spark-submit tasks do not show in the lineage view.
6. Time. Nothing before 1 September 2024 exists, and system tables roll off after a year unless you copy them out.

Fix the Workflow Before You Trace It

Lineage records what happened. It does not make what happened sensible. If four teams each build their own customer table from the CRM, lineage will faithfully show you four tangled graphs. That is a process defect, and automation just documents it faster.

This is where our AIM-IT sequence earns its place. Assess the current flow of regulated data and count the hand-offs. Innovate by removing the ones that exist only because of data silos: the extract that feeds a spreadsheet that feeds another extract. Then Model the target flow on one governed lakehouse, Implement it in pipelines and jobs, and Track it with lineage and data quality checks. Our data quality framework covers the Track step, and our Unity Catalog guide covers the governance setup underneath it.

Score Yourself Before the Auditor Does

Give yourself one point for every statement that is true today. Be strict: “mostly” scores zero.

• We have a written list of regulated fields, and those fields carry governed tags in Unity Catalog.
• Every regulated source lands in a registered bronze table, with no manual file drops.
• No production pipeline reads regulated data by storage path.
• Every transformation of regulated data runs as a Lakeflow Job or pipeline with a named owner.
• BI tools and outbound integrations that receive regulated data are registered as external assets.
• Lineage and audit system tables are archived to our own Delta tables for as long as our longest record-keeping duty.
• We have reconstructed one real decision end to end in the last quarter.

Six or seven points: you can answer an auditor with evidence. Four or five: the platform is right and the process has leaks; close the gaps named above first. Three or fewer: you are running lineage on top of data silos, and the first job is consolidation, not tooling.

If your score came in low and the regulated flows span several source systems, our data engineering team builds exactly this: governed ingestion, orchestrated pipelines and an evidence layer that holds up. This article is general information, not legal advice; your counsel or a qualified assessor decides what your obligations require.

Frequently Asked Questions (FAQs)

What is data lineage in simple terms?

Data lineage is the recorded history of a piece of data: the source it came from, every transformation and table it passed through, and every report, model or system that consumed it. For compliance it has to be captured by the platform, not drawn by hand.

Does Unity Catalog capture column-level lineage automatically?

Yes, for queries run on Databricks against Unity Catalog tables. It is not captured for path-based references, through user-defined functions, or for renamed objects, so those patterns need to be designed out of regulated pipelines.

How long does Databricks keep lineage data?

The lineage system tables hold a rolling 365 days, and lineage captured before 1 September 2024 is not available. If you have a longer record-keeping duty, copy the tables into your own archive on a schedule.

Is lineage enough to prove who accessed regulated data?

No. Lineage shows how data moved between tables and consumers. Access comes from the audit log in system.access.audit, which is in Public Preview. You need both joined on user and time to answer an access question.

How do we capture lineage for Power BI or Tableau?

Register the BI tool and its reports as external metadata in Unity Catalog so they appear as downstream assets in the lineage graph. Pair that with governed gold views so reports read from registered tables rather than extracts.