Data Lineage Tools for a Lakehouse: Unity Catalog vs the Rest

What You Need to Know

On a Databricks lakehouse, Unity Catalog already captures column-level lineage for queries run on Databricks, at no extra license. You need another data lineage tool only when lineage must cover systems outside Databricks, survive longer than the system-table window, or live in an enterprise catalog. OpenLineage, Microsoft Purview and DataHub each fill a different part of that gap.

• Native coverage: tables, views, notebooks, jobs and dashboards on Databricks, down to the column.
• Where it stops: workloads outside Databricks, path-referenced tables and column-level UDF lineage.
• Cost: native lineage ships with Unity Catalog; external tools add a license or a platform to run.
• Decision driver: how much of your pipeline runs outside the lakehouse.

Most teams shopping for data lineage tools already own one. Databricks documents that Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, across every workspace attached to the metastore. No agents to install, no parsers to maintain.

That sentence carries a boundary, and the boundary is the whole buying decision. “Queries run on Databricks” excludes the first-mile ETL tool that lands files, the BI tool reading the gold layer, and any Spark job running elsewhere. The same documentation lists further gaps inside Databricks itself. A lineage graph that drops out at either edge looks complete and is not.

The cost of a wrong choice runs both ways. Buy a catalog you did not need and you pay for a second copy of metadata Unity Catalog already holds. Skip one you did need and an auditor or an incident review finds the hole for you. Our explainer on data lineage covers the concept; this piece is about the tools.

Where Exactly Does Unity Catalog Lineage Stop?

The documented limits are specific, and worth knowing before anyone tells you lineage is “end to end”:

• Workloads outside Databricks are absent. Lineage is captured for queries run on Databricks. A source system, an external ETL job or a BI dashboard appears only if you register it as external lineage.
• Path references break column lineage. When a source or target is referenced by a storage path rather than a table name, column-level lineage is not captured.
• UDFs are table-level only. Unity Catalog captures table-level lineage for UDFs but not column-level UDF lineage, and UDF lineage does not appear in the lineage system tables.
• System tables hold a rolling year. The lineage system tables, system.access.table_lineage and system.access.column_lineage, keep a rolling one-year window. Catalog Explorer retains lineage longer, but a query-based audit reads the system tables.
• External lineage stays out of the system tables. Databricks states that external lineage is not recorded in the two lineage system tables, and caps it at 10,000 external metadata objects and 100,000 external lineage relationships per metastore.

None of these is a reason to abandon native lineage. They tell you where a second mechanism has to start.

What Can You Add Inside Unity Catalog Before Buying Anything?

Unity Catalog external lineage lets you register upstream sources such as Salesforce or MySQL and downstream tools such as Tableau or Power BI as external assets, then draw lineage to and from Databricks tables. You add it through Catalog Explorer, the External Metadata and External Lineage APIs, or the Databricks SDK for Python. Lakeflow Connect managed ingestion pipelines record source lineage automatically, so a governed ingestion path fills the first mile without extra work.

For a company whose outside-the-lakehouse estate is a handful of sources and one BI tool, this is often enough. A scheduled job that reads your BI tool’s report catalog and writes external lineage through the API closes the last mile. Remember the system-table limit above: if your audit queries read lineage from system tables, external edges need their own copy in a Delta table you own.

How Do the Main Options Compare?

The table compares options on criteria, not rank. Every capability is from the vendor’s or project’s own documentation; check current versions before you commit.

A few details matter in that table. OpenLineage describes itself as an open framework for lineage collection and analysis that tracks datasets, jobs and runs; its Spark integration works through a listener that reads Spark’s dependency graphs. Microsoft Purview’s Unity Catalog connector states that its lineage extraction is passive and ingests what the Unity Catalog system tables expose, so it extends reach across your estate while inheriting the native capture underneath. The DataHub Databricks source is marked as production-ready and notes that column lineage extraction can slow ingestion.

Which Criteria Should Decide Your Choice?

Score your own situation against six questions. The answers, not a feature list, pick the tool.

• What share of your pipeline runs outside Databricks? If most transformation already lives in the lakehouse, native lineage plus a few external registrations covers it. If dbt, Airflow or another engine does heavy work outside, an OpenLineage-based path earns its place.
• Who reads the lineage? Engineers doing impact analysis are well served by Catalog Explorer. Stewards and compliance staff across many systems usually want one enterprise catalog.
• Do you already run a catalog? A company standardized on Purview or DataHub should feed it from Unity Catalog rather than build a parallel graph.
• How long must lineage be provable? If a record rule outlives the one-year system-table window, plan a retained Delta copy whichever tool you choose.
• How disciplined is table naming? Path-referenced reads and writes are the most common silent gap. Fixing them is cheaper than any tool.
• Who maintains it? Open-source options cost engineering time; commercial catalogs cost license fees. Neither is free.

Before any of this, look at the process. As a Lean Six Sigma Master Black Belt, I find that half the lineage gaps teams want a tool for are really undocumented handoffs: a CSV emailed between teams, a manual load nobody owns. A lineage tool will faithfully draw a broken process. Map it, remove the manual steps, then capture what remains.

Build It, Buy a Catalog, or Wait: What Is Your Signal?

Build on Unity Catalog when your sources land through governed ingestion and your transformations run in the lakehouse. The deciding signal: an impact question such as “which dashboards break if this column changes?” is answerable today in Catalog Explorer for most of your estate. Add external lineage for the remaining sources and BI tools, and copy lineage to retained tables. Our Unity Catalog guide covers the setup this depends on.

Buy or adopt a cross-platform catalog when lineage must span several warehouses, engines and BI tools that Databricks does not run. The deciding signal: your governance team already maintains a catalog, or an audit asks for one lineage view across systems you do not control. Feed it from Unity Catalog so native capture stays the source of truth for lakehouse workloads.

Wait when a migration is still moving workloads into the lakehouse. The deciding signal: more than a third of your pipelines are scheduled to move within two quarters. Every pipeline that moves gains native lineage on arrival, and the gap you would buy a tool to fill shrinks on its own. If you want help sequencing that migration and its lineage on Databricks, our Databricks consulting work starts with a lineage coverage check.

Frequently Asked Questions (FAQs)

What are data lineage tools?

They capture and display how data moves from source to destination: which jobs read which tables, which columns feed which reports. Some capture lineage automatically from query execution; others ingest it from metadata, logs or emitted events.

Does Unity Catalog provide lineage out of the box?

Yes. It captures table and column lineage automatically for queries run on Databricks, with documented gaps for path-referenced tables and column-level UDF lineage. Workloads outside Databricks appear only if you register them as external lineage.

Is OpenLineage a tool or a standard?

It is an open standard and framework for collecting lineage, with libraries and integrations for engines such as Spark, Airflow, dbt and Flink. You still need a backend to store and display the events it emits.

Can Microsoft Purview show Databricks lineage?

Yes. Purview’s Unity Catalog connector scans metadata and reads lineage from the Unity Catalog system tables, so the lineage it shows depends on what Databricks captured and on every relevant workspace being scanned.

How long does Databricks keep lineage?

The lineage system tables keep a rolling one-year window. If you need lineage as evidence for longer, copy it into a Delta table with your own retention on a scheduled job.

Do I need a separate lineage tool for compliance?

Not always. If your regulated data flows run inside the lakehouse, native lineage plus retained copies often answers the question. You need more when the flow crosses systems Databricks does not run.