Delta Live Tables Is Now Lakeflow Declarative Pipelines: What Changed

The Essentials

Delta Live Tables (DLT) was renamed Lakeflow Declarative Pipelines when Databricks took Lakeflow to general availability in June 2025. Existing DLT pipelines keep running with no migration. The Python module moved to pyspark.pipelines, APPLY CHANGES gained a newer name, AUTO CDC, and the old names still work. Plan a tidy-up, not a rewrite.

• Cost: the rename does not change how pipelines are billed, and classic SKUs still carry the DLT prefix.
• Effort: zero to run what you have; a few days per repo to adopt the new imports and AUTO CDC names.
• Risk: low for the code, higher for the people and process around it, where three names now describe overlapping things.
• When to act: fold the renames into normal releases, and start with any pipeline you are already touching.

Every Databricks team I talk to asks the same thing this spring: did the move from Delta Live Tables to Lakeflow break anything? The honest answer is no, and the more useful answer is that the rename is a good excuse to look at what your pipelines cost you in time. A Fivetran benchmark of 500 senior data leaders found that 53% of engineering capacity goes to pipeline maintenance rather than new capability.

That same report puts the average time to resolve a pipeline failure at about 13 hours. On a lakehouse, a large share of those hours comes from hand-written merge logic, unclear quality rules, and pipelines nobody fully understands anymore. A rename does not fix any of that. It does give you a reason to open the code.

The real problem most teams face is confusion, not breakage. Engineers see “DLT” in billing, “Lakeflow pipelines” in the docs, and “Spark Declarative Pipelines” in conference talks, and they stop trusting what they know. This guide separates the three, lists what actually changed, and shows where the upgrade is worth your time.

Three Names, Three Different Things

Databricks now uses three related names, and collapsing them into one is the most common mistake in team wikis and job descriptions. They are layered, not interchangeable.

Lakeflow Declarative Pipelines is the Databricks product you run and pay for. It is the direct successor to Delta Live Tables, announced as part of the Lakeflow general availability at Data + AI Summit 2025, and Databricks described it as 100% backward-compatible with DLT pipelines. The current docs often shorten it to “Lakeflow pipelines.”

Apache Spark Declarative Pipelines is the open-source framework Databricks contributed to the Apache Spark project. It is community-governed, and Databricks aligns its APIs with it starting in Apache Spark 4.1. It does not carry Databricks-only features such as Unity Catalog governance or serverless billing.

Delta Live Tables is the historical name. It survives in a few places on purpose, which is why you still see it in your invoice and your event logs.

What Changed in the Python API

The Databricks page on what happened to DLT is short and direct: there is no migration required, and your code will still work. The changes are naming changes that bring the Databricks API in line with the open-source framework.

Three of them matter in day-to-day code. The import moves from import dlt to from pyspark import pipelines as dp, so the decorator prefix becomes @dp. The @view decorator is now @temporary_view. And materialized views get their own @materialized_view decorator, while @table is the way to declare streaming tables.

The original dlt APIs keep working. That means you can adopt the new names file by file as you touch them, rather than scheduling a big-bang refactor that ties up a sprint for zero business value.

APPLY CHANGES Becomes AUTO CDC

Change data capture is where most hand-written pipeline code used to live, and it is where the rename has the most practical meaning. According to the AUTO CDC documentation, the AUTO CDC APIs replace the APPLY CHANGES APIs and keep the same syntax. APPLY CHANGES is still available, and Databricks recommends the new name.

The mechanics are unchanged. You declare a target streaming table, a source stream, the key columns, and a SEQUENCE BY column, and the engine computes inserts, updates and deletes for you. Out-of-order records resolve by the sequencing column, so a late update does not overwrite a newer one. You choose SCD Type 1 to overwrite in place, or SCD Type 2 to keep full history with start and end columns.

A few details trip teams up during the rename:

• The target must be a streaming table, not a materialized view.
• The sequencing column must be sortable, and NULL sequencing values are not supported, so clean the source first.
• In Python, create_auto_cdc_flow() handles a change feed, and create_auto_cdc_from_snapshot_flow() handles sources that only give you periodic full snapshots. The snapshot variant is Python only.
• Expectations do not apply to the snapshot variant, so put your quality rules upstream of it.

Expectations Still Carry Your Quality Rules

Data quality constraints came through the rename intact. The expectations documentation describes three actions on a failed record: warn and write it anyway, drop it before it reaches the target, or fail the update so a person steps in. In SQL that is EXPECT, ON VIOLATION DROP ROW and ON VIOLATION FAIL UPDATE; in Python, @dp.expect, @dp.expect_or_drop and @dp.expect_or_fail.

Metrics for warn and drop land in the pipeline event log. Failed updates record no metrics, because the update never completes. That is the difference between a quality rule you can trend over time and one that only shows up as an outage.

This is where Lean Six Sigma earns its place next to Lakeflow. An expectation is a specification limit. If you cannot say what “good” looks like for a column, you cannot write the rule, and no framework will write it for you.

Where the Rename Causes Real Friction

The code is safe. The friction sits in the people, documents and tooling around the code. These are the spots that cost teams time after the name change:

• Search results that mix eras. Stack Overflow answers written for DLT use import dlt, and newer docs use dp. Engineers copy both into one repo and nobody can tell which convention is the standard.
• Billing that does not match the docs. Finance sees DLT SKUs on the invoice while engineering talks about Lakeflow. Without a note in your cost reporting, the mismatch triggers a wasted investigation.
• Monitoring queries tied to old names. Event log schemas keep dlt in their names, which is correct, but a well-meaning engineer who “fixes” a dashboard query to the new naming breaks it.
• The open-source name used for the paid product. Calling your Databricks pipelines “Spark Declarative Pipelines” in an architecture document hides which governance and serverless features you depend on, and that matters the day someone proposes moving off Databricks.
• Hand-written merge code nobody retired. Many teams adopted DLT for new tables but left old MERGE scripts running beside it. The rename is the moment to find them, because AUTO CDC does the same job with far less code to maintain.

A Practical Upgrade Path That Costs Almost Nothing

Because nothing is forced, the sensible path is incremental. Run it in this order and you avoid disruption.

1. Write a one-page naming standard for your team: Lakeflow Declarative Pipelines for the product, dp for new code, DLT only where Databricks still uses it.
2. Add a linting check or code review rule so new files use from pyspark import pipelines as dp.
3. Switch APPLY CHANGES to AUTO CDC when you next change a CDC flow, and test the SCD Type 2 output columns before and after.
4. Inventory any standalone MERGE scripts outside your pipelines and move the stable ones into AUTO CDC flows.
5. Record expectation metrics from the event log in a small quality table so you can trend them.

If your bundles define pipelines, note that Databricks Asset Bundles were separately renamed Declarative Automation Bundles in March 2026. That rename was also non-breaking, and the databricks bundle CLI command stayed the same. For how we structure pipeline work across the lakehouse, the Databricks Lakeflow overview walks through Connect, Pipelines and Jobs together.

Red Flags That Say Fix the Process Before You Touch the Code

Some pipelines should not get the new names yet. If any of these describe yours, the process around the pipeline is the defect, and renaming it only makes a broken thing look current.

• Nobody can state the business rule behind a table. If the gold table exists because “finance asked once,” map the value stream first. An AUTO CDC flow built on an undefined rule just automates the argument.
• The same fix gets applied by hand every week. Recurring manual patches mean a quality rule is missing. Write the expectation, measure it, and only then refactor.
• Two teams own the same table. Shared ownership without a single accountable owner is how duplicate merge scripts appear. Settle ownership before you consolidate.
• Failures get restarted, never explained. If the standard response to a failed update is to rerun it, you have no root cause analysis. Start logging failure reasons before changing the code.
• The lakehouse layers are named but not enforced. Bronze, silver and gold in folder names do not guarantee anything. If silver tables still carry raw duplicates, fix the layer contract first.

This is the AIM-IT sequence we use: Assess the workflow, Innovate the process, then Model and Implement the pipeline, and Track it afterward. It is the same discipline behind our Lakeflow and Lean Six Sigma work, where the process gets fixed before the automation gets written. The lakehouse rewards teams that define quality before they scale it.

If you want a second pair of eyes on which pipelines are worth modernizing and which need process work first, our Databricks consulting team starts with exactly that assessment.

Frequently Asked Questions (FAQs)

Do I have to migrate my Delta Live Tables pipelines to Lakeflow?

No. Databricks states that no migration is required and existing DLT code keeps working. The newer API names are recommended, so adopt them as you touch each pipeline rather than in a single project.

Is Spark Declarative Pipelines the same as Lakeflow Declarative Pipelines?

Not quite. Spark Declarative Pipelines is the open-source framework in Apache Spark. Lakeflow Declarative Pipelines is the Databricks product that extends it and runs on the Databricks Runtime, with Unity Catalog governance and serverless options the open-source version does not include.

Does APPLY CHANGES still work?

Yes. APPLY CHANGES and apply_changes() still run. AUTO CDC INTO and create_auto_cdc_flow() replace them with the same syntax and behavior, and Databricks recommends the new names for new code.