The Essentials
If nearly all your work runs on Databricks, Lakeflow Jobs (formerly Databricks Workflows) is the simplest orchestrator: native, governed and billed only through the compute your tasks use. Choose Airflow when a job must coordinate many systems outside Databricks. Choose Dagster when your team thinks in data assets and wants lineage in the orchestrator itself.
- Cost: Lakeflow Jobs adds no separate tool to run; Airflow and Dagster are open source but need hosting or a paid managed service.
- Effort: Jobs is configuration inside the workspace; Airflow and Dagster are Python code your team maintains.
- Risk: two orchestrators owning one pipeline is the most common failure pattern, whichever tools you choose.
- When a hybrid fits: an external scheduler triggers Databricks jobs, and Databricks owns everything inside the lakehouse.
Orchestration looks like a tooling choice until something breaks at 2 a.m. and three teams argue about whose scheduler owns the retry. A Fivetran benchmark of 500 senior data leaders found that pipeline failures take around 13 hours to resolve on average, and large enterprises see 4.7 of them a month.
The same survey reported that 97% of those leaders say pipeline failures slow their analytics and AI programs. A good part of that delay is not the failure itself. It is working out which tool scheduled the task, which one retried it, and which one is lying about the status.
That is why the old Databricks Workflows vs Airflow debate keeps coming back on forums, now with Dagster added to the list. The honest answer depends less on features than on where your work actually runs and who will maintain the orchestration code. This guide compares the three for a lakehouse team, shows when a hybrid is sensible, and lists the red flags that mean the tool is not your real problem.
What Each Orchestrator Is in 2026
Lakeflow Jobs is the native Databricks orchestrator. Databricks renamed Workflows to Lakeflow Jobs when it announced Lakeflow general availability in June 2025, alongside Lakeflow Connect for ingestion and Lakeflow Declarative Pipelines for transformation. So anyone searching Databricks Workflows vs Airflow today is really comparing Lakeflow Jobs.
Apache Airflow is the open-source, general-purpose scheduler. Workflows are Python DAGs, and a large library of providers connects it to almost every database, cloud service and SaaS API a company runs.
Dagster is an open-source orchestrator built around data assets rather than tasks. You declare the tables and files you want to exist, and Dagster schedules the work that produces them, with lineage and freshness tracked in the orchestrator. In July 2026, Prefect announced it is acquiring Dagster Labs. Dagster’s own announcement says the open-source project will be actively developed and that Dagster+ continues under its own brand.
How Lakeflow Jobs Works Inside the Lakehouse
According to the Lakeflow Jobs documentation, a job is a set of tasks that can run notebooks, Python scripts, SQL, or a Lakeflow Declarative Pipeline. Tasks have dependencies, retries and conditions. Control flow includes if/else branching and for-each loops, so a job can fan out over a list of tables without a separate script.
Triggers cover time-based schedules and events, including file arrival in cloud storage. That matters for lakehouse ingestion: a job can start when a vendor file lands rather than on a guessed clock time.
The biggest advantage is not a feature on that list. Because the job runs inside the workspace, the tables it produces sit in Unity Catalog with lineage captured as the work runs. You do not stitch lineage together from two tools, and your job definitions can live in source control through Declarative Automation Bundles, the renamed Databricks Asset Bundles. We compare bundles with notebook-driven development in our bundles vs notebooks guide.
How Airflow and Dagster Connect to Databricks
Airflow talks to Databricks through the Databricks provider. Operators such as DatabricksRunNowOperator start an existing job, and DatabricksSubmitRunOperator submits a one-off run. The provider’s DatabricksWorkflowTaskGroup launches a Databricks workflow and runs notebook tasks inside it, which keeps Airflow in charge of the schedule while Databricks runs the compute.
Dagster connects through its Databricks integration. With Dagster Pipes, Dagster submits Databricks jobs and streams logs and metadata back, so a Databricks run materializes a Dagster asset with its own lineage record.
Both are mature patterns. Both also mean two systems hold part of the truth about each run, and that is the trade you are making.
| Factor | Lakeflow Jobs | Apache Airflow | Dagster |
|---|---|---|---|
| Core model | Tasks and jobs inside Databricks | Task DAGs written in Python | Data assets written in Python |
| Best fit | Work that runs mostly on Databricks | Many systems across clouds and on premises | Asset-first teams that want lineage in the orchestrator |
| Where it runs | Inside the Databricks workspace | Self-hosted or a managed service | Self-hosted or Dagster+ |
| Governance and lineage | Unity Catalog, captured natively | Kept in Airflow, joined to Databricks by hand | Asset lineage in Dagster, table lineage in Unity Catalog |
| What you pay for | Compute your tasks use, in Databricks units | Hosting or a managed-service subscription, plus Databricks compute | Hosting or Dagster+, plus Databricks compute |
| Main risk | Weak reach into non-Databricks systems | Two sources of run status | Ownership change after the Prefect acquisition |
What Goes Wrong When Orchestration Lives in Two Places
Most teams do not pick one orchestrator. They inherit two. These are the failure patterns we see most often in hybrid setups.
Retries fight each other
Airflow retries a task that already retried inside Databricks. One failure becomes four runs, the compute bill doubles for the night, and a non-idempotent write lands duplicate rows in silver. Decide which layer owns retries, then turn them off in the other.
The status page lies
The external scheduler shows green because it successfully submitted a run. The run failed an hour later inside Databricks. Unless the operator waits for and reports the real terminal state, your on-call engineer is looking at the wrong screen.
Lineage breaks at the handoff
Unity Catalog records which tables a job read and wrote. The external tool records which task ran when. Nobody records both in one place, so impact analysis after a source change still takes a meeting.
Cluster start-up eats the schedule
Every external trigger that spins up fresh compute pays a start-up delay before any work happens. Chaining many small triggered runs multiplies that delay. Grouping related tasks into one Databricks job, whether triggered natively or by a task group, usually shortens the whole chain.
Nobody owns the orchestration code
Airflow DAGs and Dagster definitions are software. When the engineer who wrote them leaves, the schedule becomes something everyone is afraid to change. Lakeflow Jobs has the same risk if job definitions live only in the UI and never reach source control.
Which One Fits Which Team
For most lakehouse teams, the Databricks Workflows vs Airflow question resolves by where the work runs. Use Lakeflow Jobs alone when ingestion, transformation, machine learning and reporting all run on the lakehouse. That is the cheapest setup to operate and the easiest to govern. Our Databricks Lakeflow guide shows how Jobs chains Connect and Pipelines end to end.
Keep Airflow when the business process spans many non-Databricks systems: file transfers, on-premises databases, SaaS exports and downstream notifications. Let Airflow trigger Databricks jobs, and keep all lakehouse logic inside those jobs.
Choose Dagster when your team already models work as data assets and values freshness checks in the orchestrator. Given the Prefect acquisition, confirm the roadmap and support terms before you standardize on it for years. For a wider view of the options, including cost, see our data orchestration guide.
Red Flags That Mean the Tool Is Not Your Problem
Swapping orchestrators is expensive. Before you start, check for these signs. If two or more are true, fix the process and ownership first, then choose the tool.
- No one can draw the end-to-end flow. If mapping source to dashboard takes a workshop, a new scheduler will only reproduce the confusion in new syntax. Map the value stream first.
- Jobs run on a clock because nobody knows when data arrives. Fixed times hide late or missing source data. Agree on arrival signals with source owners before you design triggers.
- Every failure gets a manual rerun. That means there is no root cause analysis. Count failure reasons for a month and the top two causes usually are not the orchestrator.
- Transformations are not idempotent. If a rerun creates duplicates, retries are dangerous in any tool. Make writes safe to repeat, for example with AUTO CDC keys and sequencing, before you automate recovery.
- The same task exists in two schedulers. Duplicate definitions are a governance defect, not a feature gap. Pick an owner per pipeline and delete the copy.
This is the AIM-IT sequence in practice: Assess the flow, Innovate the process, then Model and Implement the orchestration, and Track it with real failure data. Databricks plus Lean Six Sigma thinking prevents most orchestration projects from becoming a rewrite of the same mess. If your Databricks Workflows vs Airflow decision keeps stalling, the flow map is usually why. If you want help deciding which orchestrator should own your lakehouse, our Databricks consulting team starts with that flow map.
Frequently Asked Questions (FAQs)
Is Databricks Workflows the same as Lakeflow Jobs?
Yes. Databricks renamed Workflows to Lakeflow Jobs in June 2025 as part of the Lakeflow general availability. Existing jobs kept working, and the capabilities carried over under the new name.
Do I still need Airflow if I use Databricks?
Not if your work runs mainly on Databricks. Lakeflow Jobs handles dependencies, retries, branching, loops and file-arrival triggers natively. Keep Airflow when a process must coordinate many systems outside Databricks.
Can Airflow trigger Databricks jobs?
Yes. The Airflow Databricks provider includes operators that run an existing job or submit a one-off run, and a task group that launches a Databricks workflow and runs notebook tasks inside it.
Should I choose Dagster after the Prefect acquisition?
It is still a reasonable choice for asset-first teams. Dagster states the open-source project will be actively developed and Dagster+ continues. Confirm roadmap and support terms directly before you commit to a long contract.
Which orchestrator is cheapest on Databricks?
Lakeflow Jobs usually has the lowest operating cost for Databricks-only work, because there is no separate orchestration platform to host or license. You pay for the compute your tasks consume, measured in Databricks units.
How do I move Airflow DAGs to Lakeflow Jobs?
Start with DAGs whose tasks all run on Databricks. Recreate each as a multi-task job, move schedules or file-arrival triggers across, define jobs in bundles for source control, then retire the DAG only after parallel runs match.