What You Need to Know
The best ETL tools for a lakehouse are the ones that land data as governed Delta tables without a second copy. On Databricks, Lakeflow Connect plus Lakeflow Declarative Pipelines covers most ingestion and transformation natively. Fivetran, Airbyte, Azure Data Factory and AWS Glue still earn a place for long-tail sources, and dbt earns one for SQL-first teams.
- Cost: native tools bill in Databricks consumption units, while third-party tools add a second meter such as monthly active rows or credits.
- Effort: a managed connector takes days to stand up; a hand-built source takes weeks and never stops needing care.
- Risk: every extra tool splits lineage, so audit trails get harder the more vendors touch a pipeline.
- Fit: pick the native path first, then add a third-party tool only for the sources it cannot reach.
Pipeline upkeep eats more of the week than most leaders admit. A 2026 Fivetran benchmark of 500 senior data leaders found that 53% of engineering capacity goes to pipeline maintenance rather than new work. That figure comes from a vendor that sells an ETL tool, so read it as a signal, not gospel.
The same survey put average time to resolve a pipeline failure at about 13 hours. Most of those hours are spent tracing a broken record back through three or four tools that each keep their own logs. That is the real cost of a stack assembled one purchase at a time.
Almost every ranking of ETL tools you will find assumes the target is a cloud warehouse. A lakehouse changes the question. You are no longer asking which tool loads Snowflake fastest. You are asking which tool writes open Delta tables, registers them in Unity Catalog, and keeps the bronze, silver and gold layers under one set of permissions.
Where Lakehouse ETL Projects Go Wrong
Before comparing products, name the failure modes. These four show up in almost every assessment we run, and none of them is solved by buying a faster connector.
Two copies of the same data
A warehouse-first tool often stages data in its own bucket, then loads it again into the lakehouse. You pay storage twice, egress once or twice, and you now have two versions of the truth that drift apart the first time a sync fails halfway.
Lineage that stops at the tool boundary
When extraction lives in one product, transformation in another and scheduling in a third, column-level lineage breaks at each handoff. An auditor asks where a revenue figure came from, and the honest answer is a spreadsheet someone maintains by hand.
Change data capture written by hand
Teams that pull from ERP databases often write their own merge logic. It works until a source table is truncated or a late record arrives out of order. Then the silver layer carries duplicates and nobody trusts the gold view feeding Power BI.
Automating a process nobody mapped
This one is not technical. A pipeline built on an undocumented business process moves bad data faster. In Lean Six Sigma terms, you have automated a defect. The AIM-IT sequence we use starts with Assess for exactly this reason: map the order-to-cash or procure-to-pay flow before a single connector is configured.
The Leading Options at a Glance
The table below compares eight ETL and data pipeline tools on the attributes that matter for a Databricks lakehouse. Pricing shows the billing model each vendor publishes, not a quote.
| Tool | Type | Where it runs | Pricing model | Lakehouse fit |
|---|---|---|---|---|
| Lakeflow Connect and Declarative Pipelines | Native ingestion and transformation | Inside the Databricks workspace | Databricks consumption (DBUs) | Highest: writes Delta tables governed by Unity Catalog |
| Fivetran | Managed ELT connectors | Fivetran SaaS | Monthly active rows, free tier available | Strong for long-tail SaaS sources |
| Airbyte | Open-source and managed ELT | Self-hosted or Airbyte Cloud | Free self-hosted Core, credits in Cloud | Good where you want to own connector code |
| dbt | SQL transformation framework | Runs SQL on Databricks SQL warehouses | dbt Core free, managed platform paid | Strong for SQL-first silver and gold models |
| Azure Data Factory | Orchestration and data movement | Azure | Per activity run and data movement hour | Common on Azure, overlaps with Lakeflow Jobs |
| AWS Glue | Serverless Spark ETL | AWS | Per DPU-hour | Workable on AWS, second catalog to govern |
| Informatica IDMC | Enterprise integration suite | Informatica cloud | Informatica Processing Units | Fits large estates already on Informatica |
| Matillion | Low-code ELT | Matillion cloud or your VPC | Consumption-based | Useful for analyst-built pipelines |
1. Lakeflow Connect and Lakeflow Declarative Pipelines
Databricks Lakeflow reached general availability in June 2025 and bundles three parts: Lakeflow Connect for ingestion, Lakeflow Declarative Pipelines (formerly Delta Live Tables) for transformation, and Lakeflow Jobs (formerly Workflows) for orchestration. Connect provides managed, serverless connectors; Salesforce and Workday were the first to reach GA in April 2025, and the catalog now spans dozens of sources across GA and preview. Auto Loader handles incremental file ingestion from cloud storage.
Declarative Pipelines is where most of the value sits. You declare tables in SQL or Python, and the engine builds the dependency graph. AUTO CDC (the newer name for APPLY CHANGES, which still works) computes inserts, updates and deletes from a change feed, with SEQUENCE BY ordering late records. Expectations enforce data quality rules at each layer. We cover the mechanics in our Databricks Lakeflow guide.
Strengths: one governance model from bronze to gold, lineage in Unity Catalog without extra tooling, serverless compute that scales to zero, one contract.
Limitations: the connector catalog is narrower than the specialist vendors, and a source outside it needs Auto Loader or custom code.
2. Fivetran
Fivetran is the reference point for managed ELT. It bills on monthly active rows (MAR), publishes a free plan, and offers a very wide connector catalog with Databricks as a supported destination. In October 2025 Fivetran and dbt Labs announced a merger, with dbt Labs stating that dbt Core stays under its current license.
Strengths: broad SaaS coverage, low setup effort, mature schema-drift handling.
Limitations: MAR billing is hard to forecast when a source table is rewritten, and lineage lives outside Unity Catalog until the data lands.
3. Airbyte
Airbyte offers a free, self-hosted edition called Airbyte Core alongside credit-based cloud plans. Its connector builder lets engineers write and maintain connectors for sources nobody else covers.
Strengths: no license cost for the self-hosted edition, full control of connector code.
Limitations: self-hosting shifts upgrades, scaling and on-call to your team, which is where the free price disappears.
4. dbt
dbt is a transformation framework, not an extractor. On Databricks it runs through the dbt-databricks adapter, and Lakeflow Jobs has a native dbt task that runs dbt Core projects from a Git folder against a serverless or pro SQL warehouse.
Strengths: version-controlled SQL models, tests and documentation that analysts can own.
Limitations: it overlaps with Declarative Pipelines, so a team running both needs a clear rule for which layer each one owns.
5. Azure Data Factory
Azure Data Factory is the default mover and scheduler in many Azure estates, with a self-hosted integration runtime for on-premises sources. It bills per activity run and per data movement hour.
Strengths: reaches on-premises systems behind a firewall, familiar to Azure teams.
Limitations: orchestration overlaps with Lakeflow Jobs, and two schedulers means two places to look when a load fails. Our data orchestration guide covers how to split the work.
6. AWS Glue
AWS Glue runs serverless Spark jobs and bills per data processing unit hour, published at $0.44 per DPU-hour for a standard ETL job in US East (N. Virginia).
Strengths: no clusters to manage, tight fit with S3 and other AWS services.
Limitations: the Glue Data Catalog becomes a second catalog to reconcile with Unity Catalog, and Spark tuning still lands on your engineers.
7. Informatica IDMC
Informatica Intelligent Data Management Cloud combines integration, data quality and master data management, priced in Informatica Processing Units.
Strengths: deep enterprise features and existing skills in many large IT shops.
Limitations: heavy for a mid-market team, and much of its transformation logic duplicates what the lakehouse now does natively.
8. Matillion
Matillion offers a low-code, browser-based pipeline designer that pushes transformation down to the target platform, Databricks included.
Strengths: visual design that analysts can pick up quickly.
Limitations: pipelines live in Matillion’s metadata, not in your repository, unless you enforce Git discipline from day one.
Your Lakehouse ETL Plan for This Quarter, Step by Step
Do these in order. Skipping the first step is how teams end up with a fast pipeline that loads the wrong numbers.
- Map the business process behind each source. Document how an order, invoice or ticket actually moves before you touch a connector. This is the Assess step, and it usually removes a third of the fields people thought they needed.
- Inventory sources against the Lakeflow Connect catalog. Mark each source as native, file-based (Auto Loader) or long-tail. Only the long-tail list needs a third-party ETL tool.
- Stand up one source end to end. Bronze, silver and gold, with AUTO CDC and expectations in place, registered in Unity Catalog. Prove lineage before you scale.
- Decide where transformation lives. Declarative Pipelines, dbt, or a written split between them. One owner per layer.
- Consolidate scheduling. Move job triggers into Lakeflow Jobs where possible, and keep an external scheduler only for systems it cannot reach.
- Track cost and failure rates weekly. Treat pipeline failures as defects with a control chart, not as tickets that close and vanish.
Tool choice is the easy part. The hard part is the process discipline around it. Talk to Analytics AIML about a lakehouse ingestion plan that starts with the process, not the purchase order, or read how we approach data engineering consulting.
Frequently Asked Questions (FAQs)
What is the difference between Lakeflow Connect and Fivetran or Airbyte?
Lakeflow Connect runs inside your Databricks workspace on serverless compute and writes straight to Delta tables governed by Unity Catalog. Fivetran and Airbyte run outside it and bill on their own meters. Most teams use Lakeflow Connect for supported sources and keep a third-party tool only for sources it does not cover.
Is dbt an ETL tool?
dbt handles the transform step only. It does not extract data from source systems, so it always pairs with an ingestion tool such as Lakeflow Connect, Fivetran or Airbyte. On Databricks it runs as a native task in Lakeflow Jobs.
Do I still need Azure Data Factory if I use Databricks?
Often not for orchestration, since Lakeflow Jobs covers scheduling, dependencies and file-arrival triggers. Data Factory still earns its place for on-premises sources reached through a self-hosted integration runtime.
What is the difference between ETL and ELT on a lakehouse?
ETL transforms data before loading it; ELT loads raw data first and transforms it inside the platform. A lakehouse naturally works as ELT: raw data lands in bronze, and the silver and gold layers carry the transformation.
How much do ETL tools cost?
Pricing models differ more than headline prices. Databricks bills in DBUs, Fivetran in monthly active rows, Airbyte Cloud in credits, and AWS Glue per DPU-hour. Model your largest and most volatile sources first, because rewrites and backfills drive most surprise bills.