Quick Answer
Lakeflow Connect is the ingestion layer of Databricks Lakeflow. It gives you managed connectors that pull data from SaaS applications and databases into Unity Catalog governed tables, incrementally and on serverless compute. Use it when your source is on the supported list and the connector’s limits fit your tables. Build your own path when neither is true.
- Cost: compute based, billed in Databricks units on serverless, with no separate connector license.
- Effort: a managed connector is configuration, not code, so a supported source lands in days rather than weeks.
- Risk: each connector has its own limits on tables, schema changes and history tracking, so read them before you commit.
- When it fits: governed, repeatable ingestion from common systems into a lakehouse you already run on Databricks.
Ingestion is where most data programs quietly bleed money. A Fivetran benchmark of 500 senior data leaders at large enterprises put the business exposure from pipeline failures at about $3 million a month, with an average of 4.7 failures a month.
The same survey found that failures take around 13 hours to resolve. Most of those hours are spent on the plumbing: API pagination, expired tokens, schema drift, and custom scripts that only one engineer understands. Lakeflow Connect is Databricks’ answer to that plumbing, and it is a good answer for the sources it covers.
The hard part is knowing which of your sources it covers well, which it covers with limits, and which it does not cover at all. Teams that skip that inventory end up with half their sources on managed connectors and half on scripts nobody planned to keep. This guide gives you the inventory method, the limits to check, and a scoring checklist for the build versus buy call.
What Lakeflow Connect Actually Is
Databricks describes Lakeflow as three parts: Connect for ingestion, Declarative Pipelines for transformation, and Jobs for orchestration. Connect is the front door. According to the Lakeflow Connect documentation, it organizes connectors by source type and produces ingestion pipelines governed by Unity Catalog, running on serverless compute by default.
The product reached general availability in April 2025, when Databricks announced that its Salesforce Platform and Workday Reports connectors were generally available. The same announcement listed SQL Server, Google Analytics, ServiceNow, SFTP, SharePoint and PostgreSQL as next on the roadmap. The catalog has grown since, but status varies by connector, so check each one rather than assuming the whole list is GA.
Managed Connectors vs Standard Connectors
The documentation splits connectors into two families, and the difference decides how much work you own.
Managed connectors are configured, not coded. You pick a source, choose the objects, point them at a target catalog and schema, and Databricks handles authentication, incremental reads and retries. SaaS connectors call the application’s API. Database connectors read changes from the source database.
Standard connectors give you more control in exchange for more code. Auto Loader handles files landing in cloud storage, and there are streaming sources such as Kafka. You write the ingestion logic and own its maintenance, but you can shape it however you need.
| Path | Best for | What you own | Where it stops |
|---|---|---|---|
| Managed SaaS connector | Salesforce, Workday and other supported apps | Configuration, object selection, schedules | Objects and fields the connector supports |
| Managed database connector | Change capture from supported databases such as SQL Server | Source setup, gateway, schedules | Per-connector table and schema-change limits |
| Auto Loader | Files dropped by an export, a vendor feed or another tool | The ingestion code and file conventions | Only as good as the files it receives |
| Custom API or JDBC code | Sources with no managed connector | Everything: auth, paging, retries, drift | Your team’s capacity to maintain it |
How the Database Connectors Work Under the Hood
Database connectors have three moving parts. An ingestion gateway runs continuously and reads changes from the source. A staging location holds those changes. An ingestion pipeline then applies them to your target tables incrementally.
That design matters for cost and for your database administrators. The gateway runs as a continuous task in its own job, so it is always on while the pipeline is active. On the source side, the SQL Server connector needs change tracking or change data capture configured, depending on which method you choose, and that is a change your DBA has to approve.
Scheduling is built in. For each schedule you add to an ingestion pipeline, Lakeflow Connect creates a job for it. On the first run the connector ingests everything you selected; on later runs it reads only what changed.
How Lakeflow Connect Is Billed
The GA announcement is explicit that Lakeflow Connect uses a compute-based pricing model. The Salesforce and Workday connectors run only on serverless infrastructure and bill as serverless pipeline usage, measured in Databricks units. There is no separate per-connector license fee published.
That changes the comparison with third-party ingestion tools. Row-based or volume-based pricing grows with how much data changes. Compute-based pricing grows with how long ingestion runs. For a stable source with modest change volume, compute billing is usually easier to forecast. For database sources, remember the always-on gateway when you model the monthly line.
Where Teams Get Stuck With Managed Connectors
The limits are specific to each connector, and they compound. Here is the order in which they usually bite.
- The source is not on the list. SAP is the common example. SAP data reaches Databricks through the SAP Business Data Cloud connector, which uses a separate sharing mechanism, not a Lakeflow Connect managed connector. Plenty of ERP and niche SaaS sources fall in the same bucket.
- The source is on the list but the status is preview. A preview connector is fine for a proof of concept and risky for a finance close. Check the status on the connector’s own page before you promise a date.
- Schema changes need a full refresh. New columns are usually handled automatically. Renamed columns and changed data types often are not, and a full refresh on a large table costs time and compute.
- History tracking is not what the business expects. Some connectors support SCD Type 2 history and some do not. If audit or finance needs point-in-time answers, confirm history support first, or plan AUTO CDC downstream.
- Per-pipeline limits force a design. Connectors cap how many tables or objects one pipeline can carry. Large sources need several pipelines, which means a naming and ownership plan before you start.
None of these is a reason to avoid Lakeflow Connect. Each is a reason to read the connector’s limitations table before you design around it.
How to Inventory Your Sources Before You Choose
Make a single list of every source feeding your lakehouse. For each one, record the system, the objects you need, the change volume, how fresh the data must be, and whether anyone needs history. Then add two columns: is there a managed connector, and what is its status.
In our work that list usually splits three ways. Mainstream SaaS and SQL databases go to managed connectors. File exports and vendor feeds go to Auto Loader. ERP systems and odd internal applications need a custom path or a dedicated integration. The goal is fewer custom paths each quarter, not zero on day one.
Process comes first here too. If a source is being fed by a manual export that someone runs on Fridays, a connector will faithfully ingest a broken process. Fix the handoff, then automate it. That is the same rule behind our data engineering consulting work.
Score Yourself: Managed Connector or Your Own Build
Take one source at a time and give it a point for each statement that is true. Total the points at the end.
- A managed Lakeflow Connect connector exists for this exact source and edition.
- That connector is generally available, not in preview.
- Every object and field you need is supported by the connector.
- The connector’s table or object limits fit this source without heavy splitting.
- Your expected schema changes are the kind the connector handles automatically.
- History support matches what audit and finance need, or you will add AUTO CDC downstream.
- Your Databricks workspace is on Unity Catalog and serverless compute is enabled.
- Nobody on the team wants to own authentication, paging and retry code for this source.
7 or 8 points: use the managed connector and move on. 4 to 6 points: use it, but write down the gaps and the workaround for each before go-live. 3 or fewer: build the path with Auto Loader or custom code, and revisit the managed option each quarter as the catalog grows.
If you are also weighing Snowflake’s ingestion options, our Snowflake Openflow vs Databricks Lakeflow comparison covers that decision, and the Databricks Lakeflow guide shows how Connect hands off to Pipelines and Jobs. When you want the inventory done with you, our Databricks consulting team runs the scoring source by source across your lakehouse.
Frequently Asked Questions (FAQs)
What is Lakeflow Connect in Databricks?
Lakeflow Connect is the ingestion component of Databricks Lakeflow. It provides managed connectors for SaaS applications and databases, plus standard connectors such as Auto Loader, and lands data in Unity Catalog governed tables on serverless compute.
Which Lakeflow Connect connectors are generally available?
Salesforce Platform and Workday Reports were the first connectors announced as generally available, in April 2025. Other connectors have been added since, with status varying between preview and GA, so confirm the label on each connector’s documentation page.
Is SAP a Lakeflow Connect connector?
No. SAP data reaches Databricks through the SAP Business Data Cloud connector, which uses a separate sharing mechanism. Treat SAP as its own integration project rather than a Lakeflow Connect configuration task.
How is Lakeflow Connect priced?
It is compute based. Managed connectors run on serverless compute and bill in Databricks units, with no separate published connector license. Database connectors also run a continuous ingestion gateway, so include that in your estimate.
How is Lakeflow Connect different from Fivetran and other third-party ingestion tools?
Third-party tools offer wider connector catalogs and run outside Databricks. Lakeflow Connect covers fewer sources but keeps ingestion inside the lakehouse, governed by Unity Catalog, on one bill. Many teams run both and move sources over as managed connectors mature.
Does Lakeflow Connect handle schema changes?
Partly. Connectors typically pick up new columns automatically, while changes such as renamed columns or new data types often require a full refresh. Each connector documents exactly which changes it handles.
Do I need Unity Catalog to use Lakeflow Connect?
Yes. Lakeflow Connect ingestion pipelines are governed by Unity Catalog, and serverless compute must be available in the workspace. Teams still on the legacy Hive metastore need to plan that upgrade first.