The Short Version
A data quality framework on a Databricks lakehouse has three layers: native Lakeflow expectations that warn, drop or fail rows inside pipelines, a rules library such as DQX for profiling and quarantine, and a DMAIC control loop that decides which rules matter and who fixes the source. Tools catch defects. The process stops them coming back.
- Cost: expectations and DQX add no licence fee on Databricks; you pay for the compute that runs the checks.
- Effort: a first set of rules for one critical domain takes weeks, not months, if the business agrees on what “good” means first.
- Risk: DQX is a Databricks Labs project with no SLA, and its licence limits use to Databricks Services.
- Fit: all-Databricks shops get the most from this stack; hybrid estates need a portable library alongside it.
Every data team I have assessed says it has a data quality problem. Very few can tell me their defect rate. A Fivetran benchmark of 500 senior data leaders found 53% of engineering capacity goes to pipeline maintenance instead of new work, with failures averaging $3M a month in business exposure at large enterprises.
The downstream cost shows up in AI programs. A Cloudera and Harvard Business Review Analytic Services survey found only 7% of enterprises call their data completely ready for AI. That gap is not a shortage of tools. It is a shortage of defined standards, measured baselines and owners.
The problems are concrete. Null customer IDs land in silver tables and break joins. A source system changes a currency code and revenue doubles in a dashboard. Duplicate orders inflate a forecast, and nobody can say which rule should have caught them. This article lays out a framework that fixes those problems on the lakehouse, using the mechanics Databricks ships today and the Lean Six Sigma discipline that keeps the fixes in place.
What a Data Quality Framework Actually Contains
A framework is not a tool. It is four agreements written down and enforced: which data elements are critical, what each one must look like, where in the pipeline each rule runs, and what happens when a rule fails. The tooling only executes those agreements.
Most teams measure six dimensions, and you should name them explicitly in each rule so reports roll up cleanly:
- Completeness: required fields are populated.
- Validity: values sit in an allowed range, format or code list.
- Uniqueness: no duplicate business keys.
- Consistency: the same fact agrees across tables and sources.
- Timeliness: data arrives inside the window the business needs.
- Accuracy: values match the real-world record, usually checked by reconciliation against the source.
On a lakehouse the medallion layers give each rule a natural home. Bronze stays permissive so you never lose raw evidence. Silver enforces structure and business keys. Gold enforces the rules that protect a decision, a regulatory report or an AI agent. If you are still settling those layers, our guide to medallion architecture covers the layout this framework assumes.
Layer One: Lakeflow Expectations Inside the Pipeline
Lakeflow Declarative Pipelines expectations (the feature you knew as Delta Live Tables expectations) are the first-party control. You declare a constraint on a table and choose one of three actions:
| Action | SQL syntax | Python | What happens to bad rows | Where to use it |
|---|---|---|---|---|
| Warn (default) | EXPECT | expect | Written to the target, counted in metrics | Bronze, new rules on probation |
| Drop | EXPECT … ON VIOLATION DROP ROW | expect_or_drop | Dropped before the write | Silver, rows that are useless if wrong |
| Fail | EXPECT … ON VIOLATION FAIL UPDATE | expect_or_fail | The update stops; someone must intervene | Gold, rules that protect a financial or regulatory number |
Pass and fail counts appear on the pipeline’s Data quality tab and in the pipeline event log, which you can query like any other table. That event log is your measurement system, and it matters more than any single rule. Databricks also documents a quarantine pattern that routes invalid records to a separate table instead of dropping them, which keeps the evidence for root-cause work.
The limit of expectations is scope. They run inside a pipeline, on a table, at write time. They do not profile a new source for you, and they are awkward for checks that span many tables. That is where the second layer comes in.
Layer Two: DQX for Profiling, Rules and Quarantine
DQX is a Databricks Labs framework for data quality checks on PySpark DataFrames, batch or streaming. It covers row-level and dataset-level checks, can profile existing data to propose candidate rules, and lets you react to a failure by dropping, marking or quarantining records. Rules can be written as code or as YAML and JSON configuration, which means they can live in version control and go through review like any other change.
That last point is the one that matters to a Black Belt. A rule stored as configuration is a controlled document. You can see who changed a threshold, when, and why. DQX also works inside Lakeflow pipelines, so the two layers are not either-or.
Two limits belong in your risk register before you standardise on it. First, the project states it is provided for exploration and is not formally supported by Databricks with SLAs. Second, the DQX licence says you may not use the materials except in connection with your use of the Databricks Services. It is not a portable library for open-source Spark or another platform. If part of your estate runs outside Databricks, pair it with a permissively licensed tool such as Great Expectations for that side, and keep the rule definitions in one catalogue so the two do not drift.
Layer Three: DMAIC Turns Rules Into a Control System
This is the part most data quality articles skip, and it is why most frameworks decay within a year. Rules without a process become alert noise. Engineers mute them, thresholds creep, and the dashboard goes green while the data gets worse. DMAIC, the Define, Measure, Analyze, Improve, Control cycle from Lean Six Sigma, is the structure that prevents that. We run it inside our AIM-IT method, and the pairing is covered in more depth in our piece on Lakeflow and Lean Six Sigma.
- Define. Pick one decision the business cares about, then list the critical-to-quality data elements behind it. Ten well-chosen columns beat four hundred generic rules.
- Measure. Profile the source with DQX, run every rule in warn mode, and record a baseline defect rate per rule from the event log. Without a baseline you cannot prove an improvement.
- Analyze. Sort failures by volume and by business impact. A Pareto chart almost always shows a handful of source systems or entry screens producing most of the defects.
- Improve. Fix the cause upstream where you can: a required field on an entry form, a code list in the source, a corrected join key. Only then promote the rule from warn to drop or fail.
- Control. Chart the daily failure rate per rule and set control limits. A rule that drifts outside its limits gets an owner and a ticket, not a mute button.
The order matters. Teams that jump straight to fail mode stop production on day one, then disable the rule on day two. Measuring first keeps the pipeline running while you learn what normal looks like.
Where Data Quality Programs Break Down
Rules With No Owner
A failing rule needs a person who can fix the source, not just the engineer who wrote the check. When ownership sits only with the data team, fixes happen downstream as patches, and the same defect arrives again next week. Assign each critical element a business owner in Unity Catalog tags or your catalogue before you promote any rule past warn.
Thresholds Set by Guesswork
“Fail if more than 5% nulls” sounds reasonable until you learn the source has run at 12% for three years. Thresholds chosen without a baseline either fire constantly or never. Use the measured baseline and a control limit instead of a round number.
Quarantine Tables Nobody Reads
Quarantine is the right pattern, and it becomes a landfill when nobody reviews it. Put the quarantine row count on the same dashboard as the pipeline SLA, and give it a service level for triage. If rows sit there for a month, the rule is either wrong or unowned.
Two Rule Stores Drifting Apart
Expectations in pipeline code, DQX rules in YAML, and a third set in a BI tool will disagree within a quarter. Keep one catalogue of rule definitions and generate the others from it, or at minimum review them together on a fixed cadence.
Checks That Stop at Structure
Null and format checks are easy, so teams stop there. The expensive defects are semantic: a valid-looking price in the wrong currency, a shipped date before the order date. Write at least a few cross-field and reconciliation rules for every gold table that feeds a decision or an agent.
Build, Buy or Wait: The Signal That Decides Each
Build on native expectations and DQX when your pipelines already run on Databricks and your rule count is in the dozens to low hundreds. The deciding signal: most of your governed tables are produced by Lakeflow pipelines or notebooks in one workspace. You get checks where the data is written, metrics in the event log, and no second platform to run.
Buy a data observability product when the estate spans several warehouses, lakes and BI tools and you need anomaly detection across all of them from one screen. The deciding signal: your worst incidents start outside Databricks, in a source or a downstream tool the lakehouse never sees. A lakehouse-only framework cannot watch what it does not touch.
Wait on tooling and fix the process first when nobody can name the ten data elements that matter most or who owns them. The deciding signal: you ask three executives what “a customer” is and get three answers. Adding rules to an undefined standard produces alerts nobody acts on. Run Define and Measure first, then pick tools against a known defect profile.
If you want help setting the baseline and wiring expectations, DQX and control charts into your pipelines, our Databricks consulting work starts with exactly that assessment.
Frequently Asked Questions (FAQs)
What is a data quality framework?
It is a documented set of standards, checks and responses for your critical data: which elements matter, what valid looks like, where each check runs and what happens on failure. On a lakehouse the checks run inside pipelines, and the framework also defines owners and how results are reviewed.
Do Lakeflow expectations replace a separate data quality tool?
For checks on tables your pipelines write, expectations cover a lot: warn, drop or fail per rule, with metrics in the event log. They do not profile new sources or span many tables well, so teams add DQX or another library for profiling, reusable rules and quarantine.
Can I use DQX outside Databricks?
No. The DQX licence restricts use to connection with the Databricks Services, so it is not a portable choice for open-source Spark or another platform. Use a permissively licensed library such as Great Expectations for that side of a hybrid estate.
Where should data quality checks run: ingestion or transformation?
Both, with different actions. Keep bronze permissive with warn-level checks so raw evidence survives. Enforce structure and keys in silver with drop or quarantine, and reserve fail for gold rules that protect a financial, regulatory or AI-facing output.
How does DMAIC apply to data quality?
Define the critical data elements, Measure a baseline defect rate, Analyze which sources cause most failures, Improve the source process, then Control with charted failure rates and owners. It turns a pile of rules into a system that keeps defects from returning.