Use a Databricks repair run when one task fails and the code and source data haven’t changed. It reruns only the failed tasks and everything downstream of them. Choose a full rerun when upstream logic changes, source records get deleted, or a streaming checkpoint is corrupted, because every layer has to be rebuilt from clean state.
I’ve spent more than 30 years building enterprise systems at IBM, Ernst & Young, Johnson & Johnson and McKesson. In that time, one habit has cost more than any single bug. A job turns red and someone reruns the whole thing. One silver-layer task drops a column, and the team recomputes bronze, silver and gold to fix it.
That habit means paying for the same cluster compute twice. It also breaks SLAs for the dashboards and ML models that read the gold layer, and it adds to a bigger bill. Gartner puts the average cost of poor data quality at $12.9 million a year per organization.
The time cost is just as real. Cross-industry studies cited in Harvard Business Review in 2017 found that analysts spend 80% of their time finding and preparing data instead of using it.
Every Databricks team runs into the same three failures: task errors, schema changes and timeouts. Below is the fix for each one and the alert that would have caught it. That’s what Secure. Solve. Scale. means on a lakehouse.
Why does a single failed task cost so much?
Without a standard recovery playbook, small failures get worse fast. The pattern runs in this order.
- Blind restarts: An engineer sees a red workflow and clicks a full rerun. The cluster recomputes bronze and silver tables that already succeeded, and you pay for that compute a second time.
- Checkpoint corruption: A streaming job fails on every restart because its checkpoint no longer matches the data. The team debugs Python for hours when the state directory is the real problem.
- Schema halts: A source system adds a column and ingestion stops. Nothing moves until a person steps in.
- Timeouts: A large merge hits a time limit. The team adds workers instead of splitting the work, and the job dies at the same point again.
- Stale dashboards: A timed-out job with no failure notification leaves Databricks SQL dashboards showing yesterday’s numbers. A business user spots the gap before the data team does.
- Late models: Feature tables, and the ML and generative AI models that read them, wait on the gold layer. A pipeline that misses its window pushes every model refresh behind it.
I’m a Lean Six Sigma Master Black Belt, and I read this list the way I’d read any defect log. That’s Databricks plus Lean Six Sigma in practice: each failure mode gets one standard response, written down before the 2 a.m. page.
Repair run or full rerun: how do you tell in a minute?
A Databricks repair run reruns only the failed tasks plus every task downstream of them. If task three of five fails, tasks one and two keep their results. You can edit job or task settings before you repair, so a fixed notebook or a bigger cluster applies to the rerun.
The rule is simple. Repair when the failure is local. Run a full rerun when something upstream of the failure changed, either the source data or the logic in a task that already succeeded.
Repair is only safe when your tasks are idempotent. An idempotent task leaves the table in the same state no matter how many times it runs. This is the core of Delta Lake error handling: write with MERGE INTO on a business key instead of a plain INSERT, and a repaired task never duplicates the rows it half-wrote before it failed.
| Failure | Recovery | Why |
|---|---|---|
| API rate limit hit during a silver-layer transformation | Repair run | Bronze data is already written. Rerunning the failed task alone saves compute. |
| Cluster ran out of memory | Repair run | Resize the cluster in the task settings, then repair. Only the failed task and its dependents run again. |
| Source system deleted historical records | Full rerun | The deletes have to flow through every medallion layer, including tasks that already succeeded. |
| Logic change in a task that already succeeded | Full rerun | A repair skips successful tasks, so the new logic never reaches their output. |
| Corrupted streaming checkpoint | New checkpoint and reprocess | The stream’s state is broken. A repair hits the same checkpoint and fails again. |
Teams that come from Airflow often miss this feature. Our Databricks Workflows vs Airflow breakdown covers the difference: Databricks keeps task state with the job run, so a repair starts from the failed run’s page with one click. Every repair that replaces a full rerun is compute you don’t pay for twice. Make it your first move unless something upstream changed.
What do you do when a streaming checkpoint is corrupted?
Structured Streaming and Auto Loader record their progress in a checkpoint directory. It tracks which files or table versions are done and where the next batch starts. Think of it as the pipeline’s memory.
Checkpoints break when someone deletes files in cloud storage by hand or restarts a query with incompatible changes. Idle time breaks them too. Delta keeps removed data files for 7 days by default before VACUUM can delete them. A stream that reads a Delta table and wakes up after that window can point at files that no longer exist.
A corrupted checkpoint can’t be repaired. Point the stream at a new checkpoint location and let it rebuild its state. Auto Loader then reprocesses every existing file in the source path by default, and your MERGE logic keeps the target table clean.
On Delta Live Tables, now part of Lakeflow, a full refresh does the same job. It clears the pipeline state and reprocesses from the source.
How does Auto Loader handle schema changes, and what is the rescued data column?
Auto Loader saves the schema it infers to the schema location you set with cloudFiles.schemaLocation. That file is its record of what the source looks like. Schema evolution is what happens when the source stops matching it.
Auto Loader schema evolution defaults to the addNewColumns mode. When a new column arrives, the stream stops with an UnknownFieldException, merges the column into the stored schema and waits for a restart. Set retries on the job and the restart happens without anyone touching it. The stop is deliberate.
Schema evolution adds columns. It doesn’t change data types. When a source sends a string where the schema expects an integer, the rescued data column catches the value.
Auto Loader writes any value that doesn’t fit the schema into _rescued_data as a JSON blob, along with the source file path. The rest of the row still loads. Auto Loader adds the column automatically when it infers the schema, and the rescuedDataColumn option lets you rename it.
The bronze table keeps loading. Your engineers look at the bad records in the morning instead of debugging at midnight.
Why do tasks die at the 15-minute mark, and how does chunking fix it?
Here’s a pattern we see often. A job pulls large JSON exports from a SaaS API and merges them into a silver table in a single task. It runs for 15 minutes and dies at 900 seconds, every time.
Most teams add workers and try again. The job fails at the same mark, because the problem is a time limit, not memory. The bigger cluster only makes each failed attempt more expensive.
Before you tune anything, find where the Databricks job timeout is set. Check the task’s timeout in the job definition, the execution timeout on the compute and any caller outside Databricks that waits on the run. Then change how the work is split, because the next export will be bigger.
That’s Databricks plus first principles thinking: treat the limit as a design constraint. Split the job into four tasks: validate, chunk1, chunk2 and chunk3. The validate task checks the files and divides them into batches. Each chunk task merges one batch and finishes well under the limit.
If chunk2 fails, repair the run. Only chunk2 and the tasks after it run again. Chunk1’s work stays committed, and because every chunk uses MERGE, the retry never duplicates data.
Which alerts would have caught each failure?
Fixing failures is reactive. Four alerts catch them before your users notice.
- Task failure: Send job failure notifications to the on-call channel with a link to the run. The engineer opens the failed run and uses the table above to choose repair or rerun.
- Duration drift: Query the job run tables in Databricks System Tables for each task’s run time. When a two-minute task starts taking twelve, alert and split the work before it reaches the timeout.
- Schema drift: Count the rows where _rescued_data is not null in each bronze table, and alert when the count goes above zero. You hear about an upstream type change before an analyst finds a broken metric.
- Checkpoint health: Flag any streaming job that hasn’t run inside the 7-day file retention window, before VACUUM deletes the files it needs.
Where should your team start?
Start with idempotency. Once every task writes with MERGE, a repair run is safe, and a fresh checkpoint costs you compute, not data quality.
Next, add the four alerts and split any task that runs close to its time limit. Hold off on bigger clusters until you’ve done both. A job that fails at the same mark on every run needs a different split, and more workers only raise the price of each retry.
Analytics AIML builds Databricks pipelines with idempotent writes, chunked tasks and drift alerts from day one. See how our Data Engineering team builds a lakehouse that recovers in minutes, not hours.
Frequently Asked Questions (FAQs)
When should I use a Databricks repair run?
Use a repair run when one task fails for a local reason, such as an API outage or an out-of-memory error, and nothing upstream changed. It reruns only the failed task and its dependents. You can change task settings, like cluster size, before you repair.
How do Databricks repair runs reduce cloud compute costs?
A repair run skips every task that already succeeded, so the cluster recomputes only the failed task and the tasks downstream of it. A silver-layer failure no longer means paying to reload bronze. Idempotent MERGE writes make that shortcut safe.
How do I fix a corrupted streaming checkpoint?
Point the stream at a new checkpoint location and restart it. The stream rebuilds its state from the source, and Auto Loader reprocesses existing files by default. Write with MERGE so that reprocessing doesn’t duplicate rows.
Why do Databricks workflows time out at 900 seconds, and how does chunking fix it?
A job that stops at the same mark on every run is hitting a configured limit, not a slow query. Check the task timeout in the job settings, the execution timeout on the compute and any external caller. Then split the work into a validate task and chunk tasks that each finish well under the limit, so a repair reruns only the chunk that failed.
How does Auto Loader schema evolution work?
In the default addNewColumns mode, Auto Loader stops the stream with an UnknownFieldException when a new column arrives and adds that column to the schema stored at your schema location. On restart, it loads the new column, and job retries handle the restart automatically. Values with the wrong data type land in the _rescued_data column instead, so the rest of the row still loads.