The Fast Version
Detection engineering is the practice of writing, testing, deploying and measuring the rules that turn security logs into alerts. On a lakehouse, each detection is a versioned SQL or Python query over OCSF-normalised silver tables, backtested against months of history before it ships, scheduled as a job, and tracked for precision like any other production process.
- Cost: detections run on compute you schedule, over storage priced for volume, so backtesting a year of data is affordable.
- Effort: the first investment is clean, normalised data. Rules are quick to write once fields mean the same thing everywhere.
- Risk: untested rules flood the SOC. Backtesting and staged rollout keep alert volume predictable.
- When it fits: your team already writes detections and wants version control, testing and history the SIEM cannot give it.
Detection content is no longer scarce. The open Sigma rule repository alone holds more than 3,000 community detection rules in a vendor-neutral YAML format. The bottleneck has moved from ideas for rules to data that can actually run them.
That bottleneck is a data problem, and it is expensive. A March 2026 Fivetran benchmark of 500 senior data leaders, reported by UK Tech News, found 53% of engineering capacity going to pipeline maintenance rather than new capability. Security pipelines are no exception.
In practice the symptoms look familiar. A rule fires on one log source and not another because the username field differs. A detection that worked last quarter goes silent after a vendor changes its format, and nobody notices for weeks. Analysts learn to ignore a noisy rule instead of fixing it.
Why Detection Engineering Is a Data Engineering Discipline
Every ranking guide on this topic treats detection engineering as a SOC skill. It is that, but it rests on data engineering. A detection is a query. Its accuracy depends on schema, freshness, completeness and join keys, which are exactly the things a data platform team manages every day.
This is why we say cybersecurity is a data engineering problem. Put the logs in a lakehouse, normalise them once, and detection engineers stop writing parsing logic inside every rule. They write logic about behaviour, which is the part only they can do.
The Lakehouse Detection Pipeline, Layer by Layer
The medallion pattern maps cleanly onto the detection lifecycle. Each layer has one job, and a failure in one layer is visible without digging through the others.
| Layer | What lives there | Detection engineering job |
|---|---|---|
| Bronze | Raw logs exactly as received | Evidence and replay source for investigations |
| Silver | Events mapped to OCSF classes, UTC time, canonical identities | The only layer detections read from |
| Gold (features) | Baselines, first-seen tables, per-entity aggregates | Context that turns noisy rules into precise ones |
| Gold (findings) | Detection hits with rule ID, version and ATT&CK technique | Alert history, triage outcomes and metrics |
Silver tables follow the OCSF event classes: Authentication, Network Activity, DNS Activity, Process Activity, API Activity and so on. A rule written against the Authentication class works whether the event came from your identity provider or a VPN concentrator.
Writing Detections as Code
Each detection lives in a Git repository as a query plus metadata: an owner, a severity, the MITRE ATT&CK technique it covers, the OCSF classes it reads, and the expected alert rate. Pull requests get peer review, exactly like application code.
Sigma rules make a useful starting library. They describe log matches in YAML and are converted to a target query language with tools such as sigma-cli or pySigma. Treat converted rules as drafts: map their field names to your OCSF silver columns and test them before trusting them.
Databricks is heading the same way. Its Lakewatch announcement in March 2026 describes detection rules defined in YAML with SQL or Python, backtested against historical data and deployed through CI/CD. Lakewatch is in Private Preview, but the workflow it describes is one you can run on any Databricks lakehouse today.
Backtesting Before a Rule Ships
This is where the lakehouse earns its place. Because silver holds months or years of normalised events, you run a candidate rule over history before it ever pages anyone. The backtest answers three questions: how often would it have fired, how many of those hits were known incidents, and which entities dominate the noise.
A rule that would have fired hundreds of times a day goes back for tuning. Add an allow-list from a gold table, join a baseline, or narrow the time window. Record the backtest result in the pull request so reviewers approve evidence, not opinion.
Scheduling, Alerting and Measuring
Approved detections run as scheduled tasks in Lakeflow Jobs or, for simpler threshold rules, as Databricks SQL alerts, which run a query on a schedule and notify when a condition is met. Hits land in a gold findings table with the rule version attached, and high-severity findings are forwarded to the SIEM or case tool the SOC already uses.
Before any of that runs unattended, fix the triage process itself. Agree what a true positive means for each rule, where analysts record the outcome, and who reviews a rule that keeps misfiring. Automating a triage workflow nobody has defined only produces faster confusion, so this step comes first in our AIM-IT sequence: Assess the current process, then Implement and Track.
Then measure the rules the way a Lean Six Sigma team measures any process. Track alert volume, true-positive rate and time to triage per rule, and treat a noisy rule as a defect with a root cause. We apply the same DMAIC thinking to pipelines in our note on Lakeflow and Lean Six Sigma; detection quality responds to it in the same way.
What Breaks Detection Programs
- Silent source failures. A log feed stops and the rules depending on it return nothing, which looks identical to a quiet day. Monitor event counts per source and alert on drops, not just on detections.
- Schema drift. A vendor renames a field and parsing fails. Pipeline expectations in silver catch it at load time instead of weeks later during an incident.
- Rules with no owner. A detection nobody maintains decays into noise. Every rule needs a named owner and a review date in its metadata.
- Coverage claimed but never tested. Mapping rules to ATT&CK techniques shows intent, not proof. Replay known attack data or emulation output through the rule and confirm it fires.
- Alert fatigue treated as a staffing problem. Hiring more analysts to clear noisy queues hides defective rules. Fix precision first.
Which Setup Fits Your Team
If your SOC is small and lives in the SIEM, keep detections there for now and move only the bulk, low-alert sources to the lakehouse for hunting and retention. Start writing new rules as code in Git so the move later is a deployment change, not a rewrite.
If you already run Databricks for business data, build the OCSF silver layer on it and move your ten most important detections first. Backtest them against history that was too expensive to keep in the SIEM, and forward only high-confidence findings.
If your detection engineers outnumber your data engineers, pair them. The rule authors own logic and severity; a data engineer owns ingestion, schema and freshness. Our Databricks Lakeflow guide covers the ingestion side they will share.
If you are evaluating Databricks Lakewatch, do the data work first. Normalised OCSF tables and a versioned rule repository carry straight into a Lakewatch pilot, and they keep their value if you choose a different tool. Governance for these tables follows the model in our guide to data security posture management.
For help standing up the OCSF layer and a first set of backtested detections, see our Databricks consulting services.
Frequently Asked Questions (FAQs)
What is detection engineering?
It is the discipline of designing, testing, deploying and maintaining the rules that turn security telemetry into alerts. Good programmes treat detections like software, with version control, peer review, testing and measured quality.
What is detection as code?
Detection as code means storing each rule and its metadata in a repository, reviewing changes through pull requests and deploying through a CI/CD pipeline. It gives you history, rollback and a clear owner for every detection.
Can we run Sigma rules on Databricks?
Sigma rules are vendor-neutral YAML that tools such as sigma-cli and pySigma convert into target query languages. Treat converted output as a draft, map fields to your OCSF silver columns, and backtest before enabling any rule.
Do we still need a SIEM if detections run on the lakehouse?
Most teams keep one for case management and the analyst workflow. The lakehouse runs heavy detections and history-based analytics, then forwards findings. Databricks Lakewatch, in Private Preview, brings SIEM features to the lakehouse itself.
How do we measure detection quality?
Track per rule: alert volume, true-positive rate from triage outcomes, time to triage and time since last review. Store triage results next to findings in gold so the numbers come from data, not from memory.