What This Comes Down To
A security data lake is governed, low-cost storage for every security log you collect, normalised to one schema so analysts can query years of history. On Databricks you build it as a lakehouse: raw logs land in bronze, get mapped to OCSF in silver, and feed hunting and detection tables in gold, while the SIEM keeps only the alerts that need a human.
- Cost: you pay object storage plus the compute you run, instead of a licence priced on daily ingest.
- Effort: the hard work is source mapping, not infrastructure. Budget most of the project for OCSF field mapping and testing.
- Risk: the SIEM stays in place during the build, so nothing breaks for the SOC while the lake fills.
- When it fits: you already run Databricks, retention requirements outgrow your SIEM budget, or you have three or more log formats nobody can join.
Security teams used to argue about which vendor schema to adopt. That argument is mostly over. The Open Cybersecurity Schema Framework grew from 17 founding companies to more than 900 contributors and joined the Linux Foundation in November 2024, according to Databahn’s OCSF overview. When the vendors who sell you logs agree on the shape of an authentication event, the storage underneath stops being a security product and becomes a data platform choice.
The platform vendors have noticed. Amazon Security Lake normalises data to OCSF in your own S3 account, Microsoft Sentinel data lake stores a single copy in open Parquet files for up to 12 years, and in March 2026 Databricks announced Lakewatch, an agentic SIEM built on OCSF that it says reduces costs by up to 80%. That last figure is a vendor claim, not a benchmark, and Lakewatch is in Private Preview.
The practical problem is less glamorous. Firewall logs arrive as syslog, cloud audit trails as nested JSON, and endpoint telemetry in a vendor’s own format. Retention is capped because every gigabyte costs licence money, and an investigation that needs last year’s DNS data stops at the retention wall.
Why Cybersecurity Is a Data Engineering Problem
Cybersecurity is a data engineering problem. Detection quality depends on whether the right fields exist, arrive on time and mean the same thing across sources. A missing source IP, a timestamp in local time, or a username that is an email in one feed and a SAM account in another will defeat any detection rule, however clever.
That is the same work a data team does for finance or supply chain: ingest, conform, govern, publish. The lakehouse was built for exactly that pattern. Treating security logs as another governed domain in the lakehouse, rather than as a separate appliance, is the whole thesis of this article.
SIEM vs Security Data Lake: Who Does What
A security data lake does not replace the SIEM on day one. It changes what the SIEM is asked to hold. The split below is the one we use when scoping a build.
| Job | SIEM | Security data lake |
|---|---|---|
| Real-time alerting and case management | Primary | Feeds it curated signals |
| Long-term retention | Short and expensive | Years, on object storage |
| Threat hunting over months of data | Limited by retention | Primary |
| Joining security with HR, asset or finance data | Hard | Native, same catalog |
| Machine learning and anomaly models | Vendor features only | Your own models on gold tables |
| Compliance evidence and audit queries | Partial | Primary |
The lake takes the volume and the history. The SIEM keeps the workflow your analysts already trust.
Step 1: Assess the Sources Before You Write a Pipeline
We start every build with the Assess phase of our AIM-IT method. List every log source, its daily volume, its format, its owner and the detections that depend on it. Then decide, source by source, whether it stays in the SIEM, moves to the lake, or goes to both.
This step usually exposes duplicated feeds, sources nobody queries and detections that silently broke months ago. Fix those first. Automating a messy collection process only makes the mess arrive faster, which is the same lesson we wrote up in our field guide on not automating chaos.
Step 2: Land Raw Logs in Bronze With Auto Loader
Bronze is an exact, append-only copy of what each source sent. Route exports to cloud storage and let Auto Loader pick up new files incrementally into Delta tables. Keep the raw payload as a string column next to parsed fields, plus ingestion time, source file and source system.
Keeping the raw record matters in security more than anywhere else. When an investigator challenges a finding, you need to show the original event, not your transformed version of it. Bronze is your chain of custody, so never update it in place and set retention to match your legal hold policy.
Step 3: Map to OCSF in Silver
Silver is where each source is translated into the matching OCSF event class. Build one Lakeflow Declarative Pipelines flow per source family, and name the target tables after the class so analysts can find them without a data dictionary.
| Source | OCSF category | OCSF class (class_uid) |
|---|---|---|
| Identity provider sign-ins | Identity & Access Management | Authentication (3002) |
| Firewall and VPC flow logs | Network Activity | Network Activity (4001) |
| DNS resolver logs | Network Activity | DNS Activity (4003) |
| Web proxy logs | Network Activity | HTTP Activity (4002) |
| Endpoint process telemetry | System Activity | Process Activity (1007) |
| Cloud control-plane audit trails | Application Activity | API Activity (6003) |
Put data quality rules in the pipeline, not in a spreadsheet. Pipeline expectations such as CONSTRAINT valid_time EXPECT (time IS NOT NULL) ON VIOLATION DROP ROW keep broken events out of silver and record how many were dropped. Normalise every timestamp to UTC and every user identity to one canonical form here, once, instead of inside each detection.
For slowly changing reference data, such as asset inventory or user-to-department mappings, use AUTO CDC with SEQUENCE BY on the source change timestamp so late updates land in the right order. Enrichment against that reference data is what turns a raw IP address into a named laptop owned by a named team.
Step 4: Build Gold Tables for Hunting and Detection
Gold holds the tables people and rules actually query: failed sign-ins per user per hour, first-seen domains, rare parent-child process pairs, API calls from new regions. These are aggregates and joins built on silver OCSF classes, so one gold table can draw on identity, network and endpoint data at once.
Schedule the refresh with Lakeflow Jobs and publish the high-confidence results to the SIEM as findings. The SIEM receives fewer, richer events. That is where most of the licence saving comes from, and it is the same bronze, silver and gold discipline we describe in our medallion architecture guide.
Step 5: Govern It Like the Sensitive Data It Is
Security logs contain usernames, IP addresses, email subjects and sometimes session tokens. Register every table in Unity Catalog, grant access by group, and use row filters and column masks so a tier-one analyst sees what they need without seeing HR fields joined in from another domain.
Lineage in Unity Catalog also answers the auditor’s favourite question: which raw sources fed this finding. Our note on data governance consulting covers the access model in more depth.
Where Security Data Lake Builds Go Wrong
Most failures we see are not technical. They are scoping and ownership failures that surface as technical symptoms.
Mapping everything before shipping anything
OCSF has dozens of classes and hundreds of attributes. Teams that try to map every field of every source before going live spend a quarter in design. Map the fields your top ten detections use, ship them, then widen.
No owner for the schema
When a vendor changes its log format, someone has to update the mapping. If that person sits in neither the SOC nor the data team, parsing breaks quietly and detections go blind. Name one owner per source family and alert on parsing failure rates.
Treating the lake as a cheaper SIEM
Moving logs is not the goal. If analysts cannot query the lake in the tools and language they already use, they will keep asking for data in the SIEM, and you will pay twice. Build the gold tables around real hunting questions, and train the team on them.
Ignoring query cost
Storage is cheap, but a badly written hunt over three years of network flows is not. Partition and cluster silver tables on event time and source, and set compute policies so exploratory queries run on right-sized warehouses.
Three Build Paths, Scored on Cost and Risk
Buyers in May 2026 have three realistic ways to get a security data lake. Here is how they compare for a mid-market or enterprise team that already runs a lakehouse.
| Path | Cost profile | Delivery risk | Lock-in risk | Best when |
|---|---|---|---|---|
| Hyperscaler security lake (AWS or Microsoft) | Usage-based on the vendor’s cloud | Low | Medium, tied to one cloud’s tooling | Your estate is mostly on one cloud |
| Databricks Lakewatch | Not yet published, Private Preview | Medium, preview features change | Low, OCSF on Delta or Iceberg | You can join a preview and accept change |
| Build on your existing Databricks lakehouse | Storage plus compute you already manage | Medium, needs data engineering skill | Low, open tables in your own storage | Security data needs to join business data |
Our recommendation: if Databricks already holds your business data, build the bronze and silver OCSF layers on it now. That work carries straight into Lakewatch or any OCSF-native tool later, so it is the one investment that stays useful whichever product wins.
If you want help scoping the source inventory and the first detections, our Databricks consulting team runs that as a fixed-scope engagement.
Frequently Asked Questions (FAQs)
What is a security data lake?
It is a central store for security logs and telemetry, kept in open formats on low-cost object storage and normalised to a common schema. Analysts use it for long-term retention, threat hunting and analytics that are too expensive to run inside a SIEM.
Is a security data lake the same as a SIEM?
No. A SIEM focuses on real-time alerting, correlation and case management for the SOC. A security data lake focuses on volume, history and analytics. Most teams run both, with the lake feeding curated findings into the SIEM.
Why use OCSF instead of our own schema?
OCSF is an open, vendor-neutral schema backed by a large contributor base, so tools and detections written against it travel between platforms. A private schema has to be documented, maintained and translated by you for every new tool.
Do we need Databricks Lakewatch to build this on Databricks?
No. Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Jobs and Unity Catalog cover ingestion, normalisation, scheduling and governance today. Lakewatch adds SIEM features on top and is in Private Preview, so treat it as an option to evaluate, not a prerequisite.
How long does a first security data lake take?
The first useful slice, meaning three to five priority sources mapped to OCSF with gold tables for a handful of detections, is typically a matter of weeks, not quarters. The timeline is driven by source mapping and testing, not by infrastructure setup.
Which logs should move to the lake first?
Start with high-volume, low-alert sources such as DNS, VPC flow and proxy logs. They cost the most to keep in a SIEM and are the most valuable for historical hunting, so they give the fastest payback.