At a Glance
Audit log retention is set by the strictest rule you fall under, not by your tools. PCI DSS asks for 12 months with the latest 3 immediately searchable. Colorado’s AI law asks for 3 years of compliance records. SEC rules set notice clocks that only fast log search can meet. Keep logs in one governed store, tiered by age.
- Cost: the expensive part is keeping everything hot in a SIEM, not keeping it at all.
- Effort: most of the work is consolidating logs out of silos into one schema.
- Risk: a platform default of one year quietly becomes your policy if nobody sets one.
- When it fits: any team under two or more of these rules at once, which is most mid-market firms.
Most retention policies I review were never written. They are whatever the SIEM contract, the cloud provider and the data platform happened to default to. As the requirement text is widely quoted, PCI DSS v4.0.1 Requirement 10.5.1 says to “retain audit log history for at least 12 months, with at least the most recent three months immediately available for analysis.” That second clause is the one teams miss.
Retention is also about speed, not only duration. The SEC’s Form 8-K guidance gives a public company four business days to file once it determines a cybersecurity incident is material. Logs that exist but take a week to restore from cold storage do not help you inside that window.
The hard part is not picking a number. It is that the logs you need live in data silos: firewall and identity logs in the SIEM, cloud control-plane logs in a storage bucket, application logs in a vendor console, platform audit logs in your data platform. Each has its own retention default, and none of them agree. This reference separates what each rule says from what we recommend building.
The Comparison: What the Rule Says vs What We Build
Read the two halves of this table differently. The middle columns report what the rule or standard states, with the strength of our sourcing. The last column is our engineering recommendation, which is deliberately more conservative than the rule.
| Rule | What it asks of your logs | Stated period or clock | Source strength | Our engineering recommendation |
|---|---|---|---|---|
| PCI DSS v4.0.1 Req. 10.5.1 | Audit log history for the cardholder data environment | At least 12 months, most recent 3 months immediately available | Widely quoted requirement text; standard PDF not opened by us | 13 months in one table, last 90 days on query-ready compute, full period queryable without a restore |
| PCI DSS v4.0.1 Req. 10.4.1 and 10.4.1.1 | Daily review of security events and critical system logs, by automated mechanisms | Daily; automation mandatory since 31 March 2025 | Widely quoted requirement text | Scheduled detection jobs that write review evidence to their own table |
| SEC Form 8-K Item 1.05 | Enough evidence to judge materiality and describe the incident | File within 4 business days of the materiality determination | SEC staff guidance, primary | Incident-scoping queries that answer in minutes across all log sources |
| SEC Regulation S-P (amended) | Evidence of whether customer information was accessed | Notice to affected individuals within 30 days of becoming aware, per law-firm summaries | Compliance dates primary (FINRA); 30-day rule reported | Access logs joined to data classification so you can name affected records |
| HIPAA Security Rule | Existing Security Rule obligations; scope and period set by counsel | Not stated in this reference; confirm with counsel | Proposed 2025 update not finalized as reported | Retain to counsel’s figure; same tiered design as PCI |
| Colorado SB 26-189 | Records that demonstrate compliance for covered automated decisions | At least 3 years; law effective 1 January 2027 | Legislature page, primary | 3 years plus the current year for decision records and the access logs behind them |
| California CCPA ADMT rules | Ability to answer access requests about automated significant decisions | Access requests by 1 January 2027, per law-firm summaries | Reported, regulator text not opened by us | Decision records keyed to the consumer, retained to counsel’s figure |
| Databricks audit system table | Platform audit events (a source, not a rule) | 365-day default retention; table in Public Preview | Vendor docs, primary | Copy into your own Delta archive; never rely on the default as policy |
Two notes on that table. First, HIPAA: the Department of Health and Human Services proposed a major update to the Security Rule in January 2025, and as reported it remains a proposal, so none of its specific new requirements are current law. We have not stated a HIPAA retention figure because it is outside what we verified for this piece. Second, the EU AI Act: its Article 50 transparency obligations have applied since 2 August 2026, and a July 2026 amendment moved Annex III high-risk systems to 2 December 2027. If you have EU exposure, add those dates to your planning, but get the retention duty from counsel.
Why Retention Plans Break in Practice
Every failed retention plan I have seen broke on one of these, usually more than one.
The default became the policy
Nobody decided on one year. The SIEM licence covered 90 days hot, the cloud bucket had a lifecycle rule someone set in 2021, and the data platform keeps system tables for 365 days. The effective retention is the shortest of those for any given event. Write the policy down, per log class, and make the platform conform to it.
Hot and cold get confused
“Immediately available for analysis” is a performance requirement. A tape archive or an unrestored cold tier fails it even if the bytes exist. On a lakehouse this is easier, because a Delta table in object storage is queryable at any age; you are choosing how much compute to point at it, not whether it can be read.
Logs are spread across silos
Firewall, identity provider, endpoint, cloud control plane, SaaS admin logs and data platform audit events all arrive in different formats with different field names for the same user. You cannot answer “what did this account touch in the last 30 days” until they share one schema. That is why we normalise to OCSF before anything else.
Retention without integrity
A log store that the people being audited can edit is not evidence. Access control on the archive matters as much as its length: a separate catalog, grants to a small security role only, and the archive’s own reads recorded in the audit log.
Keeping too much
Privacy rules push the other way. Logs full of personal data kept “forever, just in case” become their own liability. Set an end date per class and enforce it with a job, not with good intentions.
One Lakehouse Retention Design That Covers All of Them
The design below meets the strictest period in the table once, rather than configuring retention separately in five tools. It is the same pattern we describe in our security data lake guide.
- Bronze. Raw logs land through Auto Loader or Lakeflow Connect into append-only Delta tables, one per source, in a restricted catalog. Nothing is transformed and nothing is deleted early.
- Silver. Lakeflow Declarative Pipelines normalise every source to OCSF classes, so an authentication event from the identity provider and one from the data platform share field names. Unity Catalog lineage records every hop from bronze to silver.
- Gold. Detection results, daily review evidence and incident-scoping views. This is what an assessor or an incident responder reads.
- Platform audit. A scheduled Lakeflow Job copies the Databricks audit and lineage system tables into the archive daily, because their default window is a year. Our Databricks system tables guide covers the columns worth keeping.
- Retention enforcement. A second job deletes rows past each class’s end date, then runs VACUUM. Set delta.deletedFileRetentionDuration and delta.logRetentionDuration deliberately, because they decide how long deleted data stays recoverable through time travel.
The point of one governed lakehouse here is not cheaper storage alone. It is that one retention policy, one access model and one lineage graph cover every log, so the answer to “how long do you keep it” is the same for every source.
What Each Retention Choice Costs and Risks
| Approach | Main cost driver | Main risk | Fits when |
|---|---|---|---|
| Rely on each tool’s default | Low and invisible | Shortest default wins; gaps appear mid-investigation | Never, for regulated data |
| Everything hot in the SIEM for the full period | Ingest and hot storage licensing at full volume | Budget forces sources to be dropped | Small log volumes, single rule |
| SIEM for recent alerting, lakehouse for the full period | Object storage plus compute when queried | Two systems to keep in step | Most mid-market teams under several rules |
| Keep everything forever | Grows every month | Privacy exposure and discovery burden | Almost never |
Our recommendation: keep recent weeks in the SIEM for alerting, and put the full retention period, normalised to OCSF, in a governed lakehouse archive with an explicit end date per log class. Our lakehouse security team designs and builds that layer. This article is general information, not legal advice; your counsel or a qualified assessor sets the retention period you must meet.
Frequently Asked Questions (FAQs)
How long does PCI DSS require audit logs to be kept?
As Requirement 10.5.1 of PCI DSS v4.0.1 is widely quoted, at least 12 months, with the most recent three months immediately available for analysis. Your QSA confirms how that applies to your environment.
Does “immediately available” mean logs must be in the SIEM?
No. It means they can be analysed without a restore. A Delta table on a lakehouse is queryable at any age, so recent months can live there as long as the queries run when needed.
How long does Databricks keep its own audit logs?
The system.access.audit table has a default retention of 365 days and is labelled Public Preview in the docs. Copy it into your own archive if any rule you follow needs longer.
Does the SEC four-day rule start when a breach is discovered?
No. Under Form 8-K Item 1.05 the four business days run from the company’s determination that the incident is material, which must be made without unreasonable delay. Fast log search is what makes that determination possible.
Do AI laws add new logging requirements?
Colorado’s SB 26-189 requires records demonstrating compliance to be kept for at least three years, from 1 January 2027. In practice that means decision records plus the access and lineage logs behind them.
Should security logs be normalised before they are archived?
Archive the raw form in bronze and a normalised OCSF form in silver. The raw copy preserves evidence exactly as received; the normalised copy makes cross-source questions answerable.