AI Red Teaming Starts With a Gold-Layer Eval Set

What This Comes Down To

AI red teaming is structured adversarial testing of a model, agent or RAG system before and after launch. It only proves something when every attack has a known-correct expected outcome. For an assistant that answers from company data, those expected outcomes come from governed gold-layer tables, so the eval set is a lakehouse data engineering asset first.

• Cost: the expensive part is curating expected answers, not running the attacks.
• Effort: a first eval set of a few hundred cases is a sprint, not a program.
• Risk: attacks scored against vibes produce reports nobody can act on.
• Fit: essential once an agent reads internal data or can call tools.

Most writing on AI red teaming focuses on clever prompts: jailbreaks, role-play, encoded payloads. Those matter. But when an internal assistant answers “what is our refund limit for enterprise accounts,” the attack that hurts you is often quiet. It gets a confident wrong number, or a number from a table the user should never have seen.

You can only catch that if you know the right number and the right access boundary in advance. IBM’s Cost of a Data Breach Report 2026 traced the root causes of AI-related incidents to compromised connected APIs, applications and cloud misconfigurations rather than model choice. In other words, the plumbing around the model fails first, and the plumbing is data, which is why we build red-team suites inside the lakehouse rather than beside it.

Which Taxonomies Should Shape the Test Plan?

You do not need to invent attack categories. Three public references cover the ground, and each answers a different question.

The NIST AI RMF gives you the Measure function, which is where testing lives, and the Generative AI Profile (NIST AI 600-1) names the generative risks, such as confabulation and data privacy, that your cases should target. The OWASP Top 10 for Agentic Applications lists categories including agent goal hijack, tool misuse and exploitation, identity and privilege abuse, and memory and context poisoning. MITRE ATLAS adds attacker sequencing. Tag every test case with the category it exercises in each reference, and coverage becomes a query instead of an opinion.

Why Do Red-Team Results Fall Apart Without Gold Data?

Here is the pattern we see when a red-team exercise disappoints. A team writes a few hundred attack prompts, runs them against the assistant and asks a reviewer, or an LLM judge, whether each response “looks wrong.” The judge flags the obvious jailbreaks. It misses the answer that is fluent, plausible and off by a decimal place, because nothing in the test tells it what correct looks like. Then a source system changes, the gold table is rebuilt, and half the “expected” answers written in a spreadsheet three months ago are now themselves wrong, so the next run produces false failures that people learn to ignore. Access tests suffer the same fate: nobody wrote down which rows a sales user is allowed to see, so a response that leaks another region’s pipeline passes because it is accurate. The root cause is not the attack library. It is that expected outcomes were authored by hand, outside the governed data, with no version, no lineage and no owner. Fix that and the attacks start producing findings you can route to a team and close.

How Do You Build the Eval Set From the Gold Layer?

Treat the eval set as another governed product in the lakehouse, built from the same tables the agent reads. The build has five moves.

1. Start from questions with deterministic answers. Pull the business questions the assistant is meant to answer and pair each with the SQL that answers it against a gold table. The query result is the expected answer. If no query can produce an answer, that question tests refusal, not accuracy.

2. Add access expectations per persona. For each question, record the expected outcome for each role: the sales user sees their region, the analyst sees all, the contractor gets a refusal. Those expectations should come from the same row filters and column masks that Unity Catalog enforces, so the test and the control share one definition.

3. Layer adversarial variants on top. For each clean question, generate attack variants tagged to a taxonomy category: an injected instruction inside a retrieved document, a request phrased to bypass a mask, a chain of small questions aimed at reconstructing a protected field. The expected outcome stays tied to the clean answer or the refusal.

4. Store it where it is versioned. On Databricks, MLflow evaluation datasets are stored in Unity Catalog, which provides versioning, lineage, sharing and governance. Each record holds inputs, expectations and optional tags, and records can be added from existing tables or from production traces. The documented limits are 2,000 rows per dataset and 20 expectations per record, so split large suites by domain.

5. Regenerate expectations when gold changes. Because each expected answer is a query, a scheduled job reruns the queries after every gold refresh and updates the expectations. Lineage tells you which test cases depend on a changed table. This is the step that keeps the suite honest for more than a quarter.

The quality of the suite is capped by the quality of the gold layer underneath it. If gold tables carry duplicates or stale dimensions, the expected answers inherit them. Our data quality framework covers the checks that should pass before a table becomes a source of truth for testing.

What Should Run Continuously After Launch?

A red-team exercise before launch is a snapshot. Agents change when prompts, tools, models or the data underneath them change, so the suite has to run again on each of those events.

Wire the eval run into the release path: a new model version, a new tool, a prompt change or a major gold-table change triggers the suite, and a failure threshold blocks promotion. In production, route model and tool calls through a gateway that records them. Unity AI Gateway reached general availability on 4 August 2026 with guardrails for PII and prompt injection and unified tracing across models, agents, MCP servers and tools. Traces from real traffic are the best source of new test cases: when a guardrail blocks something novel, add a sanitised version to the eval set.

Guardrails and tests are two halves of one loop. Guardrails stop known bad patterns at run time; the eval set proves the guardrails still work after each change. We walk through the run-time side in our post on RAG guardrails.

What Should You Ask Before Buying a Red-Teaming Tool or Building Your Own?

Commercial red-teaming tools bring large attack libraries and good reporting. Building in-house gives you tight coupling to your data. Either way, these questions separate a useful program from a demo.

• Where do expected answers come from? If the answer is “an LLM judge decides,” ask how it knows your refund limit. Expected outcomes for company-data questions must trace to a governed table.
• Can tests carry per-role access expectations? An agent that answers correctly for the wrong person has failed. The tool or framework must test the same question under several identities.
• Is every case tagged to a taxonomy? Ask for coverage reported against OWASP agentic categories and ATLAS techniques, not a raw count of prompts.
• What triggers a rerun? If the suite only runs on demand, it will stop running. It should hook into model, prompt, tool and data changes.
• Where do results live? Results belong next to the model version and dataset version they tested, in tables your auditors can query, not in a vendor dashboard you lose at renewal.
• Who closes a finding? Every failure class needs an owner: data engineering for wrong answers from bad gold, security for access leaks, the application team for tool misuse.

If the honest answer to the first two questions is “we don’t have governed gold data yet,” that is where the work starts. Our AI on the lakehouse engagements build the gold layer and the eval set together, so testing has something true to test against.

Frequently Asked Questions (FAQs)

What is the difference between AI red teaming and AI evaluation?

Evaluation measures how well a system does its intended job on representative cases. Red teaming deliberately tries to make it fail, leak or misbehave. In practice they share one dataset: clean cases measure quality, adversarial variants measure resilience, and both need known-correct expected outcomes.

How many test cases does a first red-team eval set need?

Start with a few hundred cases spread across your highest-risk questions, each persona and each taxonomy category you care about. Coverage across categories matters more than raw volume. Grow the set from production traces once the agent is live.

Can an LLM judge score red-team results?

For tone, refusal quality and policy wording, a judge is useful. For factual answers drawn from company data, compare against the expected value from the gold table first and use the judge only where no deterministic answer exists.

Is AI red teaming legally required?

The NIST AI RMF and its Generative AI Profile are voluntary frameworks, not regulations. Customers, auditors and internal risk committees increasingly ask for testing evidence anyway, so treat it as a control you will be asked to show. This is general information, not legal advice; counsel decides what your obligations are.

Who should run red teaming in a mid-market company?

Security usually leads the attack design and data engineering owns the eval set and the expected answers. The application team fixes what fails. A small company does not need a separate red team, but it does need those three owners named.