Red Teaming Keeps Forgetting What It Already Learned

Published at HCII 2026 in Artificial Intelligence in HCI (Springer, LNCS).

Red teaming is now a central method for evaluating LLM security. It is also, in practice, episodic: a campaign runs, findings are written up, and the operational knowledge of how the attacks actually worked largely evaporates. The next campaign — new model, new version, new team — starts close to where the last one started.

That would be tolerable if attacks were simple. They are not. Multi-turn attacks consistently outperform single-turn ones, which means the valuable knowledge is not a prompt but a sequence: a way of steering a conversation across several exchanges toward a boundary the model was supposed to hold.

Automated red-teaming frameworks have no persistent memory for any of that. The result is redundant effort and weak reproducibility. Teams rediscover known tactics, and the field cannot easily tell whether a model got safer or the testers got unlucky.

Attack Memory

We introduce Attack Memory, a RAG-inspired architecture for storing adversarial experience so that it compounds.

The architecture has four core components. The Attack Memory itself pairs a relational store of persistent traces and structured metadata with a vector database holding the semantic index. A Retrieval Module selects from it. Generation & Execution turns retrieved traces into live multi-turn attempts against the target system. A Learning Mechanism feeds outcomes back in, and human oversight sits alongside for annotation and quality review.

The core design decision is what gets stored. Attack Memory does not keep a library of prompt strings; it stores structured multi-turn traces — full adversarial interactions, preserved with the sequence intact — and indexes them with a retrieval-oriented taxonomy built specifically so that the right trace can be found later.

Retrieved traces then function as procedural blueprints rather than static prompts. This is the distinction the architecture rests on. A stored prompt can only be replayed, and stops working the moment a model is patched against that exact string. A blueprint encodes the approach — the shape of the escalation, the manoeuvre being attempted — which remains applicable against a model that has been hardened against the literal wording.

Memory That Adapts as Safety Boundaries Move

An append-only archive would decay into a swamp of tactics that no longer apply to any model in production. Safety boundaries move; a memory that ignores that becomes actively misleading about a system’s current robustness.

Two mechanisms handle this. Selective retention governs what earns a place in the memory rather than admitting everything. Temporal decay lets the relevance of stored experience fade as the safety landscape shifts, so that older traces lose influence without having to be deleted outright.

Retrieval runs as a three-stage pipeline:

1. Filtering — narrow the candidate set to traces plausibly relevant to the current target.
2. Re-ranking — order the survivors by how useful they are likely to be here.
3. Diversity promotion — spread the final selection across distinct tactics, via maximal marginal relevance (MMR).

That third stage exists to counter a specific failure of naive retrieval. Pure relevance ranking returns near-duplicates of whatever worked best before, which narrows a red-teaming campaign onto one line of attack precisely when it should be broadening. Enforcing diversity keeps coverage wide.

What the Architecture Enables

Three capabilities follow from having a memory at all, none of which are practical without one:

Cumulative red teaming. Each campaign starts from accumulated adversarial experience instead of from zero.

Longitudinal robustness tracking. Because traces persist and are indexed consistently, the same tactics can be re-run against successive model versions — turning “is this model safer than the last one?” into a measurable question rather than an impression.

Transfer across deployments. Tactics discovered against one system become retrievable when testing another, instead of staying with whichever team happened to find them.

What This Is, and What It Is Not

We want to be precise about the contribution, because it is easy to over-read.

This framework is conceptual, and it does not propose a new attack algorithm. It introduces no novel jailbreak and does not claim improved attack success rates against any model. What it provides is a reusable architectural substrate — a design and a taxonomy — that systematic red-teaming workflows can be built on.

That is a deliberate choice of target. The bottleneck in red teaming is increasingly not the cleverness of individual attacks but the absence of infrastructure to retain and reuse them. Novel attacks arrive steadily from the community. The ability to accumulate them does not.

What This Means

For red teams: the reproducibility problem is structural, not a matter of discipline. If your tooling has no persistent memory of multi-turn traces, findings will keep being rediscovered. Storing procedural blueprints rather than prompt strings is what makes stored knowledge survive a model patch.

For framework developers: persistent memory is a missing component in automated red-teaming systems, and retrieval design matters — particularly the diversity stage, without which retrieval quietly collapses a campaign onto a single line of attack.

For anyone tracking model safety over time: longitudinal comparison requires that the tests themselves persist in a consistent, indexed form. Without that, claims about a model becoming safer are hard to substantiate.

Publication

Heithoff, I., Woldai, B., Schacht, S. (2026). Conceptual Design and Taxonomy of a RAG-Based Attack Memory for LLM Red Teaming. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. Lecture Notes in Computer Science. Springer Nature Switzerland, Cham, pp. 61–79.

Read the paper on SpringerLink · DOI: 10.1007/978-3-032-30860-3_5

A conceptual architecture and retrieval-oriented taxonomy for accumulating multi-turn adversarial experience across red-teaming campaigns.