Vol. XX · Safety Training Plate 4 of 10robustness, adversarial training, distribution shift, rewording, red-teaming data, defence in depthAE

Training for Robustness.

A defence that holds only for the wording it was trained on is not a defence. Train on the variations too.

Safety training sees examples, and a model may learn the surface of those examples rather than the property behind them. If harmful requests in training were plainly worded, the same requests wrapped in role-play, translation, unusual formatting or many turns of innocuous context may not be recognised — a shift in the input distribution that adversaries exploit deliberately. Research on adversarial robustness in vision found the same pattern long before language models, and the remedy has the same shape.

Adversarial training finds inputs on which the model fails — by red-teaming, automated search or a second model — and adds them to training with the correct behaviour, so the model’s judgement extends to them. It is usually combined with other layers: input and output classifiers, monitoring, rate limits. The workshop uses the stylised refusal model: a share of harmful requests is reworded so that their perceived harm drops, and adversarial training removes a chosen share of that drop while making the model a little warier of sensitive requests.

Below: the share of reworded harmful requests helped, the share of all harmful requests helped, and the share of legitimate requests declined, against the strength of adversarial training.

What you will learn
Workshop
The stylised refusal model of plate XX.2 at threshold 0.5 and noise 0.1. Half of the harmful requests are reworded, lowering their perceived harm by δ = 0.3; adversarial training of strength a removes a share a of that shift and adds 0.08·a of wariness towards sensitive legitimate requests. A model, not a trained network; no attack content is involved.

Extend the judgement, pay a little for it

Without adversarial training, the reworded harmful requests are helped 40.0% of the time — the rewording has moved them below the model’s threshold — though the plainly worded ones are mostly caught. Training at strength 0.6 brings the reworded ones down to 16.0%, and over-refusal rises from 7.8% to 10.3% as the model grows wary of sensitive requests. Full-strength training gives 0.0% and 11.7%.

The pattern — robustness bought with some over-refusal — is common in practice, and it is why the training data for robustness must include legitimate requests that look like the adversarial ones, not just the adversarial ones. No single layer has proved sufficient against determined adversaries, so deployed systems stack defences: trained refusals, separate classifiers on inputs and outputs, monitoring for misuse patterns, and the ability to update quickly when a new failure is found. Each layer is imperfect; together they make failures rarer and more costly to find.

Robustness evaluation is itself adversarial: a model is only as robust as the best search for its failures has been able to show. That makes red-teaming a process run continuously, not a test passed once (plate XX.9).

Terms

Robustness
Keeping correct behaviour under changes to the input.
Distribution shift
Inputs that differ systematically from those seen in training.
Adversarial training
Training on inputs found to cause failures, with the correct behaviour.
Defence in depth
Several independent layers of protection, so one failure is not decisive.
Adversarial training extends the model’s judgement to the inputs that fooled it, usually at some cost in over-refusal; deployed systems stack it with other layers.

Sources

  1. Goodfellow, Shlens and Szegedy, “Explaining and Harnessing Adversarial Examples”, ICLR 2015.
  2. Ganguli et al., “Red Teaming Language Models to Reduce Harms”, arXiv:2209.07858, 2022.
  3. Anthropic, “Constitutional Classifiers”, arXiv:2501.18837, 2025 (input and output classifiers as a further layer).