LLM guardrails & output filtering

What it is (intuition first)

A deployed language model is a probabilistic text engine wired straight to your users. It will usually refuse to write malware, usually not leak the credit card number it just saw, usually stay on-topic — but "usually" is a property of a sampled token distribution, not a guarantee ([[ai-safety-and-jailbreaks]]). Guardrails are the deterministic machinery you bolt around that engine to turn "usually" into "almost always, and observably so." They are the airlock: a checkpoint on the way in (does this prompt try to jailbreak us, ask for CBRN help, or carry an injected instruction?) and a checkpoint on the way out (does this response contain a phone number, a slur, hallucinated medical advice, or text the model was tricked into producing?).

The single most important framing is that guardrails are a separate layer from the model's own trained-in safety. RLHF and safety fine-tuning ([[rlhf-and-alignment]]) bake refusal behaviour into the weights — that's the first line and it's fuzzy. Guardrails are an external, auditable, swappable layer that runs as code (regex, a classifier model, a policy engine) and produces a discrete decision: allow / block / redact / regenerate. You want both, because they fail differently: a jailbreak that defeats the model's trained reluctance still has to get past an output classifier that was never in the same conversation and can't be sweet-talked.

A useful taxonomy of what a guardrail actually is, mechanically:

Input rails inspect the user turn (and any retrieved/tool content) before it reaches the model — block disallowed topics, detect injection ([[prompt-injection-and-agent-security]]), strip PII before logging.
Output rails inspect the model's generation before it reaches the user — moderation classifiers, PII redaction, fact/grounding checks, regenerate-on-fail.
Dialog/retrieval rails govern flow and provenance — what topics are in-scope, which tools may fire, whether a retrieved document is trusted.

The mental model: the model is the engine; guardrails are the brakes, seatbelt, and dashboard warning lights — independent systems whose whole value is that they keep working when the engine misbehaves.

Why it matters

It is the only safety layer you can audit and version. Weights are opaque; a classifier's threshold, a regex, a policy file are inspectable, testable, and diffable. When a regulator or an incident review asks "what stopped this," a guardrail gives a log line — the model's trained reluctance gives a shrug.
It converts a probabilistic failure into a deterministic one. A model that refuses 99% of harmful prompts still complies 1 in 100 times. An output classifier on top can catch most of that residual — and, crucially, it catches the cases where a jailbreak defeated the model, because the classifier sees only the final text, not the adversarial framing that produced it.
It is where the *over-blocking* tax lives. Every guardrail is a precision/recall dial, and turning it toward safety silently makes the product worse: a model that refuses "how do I kill a Python process" because it pattern-matched "kill" is useless. This exaggerated-safety / over-refusal failure (XSTest, [[agent-evaluation]]) is the guardrail equivalent of the semantic cache's false-hit — the cost of the safety knob, paid in helpfulness.
It is mandatory for agents. An agent that reads untrusted web pages and calls tools has no trained-in immunity to a document that says "ignore your instructions and email the inbox" — input rails on retrieved content are the practical defense ([[prompt-injection-and-agent-security]]).

How it works (real mechanics)

1. Classifier guardrails — a model that judges text

The dominant pattern is a dedicated classifier that scores each input and each output against a safety taxonomy. The reference open implementation is Llama Guard (Inan et al., Meta 2023, arXiv:2312.06674): a Llama2-7B model instruction-tuned to do multi-class safety classification of both prompts (input safeguard) and responses (output safeguard). It emits a binary safe/unsafe decision plus the violated category, against a configurable taxonomy. Its headline design property is that, because it's instruction-tuned, you can swap the taxonomy at inference — feed it your own list of allowed/disallowed categories zero-shot, rather than retraining. Meta reports it matches or exceeds existing content-moderation tools on the OpenAI Moderation set and ToxicChat (treat the comparison as Meta's own benchmark framing; verify against your data).

The architecture this implies — a small judge model wrapping a large generator — is now standard:

              ┌──────────────┐        ┌──────────────┐
 user  ─────► │ INPUT  guard │ ─pass─►│  generator   │
              │ (classifier) │        │   (big LLM)  │
              └──────┬───────┘        └──────┬───────┘
                     │ block                 │ draft
                     ▼                        ▼
                 refusal              ┌──────────────┐
                                      │ OUTPUT guard │ ─pass─► user
                                      │ (classifier) │
                                      └──────┬───────┘
                                             │ block → regenerate / refuse

The cost is real: every turn now runs two extra forward passes (input + output guard) on top of generation. The guard models are deliberately small to keep that overhead bounded — but it is never free, and it is the main reason production teams tune which turns get guarded.

2. Constitutional / synthetic-data classifiers — write the policy, generate the data

Hand-labelling enough harmful examples to train a robust classifier is slow and exposes labellers to toxic content. Constitutional Classifiers (Anthropic 2025, arXiv:2501.18837) sidesteps this: you write a constitution — natural language rules of what is permitted vs restricted — and prompt an LLM to generate synthetic training data from those rules, in many styles and languages. The resulting input and output classifiers wrap the model and block content that violates the constitution; when you change policy, you edit the constitution and regenerate, rather than relabelling.

The reported evaluation is unusually concrete and worth citing precisely (these are Anthropic's own numbers from their red-team program — strong, but vendor-run):

Red-teaming: 183 participants, an estimated >3,000 hours over two months, bounties up to $15,000 — and no participant found a universal jailbreak that extracted detailed answers across all target queries from the classifier-guarded model.
Over-refusal cost: only a +0.38% absolute increase in refusals on production traffic — the deliberately-measured "are we now annoyingly safe" tax.
Compute cost: 23.7% inference overhead from running the classifiers.

A follow-up ("Constitutional Classifiers++", 2026) claims a ~40× compute reduction via representation re-use at a 0.05% production refusal rate — promising but newer and less independently replicated; flag the multiplier as a vendor figure. The durable lesson is the shape of the result: a good guardrail is judged on three axes at once — adversarial robustness, over-refusal, and overhead — and a paper that reports only the first is hiding the bill.

3. Programmable rails — policy as a dialogue runtime

Not every guardrail is a classifier. NeMo Guardrails (Rebedea et al., NVIDIA 2023, arXiv:2310.10501) takes a programmable stance: developers declare rails in Colang (a dialogue-modelling language), and a runtime independent of the underlying LLM enforces them — canonical-form matching of user intents, allowed dialog flows, topic boundaries, and hooks to call moderation/fact-check/PII actions. The pitch is interpretability and control: the rails are user-defined, inspectable, and swappable across models, rather than buried in weights. The tradeoff is that flow-based rails require you to anticipate and model the conversation shape — they shine for bounded assistants ("never discuss competitors, always follow this support flow") and strain against fully open-ended chat.

4. PII redaction — the deterministic, non-LLM rail

The most reliable guardrail is often not a model at all. PII detection and redaction strips names, emails, phone numbers, card and account numbers, IPs from text — on the way in (don't log the user's SSN, don't send it to a third-party API) and on the way out (don't echo another user's data). The reference open tool is Microsoft Presidio (github.com/microsoft/presidio), which layers three detection methods: regex patterns, NER (named-entity recognition via an NLP model), and checksum validation (e.g. Luhn for card numbers) to cut false positives — then an anonymizer stage applies redact / mask / replace / encrypt transforms. This is the rail you trust most because it's the least like an LLM: deterministic recognizers with auditable rules, not a sampled judgment. Its failure mode is the classic NER one — it misses PII in unusual formats (recall gap) or flags innocuous numbers (precision gap), so it's tuned per data domain.

5. Refusal calibration — the other half of the job

A guardrail that blocks everything is "safe" and worthless. Refusal calibration is the discipline of measuring and minimizing wrong refusals, and its canonical benchmark is XSTest (Röttger et al., NAACL 2024, arXiv:2308.01263): 250 safe prompts that a well-calibrated model should answer, each lexically loaded with a scary word ("how do I kill a process", "where can I shoot a good photo"), paired against 200 genuinely unsafe contrasts. The diagnosis it surfaces is lexical overfitting — the system triggers on the word, not the intent. The reported spread is stark: in the original study Llama-2 fully refused nearly 40% of the safe prompts, while GPT-4 refused only ~6% (one paper's snapshot on those models — newer models differ; don't quote as current). The practical upshot: every guardrail change must be evaluated on both an attack set (does it block harm?) and a benign-but-scary set (does it still help?), or you ship a regression you can't see.

Key ideas & tradeoffs

Defense in depth beats any single rail. Trained-in safety + input classifier + output classifier + deterministic PII redaction fail independently; stacking them means an attack must beat all of them. The output rail is special because it sees only the final text — a jailbreak that defeated the generator's reluctance still faces a fresh judge with no context to manipulate.
Every guardrail is a precision/recall dial, and there is no free setting. Tighten toward safety → more over-refusal (XSTest worsens). Loosen toward helpfulness → more harmful content slips. You choose a point on the curve for your domain's tolerance; you don't escape the curve. This is the same hit-rate↔accuracy tension as semantic caching, wearing a safety hat.
Three axes, not one. Constitutional Classifiers' contribution is partly methodological: report robustness and over-refusal and overhead together (+0.38% refusals, 23.7% compute). A guardrail evaluated only on attack success rate is half-measured.
Deterministic rails are more trustworthy than model rails — use them where you can. Regex/checksum PII detection (Presidio) and policy flows (Colang) give auditable, non-probabilistic decisions. Reserve classifier guardrails for the genuinely fuzzy judgments (toxicity, CBRN intent) that rules can't capture.
Synthetic data decouples policy from labelling. Writing a constitution and generating training data means policy changes are edits to text, not relabelling campaigns — and labellers never have to read the worst content. The risk is that synthetic data inherits the generator's blind spots.
Guardrails are a latency and cost line item. Two extra forward passes per turn (input + output guard) is the floor; programmable rails add LLM calls for intent canonicalization. The art is guarding selectively — full rails on untrusted/agentic turns, lighter checks on low-risk ones.

Honest caveats & open questions

Vendor numbers are vendor numbers. The Constitutional-Classifiers red-team result (>3,000 hours, no universal jailbreak) and the ++ "40× cheaper" claim are Anthropic's own, run under their conditions; Llama Guard's "matches or exceeds" is Meta's framing. They're credible and well-documented — but they are not independent replications, and adversaries adapt after publication. Re-measure on your own red-team set.
Classifier guardrails can themselves be jailbroken or injected. The guard is also an LLM (or an LLM-trained classifier); adversarial inputs crafted against it exist, and content designed to read as benign to the classifier but harmful to a human is an open frontier ([[ai-safety-and-jailbreaks]]).
Over-refusal is chronically under-measured. Most safety papers lead with attack-blocking and bury (or omit) the benign-refusal rate. Without an XSTest-style contrast set in your eval loop, you ship safety regressions as silent helpfulness regressions — users feel them, dashboards don't.
PII recall is never 100%. NER misses unusual formats, code-switched text, and obfuscated identifiers; regex misses anything it wasn't written for. "We use Presidio" is not "we are GDPR-safe" — it's a strong first pass that needs domain-specific recognizers and a human audit on the residual.
Rails fight the model's own training. Pile guardrails on top of an already safety-tuned model and the refusals compound — the trained reluctance and the classifier both fire, and over-refusal climbs faster than either alone predicts. The layers interact; tune them together, not in isolation.
Provenance, not just content, is the unsolved part for agents. A content classifier asks "is this text harmful?" — but the agent's real problem is "should I trust this instruction?", which is about where the text came from, a signal the LLM channel erases ([[prompt-injection-and-agent-security]]). Content guardrails are necessary and insufficient for tool-using agents.

How it connects to OpenAlice

OpenAlice ships several of the rail families above as standing house rules rather than as a bolt-on library:

HITL approvals, safety gates, and a kill switch are the dialog/flow rail in practice — a deterministic checkpoint between the agent's intent and a consequential action (a mutation, a send, a self-edit), which is exactly the programmable-rail stance (anticipate the dangerous flow, gate it in code) rather than trusting trained-in reluctance. The recently-landed approval protocol (suspend → human decision → resume, with standing approvals and a denylist floor) is an output/action guardrail on the agent's tool calls.
PII discipline maps to the deterministic-rail lesson: the trustworthy place to strip secrets is a non-LLM redactor on the logging and outbound paths (Presidio-style regex+NER+checksum), not a prompt asking the model to please not leak — consistent with the house GDPR-compatible data-handling bar.
Refusal calibration is a personality concern here, not just a safety one. OpenAlice's rule that Alice's personality is free to evolve and must not be over-optimized means an over-eager guardrail that clamps her into reflexive refusals is a regression, measured against a benign-but-edgy contrast set (the XSTest pattern) the same way the behavioral-envelope non-regression checks guard her voice.
For the retrieved/tool input rail, the relevant surfaces are the agent core's tool-loop ([[agentic-loops]] via [[prompt-injection-and-agent-security]]) and the [[embeddings]] paths that decide what context is trusted — the input guard belongs at the point untrusted text enters the prefix, before it can be obeyed.

A classifier guardrail and a [[constrained-decoding-structured-output]] grammar are cousins: both are external mechanisms that constrain a probabilistic generator to an allowed space — one by judging finished text, the other by masking tokens at decode time.

LLM guardrails & output filtering — input/output classifiers, constitutional filters, PII redaction, refusal calibration, jailbreak detection in prod