Guardrails, for people building on top of an LLM
The runtime controls that keep a live system's behavior inside acceptable bounds — distinct from how the model was trained, and distinct from how you measure it offline. What to check, where to check it, and why every check here eventually needs its own testing.
Enforcement, not disposition or measurement
Three layers get conflated under "AI safety," and they're worth separating cleanly. Training-time alignment — RLHF, Constitutional AI, safety fine-tuning, covered in the LLM guide — shapes what the model wants to do; it's a property baked into the weights before deployment. Evals — the final guide in this series — are offline measurement, run against test sets before or between deployments to score quality and safety. Guardrails are neither: they're runtime, inline controls sitting in the actual request/response path of a live system, making a pass/fail/modify decision on every single production call.
A useful shorthand for the guide: training is disposition, evals are measurement, guardrails are enforcement. A guardrail rule is frequently born from an eval failure pattern — someone finds a recurring failure cluster during error analysis, and that specific pattern becomes a new input or output rule — but once written, it executes independently, on every request, forever. That's the categorical difference from an eval, which only ever runs in batches against a fixed set.
Catching the problem before the model sees it, or acts on it
Prompt injection comes in two flavors, and the difference matters. Direct injection is the user's own input trying to override system instructions ("ignore previous instructions..."). Indirect injection is more dangerous and increasingly the more discussed case: malicious instructions arrive embedded in third-party content the model is asked to process — a retrieved RAG document, a scraped webpage, a tool's return payload. This is exactly the channel RAG and agent architectures open up, which is why indirect injection is the variant most relevant to the rest of this stack.
A concrete architectural mitigation for indirect injection: keep third-party content structurally segregated — delivered only inside clearly labeled, source-tagged blocks the model can't mistake for a system instruction, ideally with a lightweight, fast classifier screening that content before the main model ever sees it. Jailbreak detection is a related but distinct check — attempts to get the model to violate its own training via role-play, hypotheticals, or encoding tricks, rather than override instructions directly — commonly handled with the same pattern: a small, cheap model pre-classifying input before it reaches the larger, more expensive one. Rounding out this layer: topic/scope restriction (keeping a support bot off legal questions, a brand assistant off competitor discussion) and PII detection before anything reaches the model or gets logged.
Checking what comes back before a user sees it
| Check | What it catches |
|---|---|
| Content moderation | Toxicity, hate, self-harm, violence, sexual content — scored via a classifier, usually with continuous confidence scores rather than a flat yes/no |
| PII redaction | The model leaking sensitive data it saw in context — a retrieved document, prior conversation history — back to a user who shouldn't see it |
| Structured output validation | Does the output actually match the expected schema? Validated against a defined format, with retry-on-invalid re-prompting when it doesn't |
| Groundedness / fact-checking | Is a RAG answer actually supported by the retrieved context, or fabricated? The runtime cousin of the "faithfulness" metric in the Evals guide — same underlying question, different cadence (every request vs. a batch run) |
| Brand-safety filtering | Competitor mentions, off-brand claims, unauthorized commitments (pricing, legal guarantees) — usually custom, business-specific rules rather than a general-purpose vendor check |
What exists, and how it's usually wired together
| Tool | What it targets |
|---|---|
| Guardrails AI | Output structure and content validation — a library of composable "Validators" (PII detection, toxicity, regex, schema conformance) wrapped in a "Guard," with retry-on-invalid re-prompting |
| NVIDIA NeMo Guardrails | A programmable-rails framework spanning input, dialog/topic, retrieval, execution, and output stages — including reference rails for jailbreak detection, fact-checking, and PII masking |
| Meta Llama Guard 3 | A dedicated classifier model (not a framework) — scores both input and output against a 14-category hazard taxonomy, deployed as a sidecar next to the main model |
| OpenAI Moderation API | Hosted, free, multimodal content classification across 13 harm categories — purely a safety classifier, no schema validation |
| Anthropic's approach | No standalone product — a documented combination of training-time exposure to injection attempts, classifiers that scan untrusted content entering context, architectural guidance, and ongoing red-teaming |
None of these are mutually exclusive — real deployments typically stack a hosted content classifier with a framework for structure and flow control. The dominant architectural pattern across all of them is the same: a small, fast, cheap model runs alongside the large "does the work" model, so the guardrail check's cost and latency stay a fraction of the main call. This is also why cheap-first ordering matters in practice — a regex or tiny classifier runs first and only escalates to a heavier, more expensive check when it's actually triggered.
Streaming complicates output-side checks in a genuinely unsolved way: a full-response classifier needs the complete text, which defeats the point of low time-to-first-token streaming. The practical patterns — buffering per-sentence rather than per-token, holding back the last unclosed chunk, or forecasting risk from partial output — all trade away some of streaming's latency benefit for safety coverage. There's no free option here.
Why tool access changes the stakes of injection completely
This is the single highest-stakes escalation in the whole space, and it's the direct link back to the AI Agent and MCP guides. Risk explodes when a system has all three of: access to private or sensitive data, exposure to untrusted content (the indirect-injection vector from §01), and a channel for external action — sending an email, making an API call, executing code. Without tool access, a successful injection produces bad text — embarrassing, maybe harmful, but reversible. With tool access, the same injection can exfiltrate real data or take a real-world action, because the model itself can't reliably tell the difference between an instruction from its actual principal and one smuggled in through content it was only asked to read.
Current best-practice mitigations, consistent across the major sources: least privilege (scope every tool and credential to the minimum the task needs — the same guidance the MCP guide gives for server permissions), human-in-the-loop for high-stakes actions (explicit confirmation before anything irreversible), and structural isolation — architectural patterns like a privileged orchestrator that never directly ingests untrusted content, paired with a quarantined process that does, or committing to an action plan before any untrusted content is read at all, so injected content can't rewrite the plan after the fact.
Every threshold is a trade, not a solved setting
Too strict, and legitimate use cases break — a security researcher asking about exploits, a novelist asking how poisons work, a doctor asking about dosages all get blocked, and users learn to route around or disable the guardrail entirely. Too loose, and genuinely harmful content, successful jailbreaks, or injected instructions get through. Neither failure is rare in practice; both show up constantly during tuning.
In practice teams tune this via threshold calibration on a classifier's continuous confidence score rather than accepting a vendor's default binary cutoff, category-specific thresholds (a stricter bar for self-harm than for mild profanity, for instance), and staged escalation — a borderline score routes to human review or a secondary, more expensive check instead of an automatic allow/block decision. The layered-defense pattern from §03 is itself a tuning mechanism: a cheap, loose first pass catches the obvious cases cheaply, and a stricter, costlier second pass only runs on the ambiguous middle band.
Static rules have a short shelf life
Guardrail rules are frequently reactive: a team runs error analysis over production transcripts or an eval suite (the discipline covered in full in the Evals guide), finds a recurring failure pattern, and that pattern becomes a new rule, regex, or classifier training example. The reverse direction matters just as much and is easier to neglect — guardrail effectiveness itself has to be evaluated, not shipped and trusted indefinitely.
The evidence for why is concrete: one adversarial study tested six production guardrail systems and found character-level obfuscation techniques — emoji smuggling, Unicode tricks, bidirectional text — achieved up to 100% evasion, while adversarial rewording techniques achieved 46–57% success by subtly perturbing phrasing to dodge a classifier's learned decision boundary. The underlying issue is structural: there's a mismatch between what a guardrail classifier was trained to recognize and what the underlying LLM is actually capable of understanding, and attackers exploit exactly that gap. This is precisely why guardrails need the same ongoing red-teaming discipline as the base model — a rule set effective today is routinely defeated within months as bypass techniques circulate; it's a continuous practice, not a one-time launch gate.
The layer that catches what training and design didn't
Guardrails exist because no model reliably follows its training on every input, and no eval set covers every input a live system will actually see. They're the compensating control for both gaps, running on live traffic rather than a test set.
AI Agent + MCP (Layers 04–05)
Tool access is what turns an injected instruction into a real-world action (§04) — guardrails are the runtime check that has to sit between the agent's decision and the tool actually executing.
Evals (Layer 07)
The final guide in this series — where guardrail rules come from (error analysis) and how guardrail effectiveness itself gets measured (red-teaming as an eval discipline).
This completes all seven layers — back to the hub to see the whole stack.
Scale the investment to the blast radius
- Map the actual worst case for each failure mode before choosing guardrail layers — a chatbot that only emits text to one user has a small blast radius; an agent that can send emails or move money has an enormous one. The investment should scale with that, not be uniform across every deployment.
- Never rely on a single layer. The red-teaming evidence in §06 is specific: any one classifier or rule set is beatable, which is the whole argument for defense-in-depth.
- Test the guardrails themselves, not just the base model — a classifier that isn't red-teamed on its own schedule decays silently as bypass techniques circulate.
- Treat agentic tool access as a stakes escalation, not an incremental risk — chatbot-era content moderation alone doesn't cover the lethal-trifecta scenario in §04.
- Close the feedback loop deliberately: route recurring production failure patterns from error analysis into new guardrail rules on a real cadence, not ad hoc.
- Decide your streaming trade-off explicitly before launch, rather than discovering the UX regression only after bolting a full-response classifier onto a system built for token streaming.
Before you ship guardrails
- List the specific failure modes for your application and their real-world blast radius, rather than adopting a generic checklist of "AI safety" checks.
- Decide input vs. output vs. both for each check — some risks (jailbreaks) are best caught early; others (groundedness) can only be checked after generation.
- Choose a threshold-tuning approach up front — category-specific cutoffs and staged escalation, rather than a single global pass/fail bar.
- If the system has any tool access, treat the lethal-trifecta check (§04) as mandatory, not optional — least privilege and human-in-the-loop for high-stakes actions specifically.
- Schedule red-teaming as an ongoing practice, not a pre-launch gate — put a date on the next one before you ship the first version.
- Read the Evals guide — the error-analysis workflow that generates new guardrail rules, and the discipline for evaluating the guardrails themselves, are both covered there in full.
A guardrail isn't a fix for an unreliable model. It's an admission that the model will occasionally be wrong or manipulated — and a bet, tested continuously, on catching it before a user or a tool call does.
What this guide draws on
Read in full and used to shape the structure and claims of this guide.
| Source | Publisher | Used for |
|---|---|---|
| OWASP Top 10 for LLM Applications 2025 | OWASP | Canonical risk taxonomy — direct/indirect prompt injection and adjacent risks (§00, §01) |
| OWASP Top 10 for Agentic Applications | OWASP | Agent-specific risk extensions — goal hijack, tool misuse, agentic supply chain (§04) |
| NVIDIA NeMo Guardrails | NVIDIA | Programmable rails architecture across input/dialog/retrieval/execution/output stages (§03) |
| Stream Smarter and Safer | NVIDIA Technical Blog | The output-guardrails-on-a-token-stream problem (§03) |
| Guardrails AI — Quickstart | Guardrails AI | Structured-output validation, Validators, Guards, retry-on-invalid pattern (§02, §03) |
| Llama Guard 3-8B Model Card | Meta | Dedicated safety-classifier design and hazard taxonomy (§03) |
| Moderation API | OpenAI | Hosted content classification categories and scoring (§02, §03) |
| Mitigating the risk of prompt injections in browser use | Anthropic | Defense-in-depth stack and reported attack-success-rate figures (§03, §06) |
| Mitigate jailbreaks and prompt injections | Anthropic Platform Docs | Direct vs. indirect threat distinction, harmlessness screens, structural segregation (§01) |
| The lethal trifecta for AI agents | Simon Willison | The trifecta framing and named mitigation design patterns (§04) |
| Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails | arXiv 2504.11168 | Empirical evasion rates against six production guardrail systems (§06) |
Compiled 2026. Specific evasion and attack-success-rate figures come from the cited studies' own test setups — treat them as evidence that static guardrails decay, not as fixed rates for any particular system.