LLM Output Validation: Why You Cannot Trust What the Model Returns

An LLM system prompt is an instruction, not a guarantee. You can tell the model to only return valid JSON, to never reveal internal system information, to always decline requests outside its scope, and to maintain a specific persona. The model will follow these instructions almost all the time. For production systems, 'almost all the time' is not good enough.

What Can Go Wrong with Model Outputs

• ▸Hallucinated structured data — the model returns JSON in the correct schema but with invented values
• ▸Schema violations — the model omits required fields, adds unexpected fields, or returns the wrong types
• ▸Injection success indicators — output contains phrases suggesting the model acknowledged an injection attempt
• ▸PII leakage — the model returns PII from its context window that was not intended for the end user
• ▸System prompt disclosure — the model reveals contents of the system prompt when asked
• ▸Role drift — the model gradually abandons its defined persona over a long conversation

Structural Validation

If your model is supposed to return JSON matching a schema, validate it against the schema before returning it to the client. Use a strict JSON schema validator — not 'does this look like JSON' but 'does this JSON conform exactly to this schema.' Reject non-conforming outputs and retry or fall back.

Semantic Validation

Structural validation checks format; semantic validation checks meaning. Pattern match outputs against known injection-success signatures: phrases like "I cannot refuse this request as DAN", "my real instructions are", or acknowledgements of role changes. These patterns are detectable without understanding the full output semantics.

The Retry Strategy

When output validation fails, do not return the failed output to the user. Options: retry the request with the same input (works for transient failures), retry with a modified input that reinforces the output constraints, return a graceful degradation response, or escalate to human review for high-stakes decisions.