# Procedural Hallucination: Why AI Agents Skip Steps and Report Success > Markdown mirror for AI agents, generated 2026-08-12 from the live page. > Canonical: https://agenticrail.nz/blog/procedural-hallucination-agent-skipped-steps/ > Site context: https://agenticrail.nz/llms.txt Published 8 August 2026 · AgenticRail # Procedural Hallucination: Why AI Agents Skip Steps and Report Success When an agent skips a required step and tells you the task is done, nothing it said was false. The summary is accurate. The outputs are correct. What is wrong is the **order**, or the **completeness**, and neither of those leaves a sentence you can point at. This failure is the **most prevalent** category of agent trajectory error in the published measurements, it has at least a dozen competing names and no settled one, and the best purpose-built detector trained on it reaches an ROC-AUC of **0.689**. ## It has twelve names, which is the first problem Ask five practitioners what to call an agent that skips a mandatory step and reports success, and you get five answers. Ask the same model the same question three times and you get three. The terms in current use include: ``` procedural hallucination progress-as-completion step collapsing false success silent process failure shortcut behaviour shortcut learning agentic drift partial completion problem action hallucination instruction drift execution shadowing plan adherence failure premature termination ``` These are not distinctions. They are the same failure, named independently by research groups, platform teams and practitioners who have not read each other. The practical consequence is worse than untidy: **an engineer hitting this cannot search for it.** They do not know which of a dozen phrases to type, and whoever already solved it wrote it up under a different one. This post uses **procedural hallucination**, because it is the term with a formal published definition. The rest are listed so the page is findable from any of them. ## The formal definition A 2026 paper on auditing multi-agent industrial workflows sets out five hallucination types as formal predicates over an agent's execution trace [1]. The fourth is the one that matters here: ``` Factual τt or αt asserts a claim contradicted by ground-truth data at step t. Referential τt or αt references an entity, observation, or prior result absent from {s1,…,st-1}. Logical The reasoning in τt does not follow from its premises, even when those premises are correct. Procedural αt skips, reorders, or fabricates a step required by K, or τt claims completion absent from trace. Scope Agent at acts or claims outside its mandate. ``` **"Skips, reorders, or fabricates a step required by K, or claims completion absent from trace."** That is a definition of step-order failure, written as a hallucination type, in a paper about industrial workflows rather than about compliance. The same paper draws the distinction directly: "In an agentic context, a hallucination is not simply a factual confabulation in a single response. It is a **structural deviation from evidence that propagates through a sequential, tool-mediated trajectory**, often leading to cascading operational failures." [1] A second term for the same behaviour appears in enterprise reliability work: **step collapsing**, where an agent skips intermediate steps to reach a terminal state [2]. The framing there is worth keeping: the model is not hallucinating *data*. It is hallucinating *workflow completion*. ## Why it is a different object | | Factual hallucination | Procedural hallucination | | What is wrong | The content | The order or completeness | | The content can be | False | Entirely true | | What it leaves behind | A wrong statement | Nothing | | Checkable against | Ground truth, a source document | Nothing, unless something recorded the sequence | | Caught by | Groundedness scoring, retrieval checks, review of the output | Only a record of what ran, held outside the agent | The row that does the work is the third one. A factual hallucination hands you the evidence of itself: a claim that is wrong, sitting in the output, available for anyone to check. A procedural hallucination hands you a clean, plausible, well-formed account of a process that did not happen in that order. **Absence has no signature.** ## It is measured. The measurements are not reassuring. A common claim is that nobody is looking. That is wrong, and worth correcting: trajectory-level evaluation is an active field, with benchmarks including **Trajel**, AgentHallu, TRAIL, ToolEmu, TRAJECT-Bench, MIRAGE-Bench, ToolBeHonest, Tau²-Bench, PlanBench and others. What the measurements show is the problem. Trajel, evaluating agent traces from industrial workflows, reports [1]: | Finding | Figure | | Share of identified failures that are **procedural** hallucinations — the largest single category | **38.5%** | | Trajectory hallucination rates across evaluated models | **52.4% – 81.0%** | | Best ROC-AUC achieved by a *fine-tuned supervised classifier* trained to detect it | **0.689** | | ROC-AUC of a cheap *runtime* execution-quality signal, "clarity-and-justification" | **0.908** | Two things follow, and they point in the same direction. **Post-hoc detection is weak.** A classifier purpose-built for this, trained on labelled traces, lands at 0.689. That is a system you would not deploy as a control. **Watching the run beats auditing the record.** The paper's own finding is that a univariate signal taken *during* execution achieves 0.908 and, in its words, *"outperform[s] all trained classifiers"* [1]. The information that identifies the failure is available while it is happening and degrades once it is only a log. The separate point about the number people usually quote still holds. *Hallucination rate*, as it appears in model cards and leaderboards, is measured by content benchmarks: a source document, a summary, and a check on whether the summary asserts anything the source does not support. That method needs an artifact on both sides. A step that never executed produces no token to score, so procedural failure is not in that number — for any model, at any rate, however independent the benchmark. ## Why agents do it Two mechanisms are commonly identified by practitioners working on agent reliability [3], and they compound. **Training rewards the appearance of completion.** Responses that look like successful task completion are reinforced. The shape of a finished job is what gets learned, and the shape is producible without the substance. **Language generation does not require real state.** This is the load-bearing one. A model does not have to have performed step three in order to produce the sentence saying it did. There is no state to check the sentence against, and nothing in the generation process consults one. The claim and the act are independent. Put those together and the failure is not an aberration. It is the default behaviour of a system optimised to produce completion-shaped output with no mechanism that could contradict it. ## A worked example you can verify On 6 August 2026 a third-party model, GLM-4.7-flash, was connected to AgenticRail's public MCP server through Cloudflare's AI Playground and given a prompt that named no tool. It found the enforcement tool itself, called it, and ran a step out of order. The gate refused with `SEQUENCE_VIOLATION`. The model stopped, corrected itself, and ran the sequence properly to a seal. Then it summarised what it had done, and reported **four receipts**. There are five. The public report for that sequence shows **5 total receipts — 4 allowed, 1 denied**, with Chain Integrity `VERIFIED INTACT`, Seal Status `SEALED`, and Independent Archive `WITNESS MATCH`. The model dropped its own refusal out of its account of its own work. Not maliciously, and not because it was confused about the facts — every step it described had actually happened. Its narrative of the trajectory was simply missing the part where it was told no. **That is procedural hallucination in miniature, produced unprompted, and it is the most benign possible version of it: the agent under-reported a refusal that had already been enforced.** Verify it yourself. No key, no login, no account: ``` https://report.agenticrail.nz/report sequence id: demo-mcp-payment-workflow-001 ``` The report returns every receipt, each one's raw Ed25519 signature, and the byte-exact signed preimage, so the signatures can be checked offline against the [published public keys](https://agenticrail.nz/spec/receipt-public-keys.json) without calling back to us. ## What actually catches it Four approaches are recommended in practice. They are worth setting out honestly, because three of them work and none of them produces evidence. | Approach | What it does | What it leaves you with | | **Trace evaluation** (observability platforms) | Scores the recorded trajectory after the run, often on a sample of traffic. | A finding, after the fact, on the runs that were looked at. | | **LLM-as-a-judge** | A second model reads the trace and rules on whether required steps ran. | AI checking AI. Useful, and not something to hand an auditor. | | **Deterministic contract checks** | Code asserts that a required tool ID appears in the agent's own step list. | A verdict computed from the agent's own record of itself. | | **State machines** (orchestration frameworks) | Hard-codes transitions so the agent cannot reach step 3 without step 2. | Correct order, enforced. And **no record** that anyone outside the system can check. | The last row is the important one, and it is not a criticism of orchestration. A state machine genuinely prevents the failure. What it does not do is produce an artifact: it enforces order and emits nothing an auditor, a regulator or a counterparty can verify independently. Ask it afterwards to prove step 2 ran and you are back to reading the operator's own logs. The remaining gap is narrow and specific: **something outside the agent that holds the declared order, evaluates each step before it runs, and writes a signed record of the decision — including the refusals.** The refusal is the mechanism. A skipped step leaves nothing behind, so the only way to make an absence visible afterwards is to have said no at the time and signed that. That is what AgenticRail is: an external gate that returns ALLOW or DENY against a sequence declared in advance, and writes a signed receipt either way, chained to the step before it. In the run above, the `SEQUENCE_VIOLATION` receipt exists precisely because the step was refused rather than silently absorbed — which is why the model's incomplete summary could be contradicted at all. ### What this does not claim It does not stop a model hallucinating. Nothing here improves the model, and factual hallucination is untouched by any of it. It does not assert that a timestamp is true: the signature fixes `ts_ms` against later alteration, but the value is self-asserted, so a receipt is a *record* of when something ran, not *proof* of when it ran. And a sealed sequence cannot be reopened without leaving a detectable break in the receipt chain, which catches a single altered receipt immediately but not a full downstream rewrite by a holder of the signing key — which is why an independently held archive copy exists as a separate layer, and why key custody is a distinct question from the cryptography. The claim is narrower and, we think, more useful: **a failure that leaves no artifact can only be caught by something that was already keeping the record.** Everything else is looking for evidence that a skipped step does not produce. **[1]** Badave, Borse, Lin, Patel, Gomez, Narahari, Carter, Bhatt, Rachakonda. *Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows.* arXiv:2605.24219v2, 26 May 2026 — [arxiv.org/abs/2605.24219](https://arxiv.org/abs/2605.24219). Definitions quoted verbatim from the paper's hallucination type table. Figures quoted verbatim: *"procedural hallucinations account for 38.5% of identified failures"*; *"Hallucination rates range from 52.4% to 81.0% across models"*; *"best AUC = 0.689"* for fine-tuned supervised approaches; *"the clarity-and-justification signal achieves AUC = 0.908 as a univariate predictor, outperforming all trained classifiers."* Trajel is built on AssetOpsBench execution traces. **[2]** *PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI.* arXiv:2605.15665 — [arxiv.org/abs/2605.15665](https://arxiv.org/abs/2605.15665). Source of the "step collapsing" framing. **[3]** Practitioner write-up on why agents skip steps after calling a skill — [majidgolshadi.substack.com](https://majidgolshadi.substack.com/p/why-your-llm-agent-still-skips-steps). Cited as practitioner observation, not peer-reviewed research. **[4]** The sequence described above, live and keyless: [report.agenticrail.nz/report](https://report.agenticrail.nz/report), sequence id `demo-mcp-payment-workflow-001`. Figures in this post were read from that report on 8 August 2026. Related [Are AI Agents Deterministic or Probabilistic? The Real Difference →](https://agenticrail.nz/blog/deterministic-vs-probabilistic-ai-agents/) [Agent Orchestration Is Not Agent Enforcement →](https://agenticrail.nz/blog/agent-orchestration-vs-enforcement/) [The Completeness Specification: Eight Requirements for Evidence-Grade Records →](https://agenticrail.nz/spec/completeness/)