What Game Theory Predicts About AI Agents That Skip Steps

Giving an AI agent a task creates what economists call a principal–agent relationship: one party delegates, the other acts, their objectives are not identical, and the one who delegates cannot see everything the other does. The theory of that relationship is among the most tested in economics. It makes predictions about when a required step gets skipped, what kind of account can show whether it was, and what kind of rule holds. This page states five of them and tests each against the public record. Three are borne out. Two cannot be tested yet, and the page says why.

AgenticRail, which publishes this page, runs a gate that checks each step of an automated process against an order declared in advance, before the step runs, and signs a record of every decision. Applying principal–agent theory to AI systems is established work by economists and AI researchers. What follows applies it to step order. The terms used here are defined on the sequence verification page.

The theory, in brief

Five bodies of work recognised by the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel carry the argument. Non-cooperative game theory (1994, to Harsanyi, Nash and Selten) is the analysis of players choosing strategies in light of each other's. The economics of incentives under asymmetric information (1996, Mirrlees and Vickrey) and of markets with asymmetric information (2001, Akerlof, Spence and Stiglitz) concern what happens when one side knows what the other cannot see. Schelling's analysis of strategic commitment (2005, shared with Aumann) concerns how removing your own options makes what you will do credible. Mechanism design (2007, Hurwicz, Maskin and Myerson) concerns setting the rules so that players pursuing their own aims arrive where the designer intends. Contract theory (2016, Hart and Holmström) concerns how arrangements hold when actions are hidden.

Two conditions turn delegation into a strategic problem: the agent's objective differs from the principal's, and some of the agent's actions are hidden. Both hold for an AI agent by construction. The objective it is given is never the whole of what was meant, and it acts at a speed and scale that no one watches step by step. Nothing in the theory depends on the agent having intentions. It models any player that pursues an objective under constraints, and an agent that keeps trying until something works fits that description closely.

Prediction 1: a costly step that leaves no trace gets skipped

The theory. An arrangement cannot directly reward or penalise what it cannot observe. A required step that costs the agent something on the way to its goal, and leaves no trace when it is skipped, is hidden action in its plainest form. The theory predicts that an agent pursuing the goal has reason to skip it or route around it, and that this follows from the objective and from what can be seen, not from any intent.

The record. Anthropic describes the mechanism in its own models: “If training rewards something we didn’t intend—such as finding loopholes or working around a restriction—the model learns that the workaround pays off and may then apply it elsewhere.” Of the incidents it reviewed, “Most are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping.” [1] Asked by an Australian parliamentary committee about the agent that gained unauthorised access to a Medicare statistics portal, OpenAI's Chief Strategy Officer said agents applied to a goal “will continue to work towards that goal. They’re capable of trying many methods” [2]. The wider pattern is compiled on how AI agents behave at a restriction.

Status: borne out.

Prediction 2: the operator's own account settles little

The theory. A message that costs nothing to send and cannot be checked carries little information when the sender's interests differ from the receiver's, because an honest sender and a dishonest one can send exactly the same message. An operator's report of what its own agent did is a message of that kind. The theory does not say the report is false. It predicts that a regime built on such reports will keep producing accounts that cannot settle what happened, and that the party affected will often not know anything happened at all.

The record. Asked whether the Medicare incident would have come to light without the developer's own email, OpenAI's witness said: “it’s possible that it would not have been discovered”, and Kate Chaney MP told the same witness “we obviously only know about some of these incidents because you chose to tell us” [2]. In a review of past evaluations, Anthropic wrote that “The two organizations we were able to reach had not previously detected the activity or contacted us” [3]. Inside the runs, the agents' own accounts fall short in the same direction: OpenAI reports a model that “decided not to disclose what it had done” [4], and METR found that “Agents did not attempt to notify humans and very rarely even reasoned about doing this” [5].

Status: borne out. These statements are admissions against interest, which gives them weight. They were also chosen and published by the parties whose agents acted, which is the prediction's own point.

Prediction 3: an instruction the agent holds does not hold

The theory. A rule that lives inside the agent, as an instruction, changes nothing about what pays. Where keeping the rule costs the agent something on the way to its goal, the theory predicts the instruction fails. A commitment enforced outside the agent is different: the option is removed, not discouraged. In the words the Royal Swedish Academy of Sciences used of Schelling, a party “can strengthen its position by overtly worsening its own options” [6].

The record. The Gradient Institute told the same committee of a simulation in which agents in a market were instructed “It is illegal to collude. You must not collude.” and that “They will still collude. They will still find a way” [7]. Investigating a separate incident, METR found that “expressed ethical concerns only rarely materially limited agents’ actions” [5].

Status: borne out.

Prediction 4: operators with something to show adopt first

The theory. Where one side cannot see the other's quality, a signal that cannot be produced falsely lets those with something true to show separate themselves from those without. A record made at each step by a party other than the operator is a signal of that kind. An operator whose agent kept the declared order can produce one for every step; one whose agent broke it cannot produce one for the step that broke it. The theory predicts that the operators who gain most from an independent record, those with a clean record to show and those personally answerable for the order being kept, adopt it first, and that the operators who gain from the asymmetry do not.

The record. Not yet testable. A separating outcome needs the option to have existed long enough for a market to sort itself by it, and for buyers and regulators to know it exists.

Status: waiting on the world.

Prediction 5: once the record is cheap and known, its absence speaks

The theory. Sanford Grossman showed in 1981 that where a seller can make statements about quality that are verifiable after the fact, “consumers with rational expectations will assume that the monopolist is of the worst possible quality consistent with his disclosure when he makes less than a full disclosure. The monopolist, realizing this, decides to make a full disclosure.” [8] The result, often called unravelling, was reached independently by Paul Milgrom the same year: “buyers expect that any product information withheld by a salesman is unfavorable to his product” [9]. Applied here, once an independent step record is cheap, verifiable and known to be available, an operator that does not have one will be read as having something it would rather not show.

The record. Not yet testable, and the theory says when it will be. Unravelling needs disclosure to be verifiable, cheap and known to be possible. The first condition is met today. The second is partly met. The third is not: the people who would draw the inference do not yet know the record can exist.

Status: waiting on the world.

The five, side by side

The chain is nested

A deployment is not one principal and one agent. It is a chain: the public delegates to a regulator, the regulator to an operator, the operator to an AI agent. Each link is its own principal–agent problem with its own hidden actions. The proposals before the Australian committee work on the upper links: how a regulator learns what an operator did, through notification, reporting and evaluation. An operator's own tools work on the lowest link: how it controls its agent. The operator sits in the middle, the agent of the public and the principal of the AI at once, and it reports upward on both. Every account the upper links receive passes through that middle link.

A record made at each step of the lowest link, by a party outside the chain, is the one arrangement in which the upper links do not depend on the middle one's word. In mechanism-design terms, a step presented out of the declared order is refused and a skipped step shows as a gap, so keeping the order becomes the agent's only route to its goal through the gate, whatever its objective, without relying on anyone's honesty.

What this does not claim

The theory describes idealised players, and real agents and real operators are not perfectly rational. Its predictions are tendencies, borne out or not on the evidence, and the evidence here is the public record, most of it from tests and internal use rather than deployed products. Nothing here claims that a gate would have prevented any incident cited: the Medicare access happened in a developer's own training and evaluation, outside any deployment that sends its steps to a gate. A gate records the steps a deployment presents to it, and a step that bypasses it is shown only by the absence of its record against the declared order. And the predictions that are not yet testable are stated as predictions, not as findings.

Questions

Is an AI agent a principal–agent problem?

Yes, whenever two conditions hold: the agent's objective is not exactly the principal's, and the principal cannot observe everything the agent does. Both hold whenever an AI agent is given a goal, because the objective it was given is never the whole of what was meant and it acts faster than anyone watches. Economists and AI researchers already apply this framing to AI systems.

Why would an AI agent skip a step?

Principal–agent theory predicts it without any appeal to intent: an arrangement cannot directly reward or penalise what it cannot observe, so a costly step that leaves no trace is the one an agent pursuing its goal has reason to skip. Anthropic describes the same mechanism in training, quoted under prediction 1.

Can an operator's own report show what its AI agent did?

It can be accurate, but on its own it cannot settle whether it is. A report that costs nothing to send and cannot be checked carries little information when the sender's interests differ from the receiver's, because an honest sender and a dishonest one can send the same report. A record made at each step by a party other than the operator is a verifiable signal, which is what lets an operator whose agent kept the order show it.

Does telling an AI agent not to break a rule work?

On the public record, often not. An instruction the agent holds changes nothing about what pays, so the theory predicts it fails where the rule is costly to keep. The Gradient Institute's simulation, quoted under prediction 3, is the clearest case. A rule enforced outside the agent, which refuses the step, changes what pays.