Designing Simulated API Endpoints That Expose Agent Decision Boundaries Not Just Response Coverage

Stateful simulators with injected faults reveal where agents overstep.

· 8 min read
Cover illustration for “Designing Simulated API Endpoints That Expose Agent Decision Boundaries Not Just Response Coverage”
AI Agent Testing · November 19, 2026 · 8 min read · 1,889 words

To use an API for agent evals without hitting production, run the agent against stateful simulated endpoints, seed conditions that force consequential choices, and grade the resulting world state. The simulator must keep dependencies between calls intact, inject failures on purpose, and stay checked against the real API so that passing tests show that the agent behaves correctly in the real world.

Why response coverage is the wrong success criterion for agent testing

An agent calls create_invoice and then list_bills. Both calls can pass their own coverage checks even when the new invoice is missing from the returned list. The responses parse, but the workflow is broken. A real API should expose the effect of the write through the subsequent read, and a test that checks each endpoint separately can miss that dependency.

This is a category error in the success criterion. Response coverage answers whether the agent called a tool and understood its response. Agents vary both in their language outputs and in their choice of tools, ordering, and parameters, so one request can produce several valid execution traces. A subtle state corruption in the first step may become visible only in the third. Endpoint checks cannot establish correctness across that causal chain.

What decision boundaries actually are and why they require manufactured conditions

A decision boundary is the edge of an agent’s correct authority. It is the API state, call sequence, or edge condition at which the agent’s behavior should change. That edge can involve decision logic, state management, or policy adherence, including business rules the API itself does not enforce. These layers intersect when the agent must choose a tool based on both the current world state and the user’s instruction. The choice depends on what has already happened.

AutomationBench, released by Zapier in April 2026, grades whether the right data ended up in the right systems across sales, marketing, operations, support, finance, and HR tasks. The benchmark requires autonomous endpoint discovery, layered business-rule interpretation, and navigation through irrelevant and sometimes misleading records. Its 2026 results put even the best frontier models below 10%. The test is not simply whether a call succeeds.

Those conditions must be manufactured because the happy path does not necessarily force a choice between correct and incorrect authority. Ambiguous records, conflicting state, and hostile responses make that choice observable. For irreversible actions such as posting to Slack, deleting records, or charging a card, invocation alone is insufficient evidence. The boundary check must also examine what the agent would submit.

Why stateless mocks structurally cannot manufacture the conditions that expose decision boundaries

Stateless mocks remain useful for checking tool selection and response parsing. The limitation appears when one call must change the meaning or result of another, because each fixture returns its response independently of prior actions. Consistency then rests on the scenario author.

For the invoice workflow, the author must manually coordinate the creation response with the later list response. Every trajectory variant requires that coordination again. If the agent takes a valid but unanticipated path, the fixtures can stop making sense, and the evaluation begins testing the author’s prediction of the trace.

Resource lifecycles make the mismatch especially clear. A payment mock that ignores the create-to-capture-to-refund sequence can return “captured” before a capture has occurred. The agent then passes against behavior the real API would reject. Without tracked world state, the grader also has no known final state to check for correctness, so someone must judge the workflow’s outcome by hand.

How stateful simulators make decision boundaries reachable and gradeable

A stateful simulator keeps the same world consistent from one call to the next. Writes change that world, later reads show those changes, and the agent must choose its next action based on them. The invoice appears in the bill list because both endpoints operate on shared state. This preserves the causal structure that isolated responses discard. Decisions now have persistent consequences within the test.

That persistence makes outcomes gradeable without prescribing one execution trace. An unexpected but valid route can pass if it leaves the correct data in the correct systems. Conversely, an agent can make every expected call and still fail because it corrupts the resulting data. AutomationBench uses programmatic assertions against simulated application state to make this distinction. Call order is evidence, but not the verdict.

The simulator must also enforce resource lifecycles. A payment in the created state should not behave like one already captured or refunded, and invalid transitions must produce the corresponding rejection. Seeded records give those rules something substantive to operate on. An empty environment removes the distractors and ambiguous states that force the agent to distinguish a permitted action from a merely available one.

Fault injection as a first-class method for forcing boundary conditions into the open

State sequences expose only part of the boundary. Rate limits, timeouts, partial responses, and schema drift can force the agent to reconsider whether it should continue, retry, or escalate. A 429 followed by appropriate recovery can represent a passed test, while looping or inappropriate escalation can represent a failure. Receiving an error is not itself the failure. The evaluation concerns the agent’s response to it.

These tests need scripted sequences that go beyond error fixtures used on their own. A rate-limit storm, an authentication token expiring mid-session, or a schema change on the third call depends on what happened earlier. Simulator-level control makes those conditions repeatable, including when an agent framework does not expose fault injection natively. It also allows adversarial API content to probe prompt-injection resistance and privilege-escalation prevention.

Fault injection exposes authority decisions by testing what the agent treats as permission when it can no longer rely on the environment. The same failure sequence can also support comparisons between agent architectures alongside measurements of a single agent’s pass rate. Simpler architectures can recover more reliably, making recovery behavior an important object of comparison across levels of complexity.

What a simulator endpoint must model to expose a specific boundary — a design checklist

For endpoint designers, the practical checklist begins before any fixture is written. What transitions does the resource support, which are invalid, what changes in the agent’s view at each stage, and which injected condition would force a consequential choice? These questions define the endpoint’s testing purpose. Without answers, the simulator may return responses that look right while never reaching the boundary it is meant to expose.

The state transition map establishes the resource lifecycle, such as created, active, suspended, and deleted. It must specify how the real API rejects an illegal transition, including the full rejection details beyond a generic 400: the status code and body. Sequence dependencies then establish which reads must reflect which writes. If POST /charges creates a charge, GET /charges must show it without someone having to line up the fixtures by hand.

Visibility needs its own check because a correct internal state can still produce a misleading response. A refunded charge that remains visible as capturable gives the agent the wrong decision context. Injection points should then target that context, including a 429 on the third sequential write, a partial list response, or a field disappearing after a state change. Each fault must force a relevant choice.

Irreversible endpoints need content assertions as well as invocation checks. Slack posts, GitHub comments, and email sends must expose what the agent would have submitted for grading. A spec-based approach provides a structural safeguard by generating the simulator’s state machine from the same spec that defines the API schema. Transitions and responses can then be validated against documented behavior and updated together. Documentation alone, however, cannot establish live fidelity.

Fidelity drift as the silent enemy of boundary testing — how simulators go wrong over time

A simulator can remain internally consistent while becoming externally wrong. The live API may change its authentication requirements, how it handles errors, or which transitions it allows while the simulator stays the same. Tests then pass because the simulated world accepts behavior the real world no longer accepts. The boundary has moved without the evaluation noticing.

The Agentverse API audit in Q1 2026 found that only a portion of documented endpoints were fully operational with a standard bearer token. Others needed more authentication than the basic token or were only partly deployed. Endpoints documented at audit time have since been modified, consolidated, or deprecated. A simulator derived solely from that documentation would therefore misrepresent operational behavior.

The simulator can enforce a transition the real API no longer enforces, or permit one the real API now rejects, changing the choices available to the agent across an entire workflow. A fidelity check at build time cannot protect later releases from that change. Continuous verification against the live API, with published fidelity scores, makes the gap visible.

Verification methodology that keeps simulated boundaries honest

Continuous verification checks how the simulator behaves against how the live API behaves before every release. The comparison must cover transitions and response shapes. It needs to establish whether both systems reject the same illegal actions, return the same error codes, and expose writes through subsequent reads with the same latency characteristics. Otherwise, a matching response body can conceal a different decision boundary.

The drift-detection method depends on how the simulator is built. Spec-to-spec diffing needs specifications that are kept up to date on both sides, while spec-to-reality monitoring checks live requests against OpenAPI. Reality-to-reality monitoring uses a baseline learned from traffic it has seen. These methods answer different comparison questions. The relevant choice depends on whether the simulator is driven by specifications or observed behavior.

Release-time verification is needed even when simulator code has not changed, because an upstream API change can alter the expected behavior independently. Published fidelity scores then give integrating teams a basis for calibrating trust. In CI, an X-World-Version header or equivalent version pin on every simulator response also makes world changes detectable, allowing boundary assertions to be re-baselined when the simulator changes.

Connecting boundary exposure to CI/CD and RL environment design

For automated agent evaluations at scale, stateful boundary simulators provide a repeatable environment without routing every test through a live service. Pinned world versions and seeded data keep the test conditions the same from one commit to the next. A simulator running locally or in a container avoids live credentials, external rate limits, and accumulating API costs during those runs. Rate limits can still be injected on purpose as test conditions. Environmental repeatability keeps environment changes from obscuring the agent’s own variability.

Reinforcement learning environments need the same foundation under stricter operating demands. World behavior must be deterministic enough for reward signals to be comparable, horizontally scalable for thousands of parallel agent instances, and stateful so that actions persist within an episode. AutomationBench demonstrates the compatible pattern of a simulated application world graded on end-state correctness. It does not require one prescribed route to the outcome. Earlier actions shape what later actions can do.

For training, boundary conditions should be introduced deliberately through seeded adversarial states, distractors, and fault sequences that give the agent encounters with consequential choices before production. This is the same design work required for useful regression tests, including shared state, controlled failures, and a gradeable outcome. A simulator built to expose decision boundaries can therefore serve development evaluation, CI regression testing, and RL environments, saving the work of building one for each.

Sources

  1. AutomationBench
  2. Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform
  3. AutomationBench
Filed underAI Agent Testing

More in AI Agent Testing