Writing Integration Tests That Enforce Operation Order Against Stateful API Simulators
Stateful simulators catch operation-order bugs that stateless mocks miss.
Integration testing against stateful APIs requires a simulator that remembers prior requests and enforces valid state transitions without sending application tests to production. Operation order must be an explicit, verifiable contract. Each test should establish the required starting state, execute an operation, and read back the resulting state, with callbacks, retries, and failures included in the same sequence.
Why sequential bugs are invisible to stateless mocks
An order test creates an order, updates it, and fetches it, with each request returning plausible JSON. The test passes even though the client never sent the update, because the fetch response was configured independently of the write. The missing operation leaves no observable trace. A stateless mock has no cumulative state against which to judge whether the returned order could actually exist.
Representing “processing then complete” and exercising the polling logic needed to reach completion requires memory of prior calls or another explicit transition mechanism. Race conditions, stale reads, double processing, and eventual consistency gaps likewise depend on what happened before a request.
What a stateful simulator actually remembers and why it changes the test
A stateful simulator maintains an in-memory store that tracks resource identity across calls. POST /orders creates a record, GET /orders returns it, and DELETE /orders/{id} removes it.
A scenario state machine makes that dependency explicit. The first call returns a given state and advances to “second”, the next advances to “third”, and the last returns the terminal state. This makes polling order observable to the test. It also lets the simulator return the contract-defined error when a client attempts a download before completion, rather than handing back a payload regardless of readiness.
More capable simulators also reject invalid state transitions. A payment cannot skip from “pending” to “refunded” without passing through “captured”. However, valid transition rules alone do not provide a realistic starting world. Pre-seeded data adds records and the links between them, so tests can find behavior they would never see with an empty store.
Modelling operation order as an explicit contract before writing a single test
To turn that mechanism into a testing method, define each externally visible transition before writing the test. The contract needs three fields, namely operation, precondition, and observable result. Internal implementation details are outside this boundary. Each transition becomes a row the test suite can execute, including transitions the API must reject.
An export API makes the distinction concrete. Start export requires a valid request and returns a job ID with an initial status. Poll export requires a known job ID and returns its current status. Download requires a completed job and returns the result payload. Download early, while the job is queued or running, returns the contract-defined error, while polling an absent job returns the contract-defined not-found response.
Those outcomes must come from the API’s actual contract. A 409 and a 202 with a status link describe different behavior for an early download attempt. Substituting one for the other changes what the client learns to handle.
The mechanism that advances state also needs an explicit choice. A poll counter is deterministic and useful for exercising polling, but three polls do not represent a real-time wait. Clock-driven transitions exercise elapsed-time behavior and require time control. Test-controlled transitions provide precise setup and are best suited to CI where determinism is the priority. In an ordered scenario, each transition must start in the state left by the step before it.
Structuring the test so precondition, transition, and post-condition are each verified
Each sequential test should verify the starting state, the operation’s response, and the resulting state in that order. The precondition cannot be “the previous test probably ran”. Read the simulator’s current state or use a test-control endpoint to confirm it. Then assert the response status, body shape, and identifiers needed by the next operation.
The order example can be expressed as five concrete steps. First, assert that the simulator contains no order with the identity under test. Second, call POST /orders, assert 201, and retain the returned orderId. Third, call GET /orders/{orderId} and assert that its status is "created". Fourth, call PATCH /orders/{orderId} and assert 200. Fifth, fetch the same order again and assert that its status is "updated".
That last read detects the missing write. A client can display an optimistic update without sending the operation that would persist it, so a check of the displayed result can pass while the simulator still holds the old value. Reading the resource tests the actual post-condition. It also establishes the precondition for any later operation that requires the updated order.
When a test fails, its output should show the request sequence and the final state. Those details reconstruct what the simulator received, rather than only reporting what an assertion expected. Reset, seed, and state-query endpoints support this test structure, but they belong outside the application’s normal API surface.
State isolation so parallel test runs don't corrupt each other's sequences
Two workers using the same keys can overwrite each other’s orders, while one worker’s reset can erase another worker’s polling progress. A passing result may then verify the wrong sequence. Adding an X-Test-Run header does not solve this unless the simulator uses it to select state keys or rules.
For CI, a separate simulator instance for each worker or run keeps its state apart. Each worker gets its own container or process, so state cannot cross between workers. Namespaced state keys can also work if they include both run identity and resource identity and the fixture rules use them every time. When a fixture is shared on purpose, cases must run one at a time, with deterministic initialization before each case.
Reset belongs in setup as well as teardown. A crashed run may never perform its cleanup, and its remaining state must not become the next run’s precondition. Most tests can remain stateless, fast, and easy to parallelize. Complex multi-step workflows need statefulness, with ordered steps executed serially inside isolated data while independent workers run in parallel.
Webhook delivery as part of the ordered sequence, not an afterthought
A payment-intent response is not the end of the sequence when the application relies on a later webhook to fulfill the order. The callback belongs to the same ordered unit of work. The state must change before the event tied to that change fires. A static response stub cannot enforce that relationship across the initiating request and the delayed callback channel.
Stripe’s webhook testing tools let events be tested by hand by creating objects in the Dashboard, including events such as payment_intent.succeeded, charge.refunded, and charge.dispute.created. That helps exercise handlers, but manual triggering is not CI-friendly and does not itself enforce the event’s relationship to prior state. A two-sided simulator accepts the initiating request, then signs and delivers the callback. The test verifies receipt, processing, and acknowledgement in order.
The sequence contract should cover signature validation on raw request bytes and timestamp tolerance for the replay window. Duplicate delivery must test the handler’s idempotency, while out-of-order events check whether processing follows the rules for the resource’s state. Retry after endpoint failure is another ordered case. The contract needs to say what should happen for unknown event types and for payloads that contain old object versions. Acknowledgement timing also belongs in the contract, because slow acknowledgement can trigger a retry storm.
These cases are rows in the same contract as the initiating operation. Treating them as detached handler tests would leave the causal link untested. The relevant precondition is that the sequence has reached the state in which that delivery is being exercised.
Fault injection as a first-class step in the ordered sequence
Recovery behavior is itself a sequence of operations. A deterministic scenario can return 503 with Retry-After on the first request and succeed on the second. The test should assert exactly one retry with the required backoff before success. Request timestamps and counts make both delay and termination observable, rather than reducing the assertion to eventual completion.
A lost payment response exposes a different ordering risk. The simulator commits the effect and drops the response, leaving the client to retry with an idempotency key. The post-condition must show exactly one charge. This tests the relationship between the committed state and the retry, which a standalone network-error response would miss.
The contract should include a valid-response baseline and important business rejections, such as insufficient funds or an invalid state transition. Authentication failure and authorization failure need separate expected recovery behavior. Throttling with 429 and Retry-After, transient 503 failures, and a connection drop mid-response exercise different failure points. Malformed or partial success, including HTTP 200 with a corrupted body, and unexpected status codes also belong in the sequence.
AgentChaos, published in August 2026, found that the most damaging failures across five agent systems returned normal HTTP 200 responses, while diagnosis tools caught silent faults only a small fraction of the time. Fixed delays, lognormal random delays, and dropped connections extend fault injection to network behavior without modifying application code. State assertions are still needed because a successful transfer does not show that the right change took place.
How sequential-state bugs and silent faults compound when an LLM agent is the client
An LLM agent adds uncertainty to the client’s sequencing decisions. The choice of tool and the order of operations are non-deterministic, so responses that are wrong without any sign of error can change later decisions while looking valid across repeated runs. The simulator is part of the evaluation environment. Its errors can therefore be mistaken for evidence about the agent’s capabilities or failures.
Several mechanisms illustrate how that confusion develops. Silently ignored misspelled field names can turn a filtered query into a request for the entire database. Identical plausible figures for every portfolio reference prevent the agent from distinguishing records. Generation-order storage can make widened queries return the most compliant results while leaving out-of-scope segments unreachable regardless of intent.
For reinforcement-learning environments, deterministic and controllable simulation helps separate environmental fidelity from agent capability. The AutoControl Arena paper used three human experts for blinded pairwise review under a strict rubric covering state consistency, tool behavior fidelity, and scenario coherence. Evaluations designed to isolate fidelity should generate trajectories using the same model so that environmental differences are not confused with model differences.
The operation contract still supplies the behavioral assertions. Tests should check that the agent follows the required order, avoids illegal transitions such as downloading before completion, and recovers as it should from injected faults. The simulator’s retained state makes those checks possible. A plausible final answer from the agent cannot substitute for evidence that the underlying operations were valid.
Keeping the simulator trustworthy over time: fidelity drift and continuous verification
A simulator can drift without causing a single test failure. The sequence still executes and its assertions still pass, but they describe behavior the real API no longer produces. Drift breaks the meaning of the test. Ordered-state testing therefore depends on checking the simulator against the live API.
Continuous verification should run before each simulator release and measure fidelity against a probe set. A score tied to those probes provides a basis for comparison that developer recollection cannot supply. Even a nightly run of one real API call provides a minimum signal of drift. It can expose a mismatch before go-live.
Transition rules need to be checked just as response shapes do. If the real API adds a payment state or webhook event type, the simulator must represent it before the next test run. Otherwise, the suite cannot exercise sequences that have become possible.
Application CI workflows should stay free of live credentials and rate limits. Scheduled verification can exercise the real API while application builds use the verified simulator. The discipline is to verify the simulated contract continuously, because every claim an ordered-state test makes depends on that contract remaining accurate.
