How Third-Party APIs Age State Across Calls and What Your Tests Must Prove
Test API integrations against simulators that retain state across ordered call sequences.
To test integrations against stateful APIs, run ordered call sequences against a simulator that remembers resources, mutations, and prior requests. Assert every transition as well as the final response, and check the simulator against the provider with scheduled real-API checks. This is how tests prove that an integration handles the provider’s state machine without making every CI run depend on a live service.
Why APIs are stateful systems and why most tests don't reflect that
A checkout integration creates an order, fetches it, cancels it, and fetches it again. The second fetch must reflect the cancellation. Each response depends on what happened before it, so four successful requests do not establish correctness unless they operate on the same resource. A fixture that always returns the original order cannot test this contract.
A payment can move through pending, authorized, captured, refunded, and disputed, while a job accepts a command, exposes progress through polling, and later sends a webhook. A fixed response makes polling terminate immediately or never terminate, depending on the fixture. It can also return an ID that corresponds to nothing created. The application and its tests then agree with each other while disagreeing with the provider.
How state actually accumulates inside third-party APIs, across four common patterns
In a CRUD lifecycle, a create call establishes a resource that a subsequent GET must retrieve. Updates must change the resource that later reads return, and deletion must change how later calls can access it as the provider’s contract requires. These calls form one connected lifecycle. Testing each endpoint independently leaves the relationship between them untested.
In asynchronous command-and-poll flows, a command returns 202 before the work it was asked to do is finished. Later polls can return pending, then completing, and then done. A mock that always returns done skips the waiting logic, while one that always returns pending exercises only the nonterminal path.
In webhook choreography, the initiating request and signed callback belong to the same flow. The provider simulation must represent both sides. Signature validation on raw bytes, timestamp tolerance, duplicate delivery, out-of-order events, retry after endpoint failure, and acknowledgment timing all affect whether the integration handles the protocol correctly.
In conflict and idempotency flows, an earlier request changes what a later request may do. A duplicate or conflicting operation can produce 409 or 422 when the provider remembers the existing resource. History determines which response is valid here. A stub can return either code, but without retained state it cannot prove that the preceding operation caused that response.
What stateless mocks conceal and why the concealment compounds over time
A stateless mock hides more than missing coverage. It supplies a false contract that the application can satisfy indefinitely. The disagreement becomes visible only when the integration reaches a live provider in staging or production. As new features depend on the untested flow, they inherit its assumptions, increasing the scope of an eventual failure.
Parallel tests can corrupt each other’s assertions when they mutate the same global resources. Use per-test namespacing, a reset endpoint, and cleanup in finally hooks. Without that isolation, a test may fail because of another test’s side effects even when the integration under test works.
Recording and replay works for read-heavy integrations when the response depends mainly on the request. Its limits depend on the workflow. Creating an order and fetching it later requires the recording tool to capture a stateful scenario linking the two responses. Teams should therefore judge replay per integration, based on whether it preserves the dependencies their assertions need to test.
The assertions a test must make at each stage of a stateful sequence
At creation, assert that the returned ID identifies a resource that a subsequent GET retrieves. Match that resource to the creation request. Assert the provider’s documented initial status as well. A successful creation response alone does not establish that the resource exists for later operations.
At each mutation, assert the prior state, execute the transition, and check the exact HTTP status code and status field the provider documents. Then test which further operations the new state restricts. For the cancellation contract described here, cancelling the order again must return 409 in place of 200. This proves that cancellation changed what the system permits.
At polling boundaries, assert that pending responses do not short-circuit the caller’s loop. Termination must follow a terminal provider state. Also hold the resource in a nonterminal state and assert that timeout and max-retry logic execute to prevent the test from waiting indefinitely.
At webhook delivery, assert signature validation on raw request bytes before deserialization and enforce the documented timestamp tolerance. Deliver a duplicate and assert idempotent handling. Deliver an out-of-order event and assert that it does not corrupt resource state, then exercise delivery retry after endpoint failure and check acknowledgment timing. For conflict and validation paths, create the conflicting resource within the sequence and assert both the documented error response and execution of the caller’s error-handling branch.
Why a simulator that doesn't age alongside the real API will make these assertions meaningless
A simulator can keep state correctly while still following an out-of-date contract. Stateful behavior alone cannot prevent contract drift. If the provider changes while the simulator stays the same, even precise assertions test the wrong system.
Kalshi’s nightly Spec Drift Detection run on October 17, 2026 found that the Python SDK contract suite no longer matched the upstream OpenAPI/AsyncAPI spec. The generated models required re-synchronization.
Spec-to-spec diffing requires specifications on both sides, while spec-to-reality monitoring compares live responses with an OpenAPI spec. Reality-to-reality monitoring instead compares live responses with a learned baseline and requires no spec. Fixtures need to represent status, headers, latency, and connection behavior too, because a change in any of those dimensions can invalidate an integration test.
How fault injection fits into stateful sequence testing and why it must be deterministic
A retry is another call in a sequence that keeps state from earlier calls. To test it, define the sequence explicitly. The first request can return 503 with a Retry-After header, and the second can succeed. The simulator must remember which request it is handling so that each test run follows the same recovery path.
A payment test needs a more demanding failure scenario. The simulator should commit the effect and then drop the response, forcing the client to retry. The test must assert that the retry eventually succeeds and remains idempotent. A configurable random error rate cannot guarantee this sequence.
Fallback assertions must also reflect the risk of the substituted data. Cached rates may be acceptable for a shipping estimate but dangerous for an account balance. If the UI shows stale data, assert a visible freshness indicator. Success at the HTTP layer is not sufficient evidence either. AgentChaos, published in August 2026, found that omission faults such as truncated or empty responses that returned HTTP 200 badly damaged agent systems, while diagnosis tools rarely caught truncation faults.
Why AI agents that call external APIs need stateful sequence testing even more than traditional integrations do
An agent can produce coherent text after executing the wrong API sequence. Output quality does not establish tool correctness. Tests must inspect decisions, endpoint selection, parameter extraction, response parsing, and state management throughout the workflow. A malformed JSON payload to a payment API can cause a silent transaction failure without making the agent’s final text visibly wrong.
AgentChaos found that pass@1 fell by up to 49.66 points across five agent systems when faults returning HTTP 200 were injected. The most damaging faults varied by system. Omission faults were among the most harmful and hardest to diagnose because the HTTP response still looked successful. PALADIN, presented at ICLR 2026, trains agents to detect and recover from tool malfunctions, including timeouts, API exceptions, and inconsistent outputs that otherwise cause cascading reasoning errors and task abandonment.
Tests must try to get agents to make unauthorized API calls, extract data, or change records in a controlled stateful environment. Sampling after deployment cannot take the place of those scenarios. For reinforcement learning, the environment must additionally support deterministic execution and horizontal scaling across thousands of containers, requirements that neither a live API nor a stateless mock satisfies.
How can stateful API integration tests run in a real CI pipeline without hitting production?
In CI, dependencies on payment gateways, email providers, and other external APIs should be replaced by simulators that retain state across calls. The application can then run full sequences without relying on live credentials, provider rate limits, or third-party uptime. Provider outages should not break this pipeline. Otherwise, a build may fail because the provider is down even when the integration works.
A tiered workflow separates rapid feedback from broader validation. Fast smoke tests run on every commit, full stateful sequence suites run on PR merges, and the full checks before deployment include fault injection. After release, synthetic monitors validate critical API paths. Scheduled checks against the provider keep the simulator’s contract up to date without turning every application test into a live request.
The simulator itself must be a reproducible test dependency. Fixtures and configuration belong in version control with the tests. Each run needs a simulator that can be versioned, reset, and namespaced so that parallel execution does not leak state between scenarios. That makes the isolated simulator environment, rather than the test function alone, the unit of isolation.
The specific assertions to add when testing against Slack, GitHub, Stripe, Kalshi, and PayPal
For Slack, a successful message post is not enough. Add assertions for event ordering, duplicate handling, and signature validation on raw webhook bytes.
For GitHub, open a PR, push a commit, trigger a status check, and merge. Assert the PR’s transition at every step. Model push, pull_request, and check_run webhook events with signature validation and delivery retry behavior, so the test covers how the integration learns about those changes.
For Stripe, assert the webhook payload shape associated with the endpoint’s api_version. A null value follows the account default, so an account upgrade can change payloads without a code deployment. Local stateful fixtures test the application’s code path. A small real-API check in a sandbox or protected environment must also establish which version the endpoint is configured to deliver. Refund and chargeback scenarios should populate the mirror tables used by the read path, while type-checking a fake against Stripe’s types can expose newly added production calls that it does not implement.
For Kalshi, assert the market lifecycle sequentially. Its internal lifecycle distinguishes active, closed, determined, and finalized, with disputed or amended states also represented. The API filter value settled maps to the finalized state. Tests of settlement must preserve that distinction, and SDK contract tests need generated models synchronized with the upstream specification.
For PayPal, model CREATED, APPROVED, and COMPLETED so that tests can assert each order transition on its own. Keep order and capture states distinct. REFUNDED is a capture-level status, distinct from Orders v2 order statuses. Exercise payment notification validation and webhook signature validation against the simulator’s signed payloads.
The minimum viable stateful test suite for a new third-party integration
Start by identifying each resource’s states and allowed transitions in the provider’s documentation. Where no explicit state machine exists, infer it from documented status codes and error conditions. Write one test per transition. Each test must assert the starting state, make the call, and assert the state after the call.
For every resource type that supports conflicts, add a test that first creates the conflicting resource and then checks the documented error code and body. Add a polling test with at least two intermediate states before termination. Add a webhook test that validates the signature, handles a duplicate idempotently, and checks acknowledgment timing.
For every external call with documented retry behavior, add a deterministic first-failure, second-success scenario and assert exactly one retry. Run the full suite against a continuously verified stateful simulator in CI, with a smaller scheduled subset against the provider’s sandbox to detect drift. This is a starting point for coverage. It is the minimum that replaces assumed state transitions with tested ones.
