Flaky Integration Tests Caused by Third-Party API Nondeterminism
Replace live API calls with a stateful simulator verified continuously against the real service.
Flaky integration tests caused by third-party APIs stem from how the tests are set up as well as how they are written. A test that reaches a live external service inherits conditions its author cannot control. The durable fix is to replace that live dependency with a controlled, stateful simulator whose behavior is continuously verified against the real API.
Why integration tests that reach live third-party APIs are structurally nondeterministic
A broken test fails consistently for a clear reason. A flaky test gives different results even when the code under test has not changed. When an integration test calls an external API, its result depends on both the application and the service’s execution conditions. A network timeout or slow response can therefore fail the test even when the application behaves correctly. The dependency makes the execution environment a variable in the result.
Ephemeral runners reset state between jobs, parallel execution and shared infrastructure make jobs compete for resources, and dynamic resource allocation changes the conditions in which tests run. Third-party APIs then add rate limits, maintenance windows, sandbox instability, and response-time variance. These are properties of infrastructure outside the test team’s control. As long as the test reaches that infrastructure, identical code and test inputs do not guarantee identical outcomes.
What nondeterministic tests do to a CI pipeline over time
Once reruns become routine, a red result stops being a clear request to investigate and becomes a reason to try again. Triaging unrelated failures, rerunning pipelines, and babysitting the suite consume engineering time without improving the application. When the full suite has to run again to get a green result, merges take longer and releases slip. The damage compounds because the same suite costs more to operate while providing less trustworthy evidence.
That loss of trust is more serious than the delay alone. A real regression goes into the same queue as failures that went away on earlier reruns. Engineers now have a familiar way to explain a new defect, because they think the test is probably flaky. An application race condition can fail intermittently and then pass on retry, just as an external timeout does. If the green rerun replaces investigation, the pipeline permits the race condition to ship.
Why rewriting tests and adding retries do not fix the underlying problem
Explicit waits replace arbitrary sleeps with checks for the condition a test actually needs. With test isolation and data cleanup, tests no longer depend on shared data or the order in which the suite runs them. These changes reduce internal timing and shared-state failures, but they do not control the external service. A properly isolated test still encounters the service’s rate limit counters, account-level quotas, and sandbox resets.
Retry budgets and quarantine queues have a similarly limited role. Retries let a test run again after transient failures, but a later pass does not explain why it failed before. Quarantine keeps an unstable test from blocking the pipeline, but that test no longer provides a dependable release signal. Environment tuning can improve the conditions around a request, but only replacing the uncontrolled dependency removes that source of nondeterminism from the test.
Why stateless mocks raise the ceiling only so far
A stateless mock takes the place of the live service and gives responses that tests can count on. That is useful in a unit test, where the purpose is to isolate one unit from its dependencies. An integration test asks a different question, namely whether a sequence of operations behaves correctly as a whole. A mock that maps request shapes to canned responses has no memory of earlier calls. It cannot establish that a resource returned by a retrieval request exists because the preceding creation request succeeded.
Consider a workflow that creates an order, polls until it is complete, and then attempts to cancel it. The creation returns 201 with an ID, while cancellation should return 409 because the order has completed. With a fixed status fixture, polling ends at once or keeps going forever, without following the order’s state transitions. Separate fixtures can also return an ID unrelated to the created order without exposing that inconsistency.
Even carefully coordinated fixtures face drift. Handwritten mocks depend on their authors keeping pace with changes in the real API, while spec-derived mocks depend on the accuracy of the OpenAPI document. Neither source of responses inherently detects when its description becomes stale. Without verification, control can hide the instability and make tests seem more trustworthy than they are.
What a stateful, continuously verified simulator does that mocks and live APIs cannot
A stateful simulator preserves the effects of calls throughout a workflow. Creating a record changes what a later list request returns, and deleting that record changes what a subsequent retrieval can find. The simulator enforces ordering and consistency constraints. This gives multi-step tests a coherent environment in which earlier actions determine later results. It combines controlled execution with the behavior that makes an integration test meaningful.
Statefulness alone does not show that the simulator behaves like the external API. The simulator must be tested against the real service before every release so that changes in behavior cause fidelity failures in those tests before they can cause surprises in production. This separates two jobs that live integration tests otherwise combine, which are testing application behavior and checking the replacement’s accuracy. Application tests run against the controlled stand-in, while continuous verification checks that it still behaves like the dependency.
The environment also needs useful starting conditions. Realistic pre-seeded data gives tests a world that looks real, so each test does not have to build it from scratch. With local hosting, development and CI runs need no live credentials and face no rate limits from the service. It also works in offline and air-gapped environments where a live sandbox or cloud-hosted mock service cannot work.
How fault injection fits into a stateful simulator workflow
Even when the happy path works every time, the client’s handling of degraded service still needs to be tested. Live APIs expose error paths accidentally, such as when a test happens to cross a rate limit. A stateless mock can return a 429 fixture, but it cannot enforce the stateful sequence from normal responses through throttling, backoff, and recovery. A stateful simulator makes that sequence deliberate. It can inject a 429 after a configurable number of calls to a specific endpoint, return 503s on demand, or add latency to selected response paths while preserving resource state.
Webhook delivery makes the need concrete. Stripe states that event order is not guaranteed and duplicate deliveries can occur. PayPal retries failed webhook deliveries up to 25 times over several days. Testing only a successful, single delivery leaves the handler’s ordering and idempotency behavior untested. Deliberate delivery faults make those behaviors part of the integration workflow rather than something a sandbox might happen to exercise. An exit-code-gated CI step can run these scenarios with functional tests so that resilience regressions get the same checks before release.
How this applies to AI agents that call external APIs as tools
An AI agent that uses external APIs inherits the same dependency problem. A multi-step task may fan out into many tool calls across several services, allowing external variability to compound within one run. Authentication failures, rate limits, and API drift can change the outcome independently of a model or prompt change. Evaluation therefore needs a repeatable environment to distinguish changes in agent behavior from changes in its tools. A live API cannot supply that control, and stateless responses cannot represent the stateful workflows the agent traverses.
A stateful simulator supplies consistent consequences for the agent’s actions while allowing faults to be introduced deliberately. The same requirement becomes more acute in RL training, where thousands of episodes depend on a consistent environment. Uncontrolled API variation contaminates the reward signal used to measure the intended behavior. Horizontal scaling also requires independent stateful instances across containers, since a shared instance introduces shared-state variability. Like traditional integration testing, these runs need to be repeatable and state-consistent, with no live credentials or rate limits.
Applying the stateful simulator approach to specific APIs: Stripe, Kalshi, PayPal, and Resend
Stripe sandboxes support simulated transactions with test credit cards and isolated settings. Stripe also provides Dashboard tools for manually triggering events including payment_intent.succeeded, charge.refunded, and charge.dispute.created. However, Stripe states that IC+ pricing cannot be tested in any Stripe sandbox, and its remote sandboxes still require credentials and coordination across runs. A local stateful simulator addresses the execution-control gap and can deliberately exercise duplicate and out-of-order webhook handling. Any claim to cover IC+ pricing still requires evidence of that specific behavioral fidelity, rather than an assumption that local execution supplies it.
Kalshi provides a demo environment with the same endpoints as production, fake money, and separate API keys. That environment is a plumbing test rather than a strategy test, supporting checks of RSA request signing, order lifecycle handling, WebSocket reconnection, and the exponential-backoff retry path required by its Developer Agreement. Because it runs remotely, it still needs credentials and depends on the infrastructure that hosts it. A stateful simulator moves repeatable order workflows and controlled failures into independently run CI environments.
PayPal uses sandbox accounts in a virtual environment that simulates production behavior. Its webhook simulator serves a narrower purpose, since simulated events are for demonstration only, are not tied to any App, and do not appear in the dashboard event viewer. That limits what those events establish about an application’s full delivery workflow. A stateful simulator can model the delivery-retry sequence deterministically within one CI run, without a live PayPal account. This lets the test exercise handler idempotency through the sequence.
Resend’s sandbox mode deals with a separate risk in shared development environments, where test databases may hold real email addresses. It removes the category of accidental email incidents. That safety property does not, by itself, establish repeatable execution or stateful fidelity for an integration workflow. The simulator’s role is to provide those broader test properties through controlled state and verified behavior.
What to verify when choosing or building a stateful API simulator
The first requirement is a verification process that tests the simulator against the real API before every release. Without it, the replacement can silently drift into the same false confidence as stale fixtures. Published fidelity scores should make clear where behavior matches and where it does not. Teams should require that transparency before committing the simulator to a CI pipeline. A deterministic result is useful only when the behavior it measures is representative.
Statefulness must also extend beyond plausible HTTP status codes. Verification should check for resource-not-found after a resource is deleted, conflict on duplicate creation, and status transitions that follow the right order. Realistic pre-seeded data should look like the data the integration will work with, so each test does not have to fill an empty world. Fault controls should operate per endpoint, with configurable latency, 429 thresholds, and on-demand 503 responses. These checks show whether the simulator enforces a workflow or just returns responses that look right on their own.
Deployment requirements complete the assessment. Local laptops, CI containers, and air-gapped environments need a simulator that does not depend on outbound access or a cloud account during execution. Agent evaluation and RL training also need independent stateful instances that can run across many containers without sharing a bottleneck. These are practical tests of whether control extends to the whole environment. The durable fix is not simply to stop calling the live API, but to replace it with a stateful dependency whose repeatability and fidelity are both maintained.
