Parallel Integration Test Runs With Isolated API State

A test suite that passes locally and fails in CI the moment a second worker joins reveals that two workers are sharing the same external API state and stepping on each other's records while the test runs. The stack trace rarely helps here, because the failure didn't happen in your application logic. It happened in a shared sandbox, where one worker's cleanup step deleted a record that another worker's test had just created and was about to read back. Run the same test alone and it passes every single time, because there's no second worker around to race it. Running it with four workers makes it fail unpredictably, because the outcome now depends on timing nobody controls. The diagnostic rule is simple: a test that fails more than once in parallel but passes every time in sequential mode is not flaky. It has a shared-state conflict in it, and that should block the merge until it's fixed. This kind of bug is harder to retrofit than almost anything else in a test suite, because it doesn't live in the test code. It lives in the external service the tests call: the same live sandbox, the same staging endpoint, the same dev Stripe account that every worker happens to be pointed at.

Isolated API state

Isolation, done properly, means each worker runs against an API environment whose state no other worker can see or touch. That's a stronger requirement than giving each worker its own test data inside a shared database. It means each worker gets a separate stateful world, where a resource created by worker 3 does not exist as far as worker 7 is concerned, no matter how many requests worker 7 sends.

Stateless mocks can't deliver that, and the reason is built into how they work. A stateless mock returns the same hardcoded payload no matter what calls came before it, so two workers can both send a "create" request for the same resource and both get back a 200, even though neither resource actually exists anywhere. The gap is clearest in a multi-step workflow. A Stripe customer gets posted in step 1, and a subscription gets created against that customer in step 3. A stateless mock will happily return a hardcoded customer object in step 1 and a hardcoded subscription object in step 3, and the two will look consistent to the test. But that consistency is a property of how someone configured the mock, not a property of the API's actual lifecycle. The mock has no memory of step 1 by the time step 3 runs, so it cannot actually tell you whether the customer in step 3 is the customer created in step 1.

Stateless mocks also drift over time. Because their responses are authored by hand, every change to the real API requires someone to go back and update the stub to match. Most teams find out about that drift in production, when a real endpoint behaves differently than the mock said it would, rather than in CI where the cost of finding out is much lower. None of this means stateless mocks are useless. They're fine for checking a single endpoint's response shape or confirming an auth rejection. They simply cannot provide the isolated, stateful environment that parallel workers running multi-step workflows need.

Shared live API environments turn speed gains into flakiness at scale

The entire point of running tests in parallel is to take a long sequential suite and cut the wall-clock time down across several workers. That gain disappears the moment those workers start contending over the same external API state, because every conflict produces a flaky result, and flaky results produce retries, reruns, and eventually a team quietly moving the offending tests back into a sequential-only queue, which erases the speed gain they were trying to get.

Three contention patterns occur repeatedly when workers share a live or shared sandbox API account. Record collision happens when two workers create a resource using the same identifier: one worker gets a 409 it wasn't expecting, or both get a 200 and the assertions that follow are checking against the wrong record. State pollution happens when one worker's test leaves a resource in a modified condition, a subscription that's been cancelled, a payment that's been captured, and a second worker's test assumes that same resource is still in its original, untouched state. Rate limit exhaustion happens simply because several workers are hitting the same credentials at once, and the account's rate limit gets saturated, producing a wave of 429 errors that look exactly like an infrastructure outage even though nothing is actually down.

Live sandboxes make all of this worse, not better. Stripe's sandbox does not guarantee that events arrive in order, it allows duplicate webhook deliveries, and it retries undelivered events only three times over a few hours, compared to live mode's three-day retry window. A parallel run sharing one sandbox account will receive webhook callbacks interleaved across workers, so a callback meant for worker 2's checkout flow can land on worker 5's listener instead. Once you see these three patterns together, the conclusion isn't really debatable: you cannot parallelize safely against a shared API surface. The unit of isolation has to be the API environment itself, not just the data sitting inside it.

The architecture that makes per-worker API isolation practical: stateful simulators

Per-worker isolation has three requirements, and all three have to hold at once. The environment has to be stateful: it remembers what earlier calls in the same session did. It has to be scoped, meaning only this worker's calls can affect its state. And it has to be cheap enough to spin up and tear down that it can be provisioned per worker in CI without someone coordinating port numbers and cleanup by hand.

A stateful simulator is built to satisfy all three at once. It keeps its own record of everything created, updated, and deleted during a worker's test session, it runs completely isolated from every other simulator instance running alongside it, and it runs locally or in a container, so it never touches live credentials or a shared rate limit. Lined up against the alternatives, the gaps close in order. A shared live sandbox is stateful, but it isn't scoped, because every worker is writing into the same account. A stateless mock is scoped to each worker, but it isn't stateful, since it doesn't remember what happened two calls ago. Recorded cassette tests are scoped and stateful in a narrow sense, but the recorded responses are fixed at the moment they were captured; a multi-step workflow that depends on state evolving across calls will break the cassette's illusion the moment the real API's behavior shifts, and the cassette will keep answering as though nothing changed. A stateful simulator is scoped, it's stateful, and it's checked continuously against the real API, so it can't quietly drift the way a cassette or a hand-written mock can.

This is the same shared-nothing principle that Testcontainers applies to databases: give every worker its own database container, with nothing shared between runs, rather than pointing every worker at one shared instance. The same logic applies to third-party API dependencies. Each worker should get its own simulator instance, not a slice of a shared one.

Spinning up one simulator instance per worker: the concrete CI setup

The setup for this mirrors how most CI platforms already handle database isolation, so teams aren't inventing a new pattern from scratch. Each worker provisions one simulator container as part of its setup step, runs its full test session against that instance, and tears the container down once the session ends.

A few configuration choices decide whether this actually works in practice. Each worker needs a simulator port derived from its worker index, something like a base port plus the worker ID, so that ports never collide and nobody has to coordinate assignments at runtime. Each simulator instance should start with realistic, pre-seeded records rather than a blank state, because an empty sandbox produces tests that only prove setup succeeded, not that the API behaves correctly against data shaped like production data. And if the simulator fails to start, the CI job should fail outright rather than quietly falling back to a real endpoint; a suite that falls back silently can report green for a reason that has nothing to do with the code being correct.

GitHub Actions matrix builds fit this pattern directly: define a matrix over worker indices, and let each matrix job provision its own simulator instance as part of its setup. The matrix gives you the parallelism, and the per-job simulator gives you the isolation, so the two reinforce each other.

Designing tests to be safe for parallelism once the simulator is in place

Per-worker simulator isolation solves the problem of workers interfering with each other. It does not solve ordering problems inside a single worker's own session, and it won't rescue a test that assumes a clean slate but leaves residue behind for the next test to trip over.

Four practices close that remaining gap. Resource identifiers, customer IDs, order numbers, webhook targets, should be generated at runtime rather than hardcoded, because two tests in the same session that both try to create customer@example.com will conflict with each other even inside a single simulator instance. Each test should set up the state it needs for itself, at the start of the test, rather than relying on a shared fixture that an earlier test happened to leave behind, so no test depends on another having run first. Multi-step workflows, a checkout that goes create, then authorize, then capture, need to run those steps in order within one worker's session; the speed gain comes from distributing independent sessions across workers, not from trying to split a single workflow's steps across workers. And the quarantine policy from the first section still applies here: a test that fails in parallel and passes sequentially has a shared-state bug in it, and the fix is to correct the test, not to retry it until it happens to pass.

Not every test needs the simulator's full machinery. Tests that check a single endpoint's response shape, an auth rejection, or a 404 can run against a minimal fixture without the simulator's memory involved. Tests built around multi-step workflows, a Stripe payment lifecycle, a GitHub pull request review chain, a PayPal capture followed by a refund, need the simulator's state and should be tagged explicitly, so the test runner schedules them against a stateful instance.

Fault injection as a first-class concern in isolated parallel environments

With per-worker isolation in place, fault injection scenarios, a deliberate 429, a latency spike, a 503 thrown mid-workflow, become safe to run inside the parallel suite without risking any interference with the correctness tests running on other workers at the same time.

This matters more for fault-injection tests than for almost any other category, because they're the tests most likely to leave an API environment stuck in a half-finished state. A test that injects a network partition in the middle of a checkout flow might leave a PaymentIntent sitting in requires_capture with no cleanup path defined for it. In a shared environment, that stranded resource is a problem waiting for the next worker that happens to touch it. In a per-worker simulator, that stranded resource simply dies with the container when the session ends.

A few concrete scenarios illustrate where this pays off. Against Stripe, inject a 429 between the POST /v1/payment_intents call and the subsequent capture call, then confirm the client retries with exponential backoff; the customer must never be double-charged. Against GitHub, inject a 503 during a check-run update and confirm the CI integration marks the run as inconclusive. Against PayPal, inject a duplicate webhook delivery, since PayPal's own retry policy allows up to 25 delivery attempts, and confirm the handler is idempotent and doesn't process the same event twice. A stateful simulator can be configured to inject a fault at a specific point in a workflow, say the third call to a given endpoint, rather than failing every call globally, and that is what makes this possible. That turns the test into something deterministic instead of something probabilistic, so it becomes a test you can trust rather than one you have to run five times to believe.

Continuous verification keeps the simulator trustworthy against the real API in CI

All of this depends on one condition holding: the simulator has to behave like the real API. If it silently diverges, the simulator reports green with total consistency while actually being wrong, leaving the team with a false confidence that's invisible and worse than the flakiness it replaced.

The only real defense against that is continuous, automated verification. Before every simulator release, the same request corpus should run against both the simulator and the live API, the responses should get diffed against each other, and the release should be blocked if the fidelity score drops below whatever threshold the team has set. That's a different exercise from contract testing, which checks a consumer's expectations against a provider's published spec. Verifying a simulator against the real API checks the simulator's actual behavior against the real API's actual observed behavior, which catches divergences that a written spec was never going to express.

Hand-maintained mocks can't be verified this way at all, because their responses are authored rather than derived from the real system. Running a verification pass against a hand-written mock would only confirm that the mock returns what someone told it to return. It says nothing about whether what it was told matches what the real API actually does. When evaluating a simulator for this kind of parallel test architecture, the standard to hold it to is straightforward: its verification methodology should be transparent, automated, and tied directly to the release cycle of the real API it's simulating, not something that was checked once at launch and then updated only after a user filed a bug report.