Simulator Resets Versus Scope Isolation for Stateful API Test Suites

Reset between independent test scenarios, isolate concurrent ones by scope.

· 8 min read
Cover illustration for “Simulator Resets Versus Scope Isolation for Stateful API Test Suites”
API Integration Testing · November 4, 2026 · 8 min read · 1,758 words

Integration testing against stateful APIs requires a simulator that preserves the effects of earlier calls without letting one test contaminate another. Reset between independent scenarios, isolate concurrent workflows, and preserve cumulative state within a workflow while separating its assertion windows.

Why stateful simulators force a state management decision that stateless mocks never did

A stateless mock returns a hardcoded body regardless of what earlier calls did. It has no memory to leak between tests. A stateful simulator instead updates its world when a resource is created, listed, or deleted, so later calls reflect earlier actions. That continuity makes it useful for integration testing, but it also introduces a state ownership problem when tests share an instance.

The distinction is about what the test proves. Redocly's sandbox requirements guide distinguishes stateless mock servers that validate HTTP contracts from sandboxes that persist state and execute business logic. A GitHub PR lifecycle, Stripe payment flow, or Slack conversation thread depends on that persistence. Testing each response independently does not prove that the application handles the transitions between them.

How state leaks across tests and why the failure signature is hard to diagnose

State contamination has several distinct signatures in a test run. A scenario may already have advanced beyond its starting position when the next test begins. Accumulated request counters can make a “never called” verification fail because an earlier test made the call. Extra registered stubs can override baseline mappings, returning a response the current test never configured. A sequence counter that did not start at zero can likewise select the wrong response.

These failures point at the wrong test. The test that runs second reports the error, while the test that left the state behind may have passed. Ordering tests to build on that residue can disguise the dependency, making a sequential run succeed while an individual or parallel run fails. A “passed on retry” result is especially suspicious because the first attempt may have prepared the state the second attempt needed.

What a full simulator reset provides and where it breaks down

A full reset gives each independent test an identical, documented baseline. No prior test's choices survive into that starting state. One way to do this is to start a disposable instance from fixed configuration and tear it down afterward. Another uses an explicit operation on a shared instance, such as POST /sandbox/reset, to wipe accumulated state. An instance created per suite establishes a suite baseline, but does not by itself separate tests inside that suite.

For independent tests, reset belongs in beforeEach. An aborted test may bypass teardown and leave the next test contaminated. Resetting before execution restores the baseline even in that case. Redocly identifies a programmatic reset as a first-class sandbox feature, allowing pipelines to start fresh and wipe or tear down the environment afterward.

That guarantee has limits when tests share a process. Reinitializing realistic transactions, repository history, or conversation threads takes time, while a global reset during parallel execution destroys another test's active state. A test must keep the world it arranged between phases to prove that later phases change it correctly.

How scope isolation keeps state separate without destroying it

Scope isolation preserves a test's world while separating it from other tests. Each test gets a unique identity, such as a user ID, resource prefix, or test-run header. Stubs, state reads, and verifications must all use that identity. This lets concurrent tests share a live instance without observing one another's requests or resources, while each test retains the history its later steps need.

Adding X-Test-Run to requests does nothing to shared storage unless the simulator uses it when selecting keys or routing responses. Isolation therefore depends on consistent handling throughout the workflow. A missing namespace or wrong key can silently return a test to shared state.

Observation boundaries can narrow assertions to a later phase without removing the fixtures that phase depends on. Test authors must all follow the scoping rules, but they can leave the environment in place. A versioned baseline configuration makes the initial conditions and available stubs explicit.

The hidden third problem with resets that conflate observation boundaries with state destruction

Consider a test that seeds a Slack workspace with channels and members, runs its first phase, and then wants to inspect only calls made during its second phase. A broad reset() may clear both the recorded calls and the seeded workspace. The next phase then runs against an empty world.

Hassette Issue #1353 captures this distinction in RecordingApi.reset(), which clears recorded calls and seeded helper definitions. The issue proposes separating observation-only boundaries from full fake-state destruction, with checkpoint-based since= queries as the leading option and clear_calls() as a minimal fallback. The distinction is semantic, not just an API naming preference. One operation changes what the test observes, while the other changes what exists.

A simulator should give those operations separate names that make clear what each one does. Clearing calls or querying since a checkpoint starts a new assertion window and keeps resource state in place. A full reset belongs at the boundary between independent scenarios. Without that separation, a test can pass while asserting against defaults instead of the prepared scenario it was intended to exercise.

How the choice criterion follows from what the test is trying to prove about cumulative behavior

The decision rule is to match state ownership to the test's claim. For an independent workflow test, use a full reset before the scenario begins because no dependency on earlier tests is intended. Preserve state throughout that scenario so later operations can depend on earlier ones. Reset again only when the next independent scenario needs its own baseline.

For integration tests that share one instance and run in parallel, use unique-identity scope isolation. Workers cannot safely share a global reset cycle. If each worker gets its own disposable instance, the instances keep their state separate, and each worker can reset its own. For multi-phase workflows, combine a test-owned scope with observation-only boundaries.

Can the test run alone, in randomized order, and in parallel with the same result? If not, its isolation strategy needs correction. Adding namespaces to a suite while retaining global resets is not a complete migration, because one worker can still destroy another worker's correctly scoped state.

How do you write integration tests for APIs where order of operations matters?

Opening, reviewing, approving, and merging a PR form one sequential workflow, so resetting between those phases destroys the history the next phase requires. A realistic “large open-source repo” scenario can also give the opening phase a plausible starting world.

Stripe payment tests have the same structural requirement. Authorization, capture, and refund depend on accumulated payment state, while independent payment scenarios can run concurrently under unique-customer-ID scoping. Stripe's sandbox separates test objects from live objects, but its settings documentation says settings and capabilities can diverge from live. A local simulator that is continuously verified allows controlled testing without having to rely on a separately configured sandbox to stay aligned with live.

Slack threads and workspace membership also last across calls. A test proving that a message was posted in a thread needs that thread to exist first, and a “design studio” scenario can make the setup reproducible. PayPal presents a different environmental constraint because its sandbox periodically cleans transaction data. A local simulator with state that stays stable and can be controlled frees long-running multi-phase tests from that dependency, but each test still needs a reset-versus-isolation decision at its boundary.

Fault injection and where it fits into the reset-vs-isolation decision

A payment timeout after the charge step but before capture can leave charged but uncaptured state, so a subsequent test may unknowingly exercise recovery. The resource state is not the only residue. A fault configuration left active can also affect calls from tests that never requested it.

Fault configuration should have the same owner as the resources. In independent tests, a full reset should wipe both, while concurrent tests need faults scoped to their unique identities. A total error rate applied globally to a downstream dependency contaminates a shared instance just as global resource state does. Safe shared execution therefore requires fault injection to be a first-class, resettable or namespace-scoped operation.

State isolation for AI agent tests and RL environments

An agent's second call may depend on changes made by its first, so clearing state between calls destroys the trajectory being evaluated. Non-deterministic outputs complicate diagnosis because the same test can follow different paths on different runs. A known simulator starting state is necessary for reproducing the conditions of a failing trajectory, even though it does not make the agent itself deterministic.

An RL episode sets a different boundary from a workflow phase. Each episode must begin with a full reset to a known, reproducible state because no reward signal should span episodes. Scope isolation alone is insufficient if the reward function can read residue from the previous episode. State must persist during the episode and be destroyed at its end or before the next episode starts.

When scaling horizontally, each container needs its own simulator that it can reset, because shared state cannot be partitioned across container boundaries. Programmatic resets, realistic seeded scenarios, and controllable transitions support deterministic starting conditions and independent execution. The criterion remains what the test must prove about cumulative behavior, but the episode boundary requires a complete break in that accumulation.

Practical guidelines for structuring a test suite around a stateful simulator

Put full resets in beforeEach for independent scenarios. Give parallel workers disposable instances when restart overhead is acceptable, or use a namespace-partitioned shared instance when it is not. Within that shared instance, each test must consistently scope its resources, stubs, reads, and verifications. No concurrent worker should retain a global reset that can erase another worker's state.

Within multi-phase tests, keep the arranged world in place from one phase to the next. Use clear_calls() or checkpoint-based queries when only the assertion window needs to change. Apply the same namespace or reset policy to fault configuration as to resource state, and use full resets at RL episode boundaries with independent instances across containers. These are different operations because they protect different boundaries.

To check the isolation design, run each test alone, randomize the order of tests in the suite, and run tests in parallel. Differences between those modes expose hidden state dependencies. Version-control the baseline configuration and document its stubs so the starting world is reviewable. Treat simulator state as something each test owns, with the ownership boundary determined by the behavior it is meant to prove.

More in API Integration Testing