Using a Commercial API Simulator as the Full State Machine for a Gym-Compatible RL Environment

A stateful API simulator can serve as a Gym environment's state machine.

· 9 min read
Cover illustration for “Using a Commercial API Simulator as the Full State Machine for a Gym-Compatible RL Environment”
AI Agent Testing · November 17, 2026 · 9 min read · 1,946 words

To use an API in an RL environment, wrap a stateful, session-scoped API simulator in a Gym-compatible interface. Map sessions to episodes, API calls to actions, and inspectable state to observations and reward inputs. A continuously verified simulator supplies the full state machine, so the wrapper connects the training loop to that machine.

Why RL environments for API-using agents need stateful infrastructure

An API-using agent’s trajectory interleaves LLM outputs with tool calls, and each call can change what later calls should return. Resource records, message threads, and payment objects must be kept from one turn to the next. Classic RL environments also maintain state, but API tasks require that state to follow resource lifecycles across several endpoints. Fixed responses cannot supply that continuity. A single coding-agent trajectory may take minutes of container time and use a large token budget, so a broken environment is costly.

Consider an agent that creates a resource, polls its ID until its status changes, then cancels it and expects a conflict error. A stateless mock returns the same fixture regardless of what happened before. The polling loop may never terminate or may terminate immediately, the returned ID may not match the created resource, and the conflict path may be unreachable. The environment causes the failure.

What a Gym-compatible environment's state machine actually requires

The MDP tuple used here has state S, actions A, transitions P, and reward R. For an API task, S contains structured records such as payment objects and invoice lines, while A contains tool calls with structured arguments. P governs what those calls do to the records. Refunding an uncaptured charge must produce the documented error. R evaluates the resulting task outcome, such as whether a requested message was added, using a task-defined reward rule.

These terms translate into four concrete engineering requirements, namely a structured observation space, a defined action space, a deterministic reset(), and a step() that executes an action and returns its result. Reset establishes the known baseline before each episode. Without it, execution order can change outcomes, turning a training environment into a shared live sandbox. Step must also distinguish task completion from a rollout cut short. Gymnasium 0.26 and later requires the five-tuple of observation, reward, terminated, truncated, and info, whereas the older four-tuple conflates termination and truncation.

The simulator can supply the transitions and state without deciding what constitutes success. That distinction preserves the task designer’s control. A correct API response says what happened, while the reward rule decides whether that helps complete the assigned task.

How a stateful API simulator already satisfies those requirements out of the box

The correspondence between a stateful simulator and a Gym state machine is direct. Session-scoped resource state supplies S, and the endpoint catalog supplies A. Resource lifecycle rules supply P, while inspectable state changes supply inputs to R. Session reset implements the reset contract, and each HTTP round-trip performs an environment transition. The wrapper packages these capabilities into Gym’s interface.

Statefulness has a concrete meaning in this mapping. A POST stores a resource, the next GET returns it, a list includes it, and a PATCH updates it. Webhooks fire when the modeled state changes. Session isolation keeps parallel rollout workers from seeing one another’s mutations, while state injection lets each task set up its own world by posting initial JSON. Different tasks can use the same simulator implementation.

CUA-Gym-Hub demonstrates state injection and session isolation as foundational requirements for an environment usable in RL. The same contract lets tasks be evaluated through the flat key-path state diff that every /go call returns. Client IDs, invoice numbers, and timestamps must agree across Slack, Stripe, PayPal, and other APIs involved in the task. Independent fixed fixtures cannot maintain that relationship as resources change.

How to wrap an external API as a Gym environment through the session lifecycle

The episode lifecycle follows the simulator session lifecycle. The wrapper’s __init__ registers the simulator URL and declares the observation and action spaces. Before an episode begins, reset() establishes a unique session ID and POSTs the task’s initial-state JSON to the simulator’s state API. The rollout then operates against that session. A reset POST restores current state to the injected baseline rather than asking the wrapper to reconstruct every resource.

Session identity must follow the episode independently of which worker runs it. Each episode should have a unique ID, such as a task ID or UUID, so concurrent episodes do not share state files. At episode end, the session can be discarded or reused through the reset contract. Reusing simulator infrastructure does not mean sharing mutable episode state.

The baseline also defines where training randomness belongs. Reset restores the exact initial state unless the task generator changes the injected JSON. This makes variation explicit and controlled. A practitioner can vary the starting world at the data-generation layer without introducing unexplained changes inside the environment.

Representing simulator state as the observation space

The simulator’s /go endpoint returns initial_state, current_state, and state_diff, making the environment inspectable throughout a rollout. These fields serve different consumers. The reward function can read the diff, while the current state provides structured material for constructing the agent’s context. Neither requires screen-scraping or manual reconstruction from a sequence of fixtures. The simulator’s documented SCHEMA maps user-visible actions to the state fields they affect.

The wrapper declares a gymnasium.spaces.Dict corresponding to that schema when the training framework requires a structured observation declaration. This declaration primarily serves compatibility with frameworks such as RLlib and Stable-Baselines3. Many LLM-based agents instead consume the JSON string directly. The representation should not be confused with permission to expose all simulator internals.

For a partially observable API task, the agent receives only what a real API caller would see, including response bodies. The reward evaluator can look at more of the state on its own. This separation preserves the POMDP framing of tool use without sacrificing reliable outcome measurement. Full current state is suitable for the agent’s context only where the task’s observation contract allows it.

Defining the action space from the simulator's endpoint catalog

The endpoint catalog lists the available operations, so there is no need to write a separate list of actions by hand. Each action combines an HTTP method, a path, and arguments constrained by the endpoint schema. The wrapper’s step(action) translates that structure into an HTTP call against the active session and captures the response. It then reads the new observation and state diff from /go. The task’s rules for reward and completion turn those results into the Gym five-tuple.

Malformed requests and calls that break state-machine guards return the simulator’s documented errors. The agent must learn which operations are valid in the current world. A create, capture, and refund sequence shows why covering the endpoints alone is not enough. Each operation depends on the state established by the previous one, so a stateless mock cannot provide coherent feedback for the sequence.

For LLM-based agents, each endpoint schema also maps naturally to a tool definition. OpenAI uses a parameters JSON Schema object, while Anthropic uses the same underlying schema structure under input_schema. The field names and wrappers differ. The simulator catalog supplies the operation and argument definitions, leaving the Gym wrapper to translate between the agent’s tool-call format and the simulator’s HTTP interface.

Using fault injection as a curriculum control mechanism

Fault injection changes how hard an episode is while keeping the assigned task the same. Latency, 429s, 503s, and malformed responses test conditions that clean API responses cannot. AgentChaos found that fault injection reduced pass@1 by as much as half in some systems. That result makes clean-response success an insufficient measure of robustness. Fault exposure belongs in training and in a final testing stage.

A particularly damaging failure occurs when an agent receives an error but proceeds as though it received results. It may fabricate data to fill the expected response shape. Injecting failures at the HTTP layer makes it possible to evaluate whether the agent acknowledges the failure. CUA-Gym-Hub’s hardened session mode reflects a related separation principle by hiding the real session ID from the browser process, preventing CLI agents from extracting task state through Chrome LevelDB files. Environment controls must be kept out of the agent’s ordinary reach.

The recommended curriculum starts with clean responses, then progressively introduces rate-limit errors and timeouts as the policy matures. Fault configuration becomes a curriculum hyperparameter. Concurrent per-session configurations also allow a rate-limit error to coincide with a tool timeout, exercising retry and state-recovery paths that isolated faults miss. These episodes reach beyond the policy itself. They exercise the orchestrator’s retry logic, timeout handling, and human-escalation paths that normal pass@1 evaluations leave untouched.

Why simulator fidelity to the real API is load-bearing for training quality

A simulator can satisfy every structural requirement and still teach the wrong behavior if its contract drifts from the provider’s API. The policy then learns a world that no longer exists. Mocks maintained by hand are especially at risk because someone must update a stub each time the provider changes. Drift can affect response structure, nested fields, documented error behavior, and resource-transition rules. A wrapper cannot correct those discrepancies after the transition has happened.

The mitigation is to treat the simulator as an executable view of a versioned contract and verify it against the real provider in CI on every release. Verification is a recurring pipeline. Its first phase captures golden request-response pairs from a real account. The next compares simulator responses against those fixtures at every nesting level, followed by automated conformance tests that assert structural parity. The final phase tracks endpoint-level coverage through a machine-readable map.

An append-only quirk log preserves the unexpected behavior that fixture analysis uncovers. The ClawsBench quirk log has 65 entries of surprising real-API behavior found through that process. Such behavior belongs in the simulator contract even when it is inconvenient or unintuitive. Leaving that behavior out can make the simulation cleaner but the deployed policy less capable. Ongoing verification closes most fidelity gaps and makes coverage and known quirks clear.

What a production-ready simulator-backed Gym environment looks like end to end

A complete wrapper leaves API behavior in the simulator and task policy in the environment adapter. Initialization registers the URL and spaces, reset establishes the session and initial world, and step executes the tool call, reads the diff, and returns the five-tuple. Rendering is optional. This interface connects the simulator-backed environment to modern training frameworks, including RLlib, Stable-Baselines3, CleanRL, and TorchRL. Framework compatibility depends on the declared spaces and return contract.

Supported APIs with documented state schemas and session APIs become trainable surfaces through the same wiring. These include Slack, GitHub, Stripe, Kalshi, PayPal, Resend, and Lob. For a “send message” task, a reward rule can assign 1 when messages.C001 grows by one entry. That evaluates a state change directly. Most such tasks do not need screenshot parsing, VLM grading, or human labels to establish the resource-level outcome.

Session isolation also provides the basis for horizontal scaling. Thousands of parallel episode containers can target the same simulator pool without coordinating their state mutations. The session is the infrastructure boundary, while injected JSON changes the task and fault configuration changes the curriculum. Continuous contract verification then connects that training world to the provider behavior the agent will encounter.

Open-ended tasks still need reward signals that resist gaming, and outcomes that cannot be reduced to a state diff need separate verifier infrastructure. Business-intent confirmation still requires human evaluation. A simulator can establish that the requested resource transition occurred, but that alone does not establish that an open-ended business goal was satisfied. The approach removes bespoke API-state infrastructure from the training project without pretending to remove the judgment required to define success.

Filed underAI Agent Testing

More in AI Agent Testing