Versioning API Simulator Configurations Alongside Application Code

The application changes while its simulated contract stays the same. Keeping both in the same repository makes that contract visible, but the decisive rule is atomic change. Whenever application code changes how it calls an external API, the simulator configuration must change in the same commit.

Why simulator configs drift when they live outside the repository

A green CI build can still break in staging. The application sends a renamed parameter or a new authentication header, but its tests passed against a simulator that still accepts the old request. The structural cause is often where the configuration lives. A separate repository, shared drive, or GUI export places it outside the pull request that governs the application change. Reviewers approve one side of the contract without seeing the other.

That separation makes drift easy to miss. An engineer can update the integration correctly while leaving its test environment behind, because nothing in the application diff exposes the stale configuration. Proprietary GUI exports compound the problem when their contents cannot be meaningfully reviewed as text. Simulator configurations must become version-controlled artifacts that travel through review and merge with the code they test.

What "co-location" means in practice: files, formats, and where they sit

Co-location means committing simulator files to the application repository and applying the same review, merge, and branching rules to them as to source files. A practical convention places simulators/ at the repository root, with separate directories such as simulators/stripe/, simulators/github/, and simulators/slack/. Each directory holds the configuration for that integration. A simulators/README.md records which simulator covers each integration and how to start it locally, while short integration-specific READMEs can explain their own files.

The behavioral artifacts should use plain JSON or YAML. Text makes a changed route, request field, or response visible in a pull-request diff, whereas a binary export hides the substance of the change. Pre-seeded records belong in the same integration directory because the data served at startup is part of the tested contract. Fault scenarios, including latency settings and error responses, should go there too. With these files together, a change to retry logic can include its fault scenario without having to manage a second repository or a separate export.

The single-commit rule: changing the application call and the simulator config together

Any commit that changes the application's use of an external API must update the relevant simulator configuration in that commit. The trigger is a contract change, including a renamed request field, a required parameter, an authentication header, a webhook payload shape, or a new endpoint call. Keeping the files near each other does not make them change together. The application diff and simulator diff must tell the same story.

Consider a change that adds a field to a Stripe charge request. Before the change, the simulator models the request without that field. Afterward, the application sends it and the configuration models its role in the revised request, with seed data adjusted where necessary. A reviewer who sees only the application change should block the pull request. The missing configuration diff is an incomplete integration change.

Published versioning governs an API provider's contract with its consumers, while simulator versioning governs fidelity to the application's current use of a third-party API. A team need not be changing its own public API for its simulator contract to require an update.

Automating the enforcement: CI hooks that catch config mismatches before merge

Human review needs a mechanical backstop in CI. The first check starts the simulator with the committed configuration and fails the job if startup fails. An unrecognized route or field structure must be caught before tests start so the test suite has the dependency it needs to run. A missing simulator is also a hard failure. Silent fallback to the real API would allow a green result to mean something other than successful testing against the committed contract.

The next check compares contract changes on every pull request. Each external API's OpenAPI document belongs in the repository with its simulator configuration, so CI can compare the proposed contract against the base branch and classify differences as additive or breaking. Additive changes pass this check. A removed field, renamed parameter, or newly required field fails the build or requires an explicit acknowledgment label. Automated classification makes the review independent of whether an engineer correctly describes a breaking change in the pull-request text.

Startup establishes that the committed simulator can run, while the contract diff exposes changes that require review. Neither makes environment boundaries optional. The simulator must be URL-scoped to the CI test environment, with explicit configuration guards preventing it from reaching a production-like environment. Those guards belong in the reviewed configuration.

Branching strategies that keep simulator state honest across environments

Suppose two engineers are working on separate feature branches and each adds a field to the Stripe integration. Each branch must contain the application change and its corresponding update to simulators/stripe/config.yaml, rather than borrowing a shared configuration from main. If those updates conflict at merge time, the conflict signals that the two integration changes need reconciliation. It exposes competing assumptions while both changes are still under review.

Seed data follows the same branching rule. A branch that adds a new order status or event type needs records of that type to test the feature end to end, without altering main's test data prematurely. Long-lived release branches also need simulator configurations that match the application version on each branch. The simulator directory branches and merges with the rest of the repository. Treating it as a global resource would recreate the separation that co-location was intended to remove.

Environment differences can remain narrow and explicit. Where needed, an environment-agnostic base configuration can have a thin overlay, such as a fault scenario enabled in staging but not in local development. Both files should be in the same integration directory and have clear names and version history. The overlay changes how the environment should behave and keeps the branch's underlying contract in place.

Stateful simulator configs require more than route definitions

A stub that returns the same order for every GET /orders/42 does not express creating an order, fetching it, cancelling it, and fetching the changed status. A stateful simulator must encode the transitions between those operations. Its versioned contract therefore includes not only the response shape but also how earlier calls affect later ones. Otherwise, a test can exercise every route without testing the sequence the application depends on.

Initial state is part of that contract. The configuration must declare realistic records present at startup, followed by rules governing how operations change them. A PATCH may set a status, while a DELETE removes a record from subsequent GET results. Asynchronous behavior requires declarations as well. A checkout flow that initiates a payment intent and expects a webhook needs the event and its delivery delay versioned with the application code that handles it.

Fault scenarios also shape a sequence of calls. A 429 followed by a retry window is not equivalent to an isolated error response, because the application's next action depends on that window. Latency, 429s, and 503s need to be versioned just as routes and state transitions are. When retry or fallback behavior changes, the simulator scenario for it must change in the same commit so the test follows the recovery path it should.

API-specific versioning behaviors that the simulator config must track

Kalshi switched to a token-bucket rate-limit model on 2026-04-23, with seven tiers and no Retry-After header. A configuration that encodes the previous behavior teaches application code or an agent to handle a response the API no longer sends. The missing header is operationally significant. A test environment that supplies it would exercise a different rate-limit response from the real integration. The configuration under simulators/kalshi/ must record that behavior, with a comment or changelog entry noting the change date.

Kalshi also rejects timestamps too far from server time with an opaque 401 response. That response can look like a credential error. A simulator running in a container with a frozen clock must explicitly model the timestamp behavior if tests are to cover that authentication failure accurately. The contract includes both the presence of an authentication field and the relationship between the request timestamp and server time.

Slack illustrates drift at a different level. Slack's documentation includes channel_type and event_ts in real message event payloads, while a simulator payload can omit fields that the application expects. A configuration missing those fields tests a different event shape, even if delivery itself works. The update must change the event declaration and any seed data that uses those fields. When an external API publishes a changed payload, required field, or rate-limit behavior, a dedicated pull request should update the configuration and affected application code together, making review the checkpoint for the revised contract.

Applying the same versioning discipline to AI agent and RL environment setups

AI agents need the same contract discipline as integration tests. Their probabilistic behavior makes hallucinated parameters, unauthorized calls, and mishandled rate limits harder to detect after deployment. An agent that uses external APIs through tools must therefore train and test against stateful configurations versioned with its code. The simulator must model what actions do as well as return text that sounds right. Placing a trade on Kalshi, posting a message to Slack, or opening a PR on GitHub must produce a state change the environment can observe.

Reinforcement learning adds an explicit reset requirement. Between episodes, the environment must return to a declared initial state without side effects on production. That reset declaration belongs in version control with the reward function and agent tool definitions. If a reward change depends on a new API resource type, the same commit must add that resource to the initial state. Otherwise, the training objective and the environment are for different tasks.

Fault injection is part of the training setup. An agent trained against a simulator that never produces 429s or 503s will not develop the retry and fallback behavior those conditions require. Fault scenarios must be co-versioned with the training curriculum so agents can learn how to handle them. Stale parameter shapes, authentication flows, or rate-limit responses can also teach malformed or unauthorized calls, and post-deployment sampling cannot reliably catch that mismatch. Versioning the environment keeps those assumptions available for review before deployment.

A reference workflow: from directory setup to merged PR

Create simulators/ at the root and a directory for each external API, containing plain-text configuration, seed data, and fault scenarios. Add the startup and coverage instructions in the READMEs. Keep state transitions, webhook behavior, and any environment overlays with the integration they govern. For agent environments, include the initial state and reset protocol with the versioned training setup.

During feature development, update the simulator on the same working branch as the application. Put each contract change and its simulator counterpart in one commit, and identify the configuration changes and their reasons in the pull-request description. Review both diffs as parts of the same integration change. Resolve simulator merge conflicts along with the source conflicts they relate to, and keep the seed data for each branch and the release configurations. No branch should depend on simulator state maintained only on main.

Require both CI checks to pass before merging. Startup must use the committed configuration and fail without fallback if the simulator cannot run, while the OpenAPI diff must classify additive and breaking changes under the acknowledgment policy. Keep the environment guards on for the whole test job. When a third-party API changes how it behaves, use a dedicated pull request to update the simulator, record the real API change date, and change the application code it affects. That workflow makes the tested contract part of the release itself, rather than a separate system someone must remember to maintain.