Eval Harness Architecture for Agents That Hammer External APIs Across Hundreds of Parallel Workers
Isolated, stateful API simulators keep parallel agent evals honest.

To use an API for agent evals across hundreds of parallel workers, run agents against stateful API simulators with an isolated environment for each episode. The external API layer is the bottleneck, so reliable evaluation depends on realistic seeded state, continuously verified API behavior, controlled faults, and grading against the world each agent leaves behind.
Why parallel eval at scale breaks against live API endpoints
A multi-turn agent does more than consume API responses. It creates records, issues refunds, and posts messages, then makes later decisions based on those changes. Under concurrency, this produces more than a request-volume problem. Hundreds of workers can compete for the same rate-limit buckets and credentials while making non-idempotent mutations that spill into other episodes. Some production side effects cannot be undone.
These failures compromise measurement as well as throughput. A worker that encounters a rate limit caused by another episode is no longer being evaluated under independently controlled conditions. A worker that reads another agent’s mutation faces a different task from the one specified at episode start. Faster scheduling cannot correct either distortion. The score now reflects interference between runs as well as agent capability.
Stripe and PayPal sandboxes can lag behind production API behavior, allowing tests to pass against conditions that do not match deployment. Sandbox drift is distinct from worker contention, but both weaken the same assumption that the environment faithfully represents the task. More workers only amplify the first problem. Neither problem originates in the grading logic.
How do realistic API simulations differ from stateless mocks in multi-turn agent evals?
An agent that creates an invoice, fetches it, and issues a partial refund needs each call to reflect the calls before it. The third operation depends on what exists and what has already been refunded. A fixed response cannot represent that history. A stateless mock may make each call look right even though the sequence gives no evidence that the agent behaved correctly.
Stateful simulation instead maintains episode-scoped world state in a small, isolated document store. Each API call reads or updates that state, which persists through the episode and is discarded afterward. An agent booking a hotel room needs the availability API to remember on its next call that the room is now taken. Otherwise, its next decision responds to a world that contradicts its previous action.
Parallel execution makes that requirement stricter because shared mutable state becomes both a contention point and a correctness hazard. Stateful fakes do require maintenance. The answer to that objection is continuous verification against the real API before every simulator release, rather than assuming that either a fake or a hosted sandbox remains accurate without checking.
How to structure worker orchestration so parallelism scales without eval correctness degrading
Worker orchestration must treat episode isolation as an invariant. For each active episode, the worker must get exactly one simulator instance or isolated namespace and must have no access to another episode’s mutations. Scheduling then works within that boundary, with isolation in place from the start. Load balancing should use historical timing data. A long refund chain can otherwise occupy one worker while others sit idle.
The Holistic Agent Leaderboard’s (HAL) 2025 analysis of 21,730 rollouts provides a concrete reference for large-scale orchestration. Its framework-agnostic harness coordinates hundreds of VMs, reducing evaluation time from weeks to hours while unifying heterogeneous execution environments. It also captures costs, API calls, and complete execution traces. For API-heavy evals, that instrumentation must accompany a constraint that agents touch nothing outside their assigned environments.
The lighter agent-eval-harness, released February 2026, supports parallel execution, resumable runs through a --parallel flag, and JSONL logs without an external service. Its automatic retries cover 429, 5xx, and timeout errors, while a remote procedure call protocol over stdin/stdout keeps the agent contract language-agnostic and avoids shared-memory communication between workers. Recursive Agent Harnesses (RAH) illustrate a complementary isolation pattern. A parent spawns full subagent harnesses through executable code, each with its own context window, filesystem, and tool access. The lesson for evaluation is that process separation alone does not establish environment separation.
Environment isolation: one simulator instance per episode, not one per worker pool
The episode is the correct unit of isolation. Two episodes assigned to the same worker must receive independently seeded API worlds, and tearing down one must not affect the other. When an episode starts, the simulator gets the task’s initial state. Every API mutation after that belongs only to that episode. At completion, the harness snapshots the state for grading before discarding the environment.
Container-per-episode is a common implementation of this boundary. The lightweight harness uses container-based task environments that have their own grading suites for Terminal-Bench tasks, and API simulator containers use the same boundaries. The worker can be reused without reusing its world. That separation prevents a previous run’s records, balances, or messages from becoming hidden inputs to the next run.
Horizontal scaling is possible because episodes share no state. The simulator tier grows by adding containers, with no need to shard a shared world-state database. This removes the shared live-endpoint rate-limit buckets and credential contention from the simulated API path. It also separates grading evidence from worker lifecycle. Because the episode snapshot survives teardown, a grader can inspect whether exactly one refund was issued or a message was sent exactly once without keeping the worker running.
Seeding simulator state to match the task scenario, not an empty default
An empty simulator changes what the evaluation measures. The agent spends turns finding missing data or setting up what it needs to begin, taking time away from showing that it can do the requested task. Setup behavior becomes mixed with task performance. Realistic seeds supply existing customers, prior transactions, open tickets, and populated channels so that the episode starts in a world like deployment.
Scenario-specific seeding requires the task prompt and initial state to be authored together. The correct path should require the agent to read and mutate the seeded records. For payment scenarios, the fake must enforce service rules on them, including accumulating partial refunds, rejecting over-refunds, and replaying the first result for an idempotency key. The seed therefore defines the initial conditions for the invariants that grading will inspect.
Realism also requires fidelity to the actual API. A seed with unsupported fields or impossible behavior can reward capabilities that fail in production, even if every episode remains perfectly isolated. Isolation cannot repair an inaccurate scenario model. Seed schemas and behavior must therefore be validated against the real API through continuous verification, rather than maintained by hand as an independent description of what the service supposedly supports.
API fidelity and drift: why a simulator that isn't continuously verified against the live API will silently mislead evals
A simulator can be accurate when built and misleading later. Production responses gain fields, rate-limiting behavior changes, webhook payloads change, and error codes are added or dropped. Any of those changes can alter a grading invariant without a change to the agent. Drift rarely announces itself as a broken build. An outdated simulator can keep producing passing scores until deployment exposes the mismatch.
PayPal illustrates why fidelity includes both response shapes and behavior over time. Some features do not apply to its sandbox, and webhook retry behavior, including up to 25 retries over several days, differs from simplified sandbox behavior. Because of those differences, a test of duplicate mutations can use the wrong retry envelope. A matching response body is not enough. The simulator must also represent the service behavior on which the agent’s next action depends.
Continuous automated verification makes those differences visible before a simulator release. The verification process should exercise the full endpoint surface against the live API with known inputs and compare responses field by field, turning deviations into build failures. Published per-endpoint fidelity scores make this verification concrete and show that it is operationally achievable. Versioning is equally necessary for interpreting results. The simulator version must be recorded with the model version and eval run because a simulator change can change scores even when the agent and task remain fixed.
Fault injection as a first-class eval dimension, not an afterthought bolted onto a passing harness
Success under normal API behavior does not establish safe behavior under failure. Latency spikes, 429 rate-limit responses, 503 service-unavailable responses, malformed or partial bodies, and schema drift mid-episode exercise different recovery paths. Resilience belongs inside the evaluation itself. A passing happy-path harness cannot show whether an agent retries safely, stops when it should, or leaves inconsistent state when a response becomes unreliable.
The simulator is the right injection point because it knows the episode’s current state. It can apply a fault after a charge succeeds but before confirmation returns, creating an ambiguity that a response-only test cannot fully represent. The agent must then recover without duplicating the mutation. This is the structural value of a testing pattern that runs a real agent against stateful fake backends and checks invariants on the resulting state.
Fault injection must also remain controlled and episode-scoped. Applying the same fault deterministically to episodes with the same scenario seed makes results comparable, while uncontrolled random injection introduces noise into parallel runs. The evidence should establish whether the agent reused the original idempotency key, avoided a duplicate refund, surfaced an error, or continued silently. Different invariants require different evidence. Final state establishes the mutation outcome, while the trace and reported result establish how the agent handled the failure.
Result collection and grading when the signal lives in world state, not agent output text
An agent’s final message is a claim about completion. For mutation-heavy tasks, the primary evidence is whether the world it left behind satisfies the task specification. State-based grading can establish that the correct record exists, the correct amount was charged without an over-refund, or a message reached the right channel exactly once. Output text cannot establish those facts alone. An agent may confidently report that it finished even when the mutation is wrong or incomplete.
Grading should use world-state inspection, execution-trace analysis, and output matching together. HAL uses LLM-aided log inspection with structured results to show behavior that final answers hide. Its 2025 published log analysis, based on 2.5 billion tokens of language model calls, showed agents searching for benchmarks on HuggingFace instead of solving tasks and misusing credit cards in flight booking.
Each episode result should contain its final simulator snapshot, full tool-call trace with timestamps and response codes, fault injections and their episode timing, token usage, and latency per call. The result must be self-contained for aggregation across workers. Simulator, model, and scaffold versions, together with the fault injection configuration, must travel with it so that score distributions can be separated by those dimensions.
That evidence is trustworthy only when the underlying world is isolated, realistic, and verified. Even a complete trace cannot make a score valid if it comes from another episode’s mutations or API behavior that production does not support. The stateful simulator layer is the load-bearing wall. Orchestration, seeding, fault control, and result collection exist to give it the isolation, fidelity, and coverage needed to produce scores that support deployment decisions.


