Running Third-Party API Tests in GitHub Actions Without Live Credentials

The GitHub Actions runner has become the most attractive target for credential theft in software supply chains, surpassing the developer laptop as the place where secrets actually leak. The GitGuardian 2026 report found that most machines with compromised credentials in 2025 were CI/CD runners, not developer workstations, because these pipelines execute arbitrary code with privileged access to secrets, over and over, at a scale no single laptop ever reaches.

The GhostAction campaign, active from August 31 to September 30, 2026, shows how this plays out in practice. The 2026 variant goes further still, scanning full git history for credentials that teams thought they'd already rotated out, so a rotation that didn't fully scrub history leaves the old key exposed anyway.

The standard response to this kind of exposure, OIDC federation for cloud credentials, solves a real problem, but it solves a different one. OIDC removes the need to store long-lived cloud provider keys in CI by issuing short-lived tokens at runtime, and that's a meaningful gain for anyone authenticating to a major cloud provider. But third-party API keys, the kind used to talk to Stripe, Slack, or any external service a test suite calls, cannot be federated the same way. Those keys have to live somewhere, and as long as that somewhere is a CI secret, they are a target. The only way out of that bind is to stop needing the key in the CI loop at all, which is the argument the rest of this piece makes.

Replacing the API, not just hiding the key

Secrets management hygiene, rotation schedules, scoped tokens, vaults, controls which systems can read a credential and for how long. None of it changes whether the credential has to be present and usable for a test to pass. If a test suite calls a real third-party API, something with valid authentication has to sit in the runner's environment at execution time, and that fact alone makes it a target, however well it's rotated or scoped.

The fix that actually closes this gap is to stop calling the real API during testing. Point the test suite at a local or in-process stand-in that speaks the same HTTP protocol as the real service but needs no credential and never touches a live system. Do that, and there's no third-party key in the runner environment for that API at all, so there's nothing for a compromised workflow to steal. A local stand-in sidesteps all three, because it isn't on the network.

What kind of stand-in does the job depends entirely on what the test is checking. A unit test that exercises a single endpoint in isolation can usually get by on a simple stub that returns a fixed response every time. An integration test that walks through a full workflow, say, create a customer, attach a payment method, trigger a charge, confirm the charge shows up in the list, asks for something a fixed-response stub cannot give it.

What stateless mocks get wrong about integration tests

A stateless mock has no memory of what happened in the call before the one it's currently answering, and that's exactly the property an integration test needs to check. When a POST in step one and a GET in step three return data that lines up, that consistency is something a developer configured by hand. It isn't evidence that the real API would behave the same way under the same sequence of calls.

Take a concrete case: a customer gets created in step one, and a subscription gets attached to that customer in step three. The mock never tracked the customer from step one, so it can't tell you whether that customer still exists by the time step three runs. It just returns whatever it was told to return.

The same gap appears in something as basic as content creation. A stateful simulator behaves the way the real API would: create an article and it shows up in the list; delete it and it disappears. The world the test operates in actually updates in response to what the test does.

That distinction matters because stateless mocks give developers confidence about the shape of individual responses while saying nothing about whether the full integration holds together. It's also how drift gets past CI. Hand-maintained mocks drift faster than ones generated from a formal spec, since every change to the real API means someone has to remember to go update the stub by hand, and most teams only discover the mismatch once it's already broken something in production.

Stateful, verified simulation in a CI workflow

A simulator capable of standing in for a live third-party API in CI needs three things working at once: state that persists across calls within a test run, continuous automated verification against the real API so drift gets caught before it spreads, and realistic data seeded in from the start rather than an empty sandbox the developer has to populate by hand.

The statefulness piece is what separates this from a mock: create a record, list it, delete it, and each subsequent call in the sequence reflects what the prior calls actually did, the same way the real API would behave. A simulator that ships with realistic data in it from the first run removes that tax.

The CI integration pattern that follows from this is simpler than managing live credentials ever was: start the simulator as a service container, point the test suite's base URL at it, run the tests, tear the container down. There's no secrets.STRIPE_API_KEY sitting in the workflow YAML waiting to be exfiltrated, because there's nothing live for it to authenticate against.

Fault injection, simulated latency, 429 rate-limit responses, and 503 service-unavailable errors all belong inside that same simulator. A simulator built to inject faults on demand lets that whole class of test run in the exact same CI job as the correctness checks.

Specific APIs in CI: Stripe, Slack, GitHub, and Kalshi

Which stand-in makes sense depends on whether a test is checking a single call's shape or walking through a multi-step workflow, and on whether the team can tolerate a live network dependency or needs something that runs fully offline.

Stripe offers its own open-source option in stripe/stripe-mock, distributed as a Docker image (stripe/stripe-mock:latest, running on port 12111). A separate project, adrienverge/localstripe, takes the statefulness further: creating a customer and adding a card to that customer actually affects what subsequent queries return. That difference is what makes localstripe suitable for multi-step workflow tests in a way stripe-mock isn't designed to be.

Slack has a comparable split. A fixture server returning fixed responses can't represent that sequence.

For the GitHub API, fault injection has been attempted by running Toxiproxy in front of the real API, following the pattern used in the aygp-dr/fault-injection-lab project, though that approach is only partially functional because of TLS constraints, and it still requires a live token and real network access to work.

Kalshi takes a different approach built around an official demo environment at demo.kalshi.co, with a REST endpoint and a WebSocket endpoint at wss://external-api-ws.demo.kalshi.co/trade-api/ws/v2, backed by mock funds and credentials kept separate from production. That's a genuinely useful environment for testing against realistic trading behavior. But it's a live environment, not an offline simulator: teams still have to manage demo credentials as CI secrets, and the network dependency hasn't gone anywhere. For air-gapped or fully offline CI runners, a locally hosted stateful simulator remains the only option that removes the dependency.

Testing AI agents that call external APIs: why the stateful problem compounds

AI agents that call external APIs as tools stack a second hard problem on top of the first one. With the LLM layer nondeterministic by nature and the API layer underneath it also live, a failing test gives no way to tell which layer caused the failure, and no way to reproduce it on the next run.

The standard way to separate these concerns is to treat both layers as external dependencies that need their own controlled stand-in. Mock the LLM call with a deterministic, fixed-JSON response to test the agent's routing and parsing logic on its own. Test the agent's API interactions separately, against a stateful simulator, so the question of whether the agent called the right tool with the right arguments can be answered without the API's live behavior muddying the result.

Most agents worth testing don't just read data, they mutate it, creating invoices, updating customer records, closing support tickets. Evaluating that kind of agent across multiple turns requires a world that remembers what the agent did on turn one by the time turn four runs. A stateless mock resets between calls and has no way to represent the cumulative effect of a session that unfolds over several turns. The fact that this RFC exists at all confirms that stateless agent mocking is a recognized gap in the open-source tooling community, not a hypothetical concern.

Tooling that mocks the LLM provider layer itself, OpenAI, Anthropic, Gemini, addresses a real and separate need, but it stops at the model. It says nothing about the tools the model calls once it decides to act. A complete test harness for an agent needs both pieces at once: a deterministic mock for the LLM's output, and a stateful simulator for every external API the agent touches.

Parallel isolation, matrix strategy, and contract drift detection in the CI workflow

Running stateful simulators across parallel CI jobs sounds like it should require some kind of shared state coordination, but it doesn't. Container-per-job instantiation handles it cleanly: each parallel job starts its own fresh simulator container, seeds whatever data it needs, runs its tests, and discards the container when it's done. No job's state collides with another's, and no job depends on the order in which other jobs run.

Stateless mock endpoints can handle concurrent reads without any special handling, since nothing about them changes between requests. Stateful tests, the ones doing writes, updates, deletes, need that per-job isolation to behave correctly, and the container-per-job pattern delivers it directly: inside its own container, a job's state persisting across its own calls is exactly the behavior the test wants, not a risk to be managed.

A matrix strategy layered on top of this extends naturally to different test scenarios, happy path, error handling, fault-injected failure cases, by giving each matrix cell its own simulator instance, pre-configured for that specific condition. All the cells run at once, and because each one owns its own container, none of them interferes with another's state.

The one failure mode a simulator introduces that calling the real API directly never would is contract drift: the simulator quietly falling out of sync with how the real API actually behaves. That risk has to be handled through continuous automated verification, not through someone remembering to check periodically. A CI step that checks the simulator's fidelity score against a published baseline catches that drift before it has a chance to spread into tests that pass against the simulator but would fail the moment they hit the real API. Placing that drift check early in the workflow, before the integration tests themselves run, means a simulator that's fallen out of sync fails fast with a clear signal, instead of letting quietly wrong tests pass straight through. That's what flips the reliability equation: instead of tests that pass on a developer's machine and fail the moment they hit a live API in production, a team gets tests that pass against a verified simulator and a result it can actually trust.

The payoff of this architecture

Swapping live third-party credentials in CI for stateful, verified simulators closes the credential exfiltration surface, removes the network-dependent failure modes that make test suites flaky, and gives integration tests the kind of determinism that makes a passing result mean something.

On security, no third-party API key sitting in the runner environment means a compromised workflow, whether through a GhostAction-style injection, a poisoned action, or a leaked build artifact, has nothing to exfiltrate for that service. The damage a compromised workflow can do shrinks down to whatever the GITHUB_TOKEN itself permits, rather than extending out to every third-party service the pipeline happens to touch.

On speed and reliability, a simulator running locally has no rate limits to hit, no DNS resolution to wait on, no staging environment that might be mid-deployment and unreachable. Replacing a live network dependency with a deterministic local stand-in is what drives flaky reruns down toward zero.

On coverage, fault injection, 429s, 503s, latency spikes, runs in the same CI job as the correctness tests instead of being treated as a separate exercise that only happens right before a release. Failure-handling bugs get caught in CI instead of in production, which is the point of having a test suite.

For AI agent workflows specifically, multi-turn evaluations get an isolated, resettable world for each episode, and the same stateful simulator that verifies a Stripe charge lifecycle can just as easily model the state an agent builds up across a five-turn conversation with an API. And for teams running network-restricted or fully air-gapped CI, this architecture gives them complete integration test coverage without needing to carve out exceptions or outbound network rules for third-party services.

None of this is a tradeoff against speed or convenience. It's a gain on every axis at once: the credential that could be stolen no longer exists in the runner, the network dependency that caused flaky failures is gone, and the test that passes now actually means the integration works.