Seeding Realistic Test Data for CI API Integration Tests

A test can pass locally and fail in CI without a single line of test code changing. The record another test deleted is gone. The webhook that was expected never arrives because the queue was drained by a prior run. Reliable CI integration tests for API-dependent code live or die on the quality of their starting state, and most teams never give that state the attention it demands. This piece walks through how to seed realistic, stateful test data so that every pipeline run begins from a known, meaningful world rather than an empty or brittle one.

Why the starting state of a CI integration test determines whether it can fail for the right reasons

A test that begins from an empty or inconsistent world cannot distinguish a real bug from a missing precondition. PRs stall because QA cannot get a stable environment, and flaky test investigations divert the team from actual delivery. These are not separate problems, and they are not solved by more retries or longer timeouts.

The root cause shared across these symptoms is that the starting state was either absent, shared and mutated by a prior run, or inconsistent with what the code under test expects to find. A test that assumes an order exists but runs against a database where that order was deleted by an earlier test will fail, and it will fail in a way that looks exactly like a regression in the order-processing code. The follow-up investigation burns time on the wrong hypothesis.

The deeper issue is that teams inherited a stateless, hardcoded fixture model from an era when APIs were simpler. That model worked when an integration test meant one request and one response. It fails as soon as the integration spans more than one sequential call, because the world the second call needs is produced by the first call's side effect, and a fixture file cannot express that chain. The quality of the starting state, not the cleverness of the test assertions, is what determines whether a failure means something.

What "seeded state" actually means for an API integration test and why it differs from a database seed

Seeding state for an API integration test means establishing a coherent, multi-entity world that the API under test can traverse, not just populating rows in a table. The relationships matter, not just the records. A table full of orders means nothing if none of them connect to a customer with a valid payment method.

The distinction matters because API integration tests exercise code paths that call external endpoints in sequence: create, then read, then update, then cancel. Take the flow of creating an order, polling its status until it reaches a terminal state, and then cancelling it if a deadline passes. Each call's precondition is the previous call's side effect. The cancel endpoint expects an order that exists in a cancellable state. The status endpoint expects an order that was created by the same test run. A seed that populates rows without respecting these dependencies produces a world where the second call fails for reasons that have nothing to do with the code under test.

A stateless mock makes the problem worse because it returns the same fixture regardless of what came before. It cannot represent the cumulative reality of sequential calls, so a polling loop either never terminates or terminates immediately, and a conflict path such as a 409 because the resource already completed is unreachable. A true stateful environment remembers: an invoice created in step one exists and can be retrieved in step three. A deletion in step two means step four correctly returns 404. That memory is what allows a test to fail for the right reason.

Three seeding layers exist, and all three matter. Reference data covers static, immutable records: currency codes, roles, pricing tiers. It is seeded once, never mutated by tests, much as a Prisma seed script example might create 20 posts as a fixed baseline. Scenario data is the meaningful world a specific suite needs: the customer with an overdue invoice, the repository with an open pull request. It is seeded per suite or per run. Each layer has a different lifecycle, and conflating them is how seeding efforts collapse into confusion.

How handcrafted fixtures rot and why schema evolution accelerates the decay

Hand-maintained fixture files are a liability that compounds with every API change. They start as a reasonable shortcut, a JSON file checked in next to the tests that saves an afternoon of setup. They become the bottleneck that slows every test initiative, because every change to the API now demands a parallel change to the fixtures, and that change is easy to get wrong.

The immediate maintenance tax shows up in two ways. When an API adds required fields, existing fixtures become invalid, and every test that reads them fails until someone updates the file. Negative testing coverage collapses under manual fixture management, because teams focus on happy-path fixtures and cover only a small fraction of the error cases that matter in an integration environment.

The spec-derived alternative is to generate test data from OpenAPI or schema definitions, so the data adapts as the spec evolves. Nothing tells you when a handwritten mock goes stale. The spec does. That is the difference between a maintenance problem that grows with every release and one that shrinks. Drift is not theoretical, and it is not confined to test fixtures. A fixture that had included billed_until for years would keep passing locally while the production behavior silently changed.

The parallel with CI/CD is exact. Ad hoc fixture maintenance is to test data what ad hoc deployments are to software delivery: both work until scale makes them untenable. A team that would never deploy by hand to production will happily maintain hundreds of fixture files by hand and wonder why their test suite rots.

Generating seed data from schemas and factories instead of maintaining it by hand

The practical alternative to hand-crafted fixtures is generating seed data programmatically from schemas and factory functions, so the data stays valid as the spec evolves and covers both happy and unhappy paths automatically. The factory pattern is the core idea. A call like makeUser with a role override returns a complete, valid user object. The test does not manage fields it does not care about.

Faker.js v10, published as the maintained successor to the original faker.js under the package name @faker-js/faker with stable API and expanded locale support, is the standard library for generating realistic field values: names, emails, UUIDs, dates, addresses. Factory libraries such as fishery, a popular TypeScript factory library, and another schema-mock data generator with more limited adoption, sit alongside plain factory functions. None of these tools is the right answer on its own. The tier model is the argument, and each tool populates a tier.

Deterministic seeds are what make generated data usable in CI. Three generation tiers match the test type and time budget. Fast, simple generation with Faker and factory functions serves integration tests that run in seconds to low minutes. Full-complexity, statistically realistic generation, using tools like Tonic.ai, which can learn production data distributions to generate statistically realistic synthetic data through Tonic Fabricate or de-identify and mask real production records while preserving their structure and distributions through Tonic Structural, serves nightly performance runs where distribution fidelity matters.

Reference data such as country codes, roles, and pricing tiers belongs in static, versioned seed scripts managed using database schema migration tools like Liquibase or Flyway, which can also apply versioned seed scripts. These are applied reliably during CI and local runs and never regenerated per test. The reference layer is immutable, and treating it as part of the schema migration pipeline is what keeps it that way.

Loading seed data into a stateful, isolated environment rather than a shared one

Generated data placed into a shared, mutable environment is nearly as fragile as hand-crafted fixtures. Isolation at the environment level is what makes seeding actually work in CI. The shared staging database antipattern is familiar: one team's test mutations break another team's tests, a failed run leaves state that poisons the next run, and flaky failures are indistinguishable from real bugs. A suite that passes against a freshly seeded database but fails against one that a previous run left half-deleted produces exactly the intermittent, unreproducible failures this piece opened with.

Each CI run should start from a known baseline and end with that baseline destroyed. Isolation strategies differ in speed and fidelity. Transaction rollback, the pattern used in Vitest with Prisma, wraps each test in a database transaction and rolls back on teardown. It is fast and requires no schema cleanup, but it only works when the ORM cooperates and the test does not span process boundaries. A test that spawns a subprocess or calls an external service that writes to the database during the transaction will break under this model.

Ephemeral containers, the Testcontainers pattern, spin up a fresh database container seeded from versioned fixtures, run the suite, and destroy the container. The choice is a speed-versus-fidelity tradeoff, not a right-or-wrong. A seed strategy that assumes serial execution will produce the same shared-state corruption it was designed to prevent.

Why a stateful API simulator is a better host for seed data than a live sandbox or a stateless mock

Seeding realistic data into a live third-party sandbox or a stateless mock wastes the investment. Sandboxes have documented parity gaps and live credential requirements, and stateless mocks cannot preserve the seeded world across sequential calls. A developer testing a payment-reversal flow against these sandboxes is testing against a world that does not behave like production in precisely the ways that matter.

A stateless mock cannot hold seeded state at all. A stateful simulator remembers. Seed a Stripe customer with an open invoice, then call the invoice list endpoint: the simulator returns the seeded invoice because it maintains state across calls, exactly as the real API would. That memory is what makes the sequential call chain testable.

The verification problem is the quiet failure mode. A cohesive starting world covering realistic entity relationships means tests can begin exercising business logic immediately without per-test setup overhead. The seed data and the stateful host are two halves of the same investment. One without the other produces either an empty world or an unverifiable one.

Fault injection as a seeding concern, not an afterthought

A realistic starting state includes not just valid data but controlled failure conditions. Chaos in CI is structured, not random. The point is not to break things randomly but to verify a specific claim about how the system behaves under a specific condition.

The injection patterns relevant to API integration tests are narrow and precise: API timeout simulation, rate-limit responses, service-unavailable responses, network latency injection. The CI pattern that follows is equally disciplined. After a workload meets a hypothesis, such as the retry logic handles a 429 correctly, the experiment is automated to run continually as a regression gate. Fault injection becomes part of the seeded starting state for resilience tests, not a separate activity that happens occasionally when someone has time.

A simulator that supports fault injection at seed time, seeding a world where a specific endpoint returns 503 for the first two calls and then succeeds, lets developers write and run resilience tests without spinning up chaos infrastructure. Fault injection against a live API risks cascading side effects that nobody can predict or clean up. Against a simulator it is deterministic and repeatable, which is what a regression gate requires.

Keeping seeded state honest as APIs evolve: versioning and continuous verification

Seeded state is only as useful as it is current. Treat seed data as a first-class artifact: version it alongside tests, review it in pull requests, and make it possible to roll forward and back. This is the same discipline that CI/CD brought to deployments, applied to the world the tests run in.

Versioning strategies differ by layer. Static reference data is managed as repeatable migration changesets in Liquibase or Flyway, applied during CI, with changes going through the same review process as schema migrations. Factory-generated data needs no separate artifact, because the factory code itself is the versioned artifact. Updating the factory propagates to all tests that use it.

Contract testing is the drift early-warning system. Implementing contract tests against both the live API and the simulator ensures both adhere to the same OpenAPI definition. Automated pipelines should deploy to the simulator first, run contract tests, and only promote if behavior matches. Developers lose trust in the simulator when it drifts, fall back to testing against the live API, reintroduce rate limits and credential risks, and the seeded CI environment degrades into the fragile state the whole approach was designed to escape. A meaningful starting state is an asset only if it is maintained. Without versioning and verification, it is just a new kind of fixture rot.