The first time you try to write a real test for an AI agent, you hit a wall that doesn't exist for normal software: the system under test is non-deterministic, and so are its dependencies. The model picks different words each run. The vendor API returns different data, rate-limits you, or has an outage during your CI run. You can't assert on exact output, and you can't rely on the upstream being there. Most teams respond by not testing the agent at all.
Pin the dependencies
The move is to make the environment deterministic, even though the agent isn't. Capture real upstream traffic once: every request the agent makes, every response it gets. Then replay that capture in CI. The agent still thinks freely; it just talks to a recording instead of the live world. Now a test failure means the agent changed, not that a third party had a bad afternoon.
Verbatim vs redacted: you need both
There's a real tension here. For replay fidelity you want the capture verbatim — exact bytes, exact ordering. For safety you can't keep customer PII and secrets sitting in a fixture file that lands in your repo. I resolve it with two surfaces:
- An in-memory, verbatim capture for the active session, where fidelity matters and the data never outlives the run.
- A redacted-by-default persisted corpus, scrubbed before it's written, that's safe to keep and share. Verbatim persistence is opt-in and scoped, never the default.
The hard part: generalization
Naive replay matches requests by exact hash. It works until the agent asks something one character different from what you recorded. Then it 404s, and your "test" is just asserting that you recorded the exact script. Real agents don't replay scripts; they improvise around a theme.
So the replay layer has to generalize: given an unseen-but-similar request, synthesize a response with the right shape and status, consistent with what it learned from the captured corpus. The discipline is to do that offline. Learn the generalization ahead of time, so replay stays deterministic and fast when the tests run. Put a live model in the replay hot path and the non-determinism comes straight back, with model latency attached.
I never try to make the agent deterministic. Just the world it runs against.
What it buys you
- Tests that fail for one reason: the agent's behavior changed.
- CI with no live vendor calls: no quota burned, no rate-limit flakes, no prod data in the run.
- A safety story a security review can accept: the corpus is redacted, single-tenant, and there's no model inference at request time.
I went deep enough on this problem that I'm now building a product around it. But the principle stands on its own: to test something non-deterministic, make its environment deterministic. Then be honest about the redaction and generalization trade-offs instead of pretending they don't exist.