The problem
Teams putting AI agents into regulated, data-sensitive environments hit the same wall: you can't test an agent reliably against a live upstream. The vendor API is slow, rate-limited, and costs money on every call. Worse, sending real customer data through it in CI is a compliance problem all by itself. So agent test suites either don't exist, or they're flaky integration tests that fall over whenever the vendor does.
The naive fix is record-and-replay. It gets you part way, then fails on the first request the agent makes that wasn't recorded verbatim. Real agents don't replay a fixed script; they ask slightly different questions every run.
What I built
A self-hosted proxy that sits in front of the agent's upstreams inside the customer's own tenant. It does three things, and the order matters:
- Captures real upstream traffic transparently while the agent runs against production once.
- Redacts sensitive fields by default before anything is written to disk. The saved corpus is safe to keep and safe to train on.
- Replays that corpus as a deterministic mock, with a learned layer that generalizes to unseen-but-similar requests instead of 404-ing on anything it didn't record byte-for-byte.
The defining constraint: no LLM in the hot path, ever. Replay is deterministic and fast. The learned generalization is trained offline from the captured corpus and never invoked live, so latency and behavior stay predictable under a security review.
Why those choices
Single-tenant self-host because the buyers are regulated. "Your data never leaves your tenant" is a stronger security story than "we encrypt your data," and the architecture is what makes it true.
Redact before persist, because a redact-on-read design leaves a window where raw sensitive data sits on disk. Moving the floor to write-time means the artifact on disk never held raw sensitive data in the first place.
Generalization learned offline. Exact-hash replay is brittle, but putting a live LLM in the loop brings back the latency and non-determinism you were trying to remove. Training on the captured corpus keeps replay deterministic while still answering requests the agent never recorded.
Most AI projects die in the gap between the demo and production. Gostly is the tooling I wish existed for closing that gap on the testing side.
The stack
Solo: I designed the architecture, made the build/buy/cut calls, and wrote the whole thing. Proxy, redaction layer, capture/replay engine, offline training path, deployment story. The interesting work was the trade-offs, not the framework choices.