Here's the litmus:
Would this test still pass against a broken implementation?
If the answer is yes, the test is theater. It's asserting that code ran, not that it did the right thing.
The tells
A test is checking an output (worthless) instead of an outcome (real) when its assertions look like:
- "returned
200" / "didn't throw" / "compiled" / "logged OK" - "the function was called" with no check on what it did
- asserting on the shape of a response while ignoring whether the value is correct
All of those pass against an implementation that's quietly wrong. They give you a green check and zero protection.
What an outcome test asserts
- A state change actually happened: the row is in the database, the balance updated.
- The value itself is right.
- A behavior holds under the case that matters — the edge, the failure path a naive version gets wrong.
This bites hardest with agents, where it's tempting to assert "the agent responded" and call it covered. It responded, to a broken prompt, with a wrong answer, and your suite is green. Assert the thing that was supposed to change, or you've tested nothing.
End a test-writing session with one adversarial pass: for each test, ask "how would I break the code and still pass this?" If you can answer easily, rewrite the assertion.