Right Answer, Wrong Work

detected 2026-02-28

trigger

"expect(result.status).toBe(400) — test passes, but the 400 comes from a different validation than the test claims to verify"

what it is

A test that asserts the correct outcome via the wrong causal path. The assertion passes, the gate is green, but nobody traces the execution path to check whether the test verifies what it claims to verify. The LLM optimises for the shape of correctness (matching expected output) without verifying which code path produced it.

what it signals

instead

Assert why it failed, not just that it failed. `expect(result.status).toBe(400)` is wrong work. `expect(result.error.code).toBe('INVALID_JSON')` shows the work.

refs

  • SD-190 (governance recursion — plausible-but-wrong tests named)
  • Bugbot finding on PR #386 V-03c (array body passes for wrong reason)

← all patterns