when my test suite showed up in my analytics

The day I shipped a first-party collector for CSP violations, its owner panel showed its first entry: script-src-elem · https://evil.example. It reads exactly like an injection attempt against production. It was my own test suite. One test deliberately POSTs a valid-looking violation report at the public collector to prove the endpoint never leaks anything, and the fixture’s blocked URL was evil.example.

The mechanism: in CI the pipeline runs with zero secrets, the store is off, and folding a report is a no-op. But locally the test runner boots the real production server, which loads my env files — and my machine has the real storage credentials, because it has to. Every local test pass, including the gate before every pull request, quietly folded fixture data into live telemetry. Analytics had the same hole: the test runner’s user-agent isn’t on any crawler deny-list, so each run counted as a visitor.

The fix is one block of configuration: the test server pins the store credentials to empty strings, beating the env files, so a local run is exactly as secretless as CI. Forced, not assumed. “The pipeline must pass with zero secrets” had an unwritten mirror clause: it must also run with zero secrets, even on a machine that has them.

Also, distinctive fixture values are a gift. evil.example confessed on sight; a realistic fixture would still be sitting in my counts, lying.