A test that shares your assumption cannot catch it
My webhook parser and my event simulator were written from the same wrong reading of Ring's payload, so a thousand replayed events were accepted and every test passed. The fix was not more tests but a source of truth neither of them wrote: Ring's documented examples, pasted in verbatim.
Jirani reacts to Ring webhooks: a door opens, a contact sensor changes, a doorbell sees motion. The Ring Developer Playground gives you a token and an API explorer, but it has no sensors, keeps no event history and delivers no webhooks. So before Jirani could react to anything, I wrote ring-routine-sim, a generator that produces weeks of one household's events and replays them signed the way Ring signs webhooks.
That worked well for a few days. It was also wrong in a way no test of mine could see.
Two pieces of code, one belief
The webhook parser in Jirani and the payload builder in the simulator were written from the same reading of an older event shape. The parser expected the fields where the simulator put them. Every replay was accepted, every signature verified, every test passed, and a thousand events flowed through the whole pipeline on AWS.
None of that told me anything about Ring. The tests were checking that two pieces of my own code agreed with each other, and they did, because one belief had produced both of them.
When I ran Jirani's client against the real Ring API and reread the webhook documentation, the shape was different. Ring's v1.1 payload puts the event name in data.type, the device in data.attributes.source with a source_type beside it, the timestamp in epoch milliseconds, and the account in meta.account_id. Contact sensors report contact_sensor_faulted when a door opens and contact_sensor_cleared when it closes. The parts I had right, like the X-Signature: sha256=<hex> header, were right because they were written down plainly. The parts I had wrong were the parts I had inferred.
The fix was a source of truth, not more tests
More tests generated by the simulator would have passed just as happily. What the suite needed was something neither my parser nor my simulator had written.
So the parser tests now include Ring's documented example payloads, pasted in verbatim, not retyped and not built by a helper. If Ring's documentation and my parser disagree, a test fails, and it fails for the right reason. The simulator was fixed in the same change; its first npm release carried the fix, and Jirani now pins that release instead of keeping a copy.
There was one more lesson waiting in production. The deployed stack was still running the old parser, so until it was redeployed, events in the correct shape were answered with "unparseable payload", politely, with a 202. Nothing crashed; it would simply have accepted nothing. After the redeploy, 1,024 signed events in the documented shape went through with none rejected, and the README's claim was re-measured instead of carried over.
Where else this hides
Once I saw it, I found the same pattern in other places a project likes to grade itself:
- A mock written by the same person as the client it mocks.
- Fixtures produced by the serializer they are meant to test.
- A schema and its validator generated from one hand-written type.
Each is fine as a convenience, and none of them is evidence about the outside world. Ring also marks its sensor events as Early Access, so the names may still change, and I have written that down instead of pretending the tests settle it.
If the same belief wrote the code and the test, the test is measuring the belief.