Your deploy "succeeded" but users were still broken
The pipeline went green, the dashboards sat flat, the error tracker stayed quiet. Meanwhile some checkout traffic spent about forty minutes writing to a feature flag that had already been rolled back. Orders looked accepted. None of them were actually placed.
Nothing errored. That was the problem.
If you’ve shipped something that passed every check and still didn’t do its job, you know this one. “Succeeded” means it shipped. Whether it worked is a different question, and we usually let the first answer stand in for the second. Hard not to: the tools are built to shout when something breaks and say nothing otherwise, so quiet starts to feel like a yes.
That gap isn’t a fluke. It’s built into what each signal can tell you, and you can close it.
Green tells you it shipped, not that it worked
Line up the signals you get after a deploy and ask what each one actually proves.
A merged PR: the code changed. Passing tests: the cases you thought of still hold. A succeeded from the pipeline: the artifact rolled out. A quiet error tracker: nothing threw an exception you already knew to catch. Not one of them says whether the thing you meant to ship actually happened, in real traffic, for real users.
That’s not a flaw. It’s the job. Monitoring catches defined failure: an exception, a latency line crossed, a health check going red. It fires on known-bad. But “the order got accepted and never placed” isn’t known-bad. Nothing crossed a line. The request returned 200. So no alert reads as success, when it really just means no noise of that one kind.
The deploys that hurt most are the ones that don’t trip anything. They sit between “nothing broke loudly” and “the thing I wanted to happen happened.”
So how do you actually check it
Not with another dashboard. Name the outcome the change was meant to produce, then watch that one thing in the minutes right after the deploy.
Decide what “worked” looks like before you ship. Something you can watch move, not “no errors.” Shipped a checkout change? Maybe it’s orders hitting placed at about the rate you’d expect this hour. A cache change? Maybe hit ratio staying above some line. Write it down while you still remember what you were after.
Size the window by traffic, not the clock. Sixty seconds might be four requests, which tells you nothing. You want enough real traffic to actually hit the path you changed. Two minutes on a busy checkout, twenty on a sleepy internal service.
Compare against normal, not zero. “Zero errors” is the trap. A placement rate at 90% of usual, with a clean error tracker, is a broken deploy that every error check waves through.
And if a path hasn’t seen traffic yet, say so. It’s unknown, not passing. They need different labels, because the moment someone reads “verified” they stop watching.
Then write the verdict where the next person will find it: worked, still failing, or not enough traffic to tell yet. So when the same alert fires again an hour later, it shows up with the original context instead of a closed ticket insisting everything is fine.
The point
succeeded and working are different claims, and the whole toolchain is tuned for the first. That’s fine, right up until you read it as the second. The evidence for “it worked” lives in the minutes after a deploy, and usually the thing collecting it is whoever left a dashboard open.
Name the outcome. Watch that one signal, sized by traffic, next to normal. That turns “nothing errored” back into a question you’re asking, instead of an answer you assumed.
This is the corner Relivio works in. It takes the raw signals your services already emit and returns one verdict on each deploy, watching just the first fifteen minutes, so “is this deploy okay” gets an answer instead of an assumption. relivio.dev