The pull request had two approvals. CI was green. The tests covered the new path, static analysis was quiet, the security scan was clean and the change request was signed off. By every measure the team had, the change was safe.
It was merged at 11:40 and deployed at 11:52.
Everything that happened next was new information.
Merge is not the end of uncertainty
Engineering teams put serious effort into deciding whether a change is safe before it ships. Review catches design problems. Tests and CI pin down known behaviour. Static analysis and security checks find mistakes people miss. Approvals make someone think about blast radius. All of it is worth doing.
But every one of those checks works from the evidence that exists before the change meets the live system. A green build honestly answers one question: is there anything we can see from here that should stop this?
It cannot answer the next one: how will this behave in production? Production has not been asked yet.
Passing review and CI tells you what could be assessed before production. It cannot tell you what production has not yet said.
Production creates new evidence
Some evidence is not hidden before deployment. It does not exist yet.
Real traffic, real dependencies, production configuration, live data, queues, downstream services and the people using the system only meet the change once it is live. No pre-production check can observe that. It is not a gap in the process; it is where the process necessarily ends.
The problem is what happens next.
In most teams the assessment is made once: the deployment is marked low risk, and the label never updates.
Production evidence lands somewhere else. A latency panel, an error count on another team's service, a queue metric, a message in Teams, a support ticket.
Each is recorded faithfully, and none is attached to the change that might explain it. The chain starts forming while the change is still labelled safe.
Risk is temporal
Operational risk is not a label attached to a pull request. It is an assessment supported by evidence, and it has a time dimension. The right answer to "how risky is this change?" at 11:52 can differ from the right answer at 12:10, because the evidence has changed.
- T+0: the deployment completes. No concerning evidence. The assessment is low.
- T+4 min: latency on the deployed service starts to move. Weak evidence; the assessment rises slightly.
- T+9 min: a downstream service starts returning unusual errors on the changed path.
- T+13 min: queue depth behind it climbs. Now there is a plausible mechanism, not just timing.
- T+16 min: an engineer reports unexpected behaviour in Teams.
- T+21 min: some checkout attempts fail. The assessment is high, with a trail showing how it got there.
On its own, each of those is an event. Latency wobbles. Services throw errors. People say things in chat.
Lined up against the same change and the same path, they become evidence of a developing incident, well before the loudest signal arrives.
Your tools see events; the connection to the change is what turns them into evidence.
The assessment has to move both ways
A risk assessment that can only go up is an alarm with extra steps.
Go back to T+9.
Suppose the downstream errors started ten minutes before the deployment, and an unrelated service is showing the same pattern. The link to the change weakens, so the assessment should fall, and say why: the errors are real, but they now point to a shared dependency rather than this deployment.
Quiet is evidence too.
A deployment that has come through an hour of real traffic, real data and a normal peak, with nothing unusual in errors, queues, conversation or support, has accumulated evidence that did not exist at merge. It is less risky than one that has only passed CI, and the assessment should reflect that.
Lowering confidence is what makes raising it believable. A system that only escalates teaches engineers to ignore it.
A changed assessment has to explain itself
When the view of a deployment moves, that movement is a claim.
Which evidence moved it? Where did that evidence come from? What is it connected to? What would strengthen or weaken the conclusion?
A label flipping from green to amber is barely better than a frozen one. Incident AI has to show its work, and an assessment that changes over time has to show it every time it changes. That also makes the record worth keeping: a timeline says what happened when, while a sequence of evidenced assessments says what was known at each point and why the view changed.
Where Ember fits
Monitoring tells you a metric moved.
Context tells you which change, service and path it relates to.
Continuous reasoning weighs that evidence and updates the assessment.
Ember is being built for the last two, on top of the signals teams already collect.
The idea is straightforward: a change carries its risk assessment into production, and that assessment keeps changing as evidence arrives, in either direction, with the reasoning attached. It does not declare root cause or replace the judgement of the engineers who know the system. It keeps an evidence-backed view current, so that "is that deployment still fine?" reflects what production has said since merge. See how it works.
Whether AI can reliably separate developing signals like these from ordinary noise was one of the questions in our feasibility study with Robert Gordon University. Read the research.
Merge is where the evidence available before production runs out. It is not where the risk does.
A deployment should stay low risk only for as long as the evidence keeps saying so.