Investigation Playbooks
Alert Fired
Use this playbook when an alert fires and you need to confirm impact, collect evidence, and choose the right response.
Situation and outcome
Use this playbook when an alert fires and you need to confirm impact, collect evidence, and choose the right response.
An alert changed to firing or recovered after a firing period. A rule points to a service, route, SLO, or threshold but needs evidence. You need to decide whether to acknowledge, mute, tune, or escalate.
By the end of this playbook, you should know whether the alert represents active customer impact, recovered impact, or noise, and what evidence justifies the response decision.
Evidence to collect
- Rule context
- Record the rule name, target, threshold, evaluation window, current state, and any mute or recovery context.
- Impact check
- Open the affected service or route and confirm whether latency, errors, or availability still look unhealthy.
- Timeline
- Line up alert state changes with traces, logs, releases, and issues before tuning the rule.
Investigation path
Follow the path in order until the evidence points to a service, dependency, release, runtime signal, or setup gap.
- 1
Open Alerts and confirm the rule, target, state, evaluation window, and current time range.
- 2
Open the related service or route and confirm whether users are still affected.
- 3
Use Traces for slow or failed request examples behind the alert.
- 4
Use Logs for timestamped events around the threshold crossing.
- 5
Check Releases if the alert started near a deployment.
- 6
After response, decide whether the rule threshold, target, or mute state needs adjustment.
Decision point
If the alert recovered and evidence is clean, close the response loop with context. If the alert is noisy, tune it only after confirming the underlying service is healthy. If the alert matches a recurring failure, connect it to an issue or owner workflow.
Choose the next action only after the evidence explains both impact and ownership. If the evidence is stale, mismatched, or filtered too narrowly, widen the investigation before assigning work.
Handoff
A useful handoff gives the next responder enough context to reproduce the evidence without replaying the whole investigation.
Evidence 1
State whether the alert is firing, recovered, muted, or suspected noisy.
Evidence 2
Link the service, trace, log, issue, or release evidence used for the decision.
Evidence 3
Document the response action: acknowledge, escalate, mute, tune, or close with context.
Evidence 4
If the rule needs tuning, note the evidence that made it noisy or too broad.