Investigation Playbooks
Error Rate Rising
Use this playbook when failures increase and you need to move from grouped symptoms to concrete traces, logs, owners, and next actions.
Situation and outcome
Use this playbook when failures increase and you need to move from grouped symptoms to concrete traces, logs, owners, and next actions.
The Errors page shows a rising group or a new high-impact failure. A service card, issue, or alert points to elevated error rate. Users report intermittent failures but the owning service is not obvious.
By the end of this playbook, you should have the active error group, affected owner path, latest request example, and enough trace or log evidence to decide whether to mitigate, roll back, or keep investigating.
Evidence to collect
- Current group
- Use latest seen time and affected service to separate active failures from historical groups that no longer need response.
- Representative trace
- Open a recent failing example and read from the root span toward the first failed span instead of starting from the final exception.
- Event sequence
- Use logs around the same timestamp to understand retries, validation failures, downstream responses, or user-visible messages.
Investigation path
Follow the path in order until the evidence points to a service, dependency, release, runtime signal, or setup gap.
- 1
Open Errors and choose the highest-impact active group.
- 2
Confirm affected service, route, environment, first seen time, and latest example.
- 3
Open a representative trace example and read the request path from the root span outward.
- 4
Open related logs around the same trace or timestamp to reconstruct what happened.
- 5
Check service dependencies when the failed span is outbound or downstream.
- 6
Compare Releases if the first seen time lines up with a version or deployment change.
Decision point
If latest examples are stale, lower priority and look for a current active group. If the same error appears across services, inspect the shared dependency or caller path. If the error is recurring, create or tune an alert after the investigation.
Choose the next action only after the evidence explains both impact and ownership. If the evidence is stale, mismatched, or filtered too narrowly, widen the investigation before assigning work.
Handoff
A useful handoff gives the next responder enough context to reproduce the evidence without replaying the whole investigation.
Evidence 1
Share the group name or error message, affected service, endpoint, environment, and latest example time.
Evidence 2
Link one or two failing traces that show the same failure shape.
Evidence 3
Call out whether the first failed span is inside the service, in a dependency, or still unclear.
Evidence 4
Mention release timing if first seen or latest seen lines up with a deployment.