Principles
Every Alert Spends Someone’s Attention
Alert fatigue is an accounting problem. Make every page name the impact, owner, first action, and recovery condition before it can draw from the team’s attention budget.
· Vight team · 6 min read
Alerting principles
Page for action, not anxiety.
An alert is an interrupt sent to a human. It arrives with a real cost: broken focus during the day, lost sleep at night, and less trust in every notification that follows.
Teams often tune alerting as if the scarce resource were evaluation capacity. It is not. The scarce resource is credible human attention, especially when several systems compete for it during an incident.
That changes the design question from “can we detect this?” to “what should someone do now that we detected it?”
01
Require four fields before a rule can page
A paging rule should name the customer or system impact, the owner, the first investigation step, and the recovery condition. If any field is unknown, the signal may still deserve a dashboard, ticket, or asynchronous notification—but it is not ready to wake someone.
These fields also expose weak rules early. An owner cannot respond to “CPU is high” without knowing which service is affected or whether requests are failing. A route-level error-budget burn, by contrast, already points toward impact and scope.
Impact
State which user journey or operational promise is at risk.
First action
Link to the scoped evidence that should be checked first.
Exit
Define when the condition has recovered and the incident can close.
02
Protect against mathematically loud nonsense
Percentages become unstable at low traffic. One failed request out of two is a 50% error rate, but it may not justify the same response as 5,000 failures in a minute. Combine rate thresholds with a minimum event count or use an SLO burn calculation that accounts for the budget being consumed.
No data needs its own meaning too. It might indicate zero traffic, broken instrumentation, an evaluator failure, or a dead service. Treating all four as healthy—or all four as paging failures—produces predictable mistakes.
03
Design firing and recovery together
A threshold that opens an incident without a stable recovery path creates flapping. Use an evaluation window long enough to reject momentary noise, then require clean evidence before resolving. The right delay depends on how quickly the underlying signal can change and how costly a false recovery would be.
Silences and maintenance windows should expire automatically and preserve who muted what and why. A mute without an owner or end time is not tuning; it is an alert being quietly deleted.
04
Review attention as an operational metric
For each rule, track notifications, acknowledged incidents, time to action, false positives, flaps, and pages that produced no change. Review the most expensive rules after incidents and on a regular cadence.
The goal is not zero alerts. It is a set of interruptions whose recipients still believe that opening the notification is worth doing immediately.
If nobody can name the impact, owner, first action, and exit condition, the system has detected a fact—not created an alert.