World map

 

BM

World O3: Alarms & Screens

Alert on symptoms, not causes

You will learn: to page people for what customers feel, not for every busy machine.

Checked with Prometheus 3 on 1 Nov 2026

The problem

3 a.m. The pager rings: CPU at 95%. You get up and check. Every order works fine. Was it worth waking you?

Watch

Drag a card into a box. Or tap the card, then tap the box.

      A symptom: What the customer feels: failed orders, slow pages. Alert on symptoms; look at causes (CPU, disk) on the screens. is what the customer feels. A cause is one reason it might happen.

      Try it

      The CPU is busy from 5m to 13m. Orders fail from 18m. Pick the rule that wakes someone only for the second.

      Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

      When (expr)
      Wait first (for)
      Labels
      Message (annotations)
      Your rule: rules.yml
      groups:
- name: kedai
rules:
- alert: KedaiPain
expr: kedai:cpu_busy:ratio > 0.9
labels:
severity: page
annotations:
summary: "{{ $value | humanizePercentage }} of orders fail"
      The value the alert watches
      00.250.50.7510s10m20m30mthreshold 0.9
      Graph as a table
      Timekedai:cpu_busy:ratio{instance="kedai-1"}
      0s0.3
      1m0.3
      2m0.3
      3m0.3
      4m0.3
      5m0.95
      6m0.95
      7m0.95
      8m0.95
      9m0.95
      10m0.95
      11m0.95
      12m0.95
      13m0.95
      14m0.35
      15m0.35
      16m0.35
      17m0.35
      18m0.35
      19m0.35
      20m0.35
      21m0.35
      22m0.35
      23m0.35
      24m0.35
      25m0.35
      26m0.35
      27m0.35
      28m0.35
      29m0.35
      30m0.35

      What the alert does over time

      Grey: quiet. Amber: pending (true, but waiting). Red: firing.

      • {instance="kedai-1", severity="page"}

        🔔 firing 5m to 13m

      At 25m:

      Nothing is pending or firing.

      Timeline as a table
      Time{instance="kedai-1", severity="page"}
      0squiet
      1mquiet
      2mquiet
      3mquiet
      4mquiet
      5m🔔 firing
      6m🔔 firing
      7m🔔 firing
      8m🔔 firing
      9m🔔 firing
      10m🔔 firing
      11m🔔 firing
      12m🔔 firing
      13m🔔 firing
      14mquiet
      15mquiet
      16mquiet
      17mquiet
      18mquiet
      19mquiet
      20mquiet
      21mquiet
      22mquiet
      23mquiet
      24mquiet
      25mquiet
      26mquiet
      27mquiet
      28mquiet
      29mquiet
      30mquiet

      Predict

      The disk will be full in 4 hours. Customers feel nothing yet. What fits best?

      Check

      Swipe right if it should page someone. Swipe left if not.

      Swipe right for good, left for needs work. Or use the buttons.

      1 / 4

      More than 5% of checkouts fail for 2 minutes

      You did it!

      You can now page people for customer pain, not for busy machines.

      Go deeper (optional)

      Causes still matter. Put CPU, memory and disk on the screens, and in the runbook. Then on-call can find the reason fast. Prometheus’s own advice: few alerts, each tied to pain a user feels.