World map

 

BM

World O3: Alarms & Screens

SLO burn-rate alerts

You will learn: to page only when the error budget burns fast, and stop soon after the fix.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Your SLO: 99.9% of orders work this month. At 20m, 5% of orders fail for 4 minutes. Then it heals. Should someone wake up?

Watch

Divide the error ratio by the budget (0.001). That is the burn rate.

Type a PromQL query and press Run. The table shows the answer at the eval time. The graph shows it over time.

Metrics in this data (tap to add):

Time 0 is when the data starts. The query looks back from the eval time.

burn rate: How fast you spend the error budget. 1 means it lasts exactly the SLO window; 14.4 means a month's budget is gone in about two days. 1 means the budget lasts exactly the month. Bigger burns faster.

Predict

The burn rate stays at 14.4 for one hour. How much of the month’s budget is gone?

Try it

Page for the big burn from 61m. Stay quiet for the 4-minute spike. Stop soon after the fix at 91m.

Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

When (expr)
Wait first (for)
Labels
Your rule: rules.yml
groups:
- name: kedai
rules:
- alert: FastBurn
expr: job:shop_error_ratio:rate5m > 0.001
for: 2m
labels:
severity: page
The value the alert watches
00.020.040.060s30m1h1h30m2hthreshold 0.001
Graph as a table
Timejob:shop_error_ratio:rate5m{job="shop"}
0s–
1m0.0005
2m0.0005
3m0.0005
4m0.0005
5m0.0005
6m0.0005
7m0.0005
8m0.0005
9m0.0005
10m0.0005
11m0.0005
12m0.0005
13m0.0005
14m0.0005
15m0.0005
16m0.0005
17m0.0005
18m0.0005
19m0.0005
20m0.011899999999999999
21m0.0219
22m0.0319
23m0.0419
24m0.037625
25m0.025249999999999998
26m0.012874999999999998
27m0.0005
28m0.0005
29m0.0005
30m0.0005
31m0.0005
32m0.0005
33m0.0005
34m0.0005
35m0.0005
36m0.0005
37m0.0005
38m0.0005
39m0.0005
40m0.0005
41m0.0005
42m0.0005
43m0.0005
44m0.0005
45m0.0005
46m0.0005
47m0.0005
48m0.0005
49m0.0005
50m0.0005
51m0.0005
52m0.0005
53m0.0005
54m0.0005
55m0.0005
56m0.0005
57m0.0005
58m0.0005
59m0.0005
1h0.0005
1h1m0.012874999999999998
1h2m0.025249999999999998
1h3m0.037625
1h4m0.049999999999999996
1h5m0.049999999999999996
1h6m0.049999999999999996
1h7m0.049999999999999996
1h8m0.049999999999999996
1h9m0.049999999999999996
1h10m0.049999999999999996
1h11m0.049999999999999996
1h12m0.049999999999999996
1h13m0.049999999999999996
1h14m0.049999999999999996
1h15m0.049999999999999996
1h16m0.049999999999999996
1h17m0.049999999999999996
1h18m0.049999999999999996
1h19m0.049999999999999996
1h20m0.049999999999999996
1h21m0.049999999999999996
1h22m0.049999999999999996
1h23m0.049999999999999996
1h24m0.049999999999999996
1h25m0.049999999999999996
1h26m0.049999999999999996
1h27m0.049999999999999996
1h28m0.049999999999999996
1h29m0.049999999999999996
1h30m0.049999999999999996
1h31m0.037625
1h32m0.025249999999999998
1h33m0.012874999999999998
1h34m0.0005
1h35m0.0005
1h36m0.0005
1h37m0.0005
1h38m0.0005
1h39m0.0005
1h40m0.0005
1h41m0.0005
1h42m0.0005
1h43m0.0005
1h44m0.0005
1h45m0.0005
1h46m0.0005
1h47m0.0005
1h48m0.0005
1h49m0.0005
1h50m0.0005
1h51m0.0005
1h52m0.0005
1h53m0.0005
1h54m0.0005
1h55m0.0005
1h56m0.0005
1h57m0.0005
1h58m0.0005
1h59m0.0005
2h0.0005

What the alert does over time

Grey: quiet. Amber: pending (true, but waiting). Red: firing.

  • {job="shop", severity="page"}

    ⏳ pending 20m to 21m · 🔔 firing 22m to 26m · ⏳ pending 1h1m to 1h2m · 🔔 firing 1h3m to 1h33m

At 1h20m:

  • FIRING FastBurn job="shop", severity="page"
Timeline as a table
Time{job="shop", severity="page"}
0squiet
1mquiet
2mquiet
3mquiet
4mquiet
5mquiet
6mquiet
7mquiet
8mquiet
9mquiet
10mquiet
11mquiet
12mquiet
13mquiet
14mquiet
15mquiet
16mquiet
17mquiet
18mquiet
19mquiet
20m⏳ pending
21m⏳ pending
22m🔔 firing
23m🔔 firing
24m🔔 firing
25m🔔 firing
26m🔔 firing
27mquiet
28mquiet
29mquiet
30mquiet
31mquiet
32mquiet
33mquiet
34mquiet
35mquiet
36mquiet
37mquiet
38mquiet
39mquiet
40mquiet
41mquiet
42mquiet
43mquiet
44mquiet
45mquiet
46mquiet
47mquiet
48mquiet
49mquiet
50mquiet
51mquiet
52mquiet
53mquiet
54mquiet
55mquiet
56mquiet
57mquiet
58mquiet
59mquiet
1hquiet
1h1m⏳ pending
1h2m⏳ pending
1h3m🔔 firing
1h4m🔔 firing
1h5m🔔 firing
1h6m🔔 firing
1h7m🔔 firing
1h8m🔔 firing
1h9m🔔 firing
1h10m🔔 firing
1h11m🔔 firing
1h12m🔔 firing
1h13m🔔 firing
1h14m🔔 firing
1h15m🔔 firing
1h16m🔔 firing
1h17m🔔 firing
1h18m🔔 firing
1h19m🔔 firing
1h20m🔔 firing
1h21m🔔 firing
1h22m🔔 firing
1h23m🔔 firing
1h24m🔔 firing
1h25m🔔 firing
1h26m🔔 firing
1h27m🔔 firing
1h28m🔔 firing
1h29m🔔 firing
1h30m🔔 firing
1h31m🔔 firing
1h32m🔔 firing
1h33m🔔 firing
1h34mquiet
1h35mquiet
1h36mquiet
1h37mquiet
1h38mquiet
1h39mquiet
1h40mquiet
1h41mquiet
1h42mquiet
1h43mquiet
1h44mquiet
1h45mquiet
1h46mquiet
1h47mquiet
1h48mquiet
1h49mquiet
1h50mquiet
1h51mquiet
1h52mquiet
1h53mquiet
1h54mquiet
1h55mquiet
1h56mquiet
1h57mquiet
1h58mquiet
1h59mquiet
2hquiet

Check

Drag a card into a box. Or tap the card, then tap the box.

      Fast burns wake someone. Slow burns wait for the morning.

      You did it!

      You can now write burn-rate alerts with a long and a short window.

      Go deeper (optional)

      These numbers come from Google’s SRE workbook, for a 99.9% SLO over 30 days. Save each window’s error ratio with a recording rule, like job:shop_error_ratio:rate1h. Then the alert stays short and fast.

      Real mission

      2. docker run --rm --entrypoint promtool -v "$PWD:/w" -w /w quay.io/prometheus/prometheus:v3.13.5 check rules rules.yml

      0 of 2 steps done · Your ticks are saved.