World O3: Alarms & Screens

Alert fatigue

You will learn: to cut noisy alerts so the real bell still gets heard.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Tap each point on the timeline.

Last night the pager buzzed again and again. Tap each buzz. How many needed a human?

Watch

Drag a card into a box. Or tap the card, then tap the box.

Too many bells, and people stop hearing them. That is .

Try it

The queue bounces around 25. The bell flaps on, off, on. Make it ring once, in one stretch.

Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

Your rule: rules.yml

1groups:
2- name: kedai
3rules:
4- alert: QueueLong
5expr: kedai_queue_length > 25

The value the alert watches

Graph as a table

Graph as a table
Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	10
1m	12
2m	30
3m	9
4m	31
5m	11
6m	29
7m	33
8m	8
9m	35
10m	36
11m	38
12m	40
13m	42
14m	41
15m	39
16m	12
17m	10
18m	9
19m	8
20m	7
21m	30
22m	9
23m	8
24m	7
25m	6
26m	5
27m	5
28m	5
29m	5
30m	5

What the alert does over time

Grey: quiet. Amber: pending (true, but waiting). Red: firing.

{stall="nasi-lemak"}
🔔 firing at 2m · 🔔 firing at 4m · 🔔 firing 6m to 7m · 🔔 firing 9m to 15m · 🔔 firing at 21m

Look at time: 12m

At 12m:

FIRING QueueLong stall="nasi-lemak"

Timeline as a table

Time	`{stall="nasi-lemak"}`
0s	quiet
1m	quiet
2m	🔔 firing
3m	quiet
4m	🔔 firing
5m	quiet
6m	🔔 firing
7m	🔔 firing
8m	quiet
9m	🔔 firing
10m	🔔 firing
11m	🔔 firing
12m	🔔 firing
13m	🔔 firing
14m	🔔 firing
15m	🔔 firing
16m	quiet
17m	quiet
18m	quiet
19m	quiet
20m	quiet
21m	🔔 firing
22m	quiet
23m	quiet
24m	quiet
25m	quiet
26m	quiet
27m	quiet
28m	quiet
29m	quiet
30m	quiet

Hint

for ignores short bounces. keep_firing_for stops the bell from blinking off too soon.

Predict

An alert fired 40 times last month. Each time, nobody did anything. What now?

Check

This week's alert report. Tap every alert that needs tuning.

0 of 3 found

Look for bells that ring often and change nothing.

You did it!

You can now cut alert noise so the real bell gets heard.

Go deeper (optional)

Where the story breaks. A real control room has one loud bell. Ours has many quiet ones, with labels, routes, groups and silences. And a real bell never learns. Your alerts should: review them often, and delete freely.

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	10
1m	12
2m	30
3m	9
4m	31
5m	11
6m	29
7m	33
8m	8
9m	35
10m	36
11m	38
12m	40
13m	42
14m	41
15m	39
16m	12
17m	10
18m	9
19m	8
20m	7
21m	30
22m	9
23m	8
24m	7
25m	6
26m	5
27m	5
28m	5
29m	5
30m	5

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	10
1m	12
2m	30
3m	9
4m	31
5m	11
6m	29
7m	33
8m	8
9m	35
10m	36
11m	38
12m	40
13m	42
14m	41
15m	39
16m	12
17m	10
18m	9
19m	8
20m	7
21m	30
22m	9
23m	8
24m	7
25m	6
26m	5
27m	5
28m	5
29m	5
30m	5

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	10
1m	12
2m	30
3m	9
4m	31
5m	11
6m	29
7m	33
8m	8
9m	35
10m	36
11m	38
12m	40
13m	42
14m	41
15m	39
16m	12
17m	10
18m	9
19m	8
20m	7
21m	30
22m	9
23m	8
24m	7
25m	6
26m	5
27m	5
28m	5
29m	5
30m	5