World O3: Alarms & Screens

Runbooks

You will learn: to give every alert a fix-it card that a sleepy person can follow.

Checked with Prometheus 3 on 1 Nov 2026

The problem

3 a.m. Your first night on call. The bell says QueueTooLong. What do you wish you had?

Watch

Top of the runbook first. Put these in order. Use the arrows, or drag.

Still bad after 15 minutes? Call the next person
Make it safe: open a second counter
Check first: which stall, since when
What this alert means

A is the fix-it card next to the bell. Put its parts in order.

Try it

Link the runbook from the rule. Add a runbook_url line under annotations.

Change the YAML. The box below shows how a computer reads it.

rules.yml11 lines

How a computer reads it

groupslist · 1 items
- item 1map · 2 keys
  - namekedaitext
  - ruleslist · 1 items
    - item 1map · 5 keys
      - alertQueueTooLongtext
      - exprkedai_queue_length > 50text
      - for3mtext
      - labelsmap · 1 keys
        severitytickettext
      - annotationsmap · 1 keys
        summaryQueue at {{ $labels.stall }} is {{ $value }}text

Hint

runbook_url lines up with summary: ten spaces in.

Check

Pick the message on-call reads. Use the slider to see it at 15m.

Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

Your rule: rules.yml

1groups:
2- name: kedai
3rules:
4- alert: QueueTooLong
5expr: kedai_queue_length > 50
6for: 3m
7labels:
8severity: ticket
9annotations:
10summary: Queue alert

The value the alert watches

Graph as a table

Graph as a table
Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	5
1m	8
2m	12
3m	10
4m	9
5m	14
6m	20
7m	35
8m	55
9m	60
10m	64
11m	70
12m	72
13m	75
14m	78
15m	80
16m	82
17m	85
18m	86
19m	88
20m	90

What the alert does over time

Grey: quiet. Amber: pending (true, but waiting). Red: firing.

{severity="ticket", stall="nasi-lemak"}
⏳ pending 8m to 10m · 🔔 firing 11m to 20m

Look at time: 15m

At 15m:

FIRING QueueTooLong severity="ticket", stall="nasi-lemak"Queue alert

Timeline as a table

Time	`{severity="ticket", stall="nasi-lemak"}`
0s	quiet
1m	quiet
2m	quiet
3m	quiet
4m	quiet
5m	quiet
6m	quiet
7m	quiet
8m	⏳ pending
9m	⏳ pending
10m	⏳ pending
11m	🔔 firing
12m	🔔 firing
13m	🔔 firing
14m	🔔 firing
15m	🔔 firing
16m	🔔 firing
17m	🔔 firing
18m	🔔 firing
19m	🔔 firing
20m	🔔 firing

Check

Swipe right for a good runbook step. Swipe left for one that needs work.

Swipe right for good, left for needs work. Or use the buttons.

1 / 4

Open the Queue panel and find the stall named in the alert.

You did it!

You can now give every alert a runbook a sleepy person can follow.

Go deeper (optional)

Keep runbooks in Git, next to the rules. After each incident, fix the step that confused you. No runbook yet? The alert may not be ready to page anyone.

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	5
1m	8
2m	12
3m	10
4m	9
5m	14
6m	20
7m	35
8m	55
9m	60
10m	64
11m	70
12m	72
13m	75
14m	78
15m	80
16m	82
17m	85
18m	86
19m	88
20m	90

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	5
1m	8
2m	12
3m	10
4m	9
5m	14
6m	20
7m	35
8m	55
9m	60
10m	64
11m	70
12m	72
13m	75
14m	78
15m	80
16m	82
17m	85
18m	86
19m	88
20m	90

Time	`kedai_queue_length{stall="nasi-lemak"}`
0s	5
1m	8
2m	12
3m	10
4m	9
5m	14
6m	20
7m	35
8m	55
9m	60
10m	64
11m	70
12m	72
13m	75
14m	78
15m	80
16m	82
17m	85
18m	86
19m	88
20m	90