World map

 

BM

World O3: Alarms & Screens

Alert rules: for, labels, annotations

You will learn: to write an alert rule that rings only for real trouble.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Press Run. Look at shop-1 on the graph.

Metrics in this data (tap to add):

Time 0 is when the data starts. The query looks back from the eval time.

At 11m, shop-1 stopped answering. Nobody was watching the screen. How can the control room ring a bell by itself?

Watch

Tap the line that says how long to wait.

0 of 1 found

An alert rule: A PromQL question Prometheus asks again and again. When the answer has rows, the control room's bell gets ready. is a question Prometheus asks every minute. Any rows in the answer? The bell gets ready.

Predict

db-1 was down for only 3 minutes. The rule waits for: 5m. What happens?

Try it

Build the rule. Page someone when a target is really down. Not for a short blip.

Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

When (expr)
Wait first (for)
Labels
Message (annotations)
Your rule: rules.yml
groups:
- name: kedai
rules:
- alert: InstanceDown
expr: up == 0
annotations:
summary: Something is down
The value the alert watches
00.250.50.7510s10m20m30mthreshold 0
Graph as a table
Timeup{instance="db-1", job="db"}up{instance="shop-1", job="shop"}up{instance="shop-2", job="shop"}
0s111
1m111
2m111
3m111
4m111
5m111
6m011
7m011
8m011
9m111
10m111
11m101
12m101
13m101
14m101
15m101
16m101
17m101
18m101
19m101
20m101
21m101
22m101
23m101
24m101
25m101
26m101
27m101
28m101
29m101
30m101
31m101

What the alert does over time

Grey: quiet. Amber: pending (true, but waiting). Red: firing.

  • {instance="db-1", job="db"}

    🔔 firing 6m to 8m

  • {instance="shop-1", job="shop"}

    🔔 firing 11m to 31m

At 12m:

  • FIRING InstanceDown instance="shop-1", job="shop"Something is down
Timeline as a table
Time{instance="db-1", job="db"}{instance="shop-1", job="shop"}
0squietquiet
1mquietquiet
2mquietquiet
3mquietquiet
4mquietquiet
5mquietquiet
6m🔔 firingquiet
7m🔔 firingquiet
8m🔔 firingquiet
9mquietquiet
10mquietquiet
11mquiet🔔 firing
12mquiet🔔 firing
13mquiet🔔 firing
14mquiet🔔 firing
15mquiet🔔 firing
16mquiet🔔 firing
17mquiet🔔 firing
18mquiet🔔 firing
19mquiet🔔 firing
20mquiet🔔 firing
21mquiet🔔 firing
22mquiet🔔 firing
23mquiet🔔 firing
24mquiet🔔 firing
25mquiet🔔 firing
26mquiet🔔 firing
27mquiet🔔 firing
28mquiet🔔 firing
29mquiet🔔 firing
30mquiet🔔 firing
31mquiet🔔 firing

Check

Drag a card into a box. Or tap the card, then tap the box.

      Labels steer the bell to the right person. Annotations are the note on it.

      Check

      bash
      promtool check rules broken.yml
      Checking broken.yml
      FAILED:
      broken.yml: 5:15: group "kedai", rule 1, "InstanceDown": could not parse expression: 1:4: parse error: unexpected "="
      promtool check rules rules.yml
      Checking rules.yml
      SUCCESS: 1 rules found

      promtool check rules tests the file before Prometheus loads it. The red line means: = is not a check. PromQL compares with ==.

      You did it!

      You can now write an alert rule that waits, then rings the right person.

      Go deeper (optional)

      keep_firing_for: 5m keeps the bell ringing a little after the problem ends. It stops on-off-on noise. In a message, {{ $labels.instance }} is the target and {{ $value }} is the number.

      Real rep: run promtool from the official image. Your own laptop is fine.

      Real mission

      2. docker run --rm --entrypoint promtool -v "$PWD:/w" -w /w quay.io/prometheus/prometheus:v3.13.5 check rules rules.yml

      0 of 3 steps done · Your ticks are saved.