World map

 

BM

World O3: Alarms & Screens

Alertmanager: route, group, silence

You will learn: how Alertmanager groups bells, picks who to call and keeps quiet when told.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Ten shop servers lost power at once. Ten alert rules fire. What should your phone get?

Watch

First step at the top. Put these in order. Use the arrows, or drag.

  • Alertmanager: bells with the same labels join one group
  • Prometheus: the rule fires
  • The receiver gets one message
  • Alertmanager: the route picks a receiver

Prometheus only rings bells. Alertmanager: The control room's dispatcher. It takes bells from Prometheus, groups them, picks who to call and stays quiet when told. is the dispatcher. It decides who hears them, and how.

Watch

bash
amtool config routes show --config.file=alertmanager.yml
Routing tree:
.
└── default-route receiver: team-chat
└── {severity="page"} receiver: on-call-phone
amtool config routes test --config.file=alertmanager.yml severity=page alertname=InstanceDown
on-call-phone

Every alert starts at the top of the route: A rule in Alertmanager that sends an alert to a receiver, by its labels. The first match wins. tree. The first child that matches its labels wins.

Predict

An alert has severity=ticket and alertname=QueueTooLong. Who gets it?

Try it

Group the bells by alertname and job. Then ten shops make one call.

Change the YAML. The box below shows how a computer reads it.

alertmanager.yml10 lines

How a computer reads it

  • routemap · 3 keys
    • receiverteam-chattext
    • group_bylist · 1 items
      • item 1alertnametext
    • routeslist · 1 items
      • item 1map · 2 keys
        • matcherslist · 1 items
          • item 1severity="page"text
        • receiveron-call-phonetext
  • receiverslist · 2 items
    • item 1map · 1 keys
      • nameteam-chattext
    • item 2map · 1 keys
      • nameon-call-phonetext

Check

bash
amtool --alertmanager.url=http://localhost:9093 silence add alertname=QueueTooLong stall=nasi-lemak --duration=2h --author=aina --comment="Fixing the queue, back by 3pm"
16587a75-aee0-4b83-b053-8781c8afcb22
amtool --alertmanager.url=http://localhost:9093 silence query
ID Matchers Ends At Created By Comment
16587a75-aee0-4b83-b053-8781c8afcb22 alertname="QueueTooLong" stall="nasi-lemak" 2026-11-01 14:56:48 UTC aina Fixing the queue, back by 3pm

Aina is fixing the queue. A silence: A note on the dispatcher's desk: don't call anyone for these alerts until a set time. It has an author and a reason. says: don’t call for this until 2 hours pass. It has an author and a reason.

Check

Drag a card into a box. Or tap the card, then tap the box.

        Which tool keeps the pager calm here?

        You did it!

        You can now route, group and silence alerts with Alertmanager.

        Go deeper (optional)

        Timers on a route: group_wait (default 30s) waits for friends before the first message. group_interval (default 5m) waits before telling about new bells in a group. repeat_interval (default 4h) resends if nothing changed. An inhibit_rules entry mutes target_matchers while source_matchers fire.

        Real mission

        2. docker run --rm --entrypoint amtool -v "$PWD:/w" -w /w quay.io/prometheus/alertmanager:v0.34.1 check-config alertmanager.yml
        3. docker run --rm --entrypoint amtool -v "$PWD:/w" -w /w quay.io/prometheus/alertmanager:v0.34.1 config routes test --config.file=alertmanager.yml severity=ticket

        0 of 3 steps done · Your ticks are saved.