World map

 

BM

World O3: Alarms & Screens · Boss challenge

Quiet the pager

You will learn: to turn a night of noisy alerts into a few that matter.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Boss! Kedai Kancil’s pager buzzed 20 times last night. The on-call cook is tired. Room 1: sort these bells.

Drag a card into a box. Or tap the card, then tap the box.

        Try it

        Room 2: the queue bell flaps all night. You may only wait 1 minute. Make it ring in one stretch.

        Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

        When (expr)
        Wait first (for)
        Labels
        Your rule: rules.yml
        groups:
- name: kedai
rules:
- alert: QueueLong
expr: kedai_queue_length > 25
labels:
severity: ticket
        The value the alert watches
        02040600s10m20m30mthreshold 25
        Graph as a table
        Timekedai_queue_length{stall="nasi-lemak"}
        0s10
        1m12
        2m30
        3m9
        4m31
        5m11
        6m29
        7m33
        8m8
        9m35
        10m36
        11m38
        12m40
        13m42
        14m41
        15m39
        16m12
        17m10
        18m9
        19m8
        20m7
        21m30
        22m9
        23m8
        24m7
        25m6
        26m5
        27m5
        28m5
        29m5
        30m5

        What the alert does over time

        Grey: quiet. Amber: pending (true, but waiting). Red: firing.

        • {severity="ticket", stall="nasi-lemak"}

          🔔 firing at 2m · 🔔 firing at 4m · 🔔 firing 6m to 7m · 🔔 firing 9m to 15m · 🔔 firing at 21m

        At 12m:

        • FIRING QueueLong severity="ticket", stall="nasi-lemak"
        Timeline as a table
        Time{severity="ticket", stall="nasi-lemak"}
        0squiet
        1mquiet
        2m🔔 firing
        3mquiet
        4m🔔 firing
        5mquiet
        6m🔔 firing
        7m🔔 firing
        8mquiet
        9m🔔 firing
        10m🔔 firing
        11m🔔 firing
        12m🔔 firing
        13m🔔 firing
        14m🔔 firing
        15m🔔 firing
        16mquiet
        17mquiet
        18mquiet
        19mquiet
        20mquiet
        21m🔔 firing
        22mquiet
        23mquiet
        24mquiet
        25mquiet
        26mquiet
        27mquiet
        28mquiet
        29mquiet
        30mquiet

        Try it

        Room 3: when the whole shop is down, mute the ticket bells it causes. Add a target_matchers line.

        Change the YAML. The box below shows how a computer reads it.

        alertmanager.yml9 lines

        How a computer reads it

        • routemap · 2 keys
          • receiverteam-chattext
          • group_bylist · 2 items
            • item 1alertnametext
            • item 2jobtext
        • receiverslist · 1 items
          • item 1map · 1 keys
            • nameteam-chattext
        • inhibit_ruleslist · 1 items
          • item 1map · 2 keys
            • source_matcherslist · 1 items
              • item 1alertname="ShopDown"text
            • equallist · 1 items
              • item 1jobtext

        Try it

        Room 4: one SLO bell replaces five error bells. Page for the fast burn only.

        Build an alert rule from the blocks. The timeline shows when it would wait and when it would fire.

        When (expr)
        Wait first (for)
        Labels
        Your rule: rules.yml
        groups:
- name: kedai
rules:
- alert: FastBurn
expr: job:shop_error_ratio:rate5m > 0.001
for: 2m
labels:
severity: ticket
        The value the alert watches
        00.020.040.060s30m1h1h30m2hthreshold 0.001
        Graph as a table
        Timejob:shop_error_ratio:rate5m{job="shop"}
        0s–
        1m0.0005
        2m0.0005
        3m0.0005
        4m0.0005
        5m0.0005
        6m0.0005
        7m0.0005
        8m0.0005
        9m0.0005
        10m0.0005
        11m0.0005
        12m0.0005
        13m0.0005
        14m0.0005
        15m0.0005
        16m0.0005
        17m0.0005
        18m0.0005
        19m0.0005
        20m0.011899999999999999
        21m0.0219
        22m0.0319
        23m0.0419
        24m0.037625
        25m0.025249999999999998
        26m0.012874999999999998
        27m0.0005
        28m0.0005
        29m0.0005
        30m0.0005
        31m0.0005
        32m0.0005
        33m0.0005
        34m0.0005
        35m0.0005
        36m0.0005
        37m0.0005
        38m0.0005
        39m0.0005
        40m0.0005
        41m0.0005
        42m0.0005
        43m0.0005
        44m0.0005
        45m0.0005
        46m0.0005
        47m0.0005
        48m0.0005
        49m0.0005
        50m0.0005
        51m0.0005
        52m0.0005
        53m0.0005
        54m0.0005
        55m0.0005
        56m0.0005
        57m0.0005
        58m0.0005
        59m0.0005
        1h0.0005
        1h1m0.012874999999999998
        1h2m0.025249999999999998
        1h3m0.037625
        1h4m0.049999999999999996
        1h5m0.049999999999999996
        1h6m0.049999999999999996
        1h7m0.049999999999999996
        1h8m0.049999999999999996
        1h9m0.049999999999999996
        1h10m0.049999999999999996
        1h11m0.049999999999999996
        1h12m0.049999999999999996
        1h13m0.049999999999999996
        1h14m0.049999999999999996
        1h15m0.049999999999999996
        1h16m0.049999999999999996
        1h17m0.049999999999999996
        1h18m0.049999999999999996
        1h19m0.049999999999999996
        1h20m0.049999999999999996
        1h21m0.049999999999999996
        1h22m0.049999999999999996
        1h23m0.049999999999999996
        1h24m0.049999999999999996
        1h25m0.049999999999999996
        1h26m0.049999999999999996
        1h27m0.049999999999999996
        1h28m0.049999999999999996
        1h29m0.049999999999999996
        1h30m0.049999999999999996
        1h31m0.037625
        1h32m0.025249999999999998
        1h33m0.012874999999999998
        1h34m0.0005
        1h35m0.0005
        1h36m0.0005
        1h37m0.0005
        1h38m0.0005
        1h39m0.0005
        1h40m0.0005
        1h41m0.0005
        1h42m0.0005
        1h43m0.0005
        1h44m0.0005
        1h45m0.0005
        1h46m0.0005
        1h47m0.0005
        1h48m0.0005
        1h49m0.0005
        1h50m0.0005
        1h51m0.0005
        1h52m0.0005
        1h53m0.0005
        1h54m0.0005
        1h55m0.0005
        1h56m0.0005
        1h57m0.0005
        1h58m0.0005
        1h59m0.0005
        2h0.0005

        What the alert does over time

        Grey: quiet. Amber: pending (true, but waiting). Red: firing.

        • {job="shop", severity="ticket"}

          ⏳ pending 20m to 21m · 🔔 firing 22m to 26m · ⏳ pending 1h1m to 1h2m · 🔔 firing 1h3m to 1h33m

        At 1h20m:

        • FIRING FastBurn job="shop", severity="ticket"
        Timeline as a table
        Time{job="shop", severity="ticket"}
        0squiet
        1mquiet
        2mquiet
        3mquiet
        4mquiet
        5mquiet
        6mquiet
        7mquiet
        8mquiet
        9mquiet
        10mquiet
        11mquiet
        12mquiet
        13mquiet
        14mquiet
        15mquiet
        16mquiet
        17mquiet
        18mquiet
        19mquiet
        20m⏳ pending
        21m⏳ pending
        22m🔔 firing
        23m🔔 firing
        24m🔔 firing
        25m🔔 firing
        26m🔔 firing
        27mquiet
        28mquiet
        29mquiet
        30mquiet
        31mquiet
        32mquiet
        33mquiet
        34mquiet
        35mquiet
        36mquiet
        37mquiet
        38mquiet
        39mquiet
        40mquiet
        41mquiet
        42mquiet
        43mquiet
        44mquiet
        45mquiet
        46mquiet
        47mquiet
        48mquiet
        49mquiet
        50mquiet
        51mquiet
        52mquiet
        53mquiet
        54mquiet
        55mquiet
        56mquiet
        57mquiet
        58mquiet
        59mquiet
        1hquiet
        1h1m⏳ pending
        1h2m⏳ pending
        1h3m🔔 firing
        1h4m🔔 firing
        1h5m🔔 firing
        1h6m🔔 firing
        1h7m🔔 firing
        1h8m🔔 firing
        1h9m🔔 firing
        1h10m🔔 firing
        1h11m🔔 firing
        1h12m🔔 firing
        1h13m🔔 firing
        1h14m🔔 firing
        1h15m🔔 firing
        1h16m🔔 firing
        1h17m🔔 firing
        1h18m🔔 firing
        1h19m🔔 firing
        1h20m🔔 firing
        1h21m🔔 firing
        1h22m🔔 firing
        1h23m🔔 firing
        1h24m🔔 firing
        1h25m🔔 firing
        1h26m🔔 firing
        1h27m🔔 firing
        1h28m🔔 firing
        1h29m🔔 firing
        1h30m🔔 firing
        1h31m🔔 firing
        1h32m🔔 firing
        1h33m🔔 firing
        1h34mquiet
        1h35mquiet
        1h36mquiet
        1h37mquiet
        1h38mquiet
        1h39mquiet
        1h40mquiet
        1h41mquiet
        1h42mquiet
        1h43mquiet
        1h44mquiet
        1h45mquiet
        1h46mquiet
        1h47mquiet
        1h48mquiet
        1h49mquiet
        1h50mquiet
        1h51mquiet
        1h52mquiet
        1h53mquiet
        1h54mquiet
        1h55mquiet
        1h56mquiet
        1h57mquiet
        1h58mquiet
        1h59mquiet
        2hquiet

        Check

        Room 5: tonight, 1 a.m. to 2 a.m., the database gets a planned upgrade. What do you do?

        Check

        Final room: swipe right for habits that keep the pager quiet.

        Swipe right for good, left for needs work. Or use the buttons.

        1 / 4

        Every paging alert links a runbook.

        You did it!

        You can now turn a noisy pager into a few bells that matter.