World map

 

BM

World O4: The Diary & One Order · Boss challenge

Find the slow step

You will learn: to follow one slow order from a graph, through logs and a trace, to its cause.

Checked with Prometheus 3 on 1 Nov 2026

The problem

Boss! Customers say paying is slow. Room 1: which service has the slowest p95 at 18m? Show only that one.

Type a PromQL query and press Run. The table shows the answer at the eval time. The graph shows it over time.

Metrics in this data (tap to add):

Time 0 is when the data starts. The query looks back from the eval time.

Try it

Room 2: find the order that failed. Build a LogQL query for kedai-api’s error lines.

Tap the pieces in the right order.

…

Try it

Room 3: the real trace of order A1041. Tap the name of the step where the time really went.

0 of 1 found

Check

bash
logcli query --quiet '{app=~"kedai-.*"} |= "3c9a51e0b7d24f6e9a18c2d4e6f80b15"'
2026-11-01T17:45:22Z {app="kedai-api", detected_level="error", service_name="kedai-api"} {"level":"error","msg":"payment timeout","order_id":"A1041","duration_ms":3000,"trace_id":"3c9a51e0b7d24f6e9a18c2d4e6f80b15"}
2026-11-01T17:45:21Z {app="kedai-pay", detected_level="warn", service_name="kedai-pay"} {"level":"warn","msg":"slow query","table":"payments","duration_ms":2800,"trace_id":"3c9a51e0b7d24f6e9a18c2d4e6f80b15"}
2026-11-01T17:45:19Z {app="kedai-api", detected_level="info", service_name="kedai-api"} {"level":"info","msg":"order received","order_id":"A1041","trace_id":"3c9a51e0b7d24f6e9a18c2d4e6f80b15"}

Room 4: search every diary for the trace ID. The pay diary agrees with the trace.

Check

Room 5: the payments table got slow after the 17:30 database change. What first?

Check

Final room: put your investigation in order.

First step at the top. Put these in order. Use the arrows, or drag.

  • Logs: the failing order and its trace ID
  • Fix: roll back the change
  • Logs again: kedai-pay says slow query
  • Trace: SELECT payments takes 2.8 s
  • Metrics: kedai-pay has the worst p95

You did it!

You can now follow one slow order from a graph to its cause.