World O2: Counters

Histograms and histogram_quantile

You will learn: how to find p95 from histogram buckets, and why an average can hide slow customers.

Checked with Prometheus 3 on 1 Nov 2026

The problem

The average checkout takes 0.39 s. Yet customers say checkout is slow. How can both be true?

Watch

A says: 95% of checkouts were this fast or faster. Build the recipe.

Tap the pieces in the right order.

…

Try it

Now run it. Then slide from 8m to 20m.

Wrap it: histogram_quantile(0.95, sum by (le) (…)). Then Run.

PromQL query

Metrics in this data (tap to add):

Eval time: 20m

Time 0 is when the data starts. The query looks back from the eval time.

Predict

At 20m, p95 is 1.86 s. What does that mean?

Try it

A teammate wrote this. Run it: no answer. Fix it so each shop gets its own p95.

Keep le next to instance in the by list, then Run.

PromQL query

Metrics in this data (tap to add):

Eval time: 20m

Time 0 is when the data starts. The query looks back from the eval time.

Hint

No le, no buckets, no answer.

Check

shop-1’s p95 is 2.29 s. shop-2’s is 0.42 s. Where do you look first?

You did it!

You can now find p95 with histogram_quantile and keep le in the sum.

Go deeper (optional)

histogram_quantile guesses inside a bucket, in a straight line. So pick bucket edges near your SLO, like 0.5 s. The average is rate(…_sum[5m]) divided by rate(…_count[5m]). Real rep: p90 for Prometheus’s own web pages.

Real mission

In your kedai-watch folder, start the stack. Wait two minutes.
docker compose up -d
At http://localhost:9090, run: histogram_quantile(0.9, sum by (le, handler) (rate(prometheus_http_request_duration_seconds_bucket[1m])))
Which handler is slowest? Open a few pages and run it again.
Stop the stack.
docker compose down

0 of 4 steps done · Your ticks are saved.