Monitoring & Observability cheat sheet

The commands you learned, in one place.

Check rules and configs

docker compose exec prometheus promtool check config /etc/prometheus/prometheus.ymlpromtool lives inside the Prometheus image. SUCCESS means the file is fine.
docker compose exec prometheus promtool check rules /etc/prometheus/rules.ymlSUCCESS: 1 rules found. A broken expr shows the parse error and the line.
amtool check-config alertmanager.ymlWill the dispatcher understand this file?

Ask Prometheus

curl localhost:9090/-/readyIt answers: Prometheus Server is Ready.
curl localhost:9090/-/healthyIs the server alive at all?
curl -s localhost:9100/metrics | grep 'node_load1 'Node exporter's 1-minute load, with its help and type lines.
curl -s localhost:9090/metrics | grep '^# TYPE' | headThe first few metric types Prometheus shows about itself.
curl -s 'localhost:9115/probe?target=http://prometheus:9090/-/ready&module=http_2xx' | grep -E 'probe_success|probe_http_status_code'probe_success 1 and status code 200 mean the page answered.
curl -s 'localhost:9090/api/v1/query?query=up'An instant vector: one value per target, at this moment.
curl -s localhost:9090/api/v1/query --data-urlencode 'query=rate(prometheus_http_requests_total[1m])'Let curl encode the brackets and spaces for you.
docker compose exec prometheus promtool query instant http://localhost:9090 'up'Each target, with 1 for up and 0 for down.

Alerts and silences

amtool config routes test --config.file=alertmanager.yml severity=pageWho gets a severity=page alert?
amtool config routes show --config.file=alertmanager.ymlDraw the whole routing tree.
amtool --alertmanager.url=http://localhost:9093 silence add alertname=QueueTooLong --duration=2h --author=aina --comment="Fixing the queue"No calls about the queue for 2 hours: Aina is on it.Risk: Careful
amtool --alertmanager.url=http://localhost:9093 silence queryList the silences: who, why, and when they end.Risk: Careful
amtool --alertmanager.url=http://localhost:9093 silence expire <id>End a silence early, when the work is done.Risk: Careful

Logs and traces

logcli query --quiet '{app="kedai-api"} |= "timeout"'Show kedai-api's lines that mention timeout.
logcli labelsWhich labels does Loki know?
logcli labels appWhich values does the app label have?
logcli query --quiet '{app=~"kedai-.*"} |= "<trace-id>"'Follow one order across every kedai diary.
docker run --rm -v "$PWD:/w" otel/opentelemetry-collector-contrib:0.164.0 validate --config=/w/config.yamlIs my Collector config sound? Run it from the official image.