BM: Baca dalam Bahasa Melayu
World K9: Cluster Admin · Boss challenge
Night of the broken node
You will learn: to fix a NotReady node and bring a lost Deployment back from an etcd snapshot, under pressure.
Checked with Kubernetes 1.37 on 1 Nov 2026
The problem
2 a.m. The pager says stall dev-worker2 is NotReady. And the shop’s order is gone. Boss mission: fix both. What do you do first?
Pick one answer.
Try it
You looked. Tap both problems.
Tap both problems.
0 of 2 found
Hint
One line says NotReady. Another says nothing was found.Try it
Room 1: bring dev-worker2 back. Look inside it first.
kind-dev namespace defaultdev-control-planeOpenManagerNo trays here.
dev-workerOpenNo trays here.
dev-worker2No news- shop·sktk6
Hint
The supervisor on that stall stopped.systemctl starts it.Predict
A snapshot waits in /opt/backup.db. It was taken before the delete. After a restore, what happens to objects made after it?
Pick one answer.
Try it
Room 2: plan the restore before you touch anything.
First step at the top. Put these in order. Use the arrows, or drag.
- Point etcd.yaml at /var/lib/etcd-restore (sed -i)
- etcdutl --write-out=table snapshot status /opt/backup.db
- ssh dev-control-plane
- exit, wait about 30 s, kubectl get deployments
- etcdutl --data-dir /var/lib/etcd-restore snapshot restore /opt/backup.db
Try it
Final room: both faults are back. Fix the stall, and bring the shop back from the snapshot. Danger: the restore rewrites etcd.
kind-dev namespace defaultdev-control-planeOpenManagerNo trays here.
dev-workerOpenNo trays here.
dev-worker2No news- shop·sktk6
Hint
Two jobs, any order. A restore alone leaves one tray waiting on the quiet stall.Check
A teammate restored first. The shop shows 2/3 ready. Why?
Pick one answer.
Check
Tomorrow’s plan. Good idea, or needs work?
Swipe right for good, left for needs work. Or use the buttons.
Save etcd snapshots on a schedule, and copy them off the node
You did it!
You can now fix a NotReady node and restore a lost Deployment.
Go deeper (optional)
Where the story breaks: our order book is one paper book. Real clusters often run 3 or 5 etcd members. Then every member is restored from the same snapshot. Also, our clock skips ahead; a real restart takes longer.
Real mission
kubectl get nodessudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.dbsudo etcdutl --write-out=table snapshot status /opt/backup.db
0 of 3 steps done · Your ticks are saved.
Bahasa Melayu
Pukul 2 pagi. Pager kata gerai dev-worker2 NotReady. Dan pesanan shop hilang. Misi boss: baiki kedua-duanya. Apa anda buat dulu?
Anda sudah lihat. Ketik kedua-dua masalah.
Bilik 1: pulihkan dev-worker2. Lihat ke dalamnya dulu.
systemctl hidupkannya.Satu snapshot menunggu di /opt/backup.db. Ia diambil sebelum pemadaman. Selepas pemulihan, apa jadi pada objek yang dibuat selepas itu?
Bilik 2: rancang pemulihan sebelum sentuh apa-apa.
Bilik terakhir: kedua-dua masalah kembali. Baiki gerai itu, dan pulihkan shop dari snapshot. Bahaya: pemulihan tulis semula etcd.
Rakan sepasukan pulihkan etcd dulu. Shop tunjuk 2/3 sedia. Kenapa?
Rancangan esok. Idea baik, atau perlu dibaiki?
Kini anda boleh baiki node NotReady dan pulihkan Deployment yang hilang.
You finished this before. Practice is always welcome!
Official docs: Kubernetes: troubleshooting clusters · Kubernetes: operating etcd clusters (backup and restore) · etcd 3.7: disaster recovery
You've learned for 25 minutes. A short break helps your brain.
Sprint done! Great focus. Stretch for a minute?