World K9: Cluster Admin

etcd backup and restore

You will learn: to photocopy the order book with etcdctl, and bring a lost object back with etcdutl.

Checked with Kubernetes 1.37 on 1 Nov 2026

The problem

At 2 a.m. someone deleted the shop’s Deployment. Its YAML was never saved in Git. What could still bring it back?

Watch

You are here: cluster kind-dev namespace default

~/shop $ ssh dev-control-planeroot@dev-control-plane:~# etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db{"level":"info","ts":"2026-10-31T01:00:22.252272Z","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/opt/backup.db.part"}{"level":"info","ts":"2026-10-31T01:00:22.255184Z","logger":"client","caller":"v3/maintenance.go:236","msg":"opened snapshot stream; downloading"}{"level":"info","ts":"2026-10-31T01:00:22.258272Z","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}{"level":"info","ts":"2026-10-31T01:00:22.260284Z","logger":"client","caller":"v3/maintenance.go:302","msg":"completed snapshot read; closing"}{"level":"info","ts":"2026-10-31T01:00:22.263372Z","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"2.4 MB","took":"9.935898ms","etcd-version":"3.7.0"}{"level":"info","ts":"2026-10-31T01:00:22.266284Z","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/opt/backup.db"}Snapshot saved at /opt/backup.dbServer version 3.7.0root@dev-control-plane:~# exitlogoutConnection to dev-control-plane closed.

An is a photocopy of the order book. etcdctl asks the running etcd for it. Keep the copy off the cluster.

Try it

Build the save command. Each piece shows what it means.

Tap the pieces in the right order.

…

Watch

You are here: cluster kind-dev namespace default

~/shop $ ssh dev-control-planeroot@dev-control-plane:~# etcdctl snapshot save /opt/backup.db{"level":"info","ts":"2026-10-31T01:00:22.252272Z","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/opt/backup.db.part"}^C# etcdctl is still waiting here: it never got in. You pressed Ctrl+C.root@dev-control-plane:~# exitlogoutConnection to dev-control-plane closed.

Trap: without certificate flags, etcdctl prints one line, then waits forever. Press Ctrl+C. Then add --cacert, --cert and --key.

Try it

Your turn. Open the office node, save a snapshot, then check it.

You are here: cluster kind-dev namespace default

dev-control-planeOpenManager

No trays here.

dev-workerOpen

shop·9448k
shop·q725x

dev-worker2Open

shop·ddlcm

Time: +0s (sped up)

etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db
etcdutl --write-out=table snapshot status /opt/backup.db

Predict

On etcd 3.7, you type etcdctl snapshot status /opt/backup.db. What happens?

Try it

The shop was deleted after the snapshot. restores the copy into a new, empty folder. Then point etcd there. Danger: newer objects are lost.

You are here: cluster kind-dev namespace default

dev-control-planeOpenManager

No trays here.

dev-workerOpen

No trays here.

dev-worker2Open

No trays here.

Time: +0s (sped up)

etcdutl --data-dir /var/lib/etcd-restore snapshot restore /opt/backup.db
sed -i 's#/var/lib/etcd$#/var/lib/etcd-restore#' /etc/kubernetes/manifests/etcd.yaml

Check

Put a restore in order.

First step at the top. Put these in order. Use the arrows, or drag.

Wait: etcd restarts, then the API answers
etcdutl --data-dir <new folder> snapshot restore
Point etcd.yaml at the new folder
kubectl get deployments
etcdutl snapshot status: is the copy healthy?

You did it!

You can now back up etcd and restore a lost object.

Go deeper (optional)

kind nodes have no etcdctl. There, run it inside the etcd pod. Only /var/lib/etcd and the certificates are shared with the node.

You are here: cluster kind-dev namespace default

~/shop $ kubectl -n kube-system exec etcd-dev-control-plane -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/backup.db{"level":"info","ts":"2026-10-31T01:00:22.252272Z","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/var/lib/etcd/backup.db.part"}{"level":"info","ts":"2026-10-31T01:00:22.255184Z","logger":"client","caller":"v3/maintenance.go:236","msg":"opened snapshot stream; downloading"}{"level":"info","ts":"2026-10-31T01:00:22.258272Z","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}{"level":"info","ts":"2026-10-31T01:00:22.260284Z","logger":"client","caller":"v3/maintenance.go:302","msg":"completed snapshot read; closing"}{"level":"info","ts":"2026-10-31T01:00:22.263372Z","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"2.4 MB","took":"9.935898ms","etcd-version":"3.7.0"}{"level":"info","ts":"2026-10-31T01:00:22.266284Z","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/var/lib/etcd/backup.db"}Snapshot saved at /var/lib/etcd/backup.dbServer version 3.7.0

Real mission

On a throwaway kind cluster, save a snapshot inside the etcd pod (command above).
kubectl -n kube-system exec etcd-<cluster>-control-plane -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/backup.db
Copy it off the node, to your laptop.
docker cp <cluster>-control-plane:/var/lib/etcd/backup.db ./backup.db

0 of 2 steps done · Your ticks are saved.