Skip to main content

Exploring etcd

· 9 min read
tanzhuo

I started writing this article because, while organizing my Kubernetes notes, I kept running into the name etcd: cluster state lives in it, service discovery depends on it, and quite a few configuration centers are built on top of it too. Rather than treating it as a black box, I decided to set it up from scratch and work through it properly. In distributed systems, you frequently run into problems like these:

  • Service nodes need to share configuration
  • The system needs service discovery
  • Distributed locks need a coordination point
  • The cluster needs a consistent state store

At their core, all of these problems call for a reliable distributed coordination system.

In the single-machine era, these needs could be met with a database table or even a plain file. But once you have multiple nodes, the question becomes: how do multiple replicas agree on the "current state"? Who has the final say? Can you still trust the data after a node dies? These are exactly the questions a distributed coordination system has to answer.

In the modern cloud-native world, the most widely used component for this is etcd.

For example:

  • Kubernetes
  • CoreDNS
  • service mesh
  • distributed configuration centers

All of these systems rely on etcd underneath.

What is etcd​

In short:

etcd is a highly reliable distributed key-value store.

But what sets it apart from an ordinary database is this:

It was designed specifically for distributed coordination.

That positioning dictates its trade-offs: it doesn't chase massive data volumes or extreme throughput; instead, it aims to make every piece of data you store trustworthy. So it's a great fit for small but critical data — metadata, configuration, state — and a poor fit as a general-purpose business database.

Its main features are:

  • Strong consistency
  • Distributed cluster support
  • A Watch mechanism
  • Transactions
  • A Lease mechanism

The combination of these features is quite interesting:

  1. Watch lets clients subscribe to changes on a key or prefix. When a config value changes, every node learns about it in real time — no polling required.

  2. A Lease is essentially a heartbeat token with a TTL. Keys can be attached to a lease, and once the client stops renewing it, those keys are deleted automatically — this is exactly how service registration and failure eviction are implemented.

  3. Transactions (Txn) provide compare-and-swap semantics: "write this key only if it doesn't exist" can be done atomically, which is the foundation of distributed locking.

Many distributed systems use etcd for:

  • Service registries
  • Configuration centers
  • Distributed locks
  • Leader election

Kubernetes, for example, stores the entire cluster state in etcd. kube-apiserver is the only component that reads and writes etcd directly; all other controllers observe state changes indirectly through the apiserver's Watch mechanism.

Strong consistency (the Raft consensus algorithm)​

Internally, etcd uses the Raft consensus algorithm.

The core idea of Raft is:

The cluster elects a single Leader node.

All write requests must go through the Leader.

The flow looks roughly like this:

Client → Leader → replicate to a majority of nodes → commit succeeds

Only when:

more than half of the nodes have acknowledged the write

is the data considered truly committed.

Let's break this down a bit further. In Raft, every node is in one of three roles: Leader, Follower, or Candidate. During normal operation there is exactly one Leader, which periodically sends heartbeats to all Followers. If a Follower doesn't receive a heartbeat within its timeout, it turns itself into a Candidate and starts an election; whichever node wins a majority of votes becomes the new Leader.

Writes go through log replication: the Leader first appends the write operation to its own log, then sends it to all Followers in parallel. Only when a majority (more than half) of nodes have persisted the log entry does the Leader mark it as committed and respond to the client.

The "majority" design is the key to strong consistency: any two majority sets must overlap, so even if the Leader crashes, the newly elected Leader is guaranteed to contain all committed data — no committed writes are lost.

This also explains why an odd number of etcd nodes (3, 5, 7) is recommended:

  • 3 nodes tolerate 1 failure; 5 nodes tolerate 2
  • 4 nodes also only tolerate 1 failure (a majority requires 3), so fault tolerance is the same as 3 nodes, with one extra replica's worth of sync overhead

Setting up an etcd cluster​

Deployment environment:

Operating system:

Debian 11.5.0

1 Download etcd​

Download the etcd binaries from GitHub:

https://github.com/etcd-io/etcd/releases

The release page provides archives per platform; just pick the one for your architecture (e.g. linux-amd64). etcd ships as a single static binary with no extra runtime dependencies — one of the reasons it's so easy to deploy.

2 Install etcd​

After extracting, place the following files into:

/usr/local/bin

Mainly these two:

etcd
etcdctl

The division of labor is clear: etcd is the server process, and etcdctl is the command-line client — all subsequent queries, writes, and maintenance go through it. Putting them in /usr/local/bin gets them onto the PATH so you can invoke them from any directory.

3 Set execute permissions​

# The extracted files may lack the executable bit; add it manually
sudo chmod +x /usr/local/bin/etcd
sudo chmod +x /usr/local/bin/etcdctl

At this point, the binaries for a single node are ready. In cluster mode, repeat the installation steps above on every node, then declare each node's name, listen addresses, and the initial cluster member list via startup flags. Nodes talk to each other over port 2380, and clients connect on port 2379. My environment has TLS enabled, with certificates stored under /opt/etcd/ssl, so the verification commands below all need certificate arguments.

4 Verify the cluster​

You can use etcdctl to inspect the data:

# List all keys in the cluster (keys only, no values)
# --cacert/--cert/--key: CA and client certificates for mutual TLS
# --endpoints: list all three nodes; the client load-balances and fails over automatically
ETCDCTL_API=3 etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--endpoints="https://172.24.93.151:2379,https://172.24.93.149:2379,https://172.24.93.150:2379" \
get / --prefix --keys-only

If the command returns results, the client-to-cluster path, the certificates, and quorum are all working. --prefix enables prefix matching; combined with / it effectively walks every key.

Delete keys:

# Delete by prefix; the empty prefix "" matches ALL keys — use with extreme caution in production
ETCDCTL_API=3 etcdctl \
--cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/server.pem \
--key=/opt/etcd/ssl/server-key.pem \
--endpoints="https://172.24.93.151:2379,https://172.24.93.149:2379,https://172.24.93.150:2379" \
del --prefix ""
warning

del --prefix "" deletes everything. If this etcd cluster is serving as the backend for Kubernetes, this command wipes the entire cluster state — double-check which environment you're targeting before running it.

Why etcd beats Redis for distributed coordination​

A common question:

Can't Redis do distributed locks too?

It can.

But Redis is an AP system, while etcd is a CP system.

Put simply:

Redis prioritizes availability and performance.

etcd prioritizes consistency and reliability.

For locking specifically, the difference shows up in failure behavior: Redis master-replica replication is asynchronous. If the master writes a lock and crashes before replicating it, the promoted replica has no record of the lock — it's simply "lost", and two clients can hold the lock at the same time. The Redlock algorithm tries to mitigate this, but its safety has long been debated in the community. etcd writes, on the other hand, must be confirmed by a majority before they succeed; a Leader failover never loses committed data, so the lock's semantics hold even under failure.

The cost, of course, is performance: every write requires a round of majority replication plus an fsync to disk, so latency and throughput are far behind Redis. So the conclusion isn't that one is better than the other — it's about matching the tool to the scenario. Use Redis for caching, counters, and lightweight locks that can tolerate occasional failure; use etcd for configuration, leader election, and locks that require strict mutual exclusion.

Pitfalls and notes​

A few things worth recording from the setup and usage process:

  1. Don't forget the ETCDCTL_API=3 environment variable. Older versions of etcdctl default to the v2 API, and v2 and v3 data are completely isolated — using the wrong API version produces the illusion that "I wrote the data but can't read it back". Newer versions default to v3, but declaring it explicitly in scripts never hurts.

  2. List all nodes in --endpoints. If you list only one node and it goes down, the client loses connectivity entirely; with all nodes listed, the client fails over to healthy nodes automatically.

  3. Certificate paths and SANs must match. With TLS enabled, the IPs/domains in the certificate must cover the addresses actually used in the endpoints, or the handshake will be rejected.

  4. Keep the node count odd. The Raft section above explains why: an even node count adds no fault tolerance, only synchronization cost.

  5. etcd is sensitive to disk latency. Every Raft log commit requires an fsync; a slow disk directly inflates write latency and can even trigger Leader election churn. Use SSDs if you can.

Wrapping up​

etcd is a textbook case of trading performance for consistency: Raft's majority commit guarantees that any committed data you read is trustworthy at any moment, and Watch, Lease, and transactions turn that trustworthy state store into a general-purpose foundation for service discovery, leader election, and distributed locking. The setup itself is simple — two binaries and a set of startup flags. What actually deserves your attention is TLS certificates, node planning, and a clear-eyed judgment of where etcd fits. Figuring out the division of labor between etcd and Redis, and using each where it excels, is far more useful than arguing about which one is better.

COMMENTS