Using etcd for elections

We recently had a requirement to implement leader elections across multiple servers (for reasons relating to avoiding contention between multiple instances of the OpenStack Neutron plugin). It’s not the first time I’ve had to implement elections, and it’s an awkward problem. You can do something pretty simple, but there are invariably lots of weird edge cases that require lots of testing and bug fixing to get right – and you never quite have the confidence it won’t break in the real world. Alternatively, if you go down the road of implementing the Raft algorithm or even Paxos you are guaranteed to be safe, but only if you get the code exactly right, and again that means a lot of testing.

Fortunately, we are using etcd, and since etcd provides both TTL on objects, and a compare and swap operation, implementing an election algorithm is trivial. Unelected instances can watch “/election” (or some other well known key) and if it is empty become elected by writing their ID to it. The written value is given a TTL that removes it after a set interval, and the elected instance must rewrite it periodically to remain elected. By the use of etcd’s atomic compare and swap operation, there is no risk of a clash between two instances being undetected. Job done, in a fraction of the time to implement it from scratch. There are some edge cases if etcd goes off line or runs catastrophically slowly, but neither of these matters for our use case.

This kind of situation is a change from the software industry even of five years ago, let alone twenty years ago. Instead of writing things from scratch and having to reinvent the wheel ourselves, we are building on top of standard third party components (in this case etcd) that do much of the work for us (here implementing the Raft algorithm in a cluster). That means that building almost any complex architecture is far cheaper than it used to be, and thus that we can build bigger and better than before.