Playing with Locks in Consul

Posted on September 13, 2015

Consul is a great piece of software, and I love building HA services with it at the heart of the distributed system. One of Consul’s features is a distributed locking mechanism, and I wanted to learn a little more about it. I also wanted to experiment with simple methods for running a one-time action, or a single instance of a service in a distributed system - potential solutions to the famous challenge: “who guards the guards?”

import sys
from time import sleep

import consul
import consul_lock
from docker import Client

'''
This is a PoC which uses consul's locking mechanism to ensure only
one instance of the named docker container is running. Requires:
  a) pip install python-consul consul-lock
  b) docker create --name my_container my/app

try it: python only-one.py container_name token

'''
pause = 3
container_name = sys.argv[1]
token = sys.argv[2]
cli = Client(base_url='unix://var/run/docker.sock')

consul_client = consul.Consul(token=token)
consul_lock.defaults.consul_client = consul_client

print 'starting up!'
while True:
    ephemeral_lock = consul_lock.EphemeralLock(('only-one/%s' % container_name),
                                               lock_timeout_seconds=3600,
                                               acquire_timeout_ms=500)
    try:
        print 'acquiring distributed lock in consul..'
        was_acquired = ephemeral_lock.acquire(fail_hard=False)
        if was_acquired:
            # do dangerous stuff here
            print 'lock acquired!'
            print ('starting container %s' % container_name)
            response = cli.start(container=container_name)
            print(response)
            print 'will now wait/block until the container exits..'
            cli.wait(container=container_name)
        else:
            print 'someone else has the lock :\ try again later'
    finally:
        print 'be sure there is no lock in consul..'
        ephemeral_lock.release()
    print ('sleeping for %d seconds' % pause)
    sleep(pause)

The original post can be seen in this gist.