Customizing botocore Waiters

It’s amazing how much thought AWS developers put into botocore. No surprise that it’s full of hidden gems.

One of them is Waiters.

The Problem

It’s a documented feature, but what’s revealed is only a tip of the iceberg. You can await completion of tasks like creation or deletion of volumes: conditions are simple, nothing can go wrong.

The story is different for tasks like attaching volumes. The builtin Waiter only allows to await attachment status, not how and where it is attached. In a project I’m working on now we have hunderds of [micro]services working with volumes asynchronously without supervision. It is vital to ensure correctness.

Let’s do just that: create custom Waiter that will await a volume to be attached to a given instance as a given device.

The Approach

Turns out, Waiters are entirely based on a JSON model and use jmespath to query AWS API responses. The library comes with all the necessary methods to work with it.

Let’s take a look at how Waiters are defined. With botocore 1.7.5 definitions can be found at botocore/data/ec2/2016–11–15/waiters-2.json. Our model will be based on VolumeInUse that does similar task:

It defines AWS API call that will be used (DescribeVolumes) and how it should be retried (40 attempts with 15s delay). The acceptors list specifies jmespath queries and how they should be interpreted: success or failure. They are evaluated in order of definition and upon match either cause Waiter to return or to raise an exception. If no match is found, call is retried.

Acceptor query is evaluated against AWS API response which for DescribeVolumes looks like this:

Fields that are imporntant to us:

• Volumes.Attachments.InstanceId must be equal to ID of our instance
• Volumes.Attachments.State must be attached
• Volumes.Attachments.Device must be device we picked
• State must be "in-use"

We only want to check status of a single volume, but the response contains a list. Thankfully, it’s possible to filter the list and then get length of the result with jmespath.

The Solution

First acceptor gets attachments of all “in-use” volumes and then checks whether their InstanceId and Device are expected. If there is one then it’s a success.

Second acceptor also gets attachments of all “in-use” volumes, but this time it checks if their InstanceId or Device are set to something else. If there is one then it’s a failure.

The last two acceptors simply fail if volume is deleted or errored.

That’s a good example of why it’s important to read the code of libraries you use. Many interesting and handy details are skimmed out of the documentation.