Handling AWS Spot Instance Termination Notices

The long and short of it is that you can now query instance metadata or the DescribeSpotInstanceRequests API to find out if a spot instance has been marked for termination. At that point, you'll have two minutes to react and shut down cleanly. This gives your application time to wrap up any work in progress.

In the post, Jeff describes the mechanism in more detail, but it is basically an instance metadata endpoint that returns a 404 HTTP status code until the instance is marked for termination. AWS recommends polling this endpoint every five seconds. This can be done in a variety of ways, but one of the easiest to start experimenting with is to periodically curl the endpoint and trigger shell scripts upon notification.

An obvious first resort for polling something and running a shell script on change is crond, but cron only provides minute increments for job scheduling. So, a simple alternative is to just background a shell script (using rc.local, supervisord, daemontools, or whatever you like) that polls every five seconds and runs your shutdown hook script when it detects the notification.

#!/bin/bash
while truedo 
if [ -z $(curl -Is http://169.254.169.254/latest/meta-data/spot/termination-time | head -1 | grep 404 | cut -d   -f 2) ] then logger "Running shutdown hook."


# Call your shutdown script here.breakelse


# Spot instance not yet marked for termination. sleep 5fi done

This is a simple implementation, but it may be just what you need if you have a simple application shutdown hook. A more robust tool, Shudder, helps with graceful shutdowns in autoscaling groups; that is expected to add support for spot termination notifications soon.

We're excited about spot termination notices, and we will definitely be using them in our Fugue product.