Mike Slinn
Mike Slinn

Working With EC2 Spot Instances From AWS CLI

Published 2020-10-24.
Time to read: about 3 minutes.

This article is categorized under 12-Factor, AWS, Bash

AWS EC2 T2.medium spot instances cost less than 2 cents per hour for Linux and can be created very easily from the command line. They self-destruct once shut down. These powerful virtual machines can do an incredible amount of work in an hour for less than 2 cents!

Permanent storage can either be on S3 or an EBS volume, which can easily be mounted. This article shows how all this can be done via the command line. I also provide an interactive bash script for automating this process.

Once again this article uses AWS CLI, the 12-factor webapp utility I wrote called mem and jq.

Create and Import a New Keypair

I want to create a new temporary ssh keypair that will just be used for this spot instance. The name of the new key pair will be of the form ~/.ssh/rsa-YYYY-MM-DD-mm-ss.

$ mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' )
AWS_KEY_PAIR_NAME=rsa-2020-11-04-43-54

$ mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME
AWS_KEY_PAIR_FILE=~/.ssh/rsa-2020-11-04-43-54

Now we can make the keypair. AWS EC2 does not accept keys longer than 2048 bits.

$ ssh-keygen -b 2048 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa
Generating public/private rsa key pair.
  Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54
  Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub
  The key fingerprint is:
  SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
  The key's randomart image is:
  +---[RSA 4096]----+
  |  ooE  .*++o+**  |
  |   =.  ooXo=.B.. |
  |  o + o +.X.  =  |
  | o = * . =.o     |
  |. + = . S o      |
  |   o +   .       |
  |    . .          |
  |                 |
  |                 |
  +----[SHA256]-----+

The new public key will be stored in ~/.ssh/2020-11-04-43-54.pub and the new private key will be stored in ~/.ssh/2020-11-04-43-54.

Now set the permissions for the key.

$ chmod 400 $AWS_KEY_PAIR_FILE

Now we can import the key pair into AWS:

$ aws ec2 import-key-pair \
  --key-name $AWS_KEY_PAIR_NAME \
  --public-key-material fileb://${AWS_KEY_PAIR_FILE}.pub
{
  "KeyFingerprint": "c7:76:90:53:17:d0:fc:ba:45:dd:93:d2:93:03:c2:19",
  "KeyName": "2020-11-04-43-54",
  "KeyPairId": "key-092a2306ec3f4aff6"
}

Select an AMI

New AMIs become available every day. You probably want your EC2 spot instance to be created from the most recent AMI that matches your needs. For most of my work I want an Ubuntu 64-bit Intel/AMD server distribution. AWS documentation is helpful and gives us a head start in automating the AMI selection.

The following incantation sets an environment variable called AWS_AMI to the details in JSON syntax of the AMI for the most recent 64-bit Ubuntu server release for Intel/AMD architecture.

$ mem AWS_AMI "$(
  aws ec2 describe-images \
    --owners 099720109477 \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \
              "Name=state,Values=available" \
    --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
  jq -r '.[0]'
)"
AWS_AMI={
  "Architecture": "x86_64",
  "CreationDate": "2020-10-30T14:07:42.000Z",
  "ImageId": "ami-0c71ec98278087e60",
  "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "ImageType": "machine",
  "Public": true,
  "OwnerId": "099720109477",
  "PlatformDetails": "Linux/UNIX",
  "UsageOperation": "RunInstances",
  "State": "available",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "DeleteOnTermination": true,
        "SnapshotId": "snap-00bf581086dd686e5",
        "VolumeSize": 8,
        "VolumeType": "gp2",
        "Encrypted": false
      }
    },
    {
      "DeviceName": "/dev/sdb",
      "VirtualName": "ephemeral0"
    },
    {
      "DeviceName": "/dev/sdc",
      "VirtualName": "ephemeral1"
    }
  ],
  "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30",
  "EnaSupport": true,
  "Hypervisor": "xen",
  "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SriovNetSupport": "simple",
  "VirtualizationType": "hvm"
}

Now let's extract the ID of the AMI image and save it as AWS_AMI_ID.

$ mem AWS_AMI_ID "$( jq -r '.[0].ImageId' <<< "$AWS_AMI" )"
AWS_AMI_ID=ami-0c71ec98278087e60

Create an EC2 Spot Instance

For my work I often want my spot instance to be created in the same VPC subnet as my other resources, with the same security group. That is why the following environment variables are defined for the Groups and SubnetId values within the network-interfaces option, as well as the AWS region.

$ mem AWS_GROUP sg-4cbc6f35
AWS_GROUP=sg-4cbc6f35

$ mem AWS_SUBNET subnet-49de033f
AWS_SUBNET=subnet-49de033f

$ mem AWS_ZONE us-east-1c
AWS_ZONE=us-east-1c

$ mem AWS_EC2_TYPE t2.medium
AWS_EC2_TYPE t2.medium

The following creates an AWS EC2 spot instance with a public IP address and runs it. Details about the newly created spot instance are stored as JSON in AWS_SPOT_INSTANCE.

$ mem AWS_SPOT_INSTANCE "$(
  aws ec2 run-instances \
    --image-id $AWS_AMI_ID \
    --instance-market-options '{ "MarketType": "spot" }' \
    --instance-type $AWS_EC2_TYPE \
    --key-name $AWS_KEY_PAIR_NAME \
    --network-interfaces "[ {
        \"DeviceIndex\": 0,
        \"Groups\": [\"$AWS_GROUP\"],
        \"SubnetId\": \"$AWS_SUBNET\",
        \"DeleteOnTermination\": true,
        \"AssociatePublicIpAddress\": true
      } ]" \
    --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \
    jq -r .Instances[0]
)"
AWS_SPOT_INSTANCE={
  "AmiLaunchIndex": 0,
  "ImageId": "ami-0dba2cb6798deb6d8",
  "InstanceId": "i-012a54aefcd333de9",
  "InstanceType": "t2.small",
  "KeyName": "rsa-2020-11-03.pub",
  "LaunchTime": "2020-11-03T23:19:50.000Z",
  "Monitoring": {
      "State": "disabled"
  },
  "Placement": {
      "AvailabilityZone": "us-east-1c",
      "GroupName": "",
      "Tenancy": "default"
  },
  "PrivateDnsName": "ip-10-0-0-210.ec2.internal",
  "PrivateIpAddress": "10.0.0.210",
  "ProductCodes": [],
  "PublicDnsName": "",
  "State": {
      "Code": 0,
      "Name": "pending"
  },
  "StateTransitionReason": "",
  "SubnetId": "subnet-49de033f",
  "VpcId": "vpc-f16a0895",
  "Architecture": "x86_64",
  "BlockDeviceMappings": [],
  "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9",
  "EbsOptimized": false,
  "EnaSupport": true,
  "Hypervisor": "xen",
  "InstanceLifecycle": "spot",
  "NetworkInterfaces": [
      {
          "Attachment": {
              "AttachTime": "2020-11-03T23:19:50.000Z",
              "AttachmentId": "eni-attach-04feb4d36cf5c6792",
              "DeleteOnTermination": true,
              "DeviceIndex": 0,
              "Status": "attaching"
          },
          "Description": "",
          "Groups": [
              {
                  "GroupName": "testSG",
                  "GroupId": "sg-4cbc6f35"
              }
          ],
          "Ipv6Addresses": [],
          "MacAddress": "0a:6d:ba:c5:65:4b",
          "NetworkInterfaceId": "eni-09ef90920cfb29dd9",
          "OwnerId": "031372724784",
          "PrivateIpAddress": "10.0.0.210",
          "PrivateIpAddresses": [
              {
                  "Primary": true,
                  "PrivateIpAddress": "10.0.0.210"
              }
          ],
          "SourceDestCheck": true,
          "Status": "in-use",
          "SubnetId": "subnet-49de033f",
          "VpcId": "vpc-f16a0895",
          "InterfaceType": "interface"
      }
  ],
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SecurityGroups": [
      {
          "GroupName": "testSG",
          "GroupId": "sg-4cbc6f35"
      }
  ],
  "SourceDestCheck": true,
  "SpotInstanceRequestId": "sir-rrs9gm3j",
  "StateReason": {
      "Code": "pending",
      "Message": "pending"
  },
  "VirtualizationType": "hvm",
  "CpuOptions": {
      "CoreCount": 1,
      "ThreadsPerCore": 1
  },
  "CapacityReservationSpecification": {
      "CapacityReservationPreference": "open"
  },
  "MetadataOptions": {
      "State": "pending",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled"
  }
}

Now extract the EC2 spot instance id and save it in AWS_SPOT_ID.

$ mem AWS_SPOT_ID "$(
  jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE"
)"
i-012a54aefcd333de9

Wait for the instance to start.

$ aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID

Connect to the Spot Instance

In order to ssh into the spot instance we first need to discover its IP address, which is saved in AWS_SPOT_IP.

$ mem AWS_SPOT_IP "$(
  aws ec2 describe-instances \
    --instance-ids $AWS_SPOT_ID | \
  jq -r .Reservations[0].Instances[0].PublicIpAddress
)"
54.242.88.254

Now we can connect to the spot instance via ssh. The default userid for Ubuntu is ubuntu.

$ ssh ubuntu@$AWS_SPOT_IP
$ # Do your work on the spot instance now

We'll disconnect and clean up next.

Disconnect from the Spot Instance and Clean Up

Once the spot instance stops it is automatically terminated. The instance will survive a reboot, but not a halt. From a prompt on the spot instance, type:

$ sudo halt

Back in the shell that launched the spot instance, wait for the spot instance to stop before cleaning up.

$ aws ec2 wait instance-stopped --instance-ids $AWS_SPOT_ID

Delete the temporary ssh keypair we created. Copies exist on AWS and the local machine; we need to remove all of them, like this:

$ aws ec2 delete-key-pair --key-name $AWS_KEY_PAIR_NAME

$ rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub

Bash Script createEc2Spot

Source Code

This script does everything discussed above, plus it prompts the user with default values for parameters unique to each invocation. Click on the name of the script and save it this script to a directory on your PATH.

#!/bin/bash

set -e

function readWithDefault {
  >&2 printf "\n$1: "
  read -e -i "$2" VALUE
  echo "$VALUE"
}

echo "Please answer a few questions so the AWS EC2 spot instance can be created."

mem AWS_GROUP "$( readWithDefault "AWS security group" sg-4cbc6f35 )"

mem AWS_SUBNET "$( readWithDefault "EC2 subnet" subnet-49de033f )"

mem AWS_ZONE "$( readWithDefault "AWS availability zone" us-east-1c )"

mem AWS_EC2_TYPE "$( readWithDefault "EC2 machine type" t2.medium )"

mem AWS_KEY_PAIR_NAME rsa-$( date '+%Y-%m-%d-%M-%S' )

mem AWS_KEY_PAIR_FILE ~/.ssh/$AWS_KEY_PAIR_NAME

ssh-keygen -b 4096 -f "$AWS_KEY_PAIR_FILE" -P "" -N "" -t rsa
chmod 400 $AWS_KEY_PAIR_FILE

aws ec2 import-key-pair \
  --key-name $AWS_KEY_PAIR_NAME \
  --public-key-material fileb://$AWS_KEY_PAIR_FILE

echo "Searching for the latest 64-bit Intel/AMD Ubuntu AMI."
mem AWS_AMI "$(
  aws ec2 describe-images \
    --owners 099720109477 \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-????????-????????-amd64-server-????????" \
              "Name=state,Values=available" \
    --query "reverse(sort_by(Images, &CreationDate))[:1]" | \
  jq -r '.[0]'
)"
source .env

echo "Obtaining the AMI image ID."
mem -d AWS_AMI_ID "$( jq -r '.ImageId' <<< "$AWS_AMI" )"
source .env

echo "Creating the EC2 spot instance."
mem AWS_SPOT_INSTANCE "$(
  aws ec2 run-instances \
    --image-id $AWS_AMI_ID \
    --instance-market-options '{ "MarketType": "spot" }' \
    --instance-type $AWS_EC2_TYPE \
    --key-name $AWS_KEY_PAIR_NAME \
    --network-interfaces "[ {
        \"DeviceIndex\": 0,
        \"Groups\": [\"$AWS_GROUP\"],
        \"SubnetId\": \"$AWS_SUBNET\",
        \"DeleteOnTermination\": true,
        \"AssociatePublicIpAddress\": true
      } ]" \
    --placement "{ \"AvailabilityZone\": \"$AWS_ZONE\" }" | \
    jq -r .Instances[0]
)"
source .env

echo "Obtaining the EC2 spot instance ID."
mem AWS_SPOT_ID "$(
  jq -r .InstanceId <<< "$AWS_SPOT_INSTANCE"
)"
source .env

echo "Awaiting for the EC2 spot instance $AWS_SPOT_ID to enter the running state."
aws ec2 wait instance-running --instance-ids $AWS_SPOT_ID

echo "Obtaining the IP address of the new EC2 spot instance $AWS_SPOT_ID."
mem AWS_SPOT_IP "$(
  aws ec2 describe-instances \
    --instance-ids $AWS_SPOT_ID | \
  jq -r .Reservations[0].Instances[0].PublicIpAddress
)"
source .env

echo "About to ssh to the EC2 spot instance as ubuntu@$AWS_SPOT_IP.
echo "When you are done, type: sudo halt."
echo "The spot instance will then terminate and be gone forever."
echo "Any predefined resources, such as volumes that you attach will be freed."
ssh ubuntu@AWS_SPOT_IP

echo "Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state."
aws ec2 wait instance-stopped --instance-ids $INSTANCE_ID

echo "The spot instance is no longer available. Deleting its keypair."
aws ec2 delete-key-pair --key-name AWS_KEY_PAIR_NAME
rm $AWS_KEY_PAIR_FILE $AWS_KEY_PAIR_FILE.pub

Make the script executable.

$ chmod a+x createEc2Spot

Sample Usage

The script is easy to use:

$ createEc2Spot
Just a few questions before the AWS EC2 spot instance can be created.

AWS security group: sg-4cbc6f35
AWS_GROUP='sg-4cbc6f35'

EC2 subnet: subnet-49de033f
AWS_SUBNET='subnet-49de033f'

AWS availability zone: us-east-1c
AWS_ZONE='us-east-1c'

EC2 machine type: t2.medium
AWS_EC2_TYPE='t2.medium'

AWS_KEY_PAIR_NAME='rsa-2020-11-04-46-00'
AWS_KEY_PAIR_FILE='/home/mslinn/.ssh/rsa-2020-11-04-46-00'

Generating public/private rsa key pair.
  Your identification has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54
  Your public key has been saved in /home/mslinn/.ssh/rsa-2020-11-04-43-54.pub
  The key fingerprint is:
  SHA256:bQScX0UMn0xGDorxSvElMZzwMyyk7hgs2FNbshBNenA mslinn@mslinn.com
  The key's randomart image is:
  +---[RSA 4096]----+
  |  ooE  .*++o+**  |
  |   =.  ooXo=.B.. |
  |  o + o +.X.  =  |
  | o = * . =.o     |
  |. + = . S o      |
  |   o +   .       |
  |    . .          |
  |                 |
  |                 |
  +----[SHA256]-----+
{
  "KeyName": "2020-11-04-43-54",
  "KeyFingerprint": "1f:51:ae:28:bf:89:e9:d8:1f:25:5d:37:2d:7d:b8:ca"
}

Searching for the latest 64-bit Intel/AMD Ubuntu AMI.
AWS_AMI='{
  "Architecture": "x86_64",
  "CreationDate": "2020-10-30T14:07:42.000Z",
  "ImageId": "ami-0c71ec98278087e60",
  "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "ImageType": "machine",
  "Public": true,
  "OwnerId": "099720109477",
  "PlatformDetails": "Linux/UNIX",
  "UsageOperation": "RunInstances",
  "State": "available",
  "BlockDeviceMappings": [
    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "DeleteOnTermination": true,
        "SnapshotId": "snap-00bf581086dd686e5",
        "VolumeSize": 8,
        "VolumeType": "gp2",
        "Encrypted": false
      }
    },
    {
      "DeviceName": "/dev/sdb",
      "VirtualName": "ephemeral0"
    },
    {
      "DeviceName": "/dev/sdc",
      "VirtualName": "ephemeral1"
    }
  ],
  "Description": "Canonical, Ubuntu, 20.10, amd64 groovy image build on 2020-10-30",
  "EnaSupport": true,
  "Hypervisor": "xen",
  "Name": "ubuntu/images/hvm-ssd/ubuntu-groovy-20.10-amd64-server-20201030",
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SriovNetSupport": "simple",
  "VirtualizationType": "hvm"
}'

Obtaining the AMI image ID.
AWS_AMI_ID='ami-0c71ec98278087e60'

Creating the EC2 spot instance.
AWS_SPOT_INSTANCE={
  "AmiLaunchIndex": 0,
  "ImageId": "ami-0dba2cb6798deb6d8",
  "InstanceId": "i-012a54aefcd333de9",
  "InstanceType": "t2.small",
  "KeyName": "rsa-2020-11-03.pub",
  "LaunchTime": "2020-11-03T23:19:50.000Z",
  "Monitoring": {
      "State": "disabled"
  },
  "Placement": {
      "AvailabilityZone": "us-east-1c",
      "GroupName": "",
      "Tenancy": "default"
  },
  "PrivateDnsName": "ip-10-0-0-210.ec2.internal",
  "PrivateIpAddress": "10.0.0.210",
  "ProductCodes": [],
  "PublicDnsName": "",
  "State": {
      "Code": 0,
      "Name": "pending"
  },
  "StateTransitionReason": "",
  "SubnetId": "subnet-49de033f",
  "VpcId": "vpc-f16a0895",
  "Architecture": "x86_64",
  "BlockDeviceMappings": [],
  "ClientToken": "026583fb-c94e-4bca-bdd2-8dcdcaa3aae9",
  "EbsOptimized": false,
  "EnaSupport": true,
  "Hypervisor": "xen",
  "InstanceLifecycle": "spot",
  "NetworkInterfaces": [
      {
          "Attachment": {
              "AttachTime": "2020-11-03T23:19:50.000Z",
              "AttachmentId": "eni-attach-04feb4d36cf5c6792",
              "DeleteOnTermination": true,
              "DeviceIndex": 0,
              "Status": "attaching"
          },
          "Description": "",
          "Groups": [
              {
                  "GroupName": "testSG",
                  "GroupId": "sg-4cbc6f35"
              }
          ],
          "Ipv6Addresses": [],
          "MacAddress": "0a:6d:ba:c5:65:4b",
          "NetworkInterfaceId": "eni-09ef90920cfb29dd9",
          "OwnerId": "031372724784",
          "PrivateIpAddress": "10.0.0.210",
          "PrivateIpAddresses": [
              {
                  "Primary": true,
                  "PrivateIpAddress": "10.0.0.210"
              }
          ],
          "SourceDestCheck": true,
          "Status": "in-use",
          "SubnetId": "subnet-49de033f",
          "VpcId": "vpc-f16a0895",
          "InterfaceType": "interface"
      }
  ],
  "RootDeviceName": "/dev/sda1",
  "RootDeviceType": "ebs",
  "SecurityGroups": [
      {
          "GroupName": "testSG",
          "GroupId": "sg-4cbc6f35"
      }
  ],
  "SourceDestCheck": true,
  "SpotInstanceRequestId": "sir-rrs9gm3j",
  "StateReason": {
      "Code": "pending",
      "Message": "pending"
  },
  "VirtualizationType": "hvm",
  "CpuOptions": {
      "CoreCount": 1,
      "ThreadsPerCore": 1
  },
  "CapacityReservationSpecification": {
      "CapacityReservationPreference": "open"
  },
  "MetadataOptions": {
      "State": "pending",
      "HttpTokens": "optional",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled"
  }
}

Obtaining the EC2 spot instance ID.
i-012a54aefcd333de9

Awaiting for the EC2 spot instance i-012a54aefcd333de9 to enter the running state.
Obtaining the IP address of the new EC2 spot instance i-012a54aefcd333de9D.
54.242.88.254
When you are done, type: sudo halt.
The spot instance will then terminate and be gone forever.
Any predefined resources, such as volumes that you attach will be freed.

Waiting for the EC2 spot instance $AWS_SPOT_ID to enter the stopped state.

The spot instance is no longer available. Deleting its keypair.