Automating EC2 With Python

$ python -m venv venv
$ source venv/bin/activate
(venv)$ pip install boto3 pprint awscli
$ aws configure
AWS Access Key ID [****************3XRQ]: **************
AWS Secret Access Key [****************UKjF]: ****************
Default region name [None]:
Default output format [None]:

Creating an EC2 Instance to Work On

In this section, I am going to go over how to create an AWS region-specific boto3 session as well as instantiate an EC2 client using the active session object. Then, using that EC2 boto3 client, I will interact with that region’s EC2 instances managing startup, shutdown, and termination.

Boto3 Session and Client

At last, I can get into writing some code! I begin by creating an empty file, a Python module, called awsutils.py and at the top, I import the library boto3 then define a function that will create a region-specific Session object.

# awsutilsimport boto3def get_session(region):
return boto3.session.Session(region_name=region)
>>> import awsutils
>>> session = awsutils.get_session('us-east-1')
>>> client = session.client('ec2')
>>> import pprint
>>> pprint.pprint(client.describe_instances())
...

Retrieving EC2 Instance Details

I can also use this same describe_instances the method along with a Filter the parameter to filter the selection by tag values. For example, if I want to get my recently created instance with the Nametag with a value of 'demo-instance', that would look like this:

>>> demo = client.describe_instances(Filters=[{'Name': 'tag:Name', 'Values': ['demo-instance']}])
>>> pprint.pprint(demo)
...

Starting and Stopping an EC2 Instance

To stop the demo-instance I use the stop_instances method of the client object, which I previously instantiated, supplying it the instance ID as a single entry list parameter to the InstanceIds the argument as shown below:

>>> demo = client.describe_instances(Filters=[{'Name': 'tag:Name', 'Values': ['demo-instance']}])
>>> pprint.pprint(client.terminate_instances(InstanceIds=[instance_id]))
{'ResponseMetadata': {'HTTPHeaders': {'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 13:59:20 GMT',
'server': 'AmazonEC2',
'transfer-encoding': 'chunked',
'vary': 'Accept-Encoding'},
'HTTPStatusCode': 200,
'RequestId': '78881a08-0240-47df-b502-61a706bfb3ab',
'RetryAttempts': 0},
'TerminatingInstances': [{'CurrentState': {'Code': 32,
'Name': 'shutting-down'},
'InstanceId': 'i-0c462c48bc396bdbb',
'PreviousState': {'Code': 16, 'Name': 'running'}}]}
>>> demo = client.describe_instances(Filters=[{'Name': 'tag:Name', 'Values': ['demo-instance']}])
>>> demo['Reservations'][0]['Instances'][0]['State']
{'Code': 80, 'Name': 'stopped'}
>>> pprint.pprint(client.start_instances(InstanceIds=[instance_id]))
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '579',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 14:10:20 GMT',
'server': 'AmazonEC2'},
'HTTPStatusCode': 200,
'RequestId': '21c65902-6665-4137-9023-43ac89f731d9',
'RetryAttempts': 0},
'StartingInstances': [{'CurrentState': {'Code': 0, 'Name': 'pending'},
'InstanceId': 'i-0c462c48bc396bdbb',
'PreviousState': {'Code': 80, 'Name': 'stopped'}}]}
>>> demo = client.describe_instances(Filters=[{'Name': 'tag:Name', 'Values': ['demo-instance']}])
>>> demo['Reservations'][0]['Instances'][0]['State']
{'Code': 16, 'Name': 'running'}

Alternative Approach to Fetching, Starting, and Stopping

In addition to the EC2.Client a class that I've been working with thus far, there is also an EC2.Instance class that is useful in cases such as this one where I only need to be concerned with one instance at a time.

>>> ec2 = session.resource('ec2')
>>> instance = ec2.Instance(instance_id)
>>> instance.state
{'Code': 16, 'Name': 'running'}
>>> pprint.pprint(instance.stop())
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '579',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 14:40:20 GMT',
'server': 'AmazonEC2'},
'HTTPStatusCode': 200,
'RequestId': 'a2f76028-cbd2-4727-be3e-ae832b12e1ff',
'RetryAttempts': 0},
'StoppingInstances': [{'CurrentState': {'Code': 64, 'Name': 'stopping'},
'InstanceId': 'i-0c462c48bc396bdbb',
'PreviousState': {'Code': 16, 'Name': 'running'}}]}
>>> instance.state
{'Code': 80, 'Name': 'stopped'}
>>> pprint.pprint(instance.start())
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '579',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 14:50:20 GMT',
'server': 'AmazonEC2'},
'HTTPStatusCode': 200,
'RequestId': '3cfc6061-5d64-4e52-9961-5eb2fefab2d8',
'RetryAttempts': 0},
'StartingInstances': [{'CurrentState': {'Code': 0, 'Name': 'pending'},
'InstanceId': 'i-0c462c48bc396bdbb',
'PreviousState': {'Code': 80, 'Name': 'stopped'}}]}
>>> instance.state
{'Code': 16, 'Name': 'running'}

Creating a Backup Image of an EC2.Instance

An important topic in server management is creating backups to fall back on in the event a server becomes corrupted. In this section, I am going to demonstrate how to create an Amazon Machine Image (AMI) backup of my demo-instance, which AWS will then store in it’s Simple Storage Service (S3). This can later be used to recreate that EC2 instance, just like how I used the initial AMI to create the demo-instance.

>>> import datetime
>>> date = datetime.datetime.utcnow().strftime('%Y%m%d')
>>> date
'20190809'
>>> name = f"InstanceID_{instance_id}_Image_Backup_{date}"
>>> name
'InstanceID_i-0c462c48bc396bdbb_Image_Backup_20181221'
>>> name = f"InstanceID_{instance_id}_Backup_Image_{date}"
>>> name
'InstanceID_i-0c462c48bc396bdbb_Backup_Image_20181221'
>>> pprint.pprint(client.create_image(InstanceId=instance_id, Name=name))
{'ImageId': 'ami-00d7c04e2b3b28e2d',
'ResponseMetadata': {'HTTPHeaders': {'content-length': '242',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 15:00:20 GMT',
'server': 'AmazonEC2'},
'HTTPStatusCode': 200,
'RequestId': '7ccccb1e-91ff-4753-8fc4-b27cf43bb8cf',
'RetryAttempts': 0}}
>>> image = instance.create_image(Name=name + '_2')

Tagging Images and EC2 Instances

A very powerful, yet extremely simple, feature of EC2 instances and AMI images are the ability to add custom tags. You can add tags both via the AWS management console, as I showed when creating the demo-instance with tags Name and BackUp, as well as programmatically with boto3 and the AWS REST API.

>>> instance.tags
[{'Key': 'BackUp', 'Value': ''}, {'Key': 'Name', 'Value': 'demo-instance'}]
>>> image.create_tags(Tags=[{'Key': 'RemoveOn', 'Value': remove_on}])
[ec2.Tag(resource_id='ami-081c72fa60c8e2d58', key='RemoveOn', value='20190809')]
>>> pprint.pprint(client.create_tags(Resources=['ami-00d7c04e2b3b28e2d'], Tags=[{'Key': 'RemoveOn', 'Value': remove_on}]))
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '221',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 15:10:20 GMT',
'server': 'AmazonEC2'},
'HTTPStatusCode': 200,
'RequestId': '645b733a-138c-42a1-9966-5c2eb0ca3ba3',
'RetryAttempts': 0}}

Creating an EC2 Instance from a Backup Image

I would like to start this section by giving you something to think about. Put yourself in the uncomfortable mindset of a system administrator, or even worse a developer pretending to be a sysadmin because the product they are working on doesn’t have one (admonition… that’s me), and one of your EC2 servers has become corrupted.

>>> pprint.pprint(client.run_instances(ImageId='ami-081c72fa60c8e2d58', MinCount=1, MaxCount=1, InstanceType='t2.micro'))
...

Removing Backup Images

Ideally, I would be making backup images on a fairly frequent interval (ie, daily at the least) and along with all these backups come three things, one of which is quite good and the other two are somewhat problematic. On the good side of things, I am making snapshots of known states of my EC2 server which gives me a point in time to fall back to if things go bad. However, on the bad side, I am creating clutter in my S3 buckets and racking up charges with each additional backup I put into storage.

>>> remove_on = '20190809'
>>> images = client.describe_images(Filters=[{'Name': 'tag:RemoveOn', 'Values': [remove_on]}])
>>> remove_on = '201812022'
>>> for img in images['Images']:
... client.deregister_image(ImageId=img['ImageId'])

Terminating an EC2 Instance

Well, having covered starting, stoping, creating, and removing backup images, and launching an EC2 instance from a backup image, I am nearing the end of this tutorial. Now all that is left to do is clean up my demo instances by calling the EC2.Client class's terminate_instances and passing in the instance IDs to terminate. Again, I will use describe_instances with a filter for the name of demo-instance to fetch the details of it and grab its instance ID. I can then use it terminate_instances to get rid of it forever.

>>> demo = client.describe_instances(Filters=[{'Name': 'tag:Name', 'Values': ['demo-instance']}])
>>> pprint.pprint(client.terminate_instances(InstanceIds=[instance_id]))
{'ResponseMetadata': {'HTTPHeaders': {'content-type': 'text/xml;charset=UTF-8',
'date': 'Fri, 09 Aug 2019 15:55:20 GMT',
'server': 'AmazonEC2',
'transfer-encoding': 'chunked',
'vary': 'Accept-Encoding'},
'HTTPStatusCode': 200,
'RequestId': '78881a08-0240-47df-b502-61a706bfb3ab',
'RetryAttempts': 0},
'TerminatingInstances': [{'CurrentState': {'Code': 32,
'Name': 'shutting-down'},
'InstanceId': 'i-0c462c48bc396bdbb',
'PreviousState': {'Code': 16, 'Name': 'running'}}]}

Pulling Things Together for an Automation Script

Now that I have walked through these functionalities issuing commands one-by-one using the Python shell interpreter (which I highly recommend readers to do at least once on their own to experiment with things) I will pull everything together into two separate scripts called ec2backup.py and amicleanup.py.

# ec2backup.pyfrom datetime import datetime, timedelta
import awsutils
def backup(region_id='us-east-1'):
'''This method searches for all EC2 instances with a tag of BackUp
and creates a backup images of them then tags the images with a
RemoveOn tag of a YYYYMMDD value of three UTC days from now
'''
created_on = datetime.utcnow().strftime('%Y%m%d')
remove_on = (datetime.utcnow() + timedelta(days=3)).strftime('%Y%m%d')
session = awsutils.get_session(region_id)
client = session.client('ec2')
resource = session.resource('ec2')
reservations = client.describe_instances(Filters=[{'Name': 'tag-key', 'Values': ['BackUp']}])
for reservation in reservations['Reservations']:
for instance_description in reservation['Instances']:
instance_id = instance_description['InstanceId']
name = f"InstanceId({instance_id})_CreatedOn({created_on})_RemoveOn({remove_on})"
print(f"Creating Backup: {name}")
image_description = client.create_image(InstanceId=instance_id, Name=name)
images.append(image_description['ImageId'])
image = resource.Image(image_description['ImageId'])
image.create_tags(Tags=[{'Key': 'RemoveOn', 'Value': remove_on}, {'Key': 'Name', 'Value': name}])
if __name__ == '__main__':
backup()
# amicleanup.pyfrom datetime import datetime
import awsutils
def cleanup(region_id='us-east-1'):
'''This method searches for all AMI images with a tag of RemoveOn
and a value of YYYYMMDD of the day its ran on then removes it
'''
today = datetime.utcnow().strftime('%Y%m%d')
session = awsutils.get_session(region_id)
client = session.client('ec2')
resource = session.resource('ec2')
images = client.describe_images(Filters=[{'Name': 'tag:RemoveOn', 'Values': [today]}])
for image_data in images['Images']:
image = resource.Image(image_data['ImageId'])
name_tag = [tag['Value'] for tag in image.tags if tag['Key'] == 'Name']
if name_tag:
print(f"Deregistering {name_tag[0]}")
image.deregister()
if __name__ == '__main__':
cleanup()

Cron Implementation

A relatively simple way to implement the functionality of these two scripts would be to schedule two cron tasks on a Linux server to run them. In an example below I have configured a cron task to run every day at 11PM to execute the ec2backup.py script then another at 11:30 PM to execute the amicleanup.py script.

0 23 * * * /path/to/venv/bin/python /path/to/ec2backup.py
30 23 * * * /path/to/venv/bin/python /path/to/amicleanup.py

AWS Lambda Implementation

A more elegant solution is to use AWS Lambda to run the two as a set of functions. There are many benefits to using AWS Lambda to run code, but for this use-case of running a couple of Python functions to create and remove backup images the most pertinent are high availability and avoidance of paying for idle resources. Both of these benefits are best realized when you compare using Lambda against running the two cron jobs described in the last section.

import boto3
import os
from datetime import datetime, timedelta
def get_session(region, access_id, secret_key):
return boto3.session.Session(region_name=region,
aws_access_key_id=access_id,
aws_secret_access_key=secret_key)
def lambda_handler(event, context):
'''This method searches for all EC2 instances with a tag of BackUp
and creates a backup images of them then tags the images with a
RemoveOn tag of a YYYYMMDD value of three UTC days from now
'''
created_on = datetime.utcnow().strftime('%Y%m%d')
remove_on = (datetime.utcnow() + timedelta(days=3)).strftime('%Y%m%d')
session = get_session(os.getenv('REGION'),
os.getenv('ACCESS_KEY_ID'),
os.getenv('SECRET_KEY'))
client = session.client('ec2')
resource = session.resource('ec2')
reservations = client.describe_instances(Filters=[{'Name': 'tag-key', 'Values': ['BackUp']}])
for reservation in reservations['Reservations']:
for instance_description in reservation['Instances']:
instance_id = instance_description['InstanceId']
name = f"InstanceId({instance_id})_CreatedOn({created_on})_RemoveOn({remove_on})"
print(f"Creating Backup: {name}")
image_description = client.create_image(InstanceId=instance_id, Name=name)
image = resource.Image(image_description['ImageId'])
image.create_tags(Tags=[{'Key': 'RemoveOn', 'Value': remove_on}, {'Key': 'Name', 'Value': name}])
  • REGION with a value of the region of the EC2 instances to backup which is us-east-1 in this example
  • ACCESS_KEY_ID with the value of the access key from the section where the boto3-user was setup
  • SECRET_KEY with the value of the secret key from the section where the boto3-user was setup
import boto3
from datetime import datetime
import os
def get_session(region, access_id, secret_key):
return boto3.session.Session(region_name=region,
aws_access_key_id=access_id,
aws_secret_access_key=secret_key)
def lambda_handler(event, context):
'''This method searches for all AMI images with a tag of RemoveOn
and a value of YYYYMMDD of the day its ran on then removes it
'''
today = datetime.utcnow().strftime('%Y%m%d')
session = get_session(os.getenv('REGION'),
os.getenv('ACCESS_KEY_ID'),
os.getenv('SECRET_KEY'))
client = session.client('ec2')
resource = session.resource('ec2')
images = client.describe_images(Filters=[{'Name': 'tag:RemoveOn', 'Values': [today]}])
for image_data in images['Images']:
image = resource.Image(image_data['ImageId'])
name_tag = [tag['Value'] for tag in image.tags if tag['Key'] == 'Name']
if name_tag:
print(f"Deregistering {name_tag[0]}")
image.deregister()

Conclusion

In this article, I have covered how to use the AWS Python SDK library Boto3 to interact with EC2 resources. I demonstrate how to automate the operational management tasks to AMI image backup creation for EC2 instances and subsequent clean up of those backup images using scheduled cron jobs on either a dedicated server or using AWS Lambda.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store