Automating Mundane Tasks with Boto3

One of the tools we have been using recently is the Boto3 Library. Boto3 is a complete open source library for talking to the AWS API. Amazon actively maintains it, which makes it highly reliable. Actually, the AWS command line interface (CLI) itself is using Botocore, which provides the low-level, core functionality of Boto3.

AWS Service Optimization with Boto3

Cloud Custodian uses Boto3 to automate mundane tasks like cleaning up unused AWS resources. Cloud Custodian will delete buckets or keys that are no longer in use on S3. It will handle deleting EBS snapshots and deleting EC2 virtual machines that are not in use anymore. Cloud Custodian can also be used to optimize service usage and therefore reduce expenses by temporarily disabling virtual machines on the weekends, for example, when no one is expected to use them.

Pagination with Boto3

Most Boto3 calls that yield a large number of results support pagination.

For example, when listing objects in an S3 bucket, Boto3 will offer up to 1,000 results, and will provide a reference key. For the next request, the reference key will be sent and Boto3 will remember what was sent before and will then provide the next page and another reference key for the page after that, and so on.

This improves the performance of individual calls to AWS, but may result in more calls overall, depening on what you’re looking for. You may run up against various limits if you’re not careful with how you craft your requests.

Example of using a reference key to grab the next page of results:

import boto3

s3client = boto3.client('s3')
results = s3client.list_objects_v2(Bucket='my-bucket')
if results['IsTruncated']:
    # there are more results
    cont = results['ContinuationToken']
    more_results = s3client.list_objects_v2(
        Bucket='my-bucket',
        ContinuationToken=cont,
    )

Of course that only gets the first two pages and is a somewhat manual process. Boto3 also provides Paginator objects that hide a lot of the messy details:

import boto3

s3client = boto3.client('s3')
paginator = s3client.get_paginator('list_objects_v2')
page_iterator = paginator.paginate(Bucket='my-bucket')
for page in page_iterator:
    stuff = page['Contents']