How to Build AWS VPCs with SaltStack Formulas

Describing your INFRASTRUCTURE

The ultimate is really to describe your new VPC and be able to have a tool that can create all the pieces. We are big users of SaltStack, as it supports sending many commands to public clouds to create infrastructure on your behalf. With Salt, you can describe your AWS VPC using YAML and then apply it to the region to create the same network each time it is run.

Salt supports doing this via its states system. The community helps out by providing example "formulas" that facilitate the creation of infrastructure in a cross-platform way. There is a nice AWS SaltStack formula available to do much of the heavy lifting for you. With many of these formulas, all you will really need to do is create a Salt Pillar that describes what you want built. The formula contains the states that read in the Pillar info to build the pieces.

Here is an example of a Salt Pillar that can generate the above VPC for you:

aws:
  region:
    us-west-1:
      profile:
        region: us-west-1
        keyid: [insert your keyid]
        key: [insert your key]
      vpc:
        {%- set vpc_name = 'demo-blog-vpc' %}
        {{ vpc_name }}:
          cidr_prefix: '172.20'
          vpc:
            name: {{ vpc_name }}
            cidr_block: 172.20.0.0/16
            instance_tenancy: default
            dns_support: 'true'
            dns_hostnames: 'true'
          internet_gateway:
            name: internet_gateway
          subnets:
            1:
              name: public_subnet
              az: b
              nat_gateway: true

You can test this out locally if you have Salt installed on your desktop. I'm using Homebrew to install Salt locally:‍

$ brew install saltstack

Note: If you do install via brew, the virtualenv that it creates will not have the boto and boto3 libraries installed. To fix this, you need to use the pip command from inside the SaltStack virtualenv.

$ /usr/local/Cellar/saltstack/2017.7.1/libexec/bin/pip install boto boto3

If you installed via pip into your own virtualenv, you will need to perform the same action as those libs aren't required dependencies of Salt.

Then you can checkout the formula locally:

$ git clone https://github.com/saltstack-formulas/aws-formula

Next you can create your Pillar file and a top.sls in your Pillar directory so it can apply it:

$ cd aws-formula
$ mkdir pillar

Inside the pillar directory, drop your YAML into a file called aws.sls that describes your AWS infrastructure. To get it to be used, you will also need a file called top.sls. This file also goes inside the pillar directory.

The contents of your top.sls will be:

base:
  '*':
    - aws

This tells salt to apply the aws Pillar to any server that matches. In our case, we will be calling it locally to apply the aws Pillar and build our infrastructure from our desktop.

Next, apply the aws state using the local definition of the states:

$ sudo salt-call state.sls aws --local --retcode-passthrough --file-root=$(pwd) --pillar-root=pillar

The result should be similar to this:

local:
----------
          ID: aws_vpc_demo-blog-vpc_create
    Function: boto_vpc.present
        Name: demo-blog-vpc
      Result: True
     Comment: VPC demo-blog-vpc created.
     Started: 13:11:36.940121
    Duration: 1623.927 ms
     Changes:
              ----------
              new:
                  ----------
                  vpc:
                      ----------
                      cidr_block:
                          172.20.0.0/16
                      dhcp_options_id:
                          dopt-431e1421
                      id:
                          vpc-df84d1bb
                      instance_tenancy:
                          default
                      is_default:
                          False
                      region:
                          us-west-1
                      state:
                          available
                      tags:
                          ----------
                          Name:
                              demo-blog-vpc
              old:
                  ----------
                  vpc:
                      None
----------
          ID: aws_vpc_demo-blog-vpc_create_internet_gateway
    Function: boto_vpc.internet_gateway_present
        Name: internet_gateway-demo-blog-vpc
      Result: True
     Comment: Internet gateway internet_gateway-demo-blog-vpc created.
     Started: 13:11:38.564276
    Duration: 630.312 ms
     Changes:
              ----------
              new:
                  ----------
                  internet_gateway:
                      igw-71ce3715
              old:
                  ----------
                  internet_gateway:
                      None
----------
          ID: aws_vpc_demo-blog-vpc_create_subnet_public_subnet
    Function: boto_vpc.subnet_present
        Name: public_subnet-demo-blog-vpc
      Result: True
     Comment: Subnet public_subnet-demo-blog-vpc created.
     Started: 13:11:39.194809
    Duration: 1089.917 ms
     Changes:
              ----------
              new:
                  ----------
                  subnet:
                      ----------
                      availability_zone:
                          us-west-1b
                      cidr_block:
                          172.20.1.0/24
                      id:
                          subnet-abc337cc
                      tags:
                          ----------
                          Name:
                              public_subnet-demo-blog-vpc
                      vpc_id:
                          vpc-df84d1bb
              old:
                  ----------
                  subnet:
                      None
----------
          ID: aws_vpc_demo-blog-vpc_create_nat_gateway_public_subnet
    Function: boto_vpc.nat_gateway_present
      Result: True
     Comment: Nat gateway created.
     Started: 13:11:40.284957
    Duration: 829.337 ms
     Changes:
              ----------
              new:
                  ----------
                  nat_gateway:
                      nat-0ba184bb33f531f69
              old:
                  ----------
                  nat_gateway:
                      None

Summary for local
------------
Succeeded: 4 (changed=4)
Failed:    0
------------
Total states run:     4
Total run time:   4.173 s

Now you can quickly create the exact same infrastructure in any region by changing the region in your Salt Pillar.

Keep in mind

When playing with the VPCs and all of their related objects. It can take time for some of these things to be deleted. If you are having issues deleting a subnet, for example, you will need to make sure that the NAT gateway has been deleted. Sometime this can take time and you will get DependancyViolation when attempting to remove the subnet or VPC to quickly.

Have fun building clouds!In a previous post, we created AWS VPCs using the Wizard and the CLI. If you are already using a tool like SaltStack in your infrastructure to handle other tasks, you can also use it to build cloud infrastructure. In the following example, we will build a VPC on our local machine and use AWS API via Salt to create our new network.