Blog

Unleashing hundreds or even thousands of developers onto a public cloud platform is almost certainly a case study in chaos engineering. The potential for giant bills, security and compliance run-out is not a matter of if, but when.

There is a huge need to enable developers to practice more modern cloud native deployment practices, but enforce guardrails to keep them from shooting themselves in the foot. Organizations can greatly increase developer satisfaction and productivity by making the cloud easy and compliant.

Bring in the Custodian

Some cloud providers are only just now starting to roll out their own security and compliance tools. For example, Amazon has just announced their AWS Security Hub product. This deals with some issues around compliance, but not necessarily the issues of cost control.

We have started using Cloud Custodian internally on customer projects to help automate simple tasks, such as office hours for development and staging instances. There is no good reason to run these instances after office hours, since most folks have personal lives and are probably not even looking at these instances outside of work.

To get started, you need Cloud Custodian installed into a virtualenv on your local computer. That can be done like this:

$ virtualenv --python python2 custodian
$ . custodian/bin/activate
(custodian) $ pip install c7n

I use virtualenvwrapper for tools like this so I can activate it while working on any project, like so:

$ mkvirtualenv --python python2 custodian
(custodian) $ pip install c7n

Later when needing it, you can just use the workon command to enable it.

$ workon custodian
(custodian) $ custodian -h
usage: custodian [-h] {run,schema,report,logs,metrics,version,validate} ...
...

Make sure to check out the Cloud Custodian Docs for some more help and examples.

Let’s jump in

This might be easier to explain with a simple example.

Let’s say that every day we only want our testing instances available for use from noon to 4pm. So now, instead of running 24 hours a day, it only runs for 4 hours. This would reduce our AWS bill for this set of servers by over 80%.

policies:
  - name: offhours-stop-ec2
    description: |
      Shutdown all running EC2 instances that have the tag
      `maid_offhours` applied.
    mode:
      type: periodic
      schedule: "rate(1 hour)"
      role: arn:aws:iam::243886768005:role/cloud_custodian
    resource: ec2
    filters:
      - type: offhour
        default_tz: America/Indiana/Indianapolis
        offhour: 16
    actions:
      - stop

  - name: onhours-start-ec2
    description: |
      Start all stopped EC2 instances that have the tag
      `maid_offhours` applied.
    mode:
      type: periodic
      schedule: "rate(1 hour)"
      role: arn:aws:iam::243886768005:role/cloud_custodian
    resource: ec2
    filters:
      - type: onhour
        default_tz: America/Indiana/Indianapolis
        onhour: 12
    actions:
      - start

Prior to running this policy, you will need to create the IAM role that you will fill in above. I just created one called cloud_custodian. It will need to have privileges that allow it to perform the operations in your policy.

Now to install this policy, you just run the custodian command like this:

$ custodian run --output-dir=. hours.yaml

This will install the Custodian policy into AWS Lambda with the schedule provided in the policy specification. It will also make a directory per policy in your YAML file that contains the log output for the run.

By default, Cloud Custodian will observe weekends and not turn your instances back on over the weekend. You can override this by specifying weekends: false in your filter specification.

The last piece of the puzzle is to tag your EC2 instances so that this policy will apply to them. The default behavior is for instance to opt-in to the hours policy. This is controlled via the filter called opt-out. You can switch that to true and use other criteria to target your instances. With the default opt-in stance, you will need to add a tag with the key maid_offhours and a value of on. Now when the lambda runs on its schedule, your instances will turn on and off throughout the workweek.

It is possible to establish those hours via the tags on your EC2 instances. The value for the maid_offhours can be set to values other than just on. For example, you can specify alternate time zones and hours for turning the instance on and off with a value like this:

off=(m-f,19);on=(m-f,7);tz=a

Another tip, you can rename what tag Custodian looks for. If you already have some tag management in place and want to target a set of instances that have a pre-existing tag, you can change it in the filter of your policy with the tag: mycustom_hours key and value.

Nice and Tidy

The project is Open Source and is open to pull requests. Enjoy keeping your AWS environment nice and tidy. In a future post, I’ll demonstrate some of the self healing compliance capabilities of the tool.