SaltStack for server configuration management and more

What is SaltStack (Salt)?

Salt is currently one of the fastest growing configuration management systems. It is written in Python and uses ZeroMQ to communicate nearly instantaneously with all of the servers in your infrastructure that it calls minions.

Some of the common tasks people use Salt for are:

• Orchestration

• Remote Execution

• Configuration Management

• and more!

Why we chose Salt

Over two years ago, we recognized that to maintain a high level of service and availability, we were going to need to deploy some mechanism to keep our servers (now mostly VMs) configured in a consistent manner. This is even more true today if you consider the rate security vulnerabilities are being reported across the entire infrastructure. In our case, this is a few hundred managed hosts. In the past, we had two false starts trying to use cfengine, but in the end we put together a list and evaluated the following choices:

• Puppet

• Chef

• cfengine

• bcfg2

• Salt

Given previous failed attempts at using cfengine, we ruled it out right away. bcfg2 was also a smaller project and didn't support FreeBSD as well as we would have liked, so it also got eliminated pretty quickly. Chef and Puppet are both written in Ruby, and writing your management details requires either using Ruby or a specific language for Puppet. Personally, I wasn't excited about learning more things to reduce my workload.

Salt was very new at this time, but seemed increasingly promising. The first big thing going for it was being written in Python, and we are a Python shop. The next big benefit for us was it uses YAML as it language for representing the states of servers and Jinja for the templating language. This was important because we already have Jinja experience and YAML keeps things nice and simple. The last big benefit for me was that it wasn't just configuration management, the remote execution was very important for being able to inventory, query and remotely fix servers en masse.

How we use Salt

We touch and use Salt everyday now at Six Feet Up, but one of the nice features was the ability to start small. We initially started with a simple set of states that just configured ntpd and our LDAP settings for all of our hosts. With each new service we needed to roll out, we would make Salt states and maintain those services strictly with Salt. You can eventually work your way up to never touching the servers directly, but always implementing the changes via the Salt states.

Salt now enables us to track the history of the state of our servers using git as our backend to the states. We are now leveraging the Salt pillar system (along with git) to simplify our states cross platform as we  manage and maintain FreeBSD, Ubuntu and CentOS.

When a new security vulnerability comes out, we can quickly audit our hosts by querying the infrastructure for vulnerable packages using Salt. We also use Salt for purposes beyond configuration management, such as attaching customer information, such as tech contacts, to the minions in Salt so we can query for contacts based on VM host or rack location.