« Back to all articles

Testing our infrastructure code using Kitchen & Testinfra

Written by Devhouse Spindle on 4th April 2018

• infrastructure
• kitchen
• testinfra

At Spindle we use SaltStack to setup our servers in a repeatable fashion. Salt gives us a lot of peace of mind because we don’t have ‘snowflake‘ servers where everyone and their grandmother makes adjustments to. The rule of thumb for us is: if it’s not in salt, it’s not on the server. It also gives us flexibility when we want to add new servers or even quickly replace complete servers in case of failure.

A fun way to think about this last scenario is a thought experiment previously coined by Henk in which a lunatic would break into your data center and throw your server out of a 10-story building. How long would it take you to get a new server up and running, exactly like the one lying butchered on the pavement? For us, this can be done in a matter of minutes.

Infrastructure as code

Using SaltStack to set up servers is also known as ‘Infrastructure as code’. There has been a big shift from having Bob your trusty Sysadmin know your infrastructure to having it defined in code. (Sorry Bob, time to make way for the new generation!) But the shift from manually setting up a server in your terminal to writing code to do the heavy lifting for you has not come easy. For example, it is considered bad practice for software developers to not write tests for their code, but leaving out tests is very common practice when writing SaltStack code. Coming from a development background, I had a hard time adjusting to writing code and only knowing if it worked the moment it was run on our production servers.

One of the reasons behind this is a lack of tooling provided by SaltStack. There is no easy way to test your salt states other than running it on a server and checking the result. This is not ideal at all while developing. Imagine writing code and not knowing if your syntax is correct until you’re running it in production!

Having salt in your Kitchen

While trying to find a solution to this problem, I found an awesome Rubygem called Kitchen. Kitchen gives you the tools to provision and test your infrastructure. It sets up a virtualized environment using technologies such as Docker, Vagrant, or Amazon EC2 and then allows you to run tests on it. When combined with kitchen-salt, it allows you to provision this VM using your own salt states. This was a godsend for local development as I now have the ability to replicate a complete production machine locally and run my salt states on it.

But this was only half the battle, as I still had to manually SSH into the servers and check if the salt states did what I expected them to do. This is when I found testinfra, a Python test module written on top of the popular Pytest testing framework. In the words of the testinfra developers:

With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Salt, Ansible, Puppet, Chef and so on.

Jackpot! Combining this with Kitchen we have a tool that:

• Automagically sets up a production-like environment;
• Provisions it using my own salt states;
• Can test my servers using Python tests.

Showtime!

So let’s see this in action. How do we put all these little pieces together? I’ve created this repository with the setup in it so you can follow along. Before you start, make sure you at least have the prerequisites installed.

We’ll be using Vagrant to spin up a Debian 8 box and install MySQL in it using a salt state. Then we’ll use testinfra to test whether MySQL is running and listening on the port we expect. First, we’ll create a .kitchen.yml file which will tell Kitchen how to do all of this:

#.kitchen.yml
driver:
  name: vagrant

platforms:
  - name: debian/contrib-jessie64

provisioner:
  name: salt_solo
  is_file_root: true
  local_salt_root: '.'
  state_top_from_file: true
  require_chef: false
  # These grains will be set in each suite (global grains)
  grains:
    kitchen: enabled

suites:
  - name: base
    verifier:
      name: shell
      command: py.test -v test/base

Now let’s create a simple state that installs MySQL for us and runs the service:

#salt/mysql/init.sls
mysql-server:
  pkg.installed

mysql-service:
  service.running:
    - name: mysql
    - enable: True

Next, we’ll create a test directory in our root and add a conftest.py file:

#test/conftest.py
import functools
import os
import pytest
import testinfra

test_host = testinfra.get_host('paramiko://{KITCHEN_USERNAME}@{KITCHEN_HOSTNAME}:{KITCHEN_PORT}'.format(**os.environ), ssh_identity_file=os.environ.get('KITCHEN_SSH_KEY'))

@pytest.fixture
def host():
    return test_host

The conftest.py file is loaded when the tests are run and will allow us to use the host variable as a mixin in our tests so we can connect to the Vagrant box. Now it’s time to write some tests using testinfra.

#test/base/test_mysql.py
def test_mysql_service_running(host):
    service = host.service('mysql')
    assert service.is_running
    assert service.is_enabled

def test_mysql_is_listening(host):
    assert host.socket('tcp://127.0.0.1:3306').is_listening

With all that in place, we can now run Kitchen and see it in action.

$ kitchen test
...removed some output for brevity...
============================= test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/ruben/dev/saltstack-testing, inifile:
plugins: testinfra-1.10.1
collecting ... collected 2 items

test/base/test_mysql.py::test_mysql_service_running PASSED               [ 50%]
test/base/test_mysql.py::test_mysql_is_listening PASSED                  [100%]
===================== 2 passed in 0.76 seconds =====================
-----> Kitchen is finished. (2m10.83s)

Within about 2 minutes Kitchen has created a Debian 8 box, applied the salt states, and tested the state of the box using testinfra.

Shortcomings

Although this is a big step forward when writing salt states, there are as always some use cases for which this won’t work.

One of the downsides is that the boxes run as a masterless minion, using salt-call to apply states. This means that some master functionalities like mines, custom grains, etc. won’t work. This could partially be overcome by being creative with your Jinja templating and using a grain set in your test suite to write slightly different states for the test suite. For example, I’ve set a grain called ‘kitchen’ in my Kitchen test suite, and use that to write if statements are in Jinja.

Another downside is that it’s difficult to test integration with other servers. If for example, you have a MySQL server that is part of a cluster, it would be difficult to test the connection to other MySQL nodes in the cluster. Tests are more focused on servers as a unit. One way to possibly overcome this is to run testinfra against your existing infrastructure. Testinfra provides a way to run tests on a server using a plain SSH connection so tests can be run on any server you’d like.

More information

I gave a presentation to our infrastructure circle to convince them to test our salt states and recorded it. If you want some more information you should watch this video. There’s a discussion at the end about some of the shortcomings I talked about in this post. Please share your thoughts in the comments below!