« Back to all articles

Server provisioning at Spindle

Written by Henk Bokhoven on 1st June 2017

• infrastructure
• provisioning

Drop it like it’s hot

Imagine: grab a random server from your infrastructure, bring it up to the 10th floor, drop it out of the window and be able to recover in 10 minutes. Can you..?

Besides ordering a new piece of hardware, do you have the means to get this done in approximately 10 minutes? Well, let’s show you some of what we at Devhouse Spindle have done in trying to overcome this so-called “10th-floor test”.

When I started as an infrastructure engineer at Spindle last January, I was given the task to set up

“a procedure, assisted with tools and infrastructure, to provision, install the OS, and configure the network, in a simple, secure, and reliable fashion.”

This is how I began my journey at the “home of the nerds” in Groningen!

Union of the snake

Before making a definitive choice, we made a small overview of tools to be considered right for the job. Here’s a shortlist of what we came up with:

• FAI (“Fully Automatic Installation”), written in Perl.
• Foreman, written in Ruby and seems to be biased towards Puppet.
• Cobbler, written in Python and originally targeted at Red Hat / CentOS.
• RackHD, a modular, API-based technology stack made by Dell EMC.
• Razor, written in Ruby, from the creators of Puppet.

Most – if not all – of these tools combine managed DHCP, DNS, TFTP, and PXE-based unattended installations. In the end, we choose Cobbler for a number of reasons:

[1] Written in Python: since Spindle’s main dev language is Python this was obvious.
[2] Experience: one of the other Infra-engineers had some positive experience with Cobbler.
[3] Easy to use and learn, no full kitchen sink included.
[4] Flexibility: Cobbler can be managed from the command line (CLI), through a web interface, or through XMLRPC API-calls (which opens the way to use third party tools like HashiCorp’s Terraform).
[5] Templating: with the Python Cheetah template engine.
[6] Debian: Cobbler can be installed on Debian too (using version 2.6.x – version 2.8.x won’t work yet, due to incompatibility issues).

Pixies

The core of a server provisioning system (like Cobbler) consists of these separate elements:

PXE (Preboot eXecution Environment, pronounced as pixie), for booting computers over a network;
DHCP, a dynamical distribution of network configuration parameters, e.g. IP-address; netmask; default gateway, etc.;
TFTP, hands the booting nodes a kernel and initrd-file (ramdisk);
HTTP(S), which serves installation files through a web-server;
DNS is used for name resolving.

Cobbler makes up the glue between these elements. Plus, it adds some additional functionality like profiles, uses different Linux-distributions and adds unique systems.

Last but not least are the kickstart- and preseed-files: kickstarts for Red Hat-based systems; preseeds for Ubuntu / Debian-based ones. When a Linux-host is booting from installation media for the first time, you will have to answer a certain amount of questions about the system’s hostname, its network settings, partitioning of the disk(s), etc. When properly crafted with predefined answers, these kickstart- / preseed-files give you the power to do a fully unattended installation of a base system. So, no questions asked (NQA) about the partitioning of the disk(s), the basic packages, the default root password, etc.

While setting up Cobbler at Spindle, I settled for “dnsmasq” for combined DHCP and DNS, instead of both ISC’s BIND and DHCPd. So “one daemon to rule them both”. This will simplify things in the long run for (new) engineers not familiar with the later two. For HTTPS I dropped the default Apache webserver in front of Cobbler and configured nginx with uWSGI instead.

Salt-n-Pepa’s here

Spindle is moving away from Puppet for configuration management: since last year we are using SaltStack instead. At provisioning time the Salt repository and a so-called “salt-minion” are being installed from a snippet in Cobbler:

#set breed = $getVar('breed','')
#set os_version = $getVar('os_version','')
#set release = $getVar('release','')

--- 8< --- cut ---

### Install salt-minion from official SaltStack-repo:
apt-key adv --fetch-keys http://repo.saltstack.com/apt/$breed/$release/amd64/latest/SALTSTACK-GPG-KEY.pub
echo "deb http://repo.saltstack.com/apt/$breed/$release/amd64/latest $os_version main" > /etc/apt/sources.list.d/saltstack.list

### Finishing up
apt-get update
apt-get -y install salt-minion

Once a newly provisioned server has rebooted for the first time, an infra engineer can apply a so called Salt server role (e.g. be an Asterisk server) to the new system and run a “highstate”. Our new server will be automagically configured with the role it had appointed.

In the end, new servers are re-installed with both a Spindle-flavored Debian Linux AND a fully functional application, all within 10 minutes!

This is where Borat would say: “Great success!”

Paint it black (or blue)

Finally, I unleashed some rusty CSS skills (pun intended) to brighten up the rather dull default Cobbler-style and came up with this instead:

These colors reflect the default Spindle brand style we use elsewhere e.g. in presentations, on our wiki pages, and in Slack.

New kids on the block

Before ending this post I’d like to show you some new tools in the server provisioning landscape that have arisen over the last months, so let’s name the two most interesting ones.

CoreOS Matchbox is one of these shiny new tools. You can use it to set up CoreOS Clusters, e.g. for a Kubernetes platform. Old fashioned PXE is boosted with chain loading to modern iPXE and gRPC for HTTP/2 TLS service communication. If you want to read more about CoreOS Matchbox, here’s a good review.

The other youngster is Digital Rebar Provision, written in Golang and presented as “a simple yet complete API-driven DHCP/PXE/TFTP provisioning system”. They even made the bold statement of being a “Cobbler replacement”! Their concept and the documentation seems to be very promising so far, so let’s keep an eye out on this one.