Automatizing tasks with Fabric
We all know that good programmers are lazy. Fabric is another trick that you should have up your sleeve, if you want to be good:
Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
Although Fabric is often mentioned in web development contexts for deploying server software to remote machines, it is general enough for all kinds of remote administration tasks. To give you an analogy: Fabric is for administrative tasks as to what a Makefile is for build tasks.
Defining Fabric tasks
Fabric tasks are defined in file called
fabfile.py, which is read by the
fab command-line tool. Each function defined in this file is accessible from
the command line as a
fab parameter. So, if you have something like
def hello(): print("Hello, world")
defined in that file, you can execute it from the command line via
$ fab hello hello Done.
It’s a great way to make Python scripts easily accessible from the command line,
but not very useful on its own. What makes Fabric shine are the
run functions. You can think of these functions as local and remote wrappers
from fabric.api import local, run def hello(): cmd = 'echo $HOSTNAME' local(cmd, shell=True) run(cmd, shell=True)
will print this on my local machine:
$ fab hello [localhost] local: echo $HOSTNAME titan No hosts found. Please specify (single) host string for connection: bloerg.net [bloerg.net] run: echo $HOSTNAME [bloerg.net] out: openSUSE-121-64-minimal [bloerg.net] out: Done. Disconnecting from bloerg.net... done.
Because, I haven’t specified a host for the
run command, Fabric will ask me
for a connection string. When a connection is established, the
echo command is
executed on the bloerg.net host.
Now, that we can run shell code remotely, it’s pretty straightforward to fetch,
build and install code. Let’s assume I have to build and install ROOT and VTK on
a couple of machines from source. The following
fabfile.py will take care of
this in an easy-to-understand and composable way (security considerations set
import os from fabric.api import run, env, cd VTK_SOURCE = 'http://www.vtk.org/files/release/5.10/vtk-5.10.1.tar.gz' ROOT_SOURCE = 'ftp://root.cern.ch/root/root_v5.34.05.source.tar.gz' ROOT_CFLAGS = ' '.join(['--enable-mathmore', '--enable-minuit2', '--enable-fftw3', '--enable-xml', '--prefix=/usr/local']) BUILD_PATH = '/tmp' def wget(url): run('wget %s' % url) def untar(filename): run('tar xfz %s' % filename) def base(path): return os.path.join(BUILD_PATH, path) def fetch(): with cd(_base('.')): wget(ROOT_SOURCE) untar(os.path.basename(ROOT_SOURCE)) wget(VTK_SOURCE) untar(os.path.basename(VTK_SOURCE)) def install(): sudo('zypper in gsl gsl-devel') fetch() n_cores = run('grep -c ^processor /proc/cpuinfo') make_cmd = 'make -j%s' % n_cores with cd(_base('root')): run('./configure %s' % ROOT_CFLAGS) run(make_cmd) sudo('make install') with cd(_base('VTK5.10.1')): run('cmake . -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release') run(make_cmd) sudo('make install') sudo('ldconfig')
To change into a certain directory for the time being, we have to use the
context manager because each command will be executed in its own SSH session.
As you can imagine,
sudo will execute a command just like
run but with super
Defining and executing tasks is dead simple and the excellent documentation helps mastering more advanced topics such as roles and parallel execution.
Now, you could implement the same procedure in a shell script, copy that to each machine and execute there. But here are some reasons why this is not the best approach:
- Compared to shell languages, Python has a relatively sane syntax and a huge standard library.
- This approach doesn’t scale beyond a couple of machines. With Fabric you have a single file in a central location.
- The strong declarative style of individual tasks documents your intents.
However, Fabric is not the end of the story. For massive infrastructure automatization Chef and Puppet are probably much better suited.