An advanced method to define tasks in fabric

One of the best things about Fabric is its simplicity. For example, in order to execute a command on a remote host, just define a task:

@task
def simple():
    """
    show uname -a
    """
    run('uname -a')

Then, call it with a fab command.

$ fab simple
[default] Executing task 'simple'
[default] run: uname -a
[default] out: Linux fabric-sample 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[default] out:

When I started to use Fabric, its simplicity of just listing commands helped me to easily transfer shell scripts or procedure manuals that I used to write.

In addition to the method to define a task with a task decorator, the document also provides a method to use Task subclasses. Let’s borrow an example from the document. If you define a subclass:

class MyTask(Task):
    name = "deploy"
    def run(self, environment, domain="whatever.com"):
        run("git clone foo")
        sudo("service apache2 restart")

instance = MyTask()

It is equivalent to below which is defined with a decorator.

@task
def deploy(environment, domain="whatever.com"):
    run("git clone foo")
    sudo("service apache2 restart")

Internally, the task decorator also creates a Task subclass called WrappedCallableTask. If you just look at the above example, the decorator method seems shorter and simpler. However, the subclass method will be useful when you want to:

• share a task across multiple services
• create a number of programmed tasks
• effectively switch between environments

Let’s look at how these functions work.

Sharing Tasks Across Multiple Services

When sharing middleware or application frameworks within a team, you may often want to use a task used for one service in other services. At Nulab, we are operating four services, Backlog, Cacoo, Typetalk and Nulab Account, and there are some tasks we want to share in those services.

In such cases, you can create a library with defined task classes.

class DeployTask(Task):
    """
    deploy application
    """
    name = 'deploy'

    def run(self, *args, **kwargs):
        execute('tomcat7_stop')
        self.pre_deploy()        
        print yellow('do deploy')
        self.post_deploy()
        execute('tomcat7_start')        

    def pre_deploy(self):
        pass

    def post_deploy(self):
        pass

Then, in every service, define the task like this:

deploy = DeployTask()

You will be able to repeat the task without copying and pasting it. If one service has specific processing, you can avoid writing duplicates by simply extending a class.

class CleanAndDeployTask(DeployTask):
    """
    clean work directory before deploy
    """
    def pre_deploy(self):
        print yellow('clean up /var/lib/tomcat7/work/')
        sudo('rm -fr /var/lib/tomcat7/work/*')

deploy = CleanAndDeployTask()

This is an advantage of object-oriented programming.

fabfile Gets Messy with Start/Stop Tasks

One of the major usages of Fabric is to start, stop and restart middleware. These tasks are simple yet they can sometimes complicate a fabfile if there are too many of them, e.g.

@task
def nginx_stop():
    """
    stop nginx
    """
    sudo('service nginx stop')

@task
def nginx_start():
    """
    start nginx
    """
    sudo('service nginx start')


@task
def nginx_restart():
    """
    restart nginx
    """
    sudo('service nginx restart')

There are only three tasks defined in this example. However, if you also have other types of middleware like memcached and tomcat, the fabfile will get messy with a number of similar tasks. As a result, you may accidentally overwrite a task or forget to copy and paste it when adding a new task.

In such a case, you can use a utility method that auto-generates tasks. In the following example, a wrapper function to generate a Task class is obtained by calling a task decorator as a function.

class Service(object):

    def __init__(self, name):
        self.name = name

    def start(self):
        sudo('service %s start' % self.name)

    def stop(self):
        sudo('service %s stop' % self.name)

    def restart(self):
        sudo('service %s restart' % self.name)

    def get_methods(self):
        return [self.start, self.stop, self.restart]

def create_tasks(name, namespace):

    service = Service(name)
    for f in service.get_methods():
        fname = f.__name__
        # task description
        f.__func__.__doc__ = 'service %s %s' % (name, fname)
        # task name
        task_name = '%s_%s' % (name, fname)
        # call task decorator as function to generate WrappedCallableTask
        wrapper = task(name=task_name)
        rand = '%d' % (time.time() * 100000)
        namespace['task_%s_%s' % (task_name, rand)] = wrapper(f)

Now, you can just call “create_tasks” method as follows, and stop/start/restart tasks will be auto-generated.

create_tasks('nginx', globals())
create_tasks('postgresql', globals())
create_tasks('tomcat7', globals())

They are now recognized as tasks.

$ fab -l
Available commands:

    nginx_restart       service nginx restart
    nginx_start         service nginx start
    nginx_stop          service nginx stop
    postgresql_restart  service postgresql restart
    postgresql_start    service postgresql start
    postgresql_stop     service postgresql stop
    tomcat7_restart     service tomcat7 restart
    tomcat7_start       service tomcat7 start
    tomcat7_stop        service tomcat7 stop

Switching Tasks for Each Environment

You may often use multiple environments like production and staging environments in a single service. In such cases, the following technique is commonly used.

$ fab switch:production sometask

As shown above, a task to switch environments is called first and then followed by the task you actually want to execute.  In a “switch” task, you can change environment specific values like host names defined in env.roles or other variables like AWS region that you have set for your own purpose.

If operations performed in “sometask” are the same in both environments, you probably will not encounter any difficulties. If there is a difference in operations between staging and production environments, you can usually use “if condition” to switch the operations like this.

@task
def sometask():
    if env.run_environment == 'production':
        production_run()
    else:
        stage_run()

However, the presence of similar processing in each task will make a program complex and difficult to read if:

• there are many tasks that need switching for each environment, or
• you want to always perform the same operation in specific environments.

If you create FactoryTask to wrap function calling, you will have the advantage of being able to:

• avoid writing “if-condition” in all tasks
• gather all the operations that are frequently used in a specific environment in FactoryTask.

class FactoryTask(Task):

    def __init__(self, runners, desc=None, *args, **kwargs):
        super(FactoryTask, self).__init__(*args, **kwargs)
        self.runners = runners
        if desc is not None:
            self.__doc__ = desc  

    def run(self, *args, **kwargs):

        if 'run_environment' not in env:
            print red('this task should be run under some environment')
            return

        runner = self.runners.get(env.run_environment)
        if runner is None:
            print red('%s is not supported this task' % env.run_environment)
        else :
            print yellow('run %s under %s environment' % (self.name, env.run_environment))
            # log in production environment
            if env.run_environment == 'production':
                args_str = ','.join('{}: {}'.format(*k) for k in enumerate(args))
                kwargs_str = ','.join('{}: {}'.format(*k) for k in kwargs.items())
                with settings(hide('running'), warn_only=True):
                    run('logger -t fabric %s args:[%s] kwargs:[%s]' % (self.name, args_str, kwargs_str))
            runner(*args, **kwargs)

def prod_run(a, b):
    print '%s %s' % (a, b)

def stage_run(a):
    print '%s' % a

@task
def switch(production=None):
    """
    switch task
    """
    env.run_environment = 'production' if production is not None else 'stage'

show_args = FactoryTask({'production' : prod_run, 'stage' : stage_run }, 'show args', name='show_args')

To call the above:

# For staging environment
$ fab switch show_args:a='1'
# For production environment
$ fab switch:production show_args:a='1',b='2'

In this case, to illustrate the difference, I used different arguments for production and staging and left the execution log with a logger for production.

As mentioned earlier, the method to subclass Task will allow you to make a program easy to understand without copying and pasting it, thus making it handy to:

• share a task across multiple services,
• create a number of programmed tasks
• effectively switch between environments.

All the sample codes provided in this post are available at:

• https://github.com/nulab/fabric-sample

All the tasks provided in this post can be executed after installing the required tools (Vagrant, Ansible and Fabric) listed in the above link and run the following commands. Try them and see how it works for yourself!

$ git clone https://github.com/nulab/fabric-sample.git
$ cd fabric-sample
$ vagrant up
$ vagrant ssh-config > ssh.config

Another advantage of Fabric is its capacity for these advanced functions in spite of its simplicity. It’s wonderful to have tools like Vagrant and Ansible which enable us to easily share a testing environment for this kind of tools.