Lazy automation

A solution to automate regular, routine, somewhat boring tasks.

Disclaimer: I am not much of a Python programmer by any means, but I have used Ansible for a number of years and using Invoke, and now runner too, has made my ansible workflow better. These are some jumbled notes to share some ideas :)

Tools

The programs I am using:

Invoke - a Python task execution tool & library.
Ansible - an IT automation and configuration management tool.
Ansible-runner - a tool and python library that helps when interfacing with Ansible directly or as part of another system.

I prefer to run all my code and tools inside a Virtual Machine, configured with Vagrant, or in a Docker container (configured with a Dockerfile, or with docker-compose).

Inside that VM or Container I use virtual python environment.

the requirements.txt with the tools:

invoke >= 1.2.0
ansible
molecule
python-vagrant
redis
docker
dnspython
ansible-runner
ansible-cmdb
ansible-lint

A setup.sh to install the tools and activate the virtual env:

#!/usr/bin/env bash
sudo apt install python3-pip
pip3 install --user virtualenv
pip3 install --user setuptools
~/.local/bin/virtualenv --system-site-packages ~/virtenv
source ~/virtenv/bin/activate
pip3 install -r requirements.txt

You can avoid typing those commands if you run them as part of Vagrant provisioning or in a Dockerfile. This post is not really about platform-independent software or code portability though, its about how useful Invoke is.

invoke

When you run invoke by default it will look for a file called tasks.py. In my ~/.bash_aliases file I have this line that will tell Invoke to look for my tasks.py in this directory, so I can list + run my tasks from any directory.

alias invoke='invoke --search-root=/mnt/code'

What I love about Invoke:

You can easily chain a bunch of commands together. This lets you integrate some CI/CD type tasks into your local workflow without adding any friction, like using a linter before running some code.
Lots of command line programs take many long and complicated parameters (like ansible), sometimes you can't quite remember them all if it has been sometime (how did I run that tool to do that setup? Hunting in ~/.bash_history or man pages to find out?).
Often you want to run the same lots of the same commands but don't want to type them all out (RSI sucks - are you scrolling up your command history to push enter again?).
Along with your code can provide someone with a command like invoke deploy integration, which seems like magic - avoid doing copy/paste of commands from a deploy_code.txt

So I like to include a tasks.py in my Ansible code directory - but any code would probably play nicely with Invoke.

If you wanted to install Invoke you could do so with sudo apt install python3-invoke, or brew install pyinvoke if that is your your thing, but the recommended way to install the latest stable release is to use pip.

code

This is a snippet of my /mnt/code/tasks.py file:

"""
Regular tasks
"""

import os
import ansible_runner
from invoke import task, run

user = os.getuid()
if user == 0:
    print ("Do not run as root.")
    quit()

print('\n --===[ Ansible Control machine tasks ]===-- \n')


#
# tasks for all/remote hosts
#

@task
def ansible_ping(c, hostname):
    """ Ansible Ping a host in your inventory. Example: invoke ansible-ping proxy001 """
    c.run("ansible %s -m ping;" % hostname)


#
# tasks for localhost (on this VM/Container)
#

# run the playbook for the Ansible Controler machine with Ansible-runner py interface
# https://ansible-runner.readthedocs.io/en/latest/python_interface.html
@task
def deployer_playbook(c):
    """ Run the playbook that configures this control machine with ansible-runner python module. """
    print("checking playbook-ansible-controller.yml")
    c.run('cd /mnt/code/ansible/ && ansible-lint playbook-ansible-controller.yml -v', pty=True)
    print("running playbook-ansible-controller.yml")
    r = ansible_runner.run(private_data_dir='/mnt/code/ansible',
        inventory='/mnt/code/ansible/localhost.ini',
        playbook='playbook-ansible-controller.yml')
    print("{}: {}".format(r.status, r.rc))
    print("Final status:")
    print(r.stats)

@task
def get_my_ip(c):
    """ Find out your IPs """
    print("My WAN IP address is: ")
    c.run('dig +short myip.opendns.com @resolver1.opendns.com', pty=False)
    print("My Internal IP is: ")
    c.run("ifconfig eth0 | grep 'inet' | cut -d: -f2 | awk '{ print $2}'", pty=False)


# These tasks run a role from "/mnt/code/ansible/roles/ <role name> /" on this local system (ansible control machine) with ansible runner or playbook.

# https://ansible-runner.readthedocs.io/en/latest/standalone.html
@task
def deployer_single_runner(c, rolename):
    """ Run a role on ansible control machine with ansible-runner """
    print("running the role %s with ansible-runner" % rolename)
    c.run('ansible-runner run --inventory /mnt/code/ansible/localhost.ini -r %s -v --roles-path /mnt/code/ansible/roles/ /mnt/code/ansible' % rolename, pty=False)

# https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html
def deployer_single_role(c, rolename):
    """ Run a role on ansible control machine with ansible-playbook """
    print("Attempting to apply %s role to localhost" % rolename)
    c.run('cd /mnt/code/ansible/ && ansible-playbook -i ./localhost.ini -e "runtherole=%s" -v playbook-run-single-role.yml' % rolename, pty=True)

The playbook-run-single-role.yml file I keep in /mnt/code/ansible/ to run a single role:

---
- name: single role
  hosts: all
  gather_facts: True
  roles:
    - "{{ runtherole }}"

using invoke

Running invoke -l to list my tasks:

(virtenv) dockeruser@eb26acd3f23b:~$ invoke -l

 --===[ Ansible Control machine tasks ]===--

Available tasks:

  ansible_ping             Ansible Ping a host in your inventory. Example: invoke ansible-ping proxy001
  deployer-playbook        Run the playbook that configures this control machine with ansible-runner python module.
  deployer-single-role     Run a role on ansible control machine with ansible-playbook
  deployer-single-runner   Run a role on ansible control machine with ansible-runner
  get-my-ip                Find out your IPs

Two different ways to run my Ansible role to install Rclone:

(virtenv) dockeruser@eb26acd3f23b:~$ invoke deployer-single-role rclone

(virtenv) dockeruser@eb26acd3f23b:~$ invoke deployer-single-runner rclone

Now I am only regularly entering 3 commands to run a task I do often, but these are just a couple of examples. You can easily have your own portable aliases with Invoke.

ansible and ansible-runner

Ansible is a great tool, the output of ansible is not so great. In Ansible.cfg you can set log_path=/var/log/ansible.log, but I find the output from runner much more useful.

Ansible-Runner is a component of AWX and Tower, and it is "responsible for running ansible and ansible-playbook tasks and gathers the output from it."

Using runner allows us to capture:

the cli flags ansible was called with
all stdout
the exit code status
the fact-cache of the system as it was

All information is logged in /mnt/code/ansible/artifacts/ < UUID >/ after a run.

There are great possibilities to do some of your own reporting by sending the output to an external interface. If you are looking for a web interface to Ansible I would suggest using Ara.

If you wanted to provide ansible with SSH-Keys, passwords, environment variables and everything to execute without any human interaction then Runner can also do this - read about "the runner input directory".

Related notes

Automation for the people - https://gist.github.com/cube-drone/9e07a36aa63624ca2dda75a1367a53c6
What is ansible-runner about - https://devops.stackexchange.com/questions/4847/what-is-ansible-runner-about
If you are developing Ansible roles you need to checkout Molecule - https://molecule.readthedocs.io/en/latest/
Three excellent Python tools to automate repetitive tasks https://pyvideo.org/pycon-us-2019/break-the-cycle-three-excellent-python-tools-to-automate-repetitive-tasks.html

More updates to come, but I hope this has been useful for someone.

craig-m/readme.md