Ansible Summary.md

Ansible

It's an automation language, engine and orchestration tool developed by Red hat written in Python using YAML format for writing instructional scripts, affectionately known as "Ansible playbook".

It has an automation engine that runs Ansible playbooks. Cloud provider is present in the ecosystem (AWS, Azure, Google, DigitalOcean, OVH, etc…).

Pros

It is an agent-less tools In most scenarios, it use ssh as a transport layer. In some way you can use it as 'bash on steroids'
It is very easy to start. If you are familiar with ssh concept - you already know Ansible (ALMOST).
It executes 'as is' - other tools (salt, puppet, chef - might execute in different scenario than you would expect)
Documentation is at the world-class standard!
Writing your own modules and extensions is fairly easy.
Ansible AWX is the open source version of Ansible Tower we have been waiting for, which provides an excellent UI.

Cons

It is an agent-less tool - every agent consumes up to 16MB ram - in some environments, it may be noticable amount.
It is agent-less - you have to verify your environment consistency 'on-demand' - there is no built-in mechanism that would warn you about some change automatically (this can be achieved with reasonable effort)
Official GUI - Ansible Tower - is great but expensive.
There is no 'small enterprice' payment plan, however Ansible AWX is the free open source version we were all waiting for.

Modules => Tasks => Plays => Playbook.

Let us see buidlding blocks about Ansible:

Inventory: It is a simple list of hosts; /etc/ansible/hosts
Playbook or commands: Execution of desired actions.
API: Cloud Services
SSH, WinRM: Work via transport layer.
Ansible roles: Special kind of playbook that are fully self-contained with tasks, variables, configurations templates and other supporting files.
Ansible Galaxy: Like git repo,collection of playbooks available as open source.
Ansible Tower: Ansible Tower is a web-based interface for managing Ansible.
Ansible AWX: It is the open source version of Ansible Tower & a web-based solution that makes Ansible even more easy to use for IT teams of all kinds. It’s designed to be the hub for all of your automation tasks. Simple, Powerful and agentless (unlike Puppet).

Module

A program (usually python) that executes, does some work and returns proper JSON output. This program performs specialized task/action (like manage instances in the cloud, execute shell command). The simplest module is called ping - it just returns a JSON with pong message.

Example of modules:

Module: ping - the simplest module that is useful to verify host connectivity
Module: shell - a module that executes shell command on a specified host(s).

$ ansible -m ping all
$ ansible -m shell -a 'date; whoami' localhost #hostname_or_a_group_name

Module: command - executes a single command that will not be processed through the shell, so variables like $HOME or operands like |` `; will not work. The command module is more secure, because it will not be affected by the user’s environment. For more complex commands - use shell module.

$ ansible -m command -a 'date; whoami' # FAILURE
$ ansible -m command -a 'date' all
$ ansible -m command -a 'whoami' all

Module: file - performs file operations (stat, link, dir, ...)
Module: raw - executes a low-down and dirty SSH command, not going through the module subsystem (useful to install python2.7)

Task

Execution of a single Ansible module is called a task. The simplest module is called ping as you could see above.

Another example of the module that allow you to execute command remotly on multiple resources is called shell. See above how you were using them already.

Playbook

Execution plan written in a form of script file(s) is called playbook. Playbook consist of multiple elements -

a list (or group) of hosts that 'the play' is executed against
task(s) or role(s) that are going to be executed
multiple optional settings (like default variables, and way more)

Playbook script language is YAML. You can think that playbook is very advanced CLI script that you are executing.

Example of the playbook

This example-playbook would execute (on all hosts defined in inventory) two tasks:

ping that would return message pong
shell that execute three commands and return the output to our terminal

- hosts: all

  tasks:
    - name: "ping all"
      ping:

    - name: "execute a shell command"
      shell: "date; whoami; df -h;"

Run the playbook with the command:

$ ansible-playbook path/name_of_the_playbook.yml

Note: Example playbook is explained in the next chapter: 'Roles'

Inventory

Inventory is a set of objects or hosts, against which we are executing our playbooks or single tasks via shell commands. For these few minutes, let's assume that we are using the default ansible inventory (which in Debian based system is placed in /etc/ansible/hosts).

localhost

[some_group]
hostA.mydomain.com
hostB.localdomain
1.2.3.4

[a_group_of_a_groups:children]
some_group
some_other_group

Additional Reading.

ansible-roles (a 'template-playbooks' with right structure)

You already know that the tasks (modules) can be run via CLI. You also know the playbooks - the execution plans of multiple tasks (with variables and logic).

A concept called role was introduced for parts of the code (playbooks) that should be reusable.

Role is a structured way to manage your set of tasks, variables, handlers, default settings, and way more (meta, files, templates). Roles allow reusing the same parts of code in multiple playbooks (you can parametrize the role 'further' during its execution). Its a great way to introduce object oriented management for your applications.

Role can be included in your playbook (executed via your playbook).

- hosts: all

  tasks:
      - name: "ping all"
        ping:
      - name: "execute a shell command"
        shell: "date; whoami; df -h;"

  roles:
      - some_role
      - { role: another_role, some_variable: 'learnxiny', tags: ['my_tag'] }

  pre_tasks:
      - name: some pre-task
        shell: echo 'this task is the last, but would be executed before roles, and before tasks'

For remaining examples we would use additional repository

This example install ansible in virtualenv so it is independend from a system. You need to initialize it into your shell-context with source environment.sh command.

We are going to use this repository with examples: https://github.com/sirkubax/ansible-for-learnXinYminutes

$ # The following example contains a shell-prompt to indicate the venv and relative path
$ git clone [email protected]:sirkubax/ansible-for-learnXinYminutes.git
user@host:~/$ cd ansible-for-learnXinYminutes
user@host:~/ansible-for-learnXinYminutes$ source environment.sh
$
$ # First lets execute the simple_playbook.yml
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_playbook.yml

Run the playbook with roles example

$ source environment.sh
$ # Now we would run the above playbook with roles
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_role.yml

Role directory structure

roles/
   some_role/
     defaults/      # contains default variables
     files/         # for static files
     templates/     # for jinja templates
     tasks/         # tasks
     handlers/      # handlers
     vars/          # more variables (higher priority)
     meta/          # meta - package (role) info

Role Handlers

Handlers are tasks that can be triggered (notified) during execution of a playbook, but they execute at the very end of a playbook. It is the best way to restart a service, check if the application port is active (successful deployment criteria), etc.

Get familiar with how you can use roles in the simple_apache_role example

playbooks/roles/simple_apache_role/
├── tasks
│   └── main.yml
└── templates
    └── main.yml

ansible - variables

Ansible is flexible - it has 21 levels of variable precedence. read more For now you should know that CLI variables have the top priority. You should also know, that a nice way to pool some data is a lookup

Lookups

Awesome tool to query data from various sources!!! Awesome! query from:

pipe (load shell command output into variable!)
file
stream
etcd
password management tools
url

# read playbooks/lookup.yml
# then run
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/lookup.yml

You can use them in CLI too

ansible -m shell -a 'echo "{{ my_variable }}"' -e 'my_variable="{{ lookup("pipe", "date") }}"' localhost
ansible -m shell -a 'echo "{{ my_variable }}"' -e 'my_variable="{{ lookup("pipe", "hostname") }}"' all

# Or use in playbook

(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/lookup.yml

Register and Conditional

Register

Another way to dynamically generate the variable content is the register command. Register is also useful to store an output of a task and use its value for executing further tasks.

(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/register_and_when.yml

---
- hosts: localhost
  tasks:
   - name: check the system capacity
     shell: df -h /
     register: root_size

   - name: debug root_size
     debug:
        msg: "{{ root_size }}"

   - name: debug root_size return code
     debug:
       msg:  "{{ root_size.rc }}"

# when: example

   - name: Print this message when return code of 'check the system capacity' was ok
     debug:
       msg:  "{{ root_size.rc }}"
     when: root_size.rc == 0
...

Conditionals - when:

You can define complex logic with Ansible and Jinja functions. Most common is usage of when:, with some variable (often dynamically generated in previous playbook steps with register or lookup)

---
- hosts: localhost
  tasks:
   - name: check the system capacity
     shell: df -h /
     when: some_variable in 'a string'
  roles:
   - { role: mid_nagios_probe, when: allow_nagios_probes }
...

ansible - tags, limit

You should know about a way to increase efficiency by this simple functionality

LIMIT

You can limit an execution of your tasks to defined hosts

ansible-playbook playbooks/simple_playbook.yml --limit localhost

--limit my_hostname
--limit groupname
--limit some_prefix*
--limit hostname:group #JM

Templates

Templates are a powerful way to deliver some (partially) dynamic content. Ansible uses Jinja2 language to describe the template.

Some static content

{{ a_variable }}

{% for item in loop_items %}
    this line item is {{ item }}
{% endfor %}

Jinja may have some limitations, but it is a powerful tool that you might like.

Please examine this simple example that installs apache2 and generates index.html from the template "playbooks/roles/simple_apache_role/templates/index.html"

$ source environment.sh
$ # Now we would run the above playbook with roles
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_role.yml --tags apache2

Jinja2 CLI

You can use the jinja in the CLI too

ansible -m shell -a 'echo {{ my_variable }}` -e 'my_variable=something, playbook_parameter=twentytwo" localhost

In fact - jinja is used to template parts of the playbooks too

# check part of this playbook: playbooks/roles/sys_debug/tasks/debug_time.yml
- local_action: shell date +'%F %T'
  register: ts
  become: False
  changed_when: False

- name: Timestamp
  debug: msg="{{ ts.stdout }}"
  when: ts is defined and ts.stdout is defined
  become: False

Jinja2 filters

Jinja is powerful. It has many built-in useful functions.

# get first item of the list
{{ some_list | first() }}
# if variable is undefined - use default value
{{ some_variable | default('default_value') }}

ansible-vault

To maintain infrastructure as code you need to store secrets. Ansible provides a way to encrypt confidential files so you can store them in the repository, yet the files are decrypted on-the-fly during ansible execution.

The best way to use it is to store the secret in some secure location, and configure ansible to use during runtime.

# Try (this would fail)
$ ansible-playbook playbooks/vault_example.yml

$ echo some_very_very_long_secret > ~/.ssh/secure_located_file

# in ansible.cfg set the path to your secret file
$ vi ansible.cfg
  ansible_vault_password_file = ~/.ssh/secure_located_file

#or use env
$ export ANSIBLE_VAULT_PASSWORD_FILE=~/.ssh/secure_located_file

$ ansible-playbook playbooks/vault_example.yml

  # encrypt the file
$ ansible-vault encrypt path/somefile

  # view the file
$ ansible-vault view path/somefile

  # check the file content:
$ cat path/somefile

  # decrypt the file
$ ansible-vault decrypt path/somefile

dynamic inventory

You might like to know, that you can build your inventory dynamically. (For Ansible) inventory is just JSON with proper structure - if you can deliver that to ansible - anything is possible.

You do not need to reinvent the wheel - there are plenty of ready to use inventory scripts for most popular Cloud providers and a lot of in-house popular usecases.

AWS example

$ etc/inv/ec2.py --refresh
$ ansible -m ping all -i etc/inv/ec2.py

ansible profiling - callback

Playbook execution takes some time. It is OK. First make it run, then you may like to speed things up. Since ansible 2.x there is built-in callback for task execution profiling.

vi ansible.cfg
# set this to:
callback_whitelist = profile_tasks

facts-cache and ansible-cmdb

You can pull some information about your environment from another hosts. If the information does not change - you may consider using a facts_cache to speed things up.

vi ansible.cfg

# if set to a persistent type (not 'memory', for example 'redis') fact values
# from previous runs in Ansible will be stored.  This may be useful when
# wanting to use, for example, IP information from one group of servers
# without having to talk to them in the same playbook run to get their
# current IP information.
fact_caching = jsonfile
fact_caching_connection = ~/facts_cache
fact_caching_timeout = 86400

I like to use jsonfile as my backend. It allows to use another project ansible-cmdb (project on github) that generates a HTML page of your inventory resources. A nice 'free' addition!

Debugging ansible [chapter in progress]

When your job fails - it is good to be effective with debugging.

Increase verbosity by using multiple -v [ -vvvvv]
If variable is undefined - grep -R path_of_your_inventory -e missing_variable
If variable (dictionary or a list) is undefined - grep -R path_of_your_inventory -e missing_variable
Jinja template debug
Strange behaviour - try to run the code 'at the destination'

Infrastructure as code

You already know, that ansible-vault allows you to store your confidential data along with your code. You can go further - and define your ansible installation and configuration as code. See environment.sh to learn how to install the ansible itself inside a virtualenv that is not attached to your operating system (can be changed by non-privileged user), and as additional benefit - upgrading version of ansible is as easy as installing new version in new virtualenv. What is more, you can have multiple versions of Ansible present at the same time.

# recreate ansible 2.x venv
$ rm -rf venv2
$ source environment2.sh

# execute playbook
(venv2)$ ansible-playbook playbooks/ansible1.9_playbook.yml # would fail - deprecated syntax

# now lets install ansible 1.9.x next to ansible 2.x
(venv2)$ deactivate
$ source environment.1.9.sh

# execute playbook
(venv1.9)$ ansible-playbook playbooks/ansible1.9_playbook.yml # works!

# please note that you have both venv1.9 and venv2 present - you need to (de)activate one - that is all

become-user, become

In Ansible - to become sudo - use the become parameter. Use become_user to specify the username.

- name: Ensure the httpd service is running
  service:
    name: httpd
    state: started
  become: true

Note: You may like to execute Ansible with --ask-sudo-pass or add the user to sudoers file in order to allow non-supervised execution if you require 'admin' privilages.

--check -C

Always make sure that your playbook can execute in 'dry run' mode (--check), and its execution is not declaring 'Changed' objects.

--diff -D

Diff is useful to see nice detail of the files changed. It compare 'in memory' the files like diff -BbruN fileA fileB.

Execute hosts with 'regex'

ansible -m ping web*

Host groups can be joined, negated, etc

ansible -m ping web*:!backend:monitoring:&allow_change

Tagging

You should tag some (not all) objects - a task in a playbook, all tasks included form a role, etc. It allows you to execute the chosen parts of the playbook.

no_logs: True

You may see, that some roles print a lot of output in verbose mode. There is also a debug module. This is the place where credentials may leak. Use no_log to hide the output.

Debug module

allows to print a value to the screen - use it!

Register the output of a task

You can register the output (stdout), rc (return code), stderr of a task with the register command.

Conditionals: when:

Loop: with, with_items, with_dict, with_together

Inventories:

Static line of servers
Ranges
Dynaminc list of servers: AWS, Azure, GCP.

Playbooks:

Plain text YAML that describe desired state of actions.
Human and machine readable
Can be build entire app environments.

Variables:

Playbooks
Files
Inventories(group vars,host vars)
Command Line
Discovered Variables(hosts)
Ansible AWX

Example

An example playbook to install apache and configure log level

---
- hosts: apache

  vars:
      apache2_log_level: "warn"

  handlers:
  - name: restart apache
    service:
      name: apache2
      state: restarted
      enabled: True
    notify:
      - Wait for instances to listen on port 80
    become: True

  - name: reload apache
    service:
      name: apache2
      state: reloaded
    notify:
      - Wait for instances to listen on port 80
    become: True

  - name: Wait for instances to listen on port 80
    wait_for:
      state: started
      host: localhost
      port: 80
      timeout: 15
      delay: 5

  tasks:
  - name: Update cache
    apt:
      update_cache: yes
      cache_valid_time: 7200
    become: True

  - name: Install packages
    apt:
      name={{ item }}
    with_items:
      - apache2
      - logrotate
    notify:
      - restart apache
    become: True

  - name: Configure apache2 log level
    lineinfile:
      dest: /etc/apache2/apache2.conf
      line: "LogLevel {{ apache2_log_level }}"
      regexp: "^LogLevel"
    notify:
      - reload apache
    become: True
...

Your first ansible command (shell execution)

# Command pings localhost (defined in default inventory: /etc/ansible/hosts)
$ ansible -m ping localhost
# You should see this output
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

For version:

ansible --version

Ping multiple hosts

fping host{0,1,2}.example.org

Copy module

With this module, you can copy a file from the controlling machine to the node. Lets say we want to copy our /etc/motd to /tmp of our target node. Please run the following command:

ansible -i hosts -m copy -a 'src=/etc/motd dest=/tmp/' host0.example.org

To check on multiple nodes

all = all servers found in inventory

We want to get some facts about the node, and know which Ubuntu version we have deployed on nodes, it's pretty easy.Please run the following:

ansible -i hosts -m shell -a 'grep DISTRIB_RELEASE /etc/lsb-release' all

Ansible will try to do here is just executing the ping module (more on modules later) on each host.

ansible -m ping all -i hosts

-m ping means use module ping

ansible -i hosts -m shell -a 'uname -a' host0.example.org

If we need information (ip addresses, RAM size, etc...). The solution comes from module setup, it specializes in node's facts gathering.

ansible -i hosts -m setup host0.example.org

To know how much memory you have on all your hosts

ansible -i hosts -m setup -a 'filter=ansible_memtotal_mb' all

There are few commands you should know about

ansible (to run modules in CLI)
ansible-playbook (to run playbooks)
ansible-vault (to manage secrets)
ansible-galaxy (to install roles from GitHub/Galaxy)

3 Ways to run Ansible:

Ad-hoc: ansible inventory -m

For ex:

-m : use module ping

-a : pass arguments

-i : inventories

ansible web -m ping
ansible web -s -m -C "yum -a "name=openssl state=latest"

Playbooks: ansible-playbook

ansible-playbook -c my_pb.yml

-c : dry run for ad-hoc commands and Playbooks.

To validate existing system:

Automation Framework: Ansible tower
Github like Ansible Galaxy

ansible_host = sets IP ansible will use to connect with host. ansible_user = connect user when using SSH.

ansible -m ping all -i hosts - To check liveness of servers

Here, hosts is the inventory file, -l limits the run only to host1.example.org and apache.yml is our playbook." ansible-playbook -i hosts -l host1.example.org apache.yml

Try to restart apache to see if it really worked ansible -i hosts -m service -a 'name=apache2 state=restarted' host1.example.org ansible-playbook -i hosts service -l host1.example.org apache.yml

Additional Resources

Servers For Hackers: An Ansible Tutorial
A system administrator's guide to getting started with Ansible - FAST!
Ansible Tower - Ansible Tower provides a web UI, dashboard and rest interface to ansible.
Ansible AWX - The Open Source version of Ansible Tower.

ramlaxman/Ansible Summary.md