Testing Ansible roles in a cluster setup with Docker and Molecule

ansible_logo_black_square

This is a follow up on the previous 2 blog posts. With the first blog we discussed some steps to test your Ansible role with some basic steps. With the 2nd blog we added some features to extend our test by using CI tooling and group vars. With this post however, we might be configuring Molecule that will not occur very common with testing Ansible roles.

This time we configure Molecule for a role that is installing and configuring a cluster on Docker, like MySQL or MongoDB. We don’t go into a specific role (as there are so many), I only give some information on how to do this. We are only configuring Molecule for this setup, I’m still busy with running some specific TestInfra tests on a specific container.

Keep in mind these actions are only needed when using the {{ ansible_eth0.ipv4.address }} isn’t enough and you need a list with all ips.

For configuring Molecule, we’ll have to change 2 files:

  1. molecule.yml
  2. playbook.yml

molecule.yml

First we update the ‘molecule.yml’ file by configuring 3 (Or more, depends on what you need) docker containers. See the following example:

docker:
  containers:
  - name: node1
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3306: 3306,
      4444: 4444
  - name: node2
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3307: 3306,
      4445: 4444
  - name: node3
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3308: 3306,
      4446: 4444

As you see, I have added the ‘port_bindings’ configuration to the instances, which is different with the examples in the previous blog posts. With the containers, we open the ports on the host (Before the ‘:’) and proxy them to the ports to the docker container (After the ‘:’ ).

In the above example, this ports configuration is used for configuring a MySQL (Or MariaDB) Galera cluster setup. You’ll have to update the ports configuration to your needs.

playbook.yml

Before we specify the roles in the ‘playbook.yml’, we add an ‘pre_tasks’. We add 2 blocks of code and will discuss them one by one. First we add the following in the ‘pre_tasks’ part:

    - name: "Get ip node 1"
      local_action: shell docker inspect --format {% raw %}'{{ .NetworkSettings.IPAddress }}'{% endraw %} node1
      register: node_ip_1
      changed_when: node_ip_1.rc == 12
    - name: "Get ip node 2"
      local_action: shell docker inspect --format {% raw %}'{{ .NetworkSettings.IPAddress }}'{% endraw %} node2
      register: node_ip_2
      changed_when: node_ip_2.rc == 12
    - name: "Get ip node 3"
      local_action: shell docker inspect --format {% raw %}'{{ .NetworkSettings.IPAddress }}'{% endraw %} node3
      register: node_ip_3
      changed_when: node_ip_3.rc == 12

We have added 3 tasks, that do the same command but for each container. We execute a ‘docker inspect’ command to get the ip address of the docker container. As the docker inspect command used the ‘{{ .NetworkSettings.IPAddress }}’ format, Ansible will try to replace it because it thinks it is a variable. Luckily, we can make use of ‘{{ raw }} {{ endraw }}’ for this and Ansible will not use this as a variable anymore.

We register a variable, because:

  1. We need to use the output, because the ‘docker inspect’ outputs the ip address
  2. We have to fool Ansible, so that the task will not get the state “changed”

What do you mean with the last one?

We have to fool Ansible with the ‘changed_when’ command for the ‘idempotence’ check. As this task will run every time, the ‘idempotence’ check will fail because of it (because it sees that there are tasks with state “Changed”).

In this case, the task will be changed when the exit code of the ‘docker inspect’ command is 12 (Just picked a random number, there is no deep background information related to this number … 😉 ).

Now we add the 2nd block of tasks to the ‘pre_tasks’, just after the first block of code we added earlier:

    - name: "Set fact"
      action:
        module: set_fact
        node_ip: "{{ node_ip_1.stdout }}"
      when: inventory_hostname  == 'node1'
    - name: "Set fact"
      action:
        module: set_fact
        node_ip: "{{ node_ip_2.stdout }}"
      when: inventory_hostname  == 'node2'
    - name: "Set fact"
      action:
        module: set_fact
        node_ip: "{{ node_ip_3.stdout }}"
      when: inventory_hostname  == 'node3'

With this block we add 3 tasks again, 1 task for each container, to create a fact. In this case, I used to create the fact with the name ‘node_ip’, but you may name it differently. In my case when I use the ‘node_ip’ in my Ansible role, it get the actual IP of the host.

You can also create a list with all the ips if you need this in your role:

cluster_host_ips:
  - "{{ node_ip_1.stdout }}"
  - "{{ node_ip_2.stdout }}"
  - "{{ node_ip_3.stdout }}"

(You have to use the corrent name for the list of course 😉 )

I don’t know if this is the correct way, but it works in my case. I have a Jenkins job that validates a role that is configured to run a 3 node setup (Elasticsearch, MariaDB and some others).

If you have a other or a better way for doing this, please let me know!

Extending Ansible Role testing with Molecule by adding group_vars, dependencies and using travis ci

ansible_logo_black_square

This blog post will extend the actions we described on this https://werner-dijkerman.nl/2016/07/10/testing-ansible-roles-with-molecule-testinfra-and-docker/ page. We only configured a very basic configuration with 2 tests in the previously mentioned page, but thats not enough to and on this page we will continue to complete the tests. Github page for Molecule (In case you forgot 😉 )

We will discuss the following in this blogpost:

  • Make the TestInfra tests OS aware
  • Use group_vars
  • Role dependencies
  • Configure Travis

Lets dive into the tasks.

TestInfra OS Aware

Ok, not really molecule related, but very important if our role needs to work on several different operating systems. In the earlier mentioned blogpost we had configured Molecule to use a Ubuntu docker image. Before we can make the tests OS aware, we have to add some docker containers to the configuration which have different operating systems.

We create the following configuration:

docker:
  containers:
  - name: zabbix-agent-centos
    ansible_groups:
      - group1
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
  - name: zabbix-agent-debian
    ansible_groups:
      - group2
    image: maint/debian-systemd
    image_version: latest
    privileged: True
  - name: zabbix-agent-ubuntu
    ansible_groups:
      - group2
    image: rastasheep/ubuntu-sshd
    image_version: latest
    privileged: True

We have 3 docker containers configured: Ubuntu, Debian and CentOS. I’ve search for Docker containers that have systemd configured, as the Zabbix Agent uses this. The official docker images of the mentioned OS’es doesn’t have the systemd enabled/configured.

We would like to run the ‘molecule test’ command, but it will fail on the ‘verify’ part.
We currently have this test:

def test_zabbix_package(Package, SystemInfo):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed

    assert zabbixagent.version.startswith("3.0")

This test will validate if Zabbix is installed and the version starts with 3.0 (Thats the default version to be installed). When running this test, it will fail on the Debian and Ubuntu host. Why?They use a slightly different version naming than CentOS.

We have to update the function by adding the SystemInfo class to the function:

def test_zabbix_package(Package, SystemInfo):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed

    if SystemInfo.distribution == 'debian':
        assert zabbixagent.version.startswith("1:3.0")
    if SystemInfo.distribution == 'centos':
        assert zabbixagent.version.startswith("3.0")

Now we have added the SystemInfo class to the function, we can use this to determine which tests we can execute on which OS. In the above example we see that if the distribution ‘debian’ is, we validate if the version starts with ‘1:3.0’. With CentOS we validate if it is ‘3.0’. When we execute the test on all 3 hosts, it will run successfully.

Using group_vars

This part also applies to using ‘host_vars’, just replace the word ‘group_vars’ with ‘host_vars’ 😉 We configure the molecule.yml file by adding some group_vars related data. We configure the 1st block in the molecule.yml like this:

ansible:
  playbook: playbook.yml
  group_vars:
    mysql:
      - database_type: mysql
        database_type_long: mysql
    postgresql:
      - database_type: pgsql
        database_type_long: postgresql
        postgresql_pg_hba_conf:
          - "host all all 127.0.0.1/32 trust"
          - "host all all ::1/128 trust"
        postgresql_pg_hba_local_ipv4: false
        postgresql_pg_hba_local_ipv6: false

This setup will “create” 2 group_var files: mysql and postgresql. When we run a molecule command, the group_vars will be created in the .molecule directory. Lets run a ‘molecule list’ command. For know we ignore the output, but lets see what is created in the .molecule directory:

[vagrant@localhost ansible-zabbix-server]$ find .molecule -type f | xargs ls -lrt
-rw-rw-r--. 1 501 games   3 Jul 17 20:31 .molecule/state
-rw-rw-r--. 1 501 games 215 Jul 17 20:31 .molecule/group_vars/postgresql
-rw-rw-r--. 1 501 games  51 Jul 17 20:31 .molecule/group_vars/mysql
[vagrant@localhost ansible-zabbix-server]$ cat .molecule/group_vars/mysql
---
database_type: mysql
database_type_long: mysql
[vagrant@localhost ansible-zabbix-server]$ cat .molecule/group_vars/postgresql
---
database_type: pgsql
database_type_long: postgresql
postgresql_pg_hba_conf:
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
postgresql_pg_hba_local_ipv4: false
postgresql_pg_hba_local_ipv6: false

Within the .molecule directory a group_vars directory is created and 2 files are present (We will ignore the ‘state’ file). The output of these files are the same as how we configured the molecule.yml file.

Role dependencies

When your role has some dependencies, we really want them to download before we execute our role in Molecule. It will fail if we don’t do this. We have to download them first.

Molecule has a simple configuration for this. Within the molecule.yml file, we add the ‘requirements_file’ option in the Ansible configuration. Our example now looks like this:

ansible:
  playbook: playbook.yml
  requirements_file: requirements.yml

  group_vars:
    mysql:
      - database_type: mysql
        database_type_long: mysql
    postgresql:
      - database_type: pgsql
        database_type_long: postgresql
        postgresql_pg_hba_conf:
          - "host all all 127.0.0.1/32 trust"
          - "host all all ::1/128 trust"
        postgresql_pg_hba_local_ipv4: false

        postgresql_pg_hba_local_ipv6: false

In the root directory of the repository, a file called ‘requirements.yml’ is found which contains the dependencies.

When we run the ‘converge’ subcommand, the roles will be downloaded and after this the role is executed:

[vagrant@localhost ansible-zabbix-server]$ molecule converge
WARNING:vagrant:The Vagrant executable cannot be found. Please check if it is in the system path.
--> Installing role dependencies ...
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/1.7.2.tar.gz
- extracting geerlingguy.apache to .molecule/roles/geerlingguy.apache
- geerlingguy.apache was installed successfully
- downloading role 'mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/2.3.0.tar.gz
- extracting geerlingguy.mysql to .molecule/roles/geerlingguy.mysql
- geerlingguy.mysql was installed successfully
- downloading role 'postgresql', owned by galaxyprojectdotorg
- downloading role from https://github.com/galaxyproject/ansible-postgresql/archive/0.9.2.tar.gz
- extracting galaxyprojectdotorg.postgresql to .molecule/roles/galaxyprojectdotorg.postgresql
- galaxyprojectdotorg.postgresql was installed successfully
--> Starting Ansible Run ...

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [zabbix-server-centos-mysql]
... 
<snip>

Configure CI

We don’t want to run manually the molecule commands every time a change occurs. To quote Apple: “There is an app for that”. Well almost. I explain 2 ways of running Molecule in a automated way: Using Travis and Using Jenkins.

Travis

A commonly known cloud platform for CI, and it is very simple to make use of it. The Molecule documentation has a very nice example on how to configure the travis setup. We create in the root directory of our role, a file named: .travis.yml

sudo: required
language: python
services:
- docker

before_install:
- sudo apt-get -qq update
- sudo apt-get install -o Dpkg::Options::="--force-confold" --force-yes -y docker-engine

install:
- pip install molecule

script:
- molecule test
notifications:
  webhooks: https://galaxy.ansible.com/api/v1/notifications/

Its a very basic travis configuration. First block is to let Travis know that we need to make use of sudo, that the language Python is and we use the ‘docker’ service.

2nd block is to update the Ubuntu’s apt cache and install Docker. When Docker is installed, we install molecule and after that, molecule test is executed. When the job is finished, we send some data via a web hook to the Ansible Galaxy page. The last step is to show the ‘passing’ (Or failing) badge on the Ansible Role page.

Jenkins

This example will show you a basic Jenkinsfile. I don’t do a single action manually in Jenkins. A  Jenkins job is basically code and we should use it as code. With Jenkins 2, we can make use of a Jenkinsfile (Also with Jenkins 1, but was named jenkins-template I believe?)

When you have a Jenkins server running, create a new Pipeline job and configure the correct git repository. (Okay, for this blogpost we do this manually, but I have a playbook for this. So this is on my part automated as well. 😉 )

A basic example:

node(){
    stage 'INFO: Checkout code'
        checkout scm
    stage 'Installing Molecule'
        sh 'sudo pip install molecule'
    stage 'Creating Containers'
        sh 'molecule create'
    stage 'Installing the Role'
        sh 'molecule converge'
    stage 'Idempotence check'
        sh 'molecule idempotence'
    stage 'Verify the application'
        sh 'molecule verify'
    stage 'Destroy the containers'
        sh 'molecule destroy'
}

When the job is executed, it will first download the Jenkinsfile from the configured git repository and the tasks will be executed one by one. This is just to get you started and the Jenkinsfile should be extended with some error handling like e-mail configuration etc.

So this was the follow up on the original blogpost for how to use Molecule with your Ansible role. The guys are really busy with Molecule so Molecule grows really fast. Maybe doing an 3rd blogpost very soon with new features.. 🙂

Testing Ansible roles with Molecule, Testinfra and Docker

ansible_logo_black_square

In some earlier posts I’ve described how you can use Test Kitchen for testing Ansible Roles (This one and the one extending it.). Test Kitchen was created for testing Chef Cookbooks and like Chef, Test Kitchen is a Ruby application. On this page we describe an other tool for the same purpose. This tool is what you might see as a Python clone of Test Kitchen, but more specific to Ansible: Molecule (Github)

Molecule isn’t that old, only few years and when I browse the internet it is not yet really known in the community. Unlike Test kitchen with the many different drivers, Molecule supports several backends, like Vagrant, Docker, and OpenStack. With Molecule you can make use of Serverspec (Like Test Kitchen), but you can also make use of ‘Testinfra’. Testinfra is like Serverspec a tool for writing unit tests, but it is written in Python.

Lets dive into Molecule and create some tests for Molecule. On this page, we make use of the Docker backend and if you following this page please install docker.

Installing Molecule is really simple:

pip install molecule

Voila, it is installed. With the installation of Molecule, Testinfra is installed to. We can configure a Ansible Role. I used my ‘zabbix-agent’ role as test case for the Test Kitchen setup, so I will use it again for Molecule.

When you haven’t created an Ansible role yet, instead of using the ansible-galaxy command you can use the following command:

molecule init --docker

This will create just like the ‘ansible-galaxy’ some default directories and files, but also gives us a starting point with a few extra files for testing this role with Molecule.

When you already have a role – like I have right now – running the molecule init command will fail and complain:

The directory ./ already exists. Cannot create new role.

No worries, we can recreate these files manually. Lets do that in a role and see what the files do.

File: <root>/molecule.yml

---
ansible:
playbook: playbook.yml

docker:
  containers:
    - name: zabbix-01
      ansible_groups:
        - group1
      image: debian
      image_version: latest
      privileged: True

This is the configuration file for Molecule. We specify which playbook Molecule will execute, in this case playbook.yml.
Next we have a docker container configuration. In this case, we only have 1 docker container specified. We use a Debian docker container with the ‘latest’ tag. We name the container ‘test-01’ and is in the group ‘group1’.

File: <root>/playbook.yml

---
- hosts: all
  roles:
    - role: ansible-zabbix-agent

The playbook that is executed in the Docker container. This is a very basic one, we only have to specify the correct .

File: <root>/tests/inventory

localhost
[group1]
zabbix-01

The Ansible inventory file. Should be a known file to you 😉

File: <root>/tests/test_default.py


from testinfra.utils.ansible_runner import AnsibleRunner
testinfra_hosts = AnsibleRunner('.molecule/ansible_inventory').get_hosts('all')

def test_hosts_file(File): 
    hosts = File('/etc/hosts')
    assert hosts.user == 'root'
    assert hosts.group == 'root' 

(Edit 2016-09-14: As of release 1.9, the first 2 lines should be present in the TestInfra script.)

This is the test infra python file, containing the tests. After the init command, we have 1 test that will check if there is a hosts file, and the user and group of the file belongs to user ‘root’. We discuss this file later on by adding some more tests.

File: <root>/tests/test.yml

---
- hosts: localhost
  remote_user: root
  roles:
    - zabbix-agent-role

Now we have discussed the files.

We add some infra test checks in the ‘test_default.py’ file. We add the following 2 tests:

def test_zabbix_package(Package):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed
    assert zabbixagent.version.startswith("1:3.0")

def test_zabbixagent_running_and_enabled(Service):
    zabbixagent = Service("zabbix-agent")
    # assert zabbixagent.is_running
    assert zabbixagent.is_enabled

These are 2 Python function which are executed with Testinfra. With the first function, we validate if the package ‘zabbix-agent’ is installed. Also we check if the version starts with: 1:3.0. If you have some experience with testing Python code, this might be familiar to you. Test Infra uses ‘PyTest‘ to execute the tests and validate the.

The 2nd function we validate the ‘zabbix-agent’ service. We make sure the service is enabled. As you see, I’ve commented the check if the service is running. When it is enabled, I get this error message:

Failed to get D-Bus connection: Unknown error -1

Strange, because I’ve configured the privileged mode on the docker container. So maybe this is a bug or misconfiguration on my part, but for now I leave it commented and need to find a solution for this.

Now we are ready to move on (I’m well aware that these 2 tests that I added will not be enough, I’ll add these myself later on).

Molecule has several subcommands, let run molecule -h and see what is available:

No handlers could be found for logger "vagrant"
Usage:
    molecule [-hv] <command> [<args>...]

Commands:
    check         check playbook syntax
    create        create instances
    converge      create and provision instances
    idempotence   converge and check the output for changes
    test          run a full test cycle: destroy, create, converge, idempotency-check, verify and destroy instances
    verify        create, provision and test instances
    destroy       destroy instances
    status        show status of instances
    list          show available platforms, providers
    login         connects to instance via SSH
    init          creates the directory structure and files for a new Ansible role compatible with molecule

Options:
    -h --help     shows this screen
    -v --version  shows the version

We first start with the ‘check’ command:

[vagrant@localhost ansible-zabbix-agent]$ molecule check
No handlers could be found for logger "vagrant"

playbook: playbook.yml
[vagrant@localhost ansible-zabbix-agent]$ echo $?
0

Seems very well, the check commands validate if the playbook.yml doesn’t have any problems/syntax errors.
We can continue with the next command: create.

[vagrant@localhost ansible-zabbix-agent]$ molecule create
No handlers could be found for logger "vagrant"
 Building ansible compatible image ...
 Step 1 : FROM debian:latest

  ---> 1b088884749b

 Step 2 : RUN bash -c 'if [ -x "$(command -v apt-get)" ]; then apt-get update && apt-get install -y python sudo; fi'

  ---> Using cache

  ---> 8ef54383599a

 Step 3 : RUN bash -c 'if [ -x "$(command -v yum)" ]; then yum makecache fast && yum update -y && yum install -y python sudo; fi'

  ---> Running in 6d3142fa72aa

...
 Finished building molecule_local/debian:latest
 Creating container zabbix-01 with base image debian:latest ...
 Container created.

[vagrant@localhost ansible-zabbix-agent]$

Now we have created a docker container where we can install our Ansible role on to, we do that with the ‘converge’ subcommand.

[vagrant@localhost ansible-zabbix-agent]$ molecule converge
No handlers could be found for logger "vagrant"

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [zabbix-01]

TASK [ansible-zabbix-agent : Include OS-specific variables] ********************
ok: [zabbix-01]

TASK [ansible-zabbix-agent : Install the correct repository] *******************
skipping: [zabbix-01]

...
RUNNING HANDLER [ansible-zabbix-agent : restart zabbix-agent] ******************
changed: [zabbix-01]

PLAY RECAP *********************************************************************
zabbix-01                  : ok=12  changed=7    unreachable=0    failed=0

Nice, the role is installed correctly without any issues on the container. With Test Kitchen we had to use BATS to validate if the Role is idempotent, but luckily molecule has just a simple sub command for it: idempotence

Well, it seems that the Role has passed the idempotence test:

[vagrant@localhost ansible-zabbix-agent]$ molecule idempotence
No handlers could be found for logger "vagrant"
Idempotence test in progress (can take a few minutes)...
Idempotence test passed.

[vagrant@localhost ansible-zabbix-agent]$

Testing the role is nicely going on right now, but we are not there yet. Now we need to use the ‘verify’ command to actually validate our role on the Docker container:

[vagrant@localhost ansible-zabbix-agent]$ molecule verify
No handlers could be found for logger "vagrant"
Trailing whitespace found in ./defaults/main.yml on lines: 35
Trailing newline found at the end of ./handlers/main.yml
Trailing whitespace found in ./library/zabbix_host.py on lines: 29
Trailing newline found at the end of ./library/zabbix_hostmacro.py
[vagrant@localhost ansible-zabbix-agent]$

Whoops, it seems it has found some issues. Let me fix that first, probably need to run the verify again after fixing it.

[vagrant@localhost ansible-zabbix-agent]$ molecule verify
No handlers could be found for logger "vagrant"

Executing testinfra tests found in tests/.
============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /git/ansible/ansible-zabbix-agent/tests, inifile:
plugins: xdist-1.14, testinfra-1.4.0
collected 3 itemss

tests/test_default.py ...

=========================== 3 passed in 0.63 seconds ===========================

No serverspec tests found in spec/.

[vagrant@localhost ansible-zabbix-agent]$

After fixing it, everything seems to work fine. Nice!

Now we are done with the container, so we can execute molecule again, but with the delete sub command and the container will be deleted.

These were the basics for testing an Ansible role with Molecule, Docker and Test Infra. This page uses the ‘Debian’ Docker image, whereas I normally use CentOS for this. I have some issues (Get the same error message when I enable the Test Infra test to validate if the service is running) to make this work on CentOS. So maybe Molecule isn’t mature enough yet, but it is getting there.

I’ll update my Ansible roles so it will use Molecule instead of Test Kitchen (No hard feelings ;-))

Extending the Ansible Test Kitchen tests with BATS tests

ansible_logo_black_square

In one of the previous blog posts (This one) I described how you can test your Ansible roles with Test Kitchen and Serverspec. With this setup we were able to execute an Ansible Role in an docker container and validate this installation with Serverspec. Server spec is a little bit limited, as we only tested the installation in an sort of technical way: Process is running, port is open, file is created and owner by user x etc. Sometimes this isn’t enough to validate your setup, BATS is the answer.

“Bats is a TAP-compliant testing framework for Bash. It provides a simple way to verify that the UNIX programs you write behave as expected.
A Bats test file is a Bash script with special syntax for defining test cases. Under the hood, each test case is just a function with a description.”
So how does this look like? Let dive in this example:
#!/usr/bin/env bats

@test "Validate status code for login page" {
  run curl -s -o /dev/null -w "%{http_code}" http://zabbix.example.com/index.php
  [[ $output = "200" ]]
}

First we let the script now this is an bats script. The 2nd line is the start of an test and this line starts with the @ sign. Each test has an description and in this case: Validate status code for login page.  Next line is the actual test, we run the curl command. The command needs to start with run, so it will know that an actual command should be executed. In this case, the output of the curl command is the http status code and this will be checked in the 4th line. The $output variable contains the output of the command and in this case, it will contain 200 (or something else, but then the test fails.)

We can also do an check to find if some string is found in the output of the test command, see the following example:

@test "Validate login page and search for \"Username\"" {
  run curl -s http://zabbix.example.com/index.php
  [[ $output =~ ">Username<" ]]
}

With this test, we do an curl of an page and checks if we can find the string “>Username<“ in the output. If this is found, this test is executed correctly, otherwise it will fail.

Just to be clear, you don’t have to use this output check for each check. You can also rely on the exit codes of the command. See this paragraph:

“Test cases consist of standard shell commands. Bats makes use of Bash’s errexit (set -e) option when running test cases. If every command in the test case exits with a 0 status code (success), the test passes. In this way, each line is an assertion of truth.”

(Source: https://github.com/sstephenson/bats)

Please check the github page, there are some nice examples on how to write your tests. But the goal for this blog post is that we have to use it with our Test Kitchen setup, so how do we continue?

As you might recall from the earlier mentioned blog post, we created the directory structure: test/integrations/default. And in this directory we created an directory named serverspec. In this “default” directory we also create an directory named bats. After this we create the file with the extension .bats.

Now we are all set. 😃Now we can execute “kitchen test” and when the Ansible role is installed, the bats suits will begin:

-----> Running bats test suite
        ✓ Validate status code for login page
        ✓ Validate login page and search for "Username"
        ✓ Validate if we can login with default credentials via API

Above example shows 3 tests and each test is executed correctly, as you can see with the checks in front of it. (Otherwise we would see an ‘x’) Right after this, the server spec will be executed.

Have fun! 😃

Installing zabbix-server with ansible

ansible_logo_black_squarezabbix_logo

Not only I have an puppet module which can be freely used from the forge, I also have some Ansible roles for Zabbix. This page will describe installing the zabbix-server with the dj-wasabi.zabbix-server role. If you want to know how you install the zabbix-agent, please check this page.

You can find the role and some information on this page: https://galaxy.ansible.com/list#/roles/2070

This role works on the 3 main Linux operating systems:

  • RedHat
  • Debian
  • Ubuntu

So, if your server has one of these operating system, you can continue. If you have however an other operating system and have some Ansible knowledge, please add some improvements and create an Pull Request on Github. I always accept Pull Requests related to the Ansible roles.

When you want to install this role, you only have to execute the following command:

ansible-galaxy install dj-wasabi.zabbix-server

Now we need to setup everything, but before we do anything we need to know what kind of database server is going to be used. Zabbix Server can work with several different databases as backend. This Ansible role only works with the following databases:

  • PostgreSQL
  • MySQL

Before we see the examples, there is one main parameter which is always needed: zabbix_url

This is the url on which the zabbix interface is available and should be an fqdn. Default it will create an Apache Virtual Host configuration file with this FQDN as ServerName. If you set this parameter as this:

zabbix_url: zabbix.example.com

the web interface will be available at: http://zabbix.example.com

PostgreSQL

Default the PostgreSQL is used as backend and before we can use this role, we need to find and download an Ansible role for PostgreSQL which can be used on your operating system. In this example we are using the following role: ‘galaxyprojectdotorg.postgresql’

The following is an example of an playbook for installing the ‘zabbix-server’ with an PostgreSQL database:

--- 
- hosts: zabbix-server
  roles: 
    - role: galaxyprojectdotorg.postgresql
      postgresql_pg_hba_conf: 
        - "host all all 127.0.0.1/32 trust"
        - "host all all ::1/128 trust"
      postgresql_pg_hba_local_ipv4: false
      postgresql_pg_hba_local_ipv6: false
    - role: dj-wasabi.zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server

This is the minimum configuration to use for this role with an PostgreSQL as database. What might help to secure everything is to use an more difficult to guess password for the ‘server_dbuser’ 😉

MySQL

Lets use MySQL as backend now. The following example is used with the following role: ‘geerlingguy.mysql’:

---
- hosts: localhost
  roles:
    - role: geerlingguy.mysql
    - role: ansible-zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      database_type: mysql
      database_type_long: mysql
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server

Same as for the example with PostgreSQL, use an different value for the server_dbpassword.

Other configurations

Don’t think that what you just saw with configuring this role is everything. There are a lot of other configuration parameters that can be set. Keep in mind, that all configuration options you’ll normally find in the ‘zabbix_server.conf’ configuration file, can also be set with this role.

Lets give an example:

When we need to set the StartPollers to value 10, we can update the MySQL playbook to look like this:

---
- hosts: localhost
  roles:
    - role: geerlingguy.mysql
    - role: ansible-zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      database_type: mysql
      database_type_long: mysql
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server
      server_startpollers: 10

When the role is executed on the ‘zabbix-server’, we see the following in the configuration file:

### option: startpollers
#	number of pre-forked instances of pollers.
#
StartPollers=10

Keep in mind to lower the property setting and prefix it with ‘server_’ and you’ll have the property for this Ansible role.

As this Ansible role isn’t perfect, please let me know if you encounter any issues by creating an issue. Pull Request for bugs or new features are always welcome!

Using Test Kitchen with Docker and serverspec to test Ansible roles

ansible_logo_black_square

Lets write some tests for our Ansible roles. When we are testing our roles, we can validate the quality of the role. With this blog item we are using the test kitchen framework and serverspec for the Ansible role: dj-wasabi.zabbix-agent.

With test kitchen we can start an vagrant box or an docker image and our Ansible role will be executed on this instance. There is an whole list of Test Kitchen drivers which can be found here. (If you have an more recent up2date list, please let me know and I update the link). When the Ansible role is installed, serverspec will be executed so we can verify if the installation and configuration is done correctly. Ideally you want to execute this every time when an change is done for the Role, so the best way is to do everything with Jenkins.

We will make use of the following tools:

  • Test Kitchen
  • docker
  • Serverspec

Installation of Jenkins is out of scope for this blog item, same as installation of docker. You’ll need to check these websites for installing Jenkins and  docker on your machine.

Before we even continue, we need to install test kitchen. We do this with the following command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:02:01 - Thu Aug 20)
 (master) > gem install test-kitchen

This is easy, it only take around 10 seconds to install. We continue with the test kitchen setup by executing the next command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:05 - Thu Aug 20)
 (master) > kitchen init --create-gemfile --driver=kitchen-docker
      create  .kitchen.yml
      create  chefignore
      create  test/integration/default
      create  .gitignore
      append  .gitignore
      append  .gitignore
      create  Gemfile
      append  Gemfile
      append  Gemfile
You must run `bundle install' to fetch any new gems.

The command creates some files and directories at default which we almost all will be using. We remove the chefignore file, as we don’t use chef in this case.

We update the Gemfile by adding the next line at the end of the file:

gem "kitchen-ansible"

Now we run the “bundle install” command, it will install the kitchen-docker and kitchen-ansible gems with their dependencies.

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:55 - Thu Aug 20)
 (master) > bundle install
Using multipart-post 2.0.0
Using faraday 0.9.1
Using highline 1.7.3
Using thor 0.19.1
Using librarian 0.1.2
Using librarian-ansible 1.0.6
Using mixlib-shellout 2.1.0
Using net-ssh 2.9.2
Using net-scp 1.2.1
Using safe_yaml 1.0.4
Using test-kitchen 1.4.2
Using kitchen-ansible 0.0.23
Using kitchen-docker 2.3.0
Using bundler 1.6.2
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:59 - Thu Aug 20)
 (master) > 

We can also set some version restrictions in this file, an example looks like this:

gem 'test-kitchen', '>= 1.4.0'
gem 'kitchen-docker', '>= 2.3.0'
gem 'kitchen-ansible'

With the above example, you’ll install test-kitchen with version 1.4.0 or higher and version 2.3.0 or higher for kitchen-docker. But for now, we use it without the versions.

We have an basic .kitchen.yml file, like this:

---
driver:
  name: docker

provisioner:
  name: chef_solo

platforms:
  - name: ubuntu-14.04
  - name: centos-7.1

suites:
  - name: default
    run_list:
    attributes:

We are changing the file, so it will look like this:

---
driver:
  name: docker
  provision_command: sed -i '/tsflags=nodocs/d' /etc/yum.conf

provisioner:
  name: ansible_playbook
#  ansible_yum_repo: "http://mirror.logol.ru/epel/6/x86_64/epel-release-6-8.noarch.rpm"
  hosts: localhost
#  requirements_path: requirements.yml

platforms:
  - name: centos-6.6

verifier:
  ruby_bindir: '/usr/bin'

suites:
  - name: default

What does it do:

We use the docker driver to run our playbook and tests. It will start an docker image and executes the playbook and after this, we execute serverspec tests to validate of everything should work as expected.

We use the “ansible_playbook” as provisioner. I have 2 lines commented in this .kitchen.yml file. My Ansible role doesn’t have any dependencies. With the “ansible_yum_repo” we can point it to for example the epel-release.rpm file. When the docker image is started, it will download and install this epel repository file so if the role needs some packages from Epel, it will succeed. Same as for the “requirements_path”. This is an yml file which can be used for downloading the role dependencies.

The Zabbix Agent role doesn’t have any dependencies, but for the Zabbix Server role, it has 3 dependencies. You could use it like this:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:22 - Thu Aug 20)
 (master) > cat requirements.yml
---
- src: geerlingguy.apache
- src: geerlingguy.mysql
- src: galaxyprojectdotorg.postgresql

There is only 1 platform in this test, centos-6.6. Same as for the “suites” there is only one. You can  specify more platforms and suits. The following example is the “suites” part of the dj-wasabi.zabbix-server role:

suites:
  - name: zabbix-server-mysql
    provisioner:
        name: ansible_playbook
        playbook: test/integration/zabbix-server-mysql.yml
  - name: zabbix-server-pgsql
    provisioner:
        name: ansible_playbook
        playbook: test/integration/zabbix-server-pgsql.yml

There are 2 suits with their own playbooks. In the above case, there is an playbook which will be executed with the “MySQL” as backend and there is an playbook with the “PostgreSQL” as backend. Both playbooks will be executed in their own docker instance. So it will start with for example the  ‘zabbix-server-mysql’ suits and when this is finished successfully, it continues with the suit ‘zabbix-server-pgsql’.

You can find on these pages some more information on the configuration of test kitchen, ansible and docker parts:

We now create our only playbook:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:25 - Thu Aug 20)
 (master) > vi test/integration/default.ym
--- 
- hosts: localhost
  roles:
    - role: ansible-zabbix-agent

We have configured the playbook which will be executed against the docker image on ‘localhost’.

But we are not there yet, we will need create some serverspec tests to. With these tests we validate if the execution of the Ansible role went successful.

First we have to create the following directory: test/integration/default/serverspec/localhost

We create the spec_helper.rb file:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:41 - Thu Aug 20)
 (master) > vi test/integration/default/serverspec/spec_helper.rb
require 'serverspec'
set :backend, :exec

We will create the serverspec file:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:08:22 - Thu Aug 20)
 (master) > vi test/integration/default/serverspec/localhost/ansible_zabbix_agent_spec.rb

The name is not that important, but it has to end with: _spec.rb. We add the following content to the file:

require 'serverspec'
require 'spec_helper'

describe 'Zabbix Agent Packages' do
    describe package('zabbix-agent') do
        it { should be_installed }
    end
end

describe 'Zabbix Agent Configuration' do
    describe file('/etc/zabbix/zabbix_agent.conf') do
        it { should be_file}
        it { should be_owned_by 'zabbix'}
        it { should be_grouped_into 'zabbix'}
    end
end

With the first “describe” we print ‘Zabbix Agent Packages’ and this is an block. When you have some rspec experience with Puppet, you would for example add the params like this: let(:params) { {:server => ‘192.168.1.1’} }
In our case, this is nothing more than printing the text.

Now we proceed with the 2nd describe: package(‘zabbix-agent’). All actions till the firsy “end” is related to this package. For the package we only have 1: it should be_installed. So, when this spec file is executed it will check if the ‘zabbix-agent’ package is installed. If not, you’ll see an error message.

We proceed with the 4th describe, file(‘/etc/zabbix/zabbix_agent.conf’). We have several checks for this file:

  • It Should be an file. (You could also choose for link, directory or even device)
  • The owner of the file needs to be user zabbix
  • The group of the file needs to be group zabbix

There are a lot of other options and checks to use in your spec file, but if we explain it here this is really gonna be an long post.

Only thing that we need to do is to run kitchen. So we execute the following command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:09:13 - Thu Aug 20)
 (master) > kitchen test

If everything goes fine, you’ll see a lot of output. At the end of the run, serverspec will be executed:

       Zabbix Agent Packages
         Package "zabbix-agent"
           should be installed
      
       Zabbix Agent Configuration
         File "/etc/zabbix/zabbix_agent.conf"
           should be file (FAILED - 1)
           should be owned by "zabbix" (FAILED - 2)
           should be grouped into "zabbix" (FAILED - 3)
      
       Failures:
      
         1) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be file
            Failure/Error: it { should be_file}
              expected `File "/etc/zabbix/zabbix_agent.conf".file?` to return true, got false
              /bin/sh -c test\ -f\ /etc/zabbix/zabbix_agent.conf
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:12:in `block (3 levels) in <top (required)>
      
         2) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be owned by "zabbix"
            Failure/Error: it { should be_owned_by 'zabbix'}
              expected `File "/etc/zabbix/zabbix_agent.conf".owned_by?("zabbix")` to return true, got false
              /bin/sh -c stat\ -c\ \%U\ /etc/zabbix/zabbix_agent.conf\ \|\ grep\ --\ \\\^zabbix\\\$
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:13:in `block (3 levels) in <top (required)>'
      
         3) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be grouped into "zabbix"
            Failure/Error: it { should be_grouped_into 'zabbix'}
              expected `File "/etc/zabbix/zabbix_agent.conf".grouped_into?("zabbix")` to return true, got false
              /bin/sh -c stat\ -c\ \%G\ /etc/zabbix/zabbix_agent.conf\ \|\ grep\ --\ \\\^zabbix\\\$
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:14:in `block (3 levels) in <top (required)>'
      
       Finished in 0.11823 seconds (files took 0.37248 seconds to load)
       4 examples, 3 failures

Whoops, it seems that my serverspec file was expecting something else. I made an typo, the file should be /etc/zabbix/zabbix_agentd.conf with an d! 🙂

We can see the docker image is created with the “kitchen list” command, the “Last Action” is “Set Up”:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:14:51- Thu Aug 20)
 (master) > kitchen list
Instance          Driver  Provisioner      Verifier  Transport  Last Action
default-centos-66  Docker  AnsiblePlaybook  Busser    Ssh        Set Up

There are 2 ways to proceed when we fix the typo:

  • We run ‘kitchen test’ again, but it will destroy our docker image and starts again from the start.
  • We run “kitchen verify’ and we only run the serverspec tests. (A lot quicker!)

We use the “kitchen verify” command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:15:12 - Thu Aug 20)
 (master) >  kitchen verify
-----> Starting Kitchen (v1.4.2)
-----> Verifying <default-centos-66>...
$$$$$$ Running legacy verify for 'Docker' Driver
       Preparing files for transfer
       Removing /tmp/verifier/suites/serverspec
       Transferring files to <default-centos-66>
-----> Running serverspec test suite
       /opt/rh/ruby193/root/usr/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.3.0/lib:/tmp/verifier/gems/gems/rspec-core-3.3.2/lib /tmp/verifier/gems/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec
      
       Zabbix Agent Packages
         Package "zabbix-agent"
           should be installed
      
       Zabbix Agent Configuration
         File "/etc/zabbix/zabbix_agentd.conf"
           should be file
           should be owned by "zabbix"
           should be grouped into "zabbix"
      
       Finished in 0.10183 seconds (files took 0.33263 seconds to load)
       4 examples, 0 failures
      
       Finished verifying <default-centos-66> (0m1.12s).
-----> Kitchen is finished. (0m1.17s)

As you see, we only run the serverspec tests and everything is ok. 4 examples with 0 failures.
We can continue with creating serverspec tests and rerun the “kitchen verify” command till we are satisfied.

In theory, you’ll create the tests first before creating the playbook. With practice ….

At the end when you are ready, you’ll create an Jenkins job which pulls for changes from your git repository. You’ll create an job which has 2 “Execute shells” steps:

  • Install bundler and test-kitchen and run bundle install
  • Execute kitchen test

Installation of the gem step:s:

gem install bundler --no-rdoc --no-ri
gem install test-kitchen --no-rdoc --no-ri
bundle install

And 2nd build step:

kitchen test

Whoot! 🙂

Using Librarian-Ansible to install Ansible roles from Gitlab

ansible_logo_black_square

I have some Ansible roles which I try to keep up2date and these are on Github and on my personal Gitlab instance. Sometimes this takes a little bit longer that I want to, but other projects needs some attention to.

For my own personal environment, I use Ansible too and this is in an seperate git repository of my Gitlab server (Repository: environment/ansible.git). There is one thing that buggers me: My Ansible roles differs from the one used in my personal Ansible setup. At moment of writing, the ‘dj-wasabi.zabbix-agent’ role is at tag 0.2.1, but I use ‘0.0.2’ in my own Ansible setup (Oh, really that old?? 🙂 ).

There should be an solution for this. But before we continue, the solution should met my goals:

  • All Ansible roles should have their own git repository in Gitlab,
  • All Ansible roles have their own Jenkins job, documentation and test cases,
  • I want to make use of tags or versions.

With this I can create specific tags/version of the role and we can run some tests via Jenkins like ‘testkitchen’. With ‘testkitchen’ we run the role on an vagrant/docker and see if everything runs fine. But for know, ‘testkitchen’ is out of scope for this.

I first looked at ‘ansible-galaxy’. It has the possibility for using an ‘requirements.yml’ file which holds all information. Like location and even an version, so we can specify the correct role. After some testing it only work when you have the repository at Github.com.
Also the repository should exists on the Galaxy itself. So for the Zabbix roles this could work, but I also have some roles created just for my own environment. These are specific and there is no need to upload them to the Galaxy or github, so the ‘ansible-galaxy’ will not work for me.

I found “librarian-ansible’. This could be something which might work for me, but didn’t found information on the web. Yes, I did found something that the Ansible creator Michael DeHaan isn’t an very big fan of this (https://groups.google.com/forum/#!msg/ansible-project/TawjChwaV08/3p6Zv24rMWgJ). So lets try it anyways, maybe it creates a fan out of me ;-).

Installation is very simple, we have to install 1 gem:

wdijkerman@curiosity [ ~/git/environment/ansible ] (14:03:56 - Sat Aug 15)
 (master) > sudo gem install librarian-ansible
Successfully installed librarian-ansible-1.0.6
1 gem installed

Now we have to create the Ansiblefile. The Ansiblefile is used for declaring the roles and where these roles can be found. We can do this with the following command:

wdijkerman@curiosity [ ~/git/environment/ansible ] (14:24:19 - Sat Aug 15)
 (master) > librarian-ansible init
      create  Ansiblefile

It creates the “Ansiblefile” in the current directory. It already has some basic roles specified, but I don’t use them.

#!/usr/bin/env ruby
#^syntax detection

site "https://galaxy.ansible.com/api/v1";

role "kunik.deploy-upstart-scripts";

role "pgolm.ansible-playbook-monit",
  github: "pgolm/ansible-playbook-monit";

With the default Ansiblefile it shows you, that you also can make use of the Ansible Galaxy. The site is configured to use Ansible Galaxy API and when you only have specified the “role” (In this case kunik.deploy-upstart-scripts”, it will be downloaded from the Galaxy.

2nd example is downloding the git repository “pgolm/ansible-playbook-monit”, which will be installed with the role name “pgolm.ansible-playbook-monit”. Nice, but I only need to make use of the “git” option. I start with the following:

#!/usr/bin/env ruby
#^syntax detection

role "zabbix-javagateway",
    git: "git@gitlab.dj-wasabi.local:ansible/zabbix-javagateway.git",
    ref: "0.1.0"

When I run “librarian-ansible install’ it will clone the git repository and checkouts the tag “0.1.0”. The role is now installed in the ‘ librarian_roles/’ directory with the name “zabbix-javagateway”. But I want it in my roles directory, so I have to run the ‘librarian-ansible’ command again, but with the config option:

librarian-ansible config path roles --global

This sets the path to my “roles” directory. This is specified in my ansible.cfg and I want them in this directory. So running again the ‘librarian-ansible install’ command and the role is installed again. But, what I didn’t know (or didn’t read in the very few sites that exists about librarian-ansible) is it will delete the content of the directory. So all my roles which were in the ‘roles/’ directory are deleted. So, ‘git checkout roles/’ and moving all roles to their own git repository and start again! 🙂

Maybe add the “roles” directory in my .Gitignore file. We don’t want to store all the roles in this repository too.

I think I’m going to be an fan for librarian-ansible. 🙂