test-kitchen

Extending the Ansible Test Kitchen tests with BATS tests

/ Werner Dijkerman / Leave a comment

ansible_logo_black_square

In one of the previous blog posts (This one) I described how you can test your Ansible roles with Test Kitchen and Serverspec. With this setup we were able to execute an Ansible Role in an docker container and validate this installation with Serverspec. Server spec is a little bit limited, as we only tested the installation in an sort of technical way: Process is running, port is open, file is created and owner by user x etc. Sometimes this isn’t enough to validate your setup, BATS is the answer.

“Bats is a TAP-compliant testing framework for Bash. It provides a simple way to verify that the UNIX programs you write behave as expected.
A Bats test file is a Bash script with special syntax for defining test cases. Under the hood, each test case is just a function with a description.”
So how does this look like? Let dive in this example:
#!/usr/bin/env bats

@test "Validate status code for login page" {
  run curl -s -o /dev/null -w "%{http_code}" http://zabbix.example.com/index.php
  [[ $output = "200" ]]
}

First we let the script now this is an bats script. The 2nd line is the start of an test and this line starts with the @ sign. Each test has an description and in this case: Validate status code for login page.  Next line is the actual test, we run the curl command. The command needs to start with run, so it will know that an actual command should be executed. In this case, the output of the curl command is the http status code and this will be checked in the 4th line. The $output variable contains the output of the command and in this case, it will contain 200 (or something else, but then the test fails.)

We can also do an check to find if some string is found in the output of the test command, see the following example:

@test "Validate login page and search for \"Username\"" {
  run curl -s http://zabbix.example.com/index.php
  [[ $output =~ ">Username<" ]]
}

With this test, we do an curl of an page and checks if we can find the string “>Username<“ in the output. If this is found, this test is executed correctly, otherwise it will fail.

Just to be clear, you don’t have to use this output check for each check. You can also rely on the exit codes of the command. See this paragraph:

“Test cases consist of standard shell commands. Bats makes use of Bash’s errexit (set -e) option when running test cases. If every command in the test case exits with a 0 status code (success), the test passes. In this way, each line is an assertion of truth.”

(Source: https://github.com/sstephenson/bats)

Please check the github page, there are some nice examples on how to write your tests. But the goal for this blog post is that we have to use it with our Test Kitchen setup, so how do we continue?

As you might recall from the earlier mentioned blog post, we created the directory structure: test/integrations/default. And in this directory we created an directory named serverspec. In this “default” directory we also create an directory named bats. After this we create the file with the extension .bats.

Now we are all set. 😃Now we can execute “kitchen test” and when the Ansible role is installed, the bats suits will begin:

-----> Running bats test suite
        ✓ Validate status code for login page
        ✓ Validate login page and search for "Username"
        ✓ Validate if we can login with default credentials via API

Above example shows 3 tests and each test is executed correctly, as you can see with the checks in front of it. (Otherwise we would see an ‘x’) Right after this, the server spec will be executed.

Have fun! 😃

Advertisement

Using Test Kitchen with Docker and serverspec to test Ansible roles

/ Werner Dijkerman / 7 Comments

ansible_logo_black_square

Lets write some tests for our Ansible roles. When we are testing our roles, we can validate the quality of the role. With this blog item we are using the test kitchen framework and serverspec for the Ansible role: dj-wasabi.zabbix-agent.

With test kitchen we can start an vagrant box or an docker image and our Ansible role will be executed on this instance. There is an whole list of Test Kitchen drivers which can be found here. (If you have an more recent up2date list, please let me know and I update the link). When the Ansible role is installed, serverspec will be executed so we can verify if the installation and configuration is done correctly. Ideally you want to execute this every time when an change is done for the Role, so the best way is to do everything with Jenkins.

We will make use of the following tools:

  • Test Kitchen
  • docker
  • Serverspec

Installation of Jenkins is out of scope for this blog item, same as installation of docker. You’ll need to check these websites for installing Jenkins and  docker on your machine.

Before we even continue, we need to install test kitchen. We do this with the following command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:02:01 - Thu Aug 20)
 (master) > gem install test-kitchen

This is easy, it only take around 10 seconds to install. We continue with the test kitchen setup by executing the next command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:05 - Thu Aug 20)
 (master) > kitchen init --create-gemfile --driver=kitchen-docker
      create  .kitchen.yml
      create  chefignore
      create  test/integration/default
      create  .gitignore
      append  .gitignore
      append  .gitignore
      create  Gemfile
      append  Gemfile
      append  Gemfile
You must run `bundle install' to fetch any new gems.

The command creates some files and directories at default which we almost all will be using. We remove the chefignore file, as we don’t use chef in this case.

We update the Gemfile by adding the next line at the end of the file:

gem "kitchen-ansible"

Now we run the “bundle install” command, it will install the kitchen-docker and kitchen-ansible gems with their dependencies.

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:55 - Thu Aug 20)
 (master) > bundle install
Using multipart-post 2.0.0
Using faraday 0.9.1
Using highline 1.7.3
Using thor 0.19.1
Using librarian 0.1.2
Using librarian-ansible 1.0.6
Using mixlib-shellout 2.1.0
Using net-ssh 2.9.2
Using net-scp 1.2.1
Using safe_yaml 1.0.4
Using test-kitchen 1.4.2
Using kitchen-ansible 0.0.23
Using kitchen-docker 2.3.0
Using bundler 1.6.2
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:03:59 - Thu Aug 20)
 (master) >

We can also set some version restrictions in this file, an example looks like this:

gem 'test-kitchen', '>= 1.4.0'
gem 'kitchen-docker', '>= 2.3.0'
gem 'kitchen-ansible'

With the above example, you’ll install test-kitchen with version 1.4.0 or higher and version 2.3.0 or higher for kitchen-docker. But for now, we use it without the versions.

We have an basic .kitchen.yml file, like this:

---
driver:
  name: docker

provisioner:
  name: chef_solo

platforms:
  - name: ubuntu-14.04
  - name: centos-7.1

suites:
  - name: default
    run_list:
    attributes:

We are changing the file, so it will look like this:

---
driver:
  name: docker
  provision_command: sed -i '/tsflags=nodocs/d' /etc/yum.conf

provisioner:
  name: ansible_playbook
#  ansible_yum_repo: "http://mirror.logol.ru/epel/6/x86_64/epel-release-6-8.noarch.rpm"
  hosts: localhost
#  requirements_path: requirements.yml

platforms:
  - name: centos-6.6

verifier:
  ruby_bindir: '/usr/bin'

suites:
  - name: default

What does it do:

We use the docker driver to run our playbook and tests. It will start an docker image and executes the playbook and after this, we execute serverspec tests to validate of everything should work as expected.

We use the “ansible_playbook” as provisioner. I have 2 lines commented in this .kitchen.yml file. My Ansible role doesn’t have any dependencies. With the “ansible_yum_repo” we can point it to for example the epel-release.rpm file. When the docker image is started, it will download and install this epel repository file so if the role needs some packages from Epel, it will succeed. Same as for the “requirements_path”. This is an yml file which can be used for downloading the role dependencies.

The Zabbix Agent role doesn’t have any dependencies, but for the Zabbix Server role, it has 3 dependencies. You could use it like this:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:22 - Thu Aug 20)
 (master) > cat requirements.yml
---
- src: geerlingguy.apache
- src: geerlingguy.mysql
- src: galaxyprojectdotorg.postgresql

There is only 1 platform in this test, centos-6.6. Same as for the “suites” there is only one. You can  specify more platforms and suits. The following example is the “suites” part of the dj-wasabi.zabbix-server role:

suites:
  - name: zabbix-server-mysql
    provisioner:
        name: ansible_playbook
        playbook: test/integration/zabbix-server-mysql.yml
  - name: zabbix-server-pgsql
    provisioner:
        name: ansible_playbook
        playbook: test/integration/zabbix-server-pgsql.yml

There are 2 suits with their own playbooks. In the above case, there is an playbook which will be executed with the “MySQL” as backend and there is an playbook with the “PostgreSQL” as backend. Both playbooks will be executed in their own docker instance. So it will start with for example the  ‘zabbix-server-mysql’ suits and when this is finished successfully, it continues with the suit ‘zabbix-server-pgsql’.

You can find on these pages some more information on the configuration of test kitchen, ansible and docker parts:

We now create our only playbook:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:25 - Thu Aug 20)
 (master) > vi test/integration/default.ym
--- 
- hosts: localhost
  roles:
    - role: ansible-zabbix-agent

We have configured the playbook which will be executed against the docker image on ‘localhost’.

But we are not there yet, we will need create some serverspec tests to. With these tests we validate if the execution of the Ansible role went successful.

First we have to create the following directory: test/integration/default/serverspec/localhost

We create the spec_helper.rb file:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:07:41 - Thu Aug 20)
 (master) > vi test/integration/default/serverspec/spec_helper.rb
require 'serverspec'
set :backend, :exec

We will create the serverspec file:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:08:22 - Thu Aug 20)
 (master) > vi test/integration/default/serverspec/localhost/ansible_zabbix_agent_spec.rb

The name is not that important, but it has to end with: _spec.rb. We add the following content to the file:

require 'serverspec'
require 'spec_helper'

describe 'Zabbix Agent Packages' do
    describe package('zabbix-agent') do
        it { should be_installed }
    end
end

describe 'Zabbix Agent Configuration' do
    describe file('/etc/zabbix/zabbix_agent.conf') do
        it { should be_file}
        it { should be_owned_by 'zabbix'}
        it { should be_grouped_into 'zabbix'}
    end
end

With the first “describe” we print ‘Zabbix Agent Packages’ and this is an block. When you have some rspec experience with Puppet, you would for example add the params like this: let(:params) { {:server => ‘192.168.1.1’} }
In our case, this is nothing more than printing the text.

Now we proceed with the 2nd describe: package(‘zabbix-agent’). All actions till the firsy “end” is related to this package. For the package we only have 1: it should be_installed. So, when this spec file is executed it will check if the ‘zabbix-agent’ package is installed. If not, you’ll see an error message.

We proceed with the 4th describe, file(‘/etc/zabbix/zabbix_agent.conf’). We have several checks for this file:

  • It Should be an file. (You could also choose for link, directory or even device)
  • The owner of the file needs to be user zabbix
  • The group of the file needs to be group zabbix

There are a lot of other options and checks to use in your spec file, but if we explain it here this is really gonna be an long post.

Only thing that we need to do is to run kitchen. So we execute the following command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:09:13 - Thu Aug 20)
 (master) > kitchen test

If everything goes fine, you’ll see a lot of output. At the end of the run, serverspec will be executed:

       Zabbix Agent Packages
         Package "zabbix-agent"
           should be installed
      
       Zabbix Agent Configuration
         File "/etc/zabbix/zabbix_agent.conf"
           should be file (FAILED - 1)
           should be owned by "zabbix" (FAILED - 2)
           should be grouped into "zabbix" (FAILED - 3)
      
       Failures:
      
         1) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be file
            Failure/Error: it { should be_file}
              expected `File "/etc/zabbix/zabbix_agent.conf".file?` to return true, got false
              /bin/sh -c test\ -f\ /etc/zabbix/zabbix_agent.conf
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:12:in `block (3 levels) in <top (required)>
      
         2) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be owned by "zabbix"
            Failure/Error: it { should be_owned_by 'zabbix'}
              expected `File "/etc/zabbix/zabbix_agent.conf".owned_by?("zabbix")` to return true, got false
              /bin/sh -c stat\ -c\ \%U\ /etc/zabbix/zabbix_agent.conf\ \|\ grep\ --\ \\\^zabbix\\\$
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:13:in `block (3 levels) in <top (required)>'
      
         3) Zabbix Agent Configuration File "/etc/zabbix/zabbix_agent.conf" should be grouped into "zabbix"
            Failure/Error: it { should be_grouped_into 'zabbix'}
              expected `File "/etc/zabbix/zabbix_agent.conf".grouped_into?("zabbix")` to return true, got false
              /bin/sh -c stat\ -c\ \%G\ /etc/zabbix/zabbix_agent.conf\ \|\ grep\ --\ \\\^zabbix\\\$
             
            # /tmp/verifier/suites/serverspec/localhost/ansible_zabbix_agent_spec.rb:14:in `block (3 levels) in <top (required)>'
      
       Finished in 0.11823 seconds (files took 0.37248 seconds to load)
       4 examples, 3 failures

Whoops, it seems that my serverspec file was expecting something else. I made an typo, the file should be /etc/zabbix/zabbix_agentd.conf with an d! 🙂

We can see the docker image is created with the “kitchen list” command, the “Last Action” is “Set Up”:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:14:51- Thu Aug 20)
 (master) > kitchen list
Instance          Driver  Provisioner      Verifier  Transport  Last Action
default-centos-66  Docker  AnsiblePlaybook  Busser    Ssh        Set Up

There are 2 ways to proceed when we fix the typo:

  • We run ‘kitchen test’ again, but it will destroy our docker image and starts again from the start.
  • We run “kitchen verify’ and we only run the serverspec tests. (A lot quicker!)

We use the “kitchen verify” command:

wdijkerman@curiosity [ ~/git/ansible/ansible-zabbix-agent ] (13:15:12 - Thu Aug 20)
 (master) >  kitchen verify
-----> Starting Kitchen (v1.4.2)
-----> Verifying <default-centos-66>...
$$$$$$ Running legacy verify for 'Docker' Driver
       Preparing files for transfer
       Removing /tmp/verifier/suites/serverspec
       Transferring files to <default-centos-66>
-----> Running serverspec test suite
       /opt/rh/ruby193/root/usr/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.3.0/lib:/tmp/verifier/gems/gems/rspec-core-3.3.2/lib /tmp/verifier/gems/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec
      
       Zabbix Agent Packages
         Package "zabbix-agent"
           should be installed
      
       Zabbix Agent Configuration
         File "/etc/zabbix/zabbix_agentd.conf"
           should be file
           should be owned by "zabbix"
           should be grouped into "zabbix"
      
       Finished in 0.10183 seconds (files took 0.33263 seconds to load)
       4 examples, 0 failures
      
       Finished verifying <default-centos-66> (0m1.12s).
-----> Kitchen is finished. (0m1.17s)

As you see, we only run the serverspec tests and everything is ok. 4 examples with 0 failures.
We can continue with creating serverspec tests and rerun the “kitchen verify” command till we are satisfied.

In theory, you’ll create the tests first before creating the playbook. With practice ….

At the end when you are ready, you’ll create an Jenkins job which pulls for changes from your git repository. You’ll create an job which has 2 “Execute shells” steps:

  • Install bundler and test-kitchen and run bundle install
  • Execute kitchen test

Installation of the gem step:s:

gem install bundler --no-rdoc --no-ri
gem install test-kitchen --no-rdoc --no-ri
bundle install

And 2nd build step:

kitchen test

Whoot! 🙂