Continuous deployment of Ansible Roles

Ansible Logo

There are a lot of articles about Ansible with continuous deployment, but these are only about using Ansible as a tool to do continuous deployment. There is not much (Well, I can’t really find none) about continuous deployment on code changes in Ansible Roles/playbooks itself. But first: Why do you want to do that?

Well, it is very easy to make changes in a role or a playbook and deploy that to a (production) machine(s). I do hope that these changes are commited into the git repository (And pushed) so that changes on the host can be tracked back to the code. Hopefully you didn’t make any errors in the playbook or role so all will be fine during deployment and no unwanted downtime is caused, because nothing is tested.

When you are part of a team, this would be a downside of using Ansible. It is very easy to make changes to a playbook or a role locally and not commit it to the repository, deploy it to a production server and continue like it didn’t happen. You can make agreements on these kinds of procedures on when and how to execute playbooks, but you always have that coworker that don’t (or partly) want to follow procedures or just because of lack of time (“It has to be working this morning!” or just any other lame excuse to not test your code before deployment).

When the team/serverpark grows bigger and/or the company you work for matures and even has an SLA, you can’t just deploy any untested code anymore. You’ll have to make sure that changes you made to code is tested, like any other code. Application developers write unit tests on their code and the application is tested by either automated tests or by using test|qa team. Application development is not any different than writing software for your infrastructure, it all needs to be tested before you use it on production.

What I will describe in this blogpost is just a suggestion on how to do this. This might not be foolproof or maybe there are other or better ways on how todo this or … (Fill in some something other reason). As this is something that works for me, it might help you to create your own pipeline. YMMV.

I haven’t looked at all at Ansible Tower or the open sourced version, so it might be that parts or maybe all of what I am describing here can be done by Tower.

Before we do anyting, I’ll first describe how my Ansible setup looks like so we have some background before we do anything. All of my roles has their own git repository, including documentation and Jenkinsfiles. A Jenkinsfile is the Jenkins job configuration file that contains all steps that Jenkins will execute. Its the .travis.yml (Of Travis CI) file equivalent of Jenkins and we will come back later to this. I also have 1 git repository that contains all ansible data, like host_vars, group_vars and the inventory file.

I have a Jenkins running with the Docker plugin and once a job is started, a Docker container will be started and the job will be executed from this container. Once the Job is done (Succeeded or Failed doesn’t matter which), the container and all data in this container is removed.

Jenkins Jobs

All my Ansible roles has 3 jenkinsfiles stored in the git repository for the following actions:

  1. Molecule Tests
  2. Staging deployment
  3. Production deployment

Molecule Tests

The first job is that the role is tested with Molecule. With Molecule we create 1 or more Docker containers and the role is deployed to these containers. Once that is done, we do an idempotent check and with TestInfra we verify if installation/configuration is done correctly. We can also execute some commands to verify that the deployed service is running correctly. Once these tests are completed, we can successfully deploy the ansible role without any problems. (On this page I have described some information on Molecule.)

How does the Jenkinsfile looks like:

node() {
    try {
        stage ("Get Latest Code") {
            checkout scm
            sh 'git rev-parse HEAD > .git/commit-id'
        }
        stage ("Install Application Dependencies") {
            sh 'sudo pip install --upgrade ansible==${ANSIBLE_VERSION} molecule==${MOLECULE_VERSION} docker'
        }
        stage ("Executing Molecule lint") {
            sh 'molecule lint'
        }
        stage ("Executing Molecule create") {
            sh 'molecule create'
        }
        stage ("Executing Molecule converge") {
            sh 'molecule converge'
        }
        stage ("Executing Molecule idemotence") {
            sh 'molecule idempotence'
        }
        stage ("Executing Molecule verify") {
            sh 'molecule verify'
        }
        stage('Tag git'){
            def commit_id = readFile('.git/commit-id').trim()
            withEnv(["COMMIT_ID=${commit_id}"]){
                sh '''#!/bin/bash
                if [[ $(git tag | grep ${COMMIT_ID} | wc -l) -eq 1 ]]
                    then    echo "Tag already exists"
                    else    echo "Tag will be created"
                            git config user.name "jenkins"
                            git config user.email "jenkins@localhost"
                            git tag -a $COMMIT_ID -m "Added tagging"
                            git push --tags
                fi
                '''
            }
        }
        stage('Start Staging Job') {
            def commit_id = readFile('.git/commit-id').trim()
            withEnv(["COMMIT_ID=${commit_id}"]){
                build job: 'ansible-access-2-staging', wait: false, parameters: [string(name: 'COMMIT_ID', value: "${COMMIT_ID}") ]
            }
        }
    } catch(all) {
        currentBuild.result = "FAILURE"
        throw err
    }
}

First stage of the Job is the checkout of the sourcecode of the git repository, so that we have data in the container. We get the latest git commit id, because I use this id to create a tag in git once the Molecule Tests succeeds.

First Molecule action is the lint. First we do some linting on the role and test files to make sure it is compliant. If it find some errors, it fails quickly and we can fix it. Then it proceeds with the Molecule actions create, converge, idemptence and verify. For those who are familiar with Molecule will notice that I use different stages for each action and not 1 stage which executes molecule test.

Stages overview of Jenkins job.

I use separate stages with single commands so I can quickly see on which part the job fails and focus on that immediately without going to the console output and scrolling down to see where it fails. After the Molecule verify stage, the Tag git stage is executed. This will use the latest commit id as a tag, so I know that this tag was triggered by Jenkins to run a build and was successful.

Last stage in the job is to start the 2nd job in Jenkins. This stage will start the job ansible-access-2-staging with the COMMIT_ID as parameter to the job and in the background (wait: false).

Currently, the Molecule configuration only has 1 “default” scenario. If I had more scenarios than the Jenkinsfile had probably a lot more stages or maybe more Jenkinsfiles.

Staging deployment

The first job was executed correctly and now this job is triggered. As mentioned before, the commit id of the previous job is passed into this job. The goal for this job is to deploy the role to an staging server and validate if everything is still working correctly. In this case we will execute the same tests on the staging staging as we did with Molecule, but we can also create an other test file and use that. In my case, there is only one staging server but it could also be a group of servers.

The Jenkinsfile for this job looks like this:

node() {
    try {
        stage ("Get the Code") {
            checkout scm: [$class: 'GitSCM', branches: [[name: "refs/tags/${params.COMMIT_ID}"]], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'ansible-access']], userRemoteConfigs: [[url: 'ssh://git@192.168.1.206:2222/ansible/access.git']]]
            checkout scm: [$class: 'GitSCM', branches: [[name: '*/master']], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'environment']], doGenerateSubmoduleConfigurations: false, userRemoteConfigs: [[url: 'ssh://git@192.168.1.206:2222/ansible/environment.git']]]
            sh 'pwd > workspace'
        }
        stage ("Install Application Dependencies") {
            sh 'sudo pip install --upgrade ansible==${ANSIBLE_VERSION} testinfra docker'
        }
        stage ("Execute role on host(s)") {
            dir("environment") {
                sh "ansible-playbook -i hosts -l staging playbooks/ansible-access.yml"
            }
        }
        stage ("Test Role execution") {
            workspace = readFile('workspace').trim()
            withEnv(["WORKSPACE_DIR=${workspace}", "MOLECULE_INVENTORY_FILE=${workspace}/environment/hosts"]){
                dir("environment/") {
                    sh "testinfra --connection=ansible --ansible-inventory=hosts --hosts=staging ${WORKSPACE_DIR}/ansible-access/molecule/default/tests/test_default.py --verbose"
                }
            }
        }
        stage('Tag git'){
            withEnv(["COMMIT_ID=${params.COMMIT_ID}_staging"]){
                dir("ansible-access/") {
                    sh '''#!/bin/bash
                    if [[ $(git tag | grep "${COMMIT_ID}" | wc -l) -eq 1 ]]
                        then    echo "Tag already exists"
                        else    echo "Tag will be created"
                                git config user.name "jenkins"
                                git config user.email "jenkins@localhost"
                                git tag -a $COMMIT_ID -m "Added tagging"
                                git push --tags
                    fi
                    '''
                }
            }
        }
        stage('Start Production Job') {
            build job: 'ansible-access-3-production', wait: false, parameters: [string(name: 'COMMIT_ID', value: "${params.COMMIT_ID}") ]
        }
    } catch(all) {
        currentBuild.result = "FAILURE"
        throw err
    }
}

The first stage is to checkout 2 git repositories: The Ansible Role and the 2nd is my “environment” repository that contains all Ansible data and both are stored in their own sub directory. With the Ansible role we checkout the provided tag refs/tags/${params.COMMIT_ID}. I also had to configure the url for the git repositories. Last step is to create a file that holds the output of the pwd file. We need this location in a later stage.

The 2nd Stage is to install the required applications, so not very interesting. The 3rd stage is to execute the playbook. In my “environment” repository (That holds all Ansible data) there is a playbooks directory and in that directory contains the playbooks for the roles. For deploying the ansible-access role, a playbook named ansible-access.yml is present and will be use to install the role on the host:

---
- hosts: all:!localhost
  become: True
  roles:
    - role: ansible-access

Very basic/simple. The 4th stage is to execute the Testinfra test script from the molecule directory to the staging server to verify the correct installation/configuration. In this case I used the same tests as Molecule, but I could also create a seperate file with some extra or other tests to verify the correct behaviour of the host.

And when all tests are complete, we create a new tag. In this job we create a new tag ${params.COMMIT_ID}_staging and push it so we know that the provided tag is deployed to our staging server.

With the last stage, we start the 3rd and last job, the job to deploy the role on the rest of the servers.

Production deployment

This is the job that deploys the Ansible role to the rest of the servers. This Jenkinsfile looks almost the same as the previous one, but with a few exceptions.

node() {
    try {
        stage ("Get the Code") {
            checkout scm: [$class: 'GitSCM', branches: [[name: "refs/tags/${params.COMMIT_ID}"]], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'ansible-access']], userRemoteConfigs: [[url: 'ssh://git@192.168.1.206:2222/ansible/access.git']]]
            checkout scm: [$class: 'GitSCM', branches: [[name: '*/master']], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'environment']], doGenerateSubmoduleConfigurations: false, userRemoteConfigs: [[url: 'ssh://git@192.168.1.206:2222/ansible/environment.git']]]
            sh 'pwd > workspace'
        }
        stage ("Install Application Dependencies") {
            sh 'sudo pip install --upgrade ansible==${ANSIBLE_VERSION} testinfra docker'
        }
        stage ("Execute role on host(s)") {
            dir("environment") {
                sh "ansible-playbook -i hosts -l 'all:!localhost:!staging' playbooks/ansible-access.yml"
            }
        }
        stage ("Test Role execution") {
            workspace = readFile('workspace').trim()
            withEnv(["WORKSPACE_DIR=${workspace}", "MOLECULE_INVENTORY_FILE=${workspace}/environment/hosts"]){
                dir("environment") {
                    sh "testinfra --connection=ansible --ansible-inventory=hosts --hosts='all:!localhost:!staging' ${WORKSPACE_DIR}/ansible-access/molecule/default/tests/test_default.py --verbose"
                }
            }
        }
        stage('Tag git'){
            withEnv(["COMMIT_ID=${params.COMMIT_ID}_production"]){
                dir("ansible-access") {
                    sh '''#!/bin/bash
                    if [[ $(git tag | grep "${COMMIT_ID}" | wc -l) -eq 1 ]]
                        then    echo "Tag already exists"
                        else    echo "Tag will be created"
                                git config user.name "jenkins"
                                git config user.email "jenkins@localhost"
                                git tag -a $COMMIT_ID -m "Added tagging"
                                git push --tags
                    fi
                    '''
                }
            }
        }
    } catch(all) {
        currentBuild.result = "FAILURE"
        throw err
    }
}

With the 3rd stage “Execute role on host(s)” we use an different limit. We now use all:!localhost:!staging to deploy to all hosts, but not to localhost and staging. Same is for the 4th stage, for executing the tests. As the last stage in the job, we create a tag ${params.COMMIT_ID}_production and we push it. Once we see this tag in our repository, we know that the changes is installed correctly on all servers.

Keep in mind that this can only be successful if you use proper and correct tests. You’ll really need to be sure that your tests is covering all of the components that is changed by your role. This deployment will fail or succeed with the quality of your tests.

Good luck and if you have suggestions please let me know.

Advertisements

Using Molecule V2 to test Ansible Roles

Its been a few weeks now since Molecule V2 was released. So lets go into some details with Molecule V2 and lets upgrade my dj-wasabi.zabbix-agent role to Molecule V2 during this blogpost.

For those who are unfamiliar with Molecule: Molecule allows you to development and test Ansible Roles. With Molecule, 1 or more Docker containers are created and the Ansible role is executed on these Docker containers (You can also configure Vagrant and some other providers). You can then verify if the role is installed/configured correctly in the container. Is the package installed by Ansible, is the service running, is the configuration file correctly placed with the correct information etc etc.

This would allow you to increase reliability and stability of your Role. For almost all of my publicly available Ansible Roles, I have tests configured. If someone makes an Pull Request on Github with a change, these tests will help me to see if the change won’t break anything and thus the Pull Request can easily been merged. If not, some more attention to the change is needed.

This might be obvious, but if you do not have Molecule installed or already have something installed lets update it to the latest version (As moment of writing 2.0.3):

pip install —upgrade molecule

When we execute the –version:

$ molecule --version
molecule, version 2.0.3

We see the version: 2.0.3

Porting

The people behind Molecule have created a page for porting a role that is already configured with Molecule V1 to port it to Molecule V2. As the page mentions about a python script and doing it manually, we use the manually option for migrating the role to Molecule V2.

I have created a git branch (port_molecule_v2) on my mac and will execute the first command that is described on the porting guide:

(environment) wdijkerman@Werners-MacBook-Pro [ ~/git/ansible/ansible-zabbix-agent -- Tue Sep 05 13:16:00 ]
(port_molecule_v2) $ molecule init scenario -r ansible-zabbix-agent -s default -d docker
--> Initializing new scenario default...
Initialized scenario in /Users/wdijkerman/git/ansible/zabbix-agent/molecule/default successfully.
(environment) wdijkerman@Werners-MacBook-Pro [ ~/git/ansible/ansible-zabbix-agent -- Tue Sep 05 13:16:02 ]
(port_molecule_v2) $

This command will create a “default” scenario. The biggest improvement of using Molecule V2 is using scenarios. You can use 1 “default” scenario or you might want to use 5 scenario’s. Its completely up to you on how you want to test your role.

The init command has created a new directory called “molecule”. This directory will contain all scenario’s:

(environment) wdijkerman@Werners-MacBook-Pro [ ~/git/ansible/ansible-zabbix-agent -- Tue Sep 05 13:22:22 ] (port_molecule_v2) $ tree molecule
molecule
└── default
    ├── Dockerfile.j2
    ├── INSTALL.rst
    ├── create.yml
    ├── destroy.yml
    ├── molecule.yml
    ├── playbook.yml
    └── tests
        └── test_default.py

2 directories, 7 files

Here you see the “default” scenario we just created earlier. This scenario contains several files. We will discuss some files later on this post.

Back to the porting guide. The 2nd option on the porting guide is to move the current testinfra tests to the file molecule/default/tests/test_default.py. So lets move all the tests (And I mean only the tests and not the other testinfra specific code) from one file to the other. Keep the contents of the new test_default.yml in place, as this is needed for Molecule.

The 3rd option on the porting guide is for ServerSpec, as we don’t use this we will skip this and continue with the 4th option. The 4th option on the porting guide is to port the old molecule.yml file to the new one. Now its get interesting.

The current default molecule.yml file in the scenario/default directory:

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: instance
    image: centos:7
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

It will end like this:

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint

platforms:
  - name: zabbix-agent-centos
    image: milcom/centos7-systemd:latest
    groups:
      - group1
    privileged: True
  - name: zabbix-agent-debian
    image: maint/debian-systemd:latest
    groups:
      - group1
    privileged: True
  - name: zabbix-agent-ubuntu
    image: solita/ubuntu-systemd:latest
    groups:
      - group1
    privileged: True
  - name: zabbix-agent-mint
    image: vcatechnology/linux-mint
    groups:
      - group1
    privileged: True
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

platforms

The platforms is a generic configuration approach to configure the instances in Molecule V2. With Molecule V1, you’ll had a docker configuration, a vagrant configuration etc etc for configuring the instances, but with V2 you only have platforms.

In the above example I have configured 4 instances, named zabbix-agent-centos, zabbix-agent-debian, zabbix-agent-ubuntu and zabbix-agent-mint. The all have an image configured and I have placed them in the group1 group. I don’t do anything with the groups with this Role, but lets add them anyways. I also added the “privileged: True”, because the role does use systemd and needs a privileged container to execute successfully. Later in this blog post we do something with dependencies and some Ansible configuration, so don’t run away just yet. 😉

The 5th option in the porting guide is to port the existing playbook.yml to the new playbook.yml in the default directory. So I’ll move the contents from one file to an other file.

As 6th and last option in the porting guide is to cleanup the old stuff. So remove the old files and directories and we can continue with the molecule test command.

Lets execute it.

(port_molecule_v2) $ molecule test
--> Test matrix
    
└── default
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── converge
    ├── idempotence
    ├── lint
    ├── side_effect
    ├── verify
    └── destroy
--> Scenario: 'default'
--> Action: 'destroy'
    
    PLAY [Destroy] *****************************************************************
    
    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=(censored due to no_log))
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=1    changed=1    unreachable=0    failed=0
    
    
--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'syntax'
    
    playbook: /Users/wdijkerman/git/ansible/zabbix-agent/molecule/default/playbook.yml
    
--> Scenario: 'default'
--> Action: 'create'

There is a lot of output now which I won’t add now, but just take a look at the beginning which I pasted above this line. During the output it shows which scenario is executing and which task. You can see that the line begins with “–> Scenario: ” and with “–> Action: “.

This is why Molecule V2 is awesome:

Molecule V2 uses Ansible itself to create the instances on which we want to install/test our Ansible role. You can see that by opening the create.yml file in the default directory. If we just place the last task in this blogpost

- name: Create molecule instance(s)
  docker_container:
    name: "{{ item.name }}"
    hostname: "{{ item.name }}"
    image: "molecule_local/{{ item.image }}"
    state: started
    recreate: False
    log_driver: syslog
    command: "{{ item.command | default('sleep infinity') }}"
    privileged: "{{ item.privileged | default(omit) }}"
    volumes: "{{ item.volumes | default(omit) }}"
    capabilities: "{{ item.capabilities | default(omit) }}"
  with_items: "{{ molecule_yml.platforms }}"

This last task in the create.yml file will create the actual Docker instance which we have configured in the molecule.yml file in the “platforms” section, which you can see at the “with_items” option. This is very cool, this means that you can configure the docker container with all the settings that Ansible allows you to use and Molecule will not limit this for you.

You can easily add for example the “oom_killer” option to the create.yml playbook and add it to the platform configuration in molecule.yml, without adding an feature request at Molecule and waiting when the feature is implemented.  Not that the waiting was long, the people behind Molecule are very fast fixing issues and adding features, so kuddo’s to them!

As you have guessed already, the create.yml file is for creating the instanced and destroy.yml will destroy those instances. You can override this if you don’t like the names.

This is an example if you really want to use other names for the playbooks (Or if you want to share playbooks when you have multiple scenario’s):

provisioner:
  name: ansible
  options:
    vvv: True
  playbooks:
    create: ../playbook/create-instances.yml
    converge: playbook.yml
    destroy:../playbook/destroy-instances.yml

Back to the molecule test command. The molecule test command fails on my first run during the lint action. (I will not show all output, as the list is very long!)

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /Users/wdijkerman/git/ansible/zabbix-agent/...
    /Users/wdijkerman/git/ansible/zabbix-agent/defaults/main.yml
      7:37      warning  too few spaces before comment  (comments)
      10:17     warning  truthy value is not quoted  (truthy)
      15:81     error    line too long (120 > 80 characters)  (line-length)
      21:81     error    line too long (106 > 80 characters)  (line-length)
      30:30     warning  truthy value is not quoted  (truthy)
      31:26     warning  truthy value is not quoted  (truthy)
    
    /Users/wdijkerman/git/ansible/zabbix-agent/handlers/main.yml
      8:11      warning  truthy value is not quoted  (truthy)
      8:14      error    no new line character at the end of file  (new-line-at-end-of-file)

Some of these messages is something I can work with, some I actually do not care. The output shows you every “failing” rule with the file. So the first file, defaults/main.yml has 6 failing rules. Per rule it shows you the following:

  • which line and character position
  • Type of error (warning or error)
  • The message

In my output of the lint actions, I see a lot of “line too long” messages. Personally I find the 80 characters limit a little bit to small these days, so lets update it to something higher. We first have to update the molecule.yml file and we have to update the lint section. First the lint section looked like this:

lint:
  name: yamllint

Now configure it like so it looks like this:

lint:
  name: yamllint
  options:
    config-file: molecule/default/yaml-lint.yml

We specify the yamllint by configuring a configuration file. Lets create the file yaml-lint.yml in the default directory and add something like this:

---

extends: default

rules:
  line-length:
    max: 120
    level: warning

We extend the current yaml-lint configuration by adding some of our own rules to overwrite the defaults. In this case, we overwrite the “line-length” rule to set the max to 120 characters and we set the level to warning (It was error). Every rule that results in an error will fail the lint action and in this case I don’t want to fail the tests because the line length was 122 characters.

When we run it again (I have fixed some other linting issues now, so output is a little different)

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /Users/wdijkerman/git/ansible/zabbix-agent/...
    /Users/wdijkerman/git/ansible/zabbix-agent/defaults/main.yml
      15:81     warning  line too long (120 > 80 characters)  (line-length)
      21:81     warning  line too long (106 > 80 characters)  (line-length)
    
    /Users/wdijkerman/git/ansible/zabbix-agent/molecule/default/create.yml
      9:81      warning  line too long (87 > 80 characters)  (line-length)
      10:81     warning  line too long (85 > 80 characters)  (line-length)
      16:81     warning  line too long (116 > 80 characters)  (line-length)
      30:81     warning  line too long (92 > 80 characters)  (line-length)
      33:81     warning  line too long (124 > 80 characters)  (line-length)

It keeps showing the “line too long”,  but as a warning and the lint action continues working. After this, the verify works too and the test is done!

Well, now I can commit my changes and push them to GitHub and lets Travis verify that it works. (Will not discuss that here).

group_vars

The zabbix-agent role doesn’t have any group_vars configured, but some of my other roles have group_vars configured in Molecule. Lets give a basic example of configuring the pizza property in the group_vars.

We have to update the provisioner section in molecule.yml:

provisioner:
  name: ansible
  lint:
    name: ansible-lint
  inventory:
    group_vars:
      group1:
        pizza: "Yes Please"

Here we “add” a property named pizza for all hosts that are in the group “group1”. If we had configured this with earlier on with the zabbix-agent role, all of the configured instances had access to the pizza property.

What if we have multiple scenarios and all use the same group_vars? We can create in the git_root directory of the role a directory named inventory and this has 1 or 2 subdirectories: group_vars and host_vars (if needed). To make the pizza property work, we create a file inventory/group_vars/group1 and add

---
pizza: "Yes Please"

Then we update the provisioner section in molecule.yml:

provisioner:
  name: ansible
  inventory:
    links:
      group_vars: ../../../inventory/group_vars/
      host_vars: ../../../inventory/host_vars/

Is this awesome or not?

Dependencies

This is almost the same as with Molecule V1, but with Molecule V2 the file should be present in the specific scenario directory (In my case molecule/default/) and should have the name requirement.yml.

The file requirement.yml is still in the same format as how it was (As this is specific to Ansible and not Molecule ;-))

---
- src: geerlingguy.apache
- src: geerlingguy.mysql
- src: geerlingguy.postgresql

If you want to add some options, you can do that by changing the dependency section of molecule.yml:

dependency:
  name: galaxy
  options:
    ignore-certs: True
    ignore-errors: True

With Molecule V1, there was a possibility to point to a requirements file, with Molecule V2 not.

ansible.cfg

This file is not needed anymore, we can all do this with the provisioner section in molecule.yml. So we don’t have to store the ansible.cfg and point it to the molecule.yml file like how it was with Molecule V1.

Lets say we have an ansible.cfg with the following contents:

[defaults]
library = Library

[ssh_connection]
scp_if_ssh = True

We can easily do this by updating the provisioner section to this:

provisioner:
  name: ansible
  config_options:
    defaults:
      library: Library
    ssh_connection:
      scp_if_ssh: True

TL;DR

Just upgrade to Molecule V2 and have fun! This is just awesome.

@Molecule coders: Thank you for this awesome version!

Testing Ansible roles in a cluster setup with Docker and Molecule

ansible_logo_black_square

This page is updated on 2017-05-01.

This is a follow up on the previous 2 blog posts. With the first blog we discussed some steps to test your Ansible role with some basic steps. With the 2nd blog we added some features to extend our test by using CI tooling and group vars. With this post however, we might be configuring Molecule that will not occur very common with testing Ansible roles.

This time we configure Molecule for a role that is installing and configuring a cluster on Docker, like MySQL or MongoDB. We don’t go into a specific role (as there are so many), I only give some information on how to do this. We are only configuring Molecule for this setup, I’m still busy with running some specific TestInfra tests on a specific container.

Keep in mind these actions are only needed when using the {{ ansible_eth0.ipv4.address }} isn’t enough and you need a list with all ips.

For configuring Molecule, we’ll have to change 2 files:

  1. molecule.yml
  2. playbook.yml

molecule.yml

First we update the ‘molecule.yml’ file by configuring 3 (Or more, depends on what you need) docker containers. See the following example:

docker:
  containers:
  - name: node1
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3306: 3306,
      4444: 4444
  - name: node2
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3307: 3306,
      4445: 4444
  - name: node3
    ansible_groups:
      - cluster_service
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
    port_bindings:
      3308: 3306,
      4446: 4444

As you see, I have added the ‘port_bindings’ configuration to the instances, which is different with the examples in the previous blog posts. With the containers, we open the ports on the host (Before the ‘:’) and proxy them to the ports to the docker container (After the ‘:’ ).

In the above example, this ports configuration is used for configuring a MySQL (Or MariaDB) Galera cluster setup. You’ll have to update the ports configuration to your needs.

playbook.yml

Before we specify the roles in the ‘playbook.yml’, we add an ‘pre_tasks’. We add 2 blocks of code and will discuss them one by one. First we add the following in the ‘pre_tasks’ part:

    - name: "Get ip node 1"
      local_action: shell docker inspect --format \{\{.NetworkSettings.IPAddress\}\} node1
      register: node_ip_1
      changed_when: False
    - name: "Get ip node 2"
      local_action: shell docker inspect --format \{\{.NetworkSettings.IPAddress\}\}  node2
      register: node_ip_2
      changed_when: False
    - name: "Get ip node 3"
      local_action: shell docker inspect --format \{\{.NetworkSettings.IPAddress\}\}  node3
      register: node_ip_3
      changed_when: False

We have added 3 tasks, that do the same command but for each container. We execute a ‘docker inspect’ command to get the ip address of the docker container. As the docker inspect command used the ‘{{ .NetworkSettings.IPAddress }}’ format, Ansible will try to replace it because it thinks it is a variable. Luckily, we can make use of ‘{{ raw }} {{ endraw }}’ for this and Ansible will not use this as a variable anymore.

We register a variable, because we need to use the output, because the ‘docker inspect’ outputs the ip address. We also have to add the property ‘changed_when: False’.

What do you mean with the last one?

We have to fool Ansible with the ‘changed_when’ command for the ‘idempotence’ check. As this task will run every time, the ‘idempotence’ check will fail because of it (because it sees that there are tasks with state “Changed”).

Now we add the 2nd block of tasks to the ‘pre_tasks’, just after the first block of code we added earlier:

    - name: "Set fact"
      set_fact:
        node_ip: "{{ node_ip_1.stdout }}"
      when: inventory_hostname  == 'node1'
    - name: "Set fact"
      set_fact:
        node_ip: "{{ node_ip_2.stdout }}"
      when: inventory_hostname  == 'node2'
    - name: "Set fact"
      set_fact:
        node_ip: "{{ node_ip_3.stdout }}"
      when: inventory_hostname  == 'node3'

With this block we add 3 tasks again, 1 task for each container, to create a fact. In this case, I used to create the fact with the name ‘node_ip’, but you may name it differently. In my case when I use the ‘node_ip’ in my Ansible role, it get the actual IP of the host.

You can also create a list with all the ips if you need this in your role:

cluster_host_ips:
  - "{{ node_ip_1.stdout }}"
  - "{{ node_ip_2.stdout }}"
  - "{{ node_ip_3.stdout }}"

(You have to use the corrent name for the list of course 😉 )

I don’t know if this is the correct way, but it works in my case. I have a Jenkins job that validates a role that is configured to run a 3 node setup (Elasticsearch, MariaDB and some others).

If you have a other or a better way for doing this, please let me know!

Extending Ansible Role testing with Molecule by adding group_vars, dependencies and using travis ci

ansible_logo_black_square

This blog post will extend the actions we described on this https://werner-dijkerman.nl/2016/07/10/testing-ansible-roles-with-molecule-testinfra-and-docker/ page. We only configured a very basic configuration with 2 tests in the previously mentioned page, but thats not enough to and on this page we will continue to complete the tests. Github page for Molecule (In case you forgot 😉 )

We will discuss the following in this blogpost:

  • Make the TestInfra tests OS aware
  • Use group_vars
  • Role dependencies
  • Configure Travis

Lets dive into the tasks.

TestInfra OS Aware

Ok, not really molecule related, but very important if our role needs to work on several different operating systems. In the earlier mentioned blogpost we had configured Molecule to use a Ubuntu docker image. Before we can make the tests OS aware, we have to add some docker containers to the configuration which have different operating systems.

We create the following configuration:

docker:
  containers:
  - name: zabbix-agent-centos
    ansible_groups:
      - group1
    image: milcom/centos7-systemd
    image_version: latest
    privileged: True
  - name: zabbix-agent-debian
    ansible_groups:
      - group2
    image: maint/debian-systemd
    image_version: latest
    privileged: True
  - name: zabbix-agent-ubuntu
    ansible_groups:
      - group2
    image: rastasheep/ubuntu-sshd
    image_version: latest
    privileged: True

We have 3 docker containers configured: Ubuntu, Debian and CentOS. I’ve searched for Docker containers that have systemd configured, as the Zabbix Agent uses this. The official docker images of the mentioned OS’es doesn’t have the systemd enabled/configured.

We would like to run the ‘molecule test’ command, but it will fail on the ‘verify’ part.
We currently have this test:

def test_zabbix_package(Package, SystemInfo):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed

    assert zabbixagent.version.startswith("3.0")

This test will validate if Zabbix is installed and the version starts with 3.0 (Thats the default version to be installed). When running this test, it will fail on the Debian and Ubuntu host. Why?They use a slightly different version naming than CentOS.

We have to update the function by adding the SystemInfo class to the function:

def test_zabbix_package(Package, SystemInfo):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed

    if SystemInfo.distribution == 'debian':
        assert zabbixagent.version.startswith("1:3.0")
    if SystemInfo.distribution == 'centos':
        assert zabbixagent.version.startswith("3.0")

Now we have added the SystemInfo class to the function, we can use this to determine which tests we can execute on which OS. In the above example we see that if the distribution ‘debian’ is, we validate if the version starts with ‘1:3.0’. With CentOS we validate if it is ‘3.0’. When we execute the test on all 3 hosts, it will run successfully.

Using group_vars

This part also applies to using ‘host_vars’, just replace the word ‘group_vars’ with ‘host_vars’ 😉 We configure the molecule.yml file by adding some group_vars related data. We configure the 1st block in the molecule.yml like this:

ansible:
  playbook: playbook.yml
  group_vars:
    mysql:
      database_type: mysql
      database_type_long: mysql
    postgresql:
      database_type: pgsql
      database_type_long: postgresql
      postgresql_pg_hba_conf:
        - "host all all 127.0.0.1/32 trust"
        - "host all all ::1/128 trust"
      postgresql_pg_hba_local_ipv4: false
      postgresql_pg_hba_local_ipv6: false

This setup will “create” 2 group_var files: mysql and postgresql. When we run a molecule command, the group_vars will be created in the .molecule directory. Lets run a ‘molecule list’ command. For know we ignore the output, but lets see what is created in the .molecule directory:

[vagrant@localhost ansible-zabbix-server]$ find .molecule -type f | xargs ls -lrt
-rw-rw-r--. 1 501 games   3 Jul 17 20:31 .molecule/state
-rw-rw-r--. 1 501 games 215 Jul 17 20:31 .molecule/group_vars/postgresql
-rw-rw-r--. 1 501 games  51 Jul 17 20:31 .molecule/group_vars/mysql
[vagrant@localhost ansible-zabbix-server]$ cat .molecule/group_vars/mysql
---
database_type: mysql
database_type_long: mysql
[vagrant@localhost ansible-zabbix-server]$ cat .molecule/group_vars/postgresql
---
database_type: pgsql
database_type_long: postgresql
postgresql_pg_hba_conf:
- host all all 127.0.0.1/32 trust
- host all all ::1/128 trust
postgresql_pg_hba_local_ipv4: false
postgresql_pg_hba_local_ipv6: false

Within the .molecule directory a group_vars directory is created and 2 files are present (We will ignore the ‘state’ file). The output of these files are the same as how we configured the molecule.yml file.

Role dependencies

When your role has some dependencies, we really want them to download before we execute our role in Molecule. It will fail if we don’t do this. We have to download them first.

Molecule has a simple configuration for this. Within the molecule.yml file, we add the ‘requirements_file’ option in the Ansible configuration. Our example now looks like this:


dependency:
  name: galaxy
  requirements_file: requirements.yml 
  options:
    ignore-certs: True
    ignore-errors: True

ansible:
  playbook: playbook.yml
  group_vars:
    mysql:
      database_type: mysql
      database_type_long: mysql
    postgresql:
      database_type: pgsql
      database_type_long: postgresql
      postgresql_pg_hba_conf:
        - "host all all 127.0.0.1/32 trust"
        - "host all all ::1/128 trust"
      postgresql_pg_hba_local_ipv4: false
      postgresql_pg_hba_local_ipv6: false

In the root directory of the repository, a file called ‘requirements.yml’ is found which contains the dependencies.

When we run the ‘converge’ subcommand, the roles will be downloaded and after this the role is executed:

[vagrant@localhost ansible-zabbix-server]$ molecule converge
WARNING:vagrant:The Vagrant executable cannot be found. Please check if it is in the system path.
--> Installing role dependencies ...
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/1.7.2.tar.gz
- extracting geerlingguy.apache to .molecule/roles/geerlingguy.apache
- geerlingguy.apache was installed successfully
- downloading role 'mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/2.3.0.tar.gz
- extracting geerlingguy.mysql to .molecule/roles/geerlingguy.mysql
- geerlingguy.mysql was installed successfully
- downloading role 'postgresql', owned by galaxyprojectdotorg
- downloading role from https://github.com/galaxyproject/ansible-postgresql/archive/0.9.2.tar.gz
- extracting galaxyprojectdotorg.postgresql to .molecule/roles/galaxyprojectdotorg.postgresql
- galaxyprojectdotorg.postgresql was installed successfully
--> Starting Ansible Run ...

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [zabbix-server-centos-mysql]
... 
<snip>

Configure CI

We don’t want to run manually the molecule commands every time a change occurs. To quote Apple: “There is an app for that”. Well almost. I explain 2 ways of running Molecule in a automated way: Using Travis and Using Jenkins.

Travis

A commonly known cloud platform for CI, and it is very simple to make use of it. The Molecule documentation has a very nice example on how to configure the travis setup. We create in the root directory of our role, a file named: .travis.yml

sudo: required
language: python
services:
- docker

before_install:
- sudo apt-get -qq update
- sudo apt-get install -o Dpkg::Options::="--force-confold" --force-yes -y docker-engine

install:
- pip install molecule ansible docker

script:
- molecule test
notifications:
  webhooks: https://galaxy.ansible.com/api/v1/notifications/

Its a very basic travis configuration. First block is to let Travis know that we need to make use of sudo, that the language Python is and we use the ‘docker’ service.

2nd block is to update the Ubuntu’s apt cache and install Docker. When Docker is installed, we install molecule and after that, molecule test is executed. When the job is finished, we send some data via a web hook to the Ansible Galaxy page. The last step is to show the ‘passing’ (Or failing) badge on the Ansible Role page.

Jenkins

This example will show you a basic Jenkinsfile. I don’t do a single action manually in Jenkins. A  Jenkins job is basically code and we should use it as code. With Jenkins 2, we can make use of a Jenkinsfile (Also with Jenkins 1, but was named jenkins-template I believe?)

When you have a Jenkins server running, create a new Pipeline job and configure the correct git repository. (Okay, for this blogpost we do this manually, but I have a playbook for this. So this is on my part automated as well. 😉 )

A basic example:

node(){
    stage 'INFO: Checkout code'
        checkout scm
    stage 'Installing Molecule'
        sh 'sudo pip install molecule'
    stage 'Creating Containers'
        sh 'molecule create'
    stage 'Installing the Role'
        sh 'molecule converge'
    stage 'Idempotence check'
        sh 'molecule idempotence'
    stage 'Verify the application'
        sh 'molecule verify'
    stage 'Destroy the containers'
        sh 'molecule destroy'
}

When the job is executed, it will first download the Jenkinsfile from the configured git repository and the tasks will be executed one by one. This is just to get you started and the Jenkinsfile should be extended with some error handling like e-mail configuration etc.

So this was the follow up on the original blogpost for how to use Molecule with your Ansible role. The guys are really busy with Molecule so Molecule grows really fast. Maybe doing an 3rd blogpost very soon with new features.. 🙂

Testing Ansible roles with Molecule, Testinfra and Docker

ansible_logo_black_square

“On 2017-05-01 I’ve updated this post to the current situation. Some things where outdated and where removed.”

In some earlier posts I’ve described how you can use Test Kitchen for testing Ansible Roles (This one and the one extending it.). Test Kitchen was created for testing Chef Cookbooks and like Chef, Test Kitchen is a Ruby application. On this page we describe an other tool for the same purpose. This tool is what you might see as a Python clone of Test Kitchen, but more specific to Ansible: Molecule (Github)

Molecule isn’t that old, only few years and when I browse the internet it is not yet really known in the community. Unlike Test kitchen with the many different drivers, Molecule supports several backends, like Vagrant, Docker, and OpenStack. With Molecule you can make use of Serverspec (Like Test Kitchen), but you can also make use of ‘Testinfra’. Testinfra is like Serverspec a tool for writing unit tests, but it is written in Python.

Lets dive into Molecule and create some tests for Molecule. On this page, we make use of the Docker backend and if you following this page please install docker.

Installing Molecule is really simple:

pip install molecule docker

Voila, it is installed. With the installation of Molecule, Testinfra is installed to. We had to provide the docker module as well, otherwise molecule doesn’t know how to connect to the docker daemon. We can configure a Ansible Role. I used my ‘zabbix-agent’ role as test case for the Test Kitchen setup, so I will use it again for Molecule.

When you haven’t created an Ansible role yet, instead of using the ansible-galaxy command you can use the following command:

molecule init --driver docker --role role_name

This will create just like the ‘ansible-galaxy’ some default directories and files, but also gives us a starting point with a few extra files for testing this role with Molecule.

When you already have a working module and want to make use of Molecule, please execute the following command:

molecule init --driver docker

This will install several files specific for Molecule. No worries, we can recreate these files manually. Lets do that in a role and see what the files do.

File: <root>/molecule.yml

---
ansible:
  playbook: playbook.yml

driver:
  docker
docker:
  containers:
    - name: zabbix-01
      ansible_groups:
        - group1
      image: debian
      image_version: latest
      privileged: True

verifier:
  name: testinfra

This is the configuration file for Molecule. We specify which playbook Molecule will execute, in this case playbook.yml.
We specify that we want to make use of the Docker driver and that we have a docker container configuration. In this case, we only have 1 docker container specified. We use a Debian docker container with the ‘latest’ tag. We name the container ‘test-01’ and is in the group ‘group1’. And at last we configure molecule to use testinfra as the testtool.

File: <root>/playbook.yml

---
- hosts: all
  roles:
    - role: ansible-zabbix-agent

The playbook that is executed in the Docker container. This is a very basic one, we only have to specify the correct .

File: <root>/tests/inventory

localhost
[group1]
zabbix-01 ansible_connection=docker

The Ansible inventory file. Should be a known file to you 😉

File: <root>/tests/test_default.py


from testinfra.utils.ansible_runner import AnsibleRunner
testinfra_hosts = AnsibleRunner('.molecule/ansible_inventory').get_hosts('all')

def test_hosts_file(File):
    hosts = File('/etc/hosts')
    assert hosts.user == 'root'
    assert hosts.group == 'root' 

(Edit 2016-09-14: As of release 1.9, the first 2 lines should be present in the TestInfra script.)

This is the test infra python file, containing the tests. After the init command, we have 1 test that will check if there is a hosts file, and the user and group of the file belongs to user ‘root’. We discuss this file later on by adding some more tests.

File: <root>/tests/test.yml

---
- hosts: localhost
  remote_user: root
  roles:
    - zabbix-agent-role

Now we have discussed the files.

We add some infra test checks in the ‘test_default.py’ file. We add the following 2 tests:

def test_zabbix_package(Package):
    zabbixagent = Package('zabbix-agent')
    assert zabbixagent.is_installed
    assert zabbixagent.version.startswith("1:3.0")

def test_zabbixagent_running_and_enabled(Service):
    zabbixagent = Service("zabbix-agent")
    # assert zabbixagent.is_running
    assert zabbixagent.is_enabled

These are 2 Python function which are executed with Testinfra. With the first function, we validate if the package ‘zabbix-agent’ is installed. Also we check if the version starts with: 1:3.0. If you have some experience with testing Python code, this might be familiar to you. Test Infra uses ‘PyTest‘ to execute the tests and validate the.

The 2nd function we validate the ‘zabbix-agent’ service. We make sure the service is enabled. As you see, I’ve commented the check if the service is running. When it is enabled, I get this error message:

Failed to get D-Bus connection: Unknown error -1

Strange, because I’ve configured the privileged mode on the docker container. So maybe this is a bug or misconfiguration on my part, but for now I leave it commented and need to find a solution for this.

Within the molecule.yml we have to update the docker container configuration by adding the following property for all the docker containers:

    required: True

Now we added it, we have to do a “molecule destroy” and start again. The container will be recreated and we won’t get an error message about the dbus.

Now we are ready to move on (I’m well aware that these 2 tests that I added will not be enough, I’ll add these myself later on).

Molecule has several subcommands, let run molecule -h and see what is available:

No handlers could be found for logger "vagrant"
Usage:
    molecule [-hv] &amp;amp;lt;command&amp;amp;gt; [&amp;amp;lt;args&amp;amp;gt;...]

Commands:
    check         check playbook syntax
    create        create instances
    converge      create and provision instances
    idempotence   converge and check the output for changes
    test          run a full test cycle: destroy, create, converge, idempotency-check, verify and destroy instances
    verify        create, provision and test instances
    destroy       destroy instances
    status        show status of instances
    list          show available platforms, providers
    login         connects to instance via SSH
    init          creates the directory structure and files for a new Ansible role compatible with molecule

Options:
    -h --help     shows this screen
    -v --version  shows the version

We first start with the ‘check’ command:

[vagrant@localhost ansible-zabbix-agent]$ molecule check
No handlers could be found for logger "vagrant"

playbook: playbook.yml
[vagrant@localhost ansible-zabbix-agent]$ echo $?
0

Seems very well, the check commands validate if the playbook.yml doesn’t have any problems/syntax errors.
We can continue with the next command: create.

[vagrant@localhost ansible-zabbix-agent]$ molecule create
No handlers could be found for logger "vagrant"
 Building ansible compatible image ...
 Step 1 : FROM debian:latest

  ---&amp;amp;gt; 1b088884749b

 Step 2 : RUN bash -c 'if [ -x "$(command -v apt-get)" ]; then apt-get update &amp;amp;amp;&amp;amp;amp; apt-get install -y python sudo; fi'

  ---&amp;amp;gt; Using cache

  ---&amp;amp;gt; 8ef54383599a

 Step 3 : RUN bash -c 'if [ -x "$(command -v yum)" ]; then yum makecache fast &amp;amp;amp;&amp;amp;amp; yum update -y &amp;amp;amp;&amp;amp;amp; yum install -y python sudo; fi'

  ---&amp;amp;gt; Running in 6d3142fa72aa

...
 Finished building molecule_local/debian:latest
 Creating container zabbix-01 with base image debian:latest ...
 Container created.

[vagrant@localhost ansible-zabbix-agent]$

Now we have created a docker container where we can install our Ansible role on to, we do that with the ‘converge’ subcommand.

[vagrant@localhost ansible-zabbix-agent]$ molecule converge
No handlers could be found for logger "vagrant"

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [zabbix-01]

TASK [ansible-zabbix-agent : Include OS-specific variables] ********************
ok: [zabbix-01]

TASK [ansible-zabbix-agent : Install the correct repository] *******************
skipping: [zabbix-01]

...
RUNNING HANDLER [ansible-zabbix-agent : restart zabbix-agent] ******************
changed: [zabbix-01]

PLAY RECAP *********************************************************************
zabbix-01                  : ok=12  changed=7    unreachable=0    failed=0

Nice, the role is installed correctly without any issues on the container. With Test Kitchen we had to use BATS to validate if the Role is idempotent, but luckily molecule has just a simple sub command for it: idempotence

Well, it seems that the Role has passed the idempotence test:

[vagrant@localhost ansible-zabbix-agent]$ molecule idempotence
No handlers could be found for logger "vagrant"
Idempotence test in progress (can take a few minutes)...
Idempotence test passed.

[vagrant@localhost ansible-zabbix-agent]$

Testing the role is nicely going on right now, but we are not there yet. Now we need to use the ‘verify’ command to actually validate our role on the Docker container:

[vagrant@localhost ansible-zabbix-agent]$ molecule verify
No handlers could be found for logger "vagrant"
Trailing whitespace found in ./defaults/main.yml on lines: 35
Trailing newline found at the end of ./handlers/main.yml
Trailing whitespace found in ./library/zabbix_host.py on lines: 29
Trailing newline found at the end of ./library/zabbix_hostmacro.py
[vagrant@localhost ansible-zabbix-agent]$

Whoops, it seems it has found some issues. Let me fix that first, probably need to run the verify again after fixing it.

[vagrant@localhost ansible-zabbix-agent]$ molecule verify
No handlers could be found for logger "vagrant"

Executing testinfra tests found in tests/.
============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /git/ansible/ansible-zabbix-agent/tests, inifile:
plugins: xdist-1.14, testinfra-1.4.0
collected 3 itemss

tests/test_default.py ...

=========================== 3 passed in 0.63 seconds ===========================

No serverspec tests found in spec/.

[vagrant@localhost ansible-zabbix-agent]$

After fixing it, everything seems to work fine. Nice!

Now we are done with the container, so we can execute molecule again, but with the delete sub command and the container will be deleted.

These were the basics for testing an Ansible role with Molecule, Docker and Test Infra. This page uses the ‘Debian’ Docker image, whereas I normally use CentOS for this. I have some issues (Get the same error message when I enable the Test Infra test to validate if the service is running) to make this work on CentOS. So maybe Molecule isn’t mature enough yet, but it is getting there.

I’ll update my Ansible roles so it will use Molecule instead of Test Kitchen (No hard feelings ;-))