Book review: AWS Security by Dylan Shields

(Image comes from manning.com)

Security is one of these things that people/projects/companies only do with it, after they have build something or somewhere at the end of the project. And I believe even the author mentions something similar in this book as well, but compares it with baking a cake. Security is one of the ingredients of what you will be making, like it is a fundamental part of the thing (cake) you build (bake). And not that cherry on top of it. How many times do we see data breaches (Just follow Troy Hunt on twitter to get an idea).

The book is very well written and explains a lot of the AWS Security related services very well. Especially the IAM part if very good, clear and actually goes very deep in explaining the possibilities and what you can do with it. Also very nice to see about usecases when working with multiple AWS accounts and how IAM can work with that.

Altough other services like Cloudtrail, Config and Guardduty are explained in the book, I think it would be more interesting if it went a bit deeper as the chapters seems more high level and explaining some basic things. I would have loved to see some more in depts use cases with these services and even how it all works together with for example a Landing Zone setup where you have multiple AWS Accounts (Cross accounts). And an Landing Zone is already preferred from a security p.o.v., so unfortunately this is not part of the book. If you have a single AWS Account, then this book is definitively for you!

Even though the wanted-to-see-things, it was a very good book and helped me with my work. Happy to have bought it! An 7/10.

You can buy the book here on Amazon or on the site of Manning.

Advertisement

Book review: Infrastructure as Code, Patterns and Practices: With examples in Python and Terraform by Rosemary Wang

(Image comes from manning.com)

I am a huge supporter for automation, in any of the IT parts and I am personally focused and specialised on automation of Cloud Infrastructure (Most specifically towards AWS). So i was very eager to read the book, as I am always curious on if I could do anything better or learn something new that will make my work or project better.

The book was very nice and in some ways easy to read. The book is very well written and explains things very deeply. Like there are complete chapters about testing and branching models, where all kinds of test strategies are explained in detail. Also about the various kinds of branching models that exist and what the benefits are for these (of course with lot of information why you should choose this or not). I don’t think I have read this in any of the other automation books (yet). And actually to my suprise while reading it (of course later on I was like, yeah I could have expected such a topic) about tracking changes via the appropiate tools when you want to deploy to production. You can not make changes in production when you want to, unless you work with a CI/CD with automatic deploys to production. Mostly you need to have approvals to deploy something on production, which is also nicely described as well.

But, I do have an issue with the Python. I am all for Python, but I think the Python examples to generate the Terraform seems a bit to much and not KISS. I think writing the Terraform code itself is easier to understand and show.

It was a really nice book to read and it made sure to freshen some things that I have done before, I did not actual learn something that I could have used in my day-to-day work. Nevertheless, a decent 7/10.

You can buy the book here on Amazon or on the site of Manning.

Write your own pre-commit hooks

In one of my previous blog posts I described that using pre-commit hooks makes life easier, it helps you writing better code. When you want to commit your changes, you immediately get a result if your code has met the various criteria the owner of the repository has set. When you google around, you will find various scripts that you can make use in your own setup. But what if you can not find that specific one? Then you will need to create your own hook.

I had to do this as well, not because I had a rare case to solve, I just wanted to learn how to write a hook myself so I can easily expand my own pre-commit library. And it seems out to be very simple. 🙂

Use case

My very basic use case: I want to validate a yaml file and make sure that it is properly formatted.

I found a very basic tool that helped me format an yaml file, named “yamlfmt“. I need to use this tool in a bash script, which you can see here:

#!/usr/bin/env bash
# Will pretty print YAML files.

if which yamlfmt &> /dev/null $? != 0 ; then
    echo "yamlfmt must be installed: pip install yamlfmt"
    exit 1
fi

EXIT_CODE=0

for file in $@; do
  yamlfmt ${file} -w
  if [[ $? -ne 0 ]]
    then  EXIT_CODE=1
          echo $file
  fi
done
exit $EXIT_CODE

First we need to make sure that we check that the tool is installed, otherwise we print an statement on how the user can install the tool. We need to have that tool installed, so we immediately exit the script with an exit code of 1. With that, the “git commit” action will also fail and thus the user needs to take action to install the tool before it can try to commit the changes.

The last part of the script is doing a for loop and will run the “yamlfmt” command on each file that is passed as an argument to the script. With each pre-commit hook, all of the files that are part of the commit are passed as an argument to the script (Not entirely, but we will discuss this a bit further on ;-)). We collect the exit code of the “yamlfmt” command and checks if that will not equals to 0. If that happens, then there will be an issue with the yaml file and we print the name of the file to stdout.

But how does the script know that it should only do yaml files? If we have a text file or png file, this will fail!

First, we need to add this script into a repository that contains other pre-commit hooks scripts. If you don’t have one, or don’t have other scripts is also fine. I have a dedicated repository for that, which you can find here https://github.com/dj-wasabi/pre-commit-hooks. This repository contains all the pre-commit hooks that I use in 1 or more (public) available repositories. We just need to have a git repository where we can store this script and most importantly, we need to store a “.pre-commit-hooks.yaml” file. This file will contain some information about the scripts which we can use in the rest of our repositories.

In order to get the above menionted script work, we store this script inside the “bin” directory in the git repository and name the file “verify-yaml.sh“. It doesn’t have to be specific in the “bin” directory, you can also just place it in the “ROOT” or some other directory, whatever you please. But in the “ROOT” of the git repository we will have to create the “.pre-commit-hooks.yaml” file and we will include the following contents:

- id: verify-yaml
  name: Pretty Print YAML files
  description: Checks YAML files and pretty prints them
  entry: bin/verify-yaml.sh
  language: script
  files: \.(ya?ml)$

There are 2 important keys that requires some additional information (I won’t have to tell you that the “entry” key is the location to the script right? Oh wait, I just did. :)) These are the “id” and the “files“.

The “id” is the value that you need to use in your repositories where you want to make use of the pre-commit hooks. Like with the https://github.com/dj-wasabi/dj-wasabi-release/blob/main/.pre-commit-config.yaml you will see the following:

repos:
- repo: https://github.com/dj-wasabi/pre-commit-hooks
  rev: master
  hooks:
  ...
  - id: verify-yaml
  ...

So every time I do a commit in my “dj-wasabi/dj-wasabi-release” repository, it will execute the pre-commit hook with id “verify-yaml“.

The “files” key in the “.pre-commit-hooks.yaml” entry is when the script should be executed. We only want the script to be executed when the file is a yaml file and not with a text or png file. The “\.(ya?ml)$” makes sure that it only affect files with the file extention “.yml” and “.yaml“.

Thats all folks!

It looks very easy right? Yes it sure is and when you start to work on your own hooks, you will create more. Because I am in a writing mood right now, lets take a look at the following script that I wrote:

#!/usr/bin/env bash
# Do not allow commits on provided branches.


EXITCODE=0
while getopts b: flag
do
    case "${flag}" in
        b) BRANCHES=${OPTARG};;
        *) echo "Unsupported option provided."
           exit 1;;
    esac
done

for BRANCH in $(echo ${BRANCHES} | sed 's/,/ /g');
    do
        if git rev-parse --abbrev-ref HEAD | grep -e ${BRANCH} > /dev/null 2>&1
            then  echo "You are not allowed to commit on ${BRANCH} branch."
                  EXITCODE=1
        fi
done
exit ${EXITCODE}

I don’t want to commit my changes on “master” or on “main“. And yes you can configure in most cases on the remote site (Read: the Git Server, like Bitbucket or Github) that it will not allow pushes to “master” or “main“, but I just want to prevent the commiting to actual take place. Everytime when I commit on a branch that the remote server is not accepting, I have to google for “git commit undo” and look for the command to undo my commit (Because for some reason I can not remember the git undo command :)).

Once I have undo my commit, I will create a proper branch and do the commit and push again. So if I can prevent committing to “master” or “main” with a pre-commit hook, my life will get easier because I don’t have to undo my commit etc! (Wait, wasn’t that the title of the blog post that I have written before this blog post? ) 🙂

If you have any question and or remarks, please let me know. If you have written a hook yourself and you want to show it, let me know as well!

May the hooks be with you!

Using pre-commit hooks makes software development life easier

Pull requests, you either love and see that they do provide benefits in the way of working, or you dislike them and see no purpose in them at all. I discussed the Pull Requests process in these “Author“, “Reviewer” and “Process” blogposts, so check these out as well. No matter which side you are on, but I think you want to create quality code and be consistent with each change. And if you are like me, then you would expect this as well from your teammates/co-workers. But how can we make that happen?

Pre-commit hooks can help with that. A pre-commit is one of the several hooks that are available in git that we can use to execute scripts, while running an git command. A full list with hooks can be found on this page https://githooks.com/

With a pre-commit hook, we can execute scripts as part of the “git commit” command. If the exit code of all of the scripts result in a zero (successful execution) then the commit will take place. If 1 or more of the scripts fail, the commit will be unsuccessful and fails to complete. Once that is happening we need to resolve the issue and try again.

When you have a git repository somewhere cloned on your host, you are already able to use pre-commit hooks or any of the other hooks that Git has. In the “.git/hook” directory you will find a bunch of “.sample” scripts. You can use these as an example and if you remove the “.sample” in the filename, then the script is active and triggered when the stage is executed.

$  ls -l .git/hooks
total 112
-rwxr-xr-x  1 wdijkerman  staff   478 Jun 14 21:01 applypatch-msg.sample
-rwxr-xr-x  1 wdijkerman  staff   896 Jun 14 21:01 commit-msg.sample
-rwxr-xr-x  1 wdijkerman  staff  3327 Jun 14 21:01 fsmonitor-watchman.sample
-rwxr-xr-x  1 wdijkerman  staff   189 Jun 14 21:01 post-update.sample
-rwxr-xr-x  1 wdijkerman  staff   424 Jun 14 21:01 pre-applypatch.sample
-rwxr-xr-x  1 wdijkerman  staff  1638 Jun 14 21:01 pre-commit.sample
-rwxr-xr-x  1 wdijkerman  staff   416 Jun 14 21:01 pre-merge-commit.sample
-rwxr-xr-x  1 wdijkerman  staff  1348 Jun 14 21:01 pre-push.sample
-rwxr-xr-x  1 wdijkerman  staff  4898 Jun 14 21:01 pre-rebase.sample
-rwxr-xr-x  1 wdijkerman  staff   544 Jun 14 21:01 pre-receive.sample
-rwxr-xr-x  1 wdijkerman  staff  1492 Jun 14 21:01 prepare-commit-msg.sample
-rwxr-xr-x  1 wdijkerman  staff  3610 Jun 14 21:01 update.sample

But we are focussing on to pre-commit hooks and there is an easier way to maintain the pre-commit hooks. Like you already have seen, we can only use 1 pre-commit file, and we want to use a lot more than just one. Sure you can call other scripts, but that makes the maintenance a bit more complicated. Especially when you have more than just a single git repository. There is a package that helps us to have a more maintainable soultion with regarding to pre-commit hooks and using them.

Installation

We have to install it first and we have to install it on our development workstation/laptop. Make sure you have Python and Pip installed before executing the following command:

pip install pre-commit

This will install the pre-commit application on our system.

“That is all nice, but I have no idea what kind of scripts we can execute as part of a pre-commit?”

– you?

No worries, lets use an example. This Github repository https://github.com/dj-wasabi/dj-wasabi-release/ contains some Python scripts that I use for maintaining my Github repositories. We will not discuss these scripts, but there is 1 important file in this repository and is named “.pre-commit-config.yaml“. You will probably find this file also on other repositories that are on my Github account. This file contains all the scripts that will be executed when we do a “git commit”. Lets take a look at the file.

repos:
- repo: https://github.com/dj-wasabi/pre-commit-hooks
  rev: master
  hooks:
  - id: shellcheck
  - id: markdown-toc
  - id: verify-yaml
  - id: no-commit-on-branch
    args: ['-b master,main']

It contains a “repos” key which is a list, this means you can have multiple entries configured in the file (As this is currently the case). Each of these entries needs the following 3 properties to work properly:

  1. repo: The git location where the scripts are stored;
  2. rev: The version of the repository, can be a tag, branch or git commit;
  3. 1 or more hooks, which has 1 or 2 keys: id (and args).

In the code block above, we see that there is a Github repository mentioned, namely https://github.com/dj-wasabi/pre-commit-hooks and on master we have several scripts available. The pre-commit hook will see that in this repository a file named “.pre-commit-hooks.yaml” exist. This file knows exactly what to do when the pre-commit hook is executed with the provided id’s. In the “.pre-commit-hooks.yaml” file, there is a configuration for the “shellcheck” id.

- id: shellcheck
  name: Shellcheck Bash Linter
  description: Performs linting on bash scripts
  entry: bin/shellcheck.sh
  language: script

The pre-commit hook now knows, that with the “shellcheck” id a script “bin/shellcheck.sh” in the https://github.com/dj-wasabi/pre-commit-hooks reposity needs to be executed. But in all fairness, you are not ready yet. 9 out of 10 times you will need to check the README on the mentioned repository to see if you need to install any other tool to make the script work. Because I am working on a Mac, I do need to make sure that shellcheck is installed:

brew install shellcheck

And I need to make sure that the dependencies that the other scripts in the “.pre-commit-hook.yaml” uses are installed on my Mac.

Once I have done that, we need to tell our “.git/hooks” directory that we want to make use of the pre-commit hooks. We will do that by executing the following command in the root of our git repository:

pre-commit install

This will give us the following output:

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

You can do now an “ls -l .git/hooks” and compare the output with before. When you execute a “git commit” from now on in this repository, it will execute the pre-commit hooks. Remember that for each git repository you have cloned on your host, it will contains their own configuration.

But why?

“But I still don’t know what kind of scripts I can execute as part of a pre-commit hook?”

– you?

The problem I see and/or have experienced a lot is that when people gets invited to be a reviewer on a PR, is that they focus on the small nitpicky things that don’t really that much. Things like, wrong identation, to many empty lines, missing space or trailing space (well, I do find this one annoying ;-)). They should be focussing on the actual change like I mentioned in one of my earlier blogposts. And 9/10 times you can prevent things like this by executing a linter or formatter. This linter will provide an overview of warnings and errors on what needs to be fixed before the linter is happy, so an ideal candidate to execute it as a pre-commit hook. So with pre-commit hooks we can sort of prevent “garbage” to be committed into a Git repository.

When you write for example Python scripts, like what I do with the https://github.com/dj-wasabi/dj-wasabi-release/ repository, you can make use of the “flake8” tool and with every “git commit”, the linter will be executed and provide errors. And that can be achieved by adding the following few lines in the “.pre-commit-hooks.yaml“:

- repo: https://gitlab.com/pycqa/flake8
  rev: 3.8.4
  hooks:
  - id: flake8
    additional_dependencies: [flake8-typing-imports==1.7.0]

If I now do a “git commit” (or when you just want to execute it without doing an actual commit: “pre-commit run -a“):

$ git commit -am "Some message that I know that it will not make it to the log"
Shellcheck Bash Linter...................................................Passed
Generate Markdown toc....................................................Passed
Pretty Print YAML files..................................................Passed
No commit on master or main..............................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

lib/djWasabi/git.py:59:1: E302 expected 2 blank lines, found 1
lib/djWasabi/git.py:79:22: E231 missing whitespace after ','
lib/djWasabi/git.py:80:52: W291 trailing whitespace
lib/djWasabi/git.py:83:53: E231 missing whitespace after ','

Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing lib/djWasabi/git.py

Check for merge conflicts................................................Passed
$ echo $?
1

Look, the pre-commit “Failed” (Or succeeded, depends on how you look at it ;-)) and it prevented to do the actual commit in git. You can see that there where 4 lines in the “lib/djWasabi/git.py” file that needs to be fixed. But you will also see a 2nd script has failed, namely the “Trim Trailing Whitespace“.

A pre-commit script can also update files when needed (which isn’t a bad thing!) and not just fail on when certain criteria is met. The “Trim Trailing Whitespace” script has updated a single file “lib/djWasabi/git.py” and removed the trailing space. When a script updates a file, the script should exit with an exit code of 1 and provide info on what file has been updated. If I would run the git commit again, then you will see that the “Trim Trailing Whitespace” is “Passed” and that the “flake8” script only has 3 errors:

Shellcheck Bash Linter...................................................Passed
Generate Markdown toc....................................................Passed
Pretty Print YAML files..................................................Passed
No commit on master or main..............................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

lib/djWasabi/git.py:59:1: E302 expected 2 blank lines, found 1
lib/djWasabi/git.py:79:22: E231 missing whitespace after ','
lib/djWasabi/git.py:83:53: E231 missing whitespace after ','

Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed

Nice right?

Updating files as part of the pre-commit hooks are very common and I do make use of it a lot. When you work with Terraform, you will probably know the “terraform fmt” command as well. And I do too make use of the “terraform fmt” command as part of the pre-commit hook, it will make sure that all my terraform files are formatted in one way. So people that will review my Pull Requests, can not write any comment on formatting issues. 🙂

Before I end this blog post, I want to say one last things that really helps me with pre-commit hooks. I really like writing documentation in Asciidoctor format, i personally thinks it is a bit better that any other “language” and I don’t want to start a flamewar, but just writing on top of the document “:toc: left” and I have my Table of Content on the left side of the page. With Markdown, you’ll have to manually write one and keep the Table of Content up 2 date. Or, you install a tool with this command:

pip install md-toc

Then you can make use of the following line in your markdown file:

<!--TOC-->

And make sure that an id with “markdown-toc” is added to the “.pre-commit-hooks.yaml” file like shown in the beginning of this blog post and you are done! (small note is that you need to commit your changes to see that it is generated :))

I can now see the benefits of using pre-commit hooks and will use them as well!

– you?

I hope so? I hope I showed you how awesome pre-commit hooks are that they help you write better code. Not only that, it also helps the reviewers to focus on a PR that matters instead of looking for the nitpicky things. And with this blog post I only mentioned linting and formatting scripts, but you basically can do anything with a pre-commit hook. One small tip with using pre-commit hooks, don’t go wild on it. My commits will take some seconds to complete, but you can even run unit tests as well but that will result in longer duration of doing a commit. And when you do a “git commit” you basically are waiting on it to finish correctly, so don’t add lots and lots of tests because that will increase the duration of a commit.

But if you do have any questions and or remarks please let me know and I will happily help you.

In this post I will describe how to create your own pre-commit hook.

May the commits be with you!

Jenkins as code, part 2: Setting up the Jenkins job

This is the follow up of the previous blogpost about Jenkins as code, which you can find here. With the previous blogpost we discussed how we have created a Docker image, containing Jenkins and several files needed to correctly install the plugins and configure Jenkins with several yaml files. The yaml configuration files are used by the configuration-as-code plugin to configure the Jenkins environment how we want to have it configured. This means that no manual changes are needed in the UI.

Just like mentioned in the first blogpost:

Before we do anything I just want to remind you that this is just 1 way to achieve a Jenkins as code setup. It does not mean that this is the only or best way, it is just one way. Next to that, these blogposts and the code in the Github repository will help you kickstart your own setup and by no means you can just run it on a production environment and blame me if something is not working fine. During the blogposts I will tell you how I was able to do things, so you can redo it all yourself (and compare it with the code in the Github repository).

Seed All DSL Jobs

With this blogpost, we will continue the job-dsl.yaml file. This is the yaml file which will make sure that once Jenkins is running, a “Seed All DSL jobs” job” is configured and loads groovy files from a specific directory in a git repository. So lets continue with these groovy files. It basically consists of 2 parts, we set some variables and have several class and functions to generate the jobs and list views. 

The example.groovy file top part are some variables we need to configure.

@Field String jenkinsCredentialId = "SSH_GIT_KEY"
@Field String basePath = 'example'
@Field String defaultPollingScm = 'H/5 * * * *'

JobConstructor[] jobList = [
        [
                "example-repo",
                "https://bitbucket.org/wernerdijkerman/this-is-some-test.git",
                defaultPollingScm
        ]
]

The first one is easy, that is the name of our Jenkins credential that we are using for authorizing to the Git server. The basePath is the value which is used for creating a “directory” and a specific “tab” with this name. See the screenshot. Then we have the defaultPollingScm, which we will configure that each job will check once every 5 minutes for changes. Ideally there should be webhooks configured on the Git server, but this only works if Jenkins is available from the Git server (which is in my situation not the case)

The jobList is basically a list with all of the repositories that belong to this “example” group. For my Terraform repositories, I would have a file named terraform.groovy, with a basePath set to ‘terraform‘ and in the jobList variable, all repositories configured related to Terraform. And if you need to add (or remove) a (terraform) repository, you will update this jobList variable by either adding or removing the repository information. Once the change is merged into main (or master), with a max of 15 minutes the new job is added (or deleted) because the Seed DSL job has ran.

The rest of the file contains some specific functions to actual create the Jobs and listviews. You can see on this page https://jenkinsci.github.io/job-dsl-plugin/ with all the possible functions (and the parameters) you can use to make it your own.

Shared Library

A shared library allows you to use code that can be used in Jenkinsfile’s. The problem when you have lets say 50 Spring Boot java microservices, you probably have 50 Jenkinsfiles that are the same, with the exception of the name of the microservice. So that is a lot of duplicate code in the various repositories, especially if these Jenkinsfiles are really large. So the goal is to create small(er) Jenkinsfiles as we still need to have a Jenkinsfile in the repositories. Because we create smaller Jenkinsfile, we can even provide arguments that can be used for correctly generating the Jenkinsfile.

This is how our Jenkinsfile will look like, for our example service:

@Library('djwasabi') _
def run = new com.djwasabi.common.examplePipeline()
def NAME = "Wasabi"
run.pipeline('test-repo', NAME)

With the first line we specify which Shared Library we want to make use. You need to go to “Manage“, “Configure system” and then look for “Global Pipeline Libraries“. When you did a docker compose up from my repository, you would see that there is a Shared Library named “djwasabi“. This shared library is from a git repository and is using the master branch as the version (You could also specify a tag or branch or even an git commit). 

To understand the 2nd line, you will need to go to https://bitbucket.org/wernerdijkerman/jenkins-shared-library/src/master/ (or look into the already earlier provided github repository in the “library” folder), you will see a specific directory structure which is used within the groovy language. You will see an examplePipeline.groovy file, which will be loaded to create a new groovy object.

The 3rd line we set a variable named NAME and set it to “Wasabi“. With the 4th line we will execute the function inside the examplePipeline.groovy file named ‘pipeline‘. This function accepts multiple parameters and we only provide 1 called “NAME“.

Let us take a look at the examplePipeline.groovy file:

package com.djwasabi.common

import com.djwasabi.common.workers.*
import groovy.transform.Field

def pipeline(jobName, name = "world", agentNode = "worker") {
    def command = new Command(this)
    def git = new Git(this)
    def resultsGetter = new ResultsGetter(this)

    node(agentNode) {
        try {
            def lastResult = currentBuild.rawBuild.getPreviousBuild()?.result

            stage("Checkout") {
                git.checkout()
                tagged = git.isCurrentCommitAlreadyTagged()
            }
            if (!tagged || hudson.model.Result.SUCCESS != lastResult) {
                stage("Run command") {
                    command.echo("hello " + name + " via job " + jobName)
                }
            } else {
                resultsGetter.repeatPreviousBuildResult(currentBuild)
            }
        } catch (all) {
            stage('Destroy it') {
                command.echo('lets run when things go wrong.')
            }
        }
    }
}

As you see with the examplePipeline.groovy file, the function “pipeline” accepts multiple arguments. The first one “jobName” does not have a default and thus it is always required, but the other 2 has default (resp. “world” and “worker“). Have you noticed that the “agentNode” parameter contains the value “worker“, which is also the value for our configuration we used for starting the Docker container in the previous blog post? So if you write your own shared library, you can make 1 or more of these node definitions and based on parameters that are set in the Jenkinsfile, certain actions can be executed.

When we run the job in Jenkins, you will see something like the above showing up. Almost at the end of the log, you will see the following line:
hello Wasabi via job test-repo

That is basically logged via this piece of code that was generated in the examplePipeline.groovy file:

                stage("Run command") {
                    command.echo("hello " + name + " via job " + jobName)
                }

Summary

In these 2 parts I helped you to explain what the possibilities are for an Jenkins as code approach. This approach will have everything set into code and when someone wants to make a change in Jenkins, the person should be creating a change into the code. When you have properly setup your Git server, this person will create a branch, make the changes and will create a Pull Request before it is merged into master|main. With everything set into code, people have an audit log of what has been changed (and by whom), so it will force you into having a discussion when things are needed (or when something went wrong).

And what I personally also like, because everything is set into code you can deploy a Jenkins server somewhere else and you can almost immediately use it. Especially when you make the Configuration as Code yaml files configurable with environment variables, you can run multiple Jenkins setups from the same code base. And because everything is in code, you don’t have to worry about backing up Jenkins (It is already on the Git server and part of their backup mechanism).

I hope you enjoyed it and that it showed you that you can too automate the configuration of any Jenkins environment you want to run in your organisation. If you have any improvements and/or remarks, please let me know.

May the force (or source) be with you!

Jenkins as code, part 1: Setting up Jenkins in Docker

I hate doing things manually, I really do.

Log into an UI, do some clicks here and there to be able to have something created or configured. It is error prone (you can easily forget something or make a typo) and it is stupid and/or boring (Especially if you need to do this on a routine basis). If you can change something in a UI, then someone is able to change that as well and even do that without you knowing that it is changed (Or viceversa ;-)). So doing things manually is not the way forward and we should focus on automation. Automation is one of the pillars of doing DevOps, so we should always automate things right?

What people do probably not know, Jenkins is a tool that can be fully automated, you only have to know how. (And based on some posts on for example Reddit, I don’t think people knows that this is even possible).

Jenkins as code

So lets dive into, what I would say as: Jenkins as code.

This will basically be a 2 part blog post where we will discuss the following:

  1. This part where we will create a Docker image, containing Jenkins with its configuration files and plugins;
  2. The next part, where we will create a shared library and use that in a Jenkinsfile, with jobs we load via a “specifications” repository;

Before we do anything I just want to remind you that this is just 1 way to achieve a Jenkins as code setup. It does not mean that this is the only or the best way, it is just one way. Just like that there are 1000 ways to go to Rome. Next to that, these blog posts and the code in the Github repository will help you kickstart your own setup and by no means you can just run it on a production environment and blame me if something is not working fine. I am not a groovy expert and I can do some basic things, so don’t expect a new world wonder. During the blog posts I will tell you how I was able to do things, so you can redo it all yourself (and compare it with the code in the Github repository) and build it on your own terms/setup.

In both blog posts, you will see a lot of code and commands popping up. But no worries, all code is available on my Github in repository https://github.com/dj-wasabi/blog-jenkins-as-code . So lets start with the first part: Setting up Jenkins.

Docker(file)

We will create a Docker image, based on “jenkins/jenkins:lts” Docker image, install Docker and configure it with the configuration-as-code plugin included with several yaml files that are used for configuring Jenkins. And as part of the Github repository, we have a docker-compose.yaml file which we can use to boot our setup.

Lets start with the Dockerfile.

USER root
RUN groupadd docker && \
    curl -fsSL https://get.docker.com -o get-docker.sh && \
    sh get-docker.sh && \
    usermod -aG root jenkins && \
    usermod -aG docker jenkins
USER jenkins

Lets discuss this first, the “jenkins/jenkins:lts” Docker image does not contain the docker application, so we need to install that and make sure that the “jenkins” user is part of the “root” and “docker” group. We need Docker in this image, as each Jenkins job will run in its own Docker container.

ENV CASC_JENKINS_CONFIG=/var/jenkins_home/casc

We need set an environment variable named CASC_JENKINS_CONFIG, which we basically tell Jenkins where the configuration-as-code yaml files can be found.

COPY casc/ /var/jenkins_home/casc
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt

Then we will copy the contents on the casc directory to the earlier mentioned directory and we place the file with all of our plugins into a specific directory. Then we run the install-plugins.sh script so we can download and install all the plugins that we need in our setup.

And that is our Dockerfile, easy right? This will allow us to build a Jenkins Docker image, with all of our files and configuration that will lead to an Jenkins environment we want to have. We can deploy this on some host running Docker or even make some additional changes to make it work in Kubernetes.

Plugins

Lets go to the plugins.txt file, as this one is a bit easier to explain than the casc files.

We need to create a plugins.txt that contains all of the plugins we want to make use, so how do we do that. I manually (oh yes, sorry! :)) started a Jenkins container and followed the installation steps and then I picked several plugins to install during the installation steps. When Jenkins is running and I have finnished nstalling all plugins (don’t forget the “configuration-as-code” plugin), I went to “manage” and then clicked on “Script Console“. There you see an textfield to execute groovy scripts and I used the following script:

def plugins = jenkins.model.Jenkins.instance.getPluginManager().getPlugins()
plugins.each {println "${it.getShortName()}:latest"}

This is a “script” that provides an overview of all plugins that are currenlty installed in Jenkins. I have used the “latest” version of the plugin which is fine for demo purposes, but you could also update the “latest” to ${it.getVersion()}. This will show the actual version of the installed plugin. I would suggest to use the versions in the plugins.txt file. This helps you in the future when someone create’s an PR that it shows you that there is an update in the version of a plugin, which you won’t see when it is using “latest”.

Then you hit the “Run” button and you will see some output appear. Select the output and place that in the plugins.txt file and you are done (I would also sort the contents of the file, so all plugins are order alphabetical).

Configuration as code

Let us first explain how we can get a yaml file. Go to “Manage” and then you will see “Configuration as Code” (And click on it please) in the “System Configuration” lane. There is a button called “Download Configuration” which will download the yaml file and with “View Configuration” you can see the yaml file in your browser. When you have downloaded the yaml configuration file, you can make use of it in your Docker image by placing it in the casc/ directory. I would suggest you split it into seperate files, so you won’t have 1 large file but smaller ones with each a specific set of configuration. For example, create a credentials.yaml file containing all Jenkins credentials. 

But before you commit all your changes, you can also update some values by using environment variables, see the following:

  securityRealm:
    local:
      allowsSignup: false
      enableCaptcha: false
      users:
      - id: "${JENKINS_ADMIN_USERNAME:-admin}"
        name: "${JENKINS_ADMIN_NAME:-Administrator}"
        password: "${JENKINS_ADMIN_PASSWORD}"

This piece of the configuration you see now is responsible for creating an admin user. I don’t want to hardcode the username and definitely not the password in this file, so I use environment variables for that. And this is also the case for the credentials that Jenkins use, see the following example of the Jenkins “credentials”:

credentials:
  system:
    domainCredentials:
      - credentials:
          - basicSSHUserPrivateKey:
              scope: GLOBAL
              id: "SSH_GIT_KEY"
              username: "git"
              description: "SSH Credentials for jenkins"
              privateKeySource:
                directEntry:
                  privateKey: ${JENKINS_SSH_GIT_KEY}

With the above credential configuration I won’t have to hardcode the SSH Private key in the Docker image, but can use it as an environment variable. Nice right? 🙂

When everything is done via code, we can also already configure the Security Matrix and allowing what user can and most importantly can’t do in Jenkins. As my Jenkins is running on premise and don’t allow traffic from outside the environment, I will allow people to start jobs (if they don’t want to wait on the triggering). So I will allow anonymous people to have read, build and cancel rights for the jobs. Why go for all that trouble for letting people authenticate against some source, so we can see that this person has started or cancelled a job? Most importantly, they can’t change anything unless they know the admin password. (But will be undone when Jenkins is restarted! :))

  authorizationStrategy:
    globalMatrix:
      permissions:
      - "Job/Build:anonymous"
      - "Job/Cancel:anonymous"
      - "Job/Read:anonymous"
      - "Overall/Administer:admin"
      - "Overall/Read:anonymous"

Now we are able to fully do Jenkins as code, as we will store the yaml files in the casc/ directory which are loaded when Jenkins is started. But when Jenkins is running, we also need to make sure that we will load the jobs from somewhere. We will do this with a “Seed Job“, which you can see in the “dsl-jobs.yaml” file in the casc/ directory in the Github repository.

            git {
                remote { 
                    url "${JENKINS_JOB_DSL_URL}"
                    credentials 'SSH_GIT_KEY' 
                }
                branch '*/main'
              }
        }
        triggers {
            scm('H/15 * * * *')
        }
        steps {
          dsl {
            external('${JENKINS_JOB_DSL_PATH:-jobs}/*.groovy')
            removeAction('DELETE')
          }
        }
      }

When Jenkins is started, we will automatically create the “Seed all DSL jobs” Jenkins job. And what it does is basically the following (Snippet is incomplete, see for the compete file on Github for full version):

  1. We use the credential ‘SSH_GIT_KEY‘ to checkout the repository mentioned in ${JENKINS_JOB_DSL_URL} (See the docker-compose.yaml file)
  2. We use the ‘main’ branch;
  3. The job is executed every 15 minutes;
  4. In the directory named ${JENKINS_JOB_DSL_PATH} we will find groovy files and if Jenkins has jobs which aren’t configured in these groovy files, we delete the jobs from Jenkins.

Before we finalise the Configuration as code part, we need to discuss one last file (and action). When we have the Jenkins server running, we will run each job in its own Docker container. So the Jenkins server will start a Docker container and do all of its action inside that container and the configuration is what follows:

jenkins:
  clouds:
    - docker:
        name: "docker"
        dockerApi:
          dockerHost:
            uri: "${DOCKER_HOST:-unix:///var/run/docker.sock}"
        templates:
        - connector:
            attach:
              user: "jenkins"
          dockerTemplateBase:
            bindAllPorts: true
            image: "jenkins/agent:latest"
            privileged: true
            environment:
              - "TZ=Europe/Amsterdam"
          instanceCapStr: "99"
          labelString: "worker"
          name: "worker"
          remoteFs: "/home/jenkins/agent"

This is also seen in the file “docker.yaml“. Here we have placed 1 template which we named “worker“, with the “jenkins/agent:latest” Docker image. As you know, this is just an example so you can modify this to your needs and use a Docker image that suits your needs. This Docker image should contain all the tools needed to run your jobs, so the “jenkins/agent:latest” might not be fit for your setup. And do know, as the “templates” key is a list, you can add a lot more templates with a unique name, settings and Docker image. For the dockerHost.uri, you will see the usage of a environment variable “DOCKER_HOST“. This is an variable we use in docker-compose.yaml file and if we don’t provide one, the default unix:///var/run/docker.sock is used.

You can go to “manage“, “Systems configuration” and scroll all the way down until you will see “Cloud“. It provides a link and when clicking on it, you’ll get the page where you can configure the “Cloud” configuration. When you make changes, don’t forget to export the yaml file on the “Configuration as Code” page mentioned earlier.

Build and ship it

So far we have discussed some basics on how we get our configuration, so lets build a Docker image. During the rest of this blog post, I will assume you will have the same layout as my Github repository. So lets go to the directory where we have the “Dockerfile“, “plugins.txt“, “docker-compose.yaml” file and the “casc/” directory. Here we will run the docker build command, to build the new Docker image.

cd server
docker build -t jenkins-as-code . --pull

I named it ‘jenkins-as-code‘ which works locally fine and if you want to push it into a Docker registry, you should prefix it with the correct registry name. If you prefix it with a registry or you named it differently, don’t forget to update the docker-compose.yml file with your new name. The –pull is there so if you already have a “jenkins/jenkins:lts” Docker image, you will get the latest one.

I think it is build now, otherwise we will wait a minute before we continue.

sleep 60 🙂

Ok, the Docker image is build and we can start it. If you see the docker-compose.yaml file, you will notice 2 ‘services’.

  1. socat;
  2. jenkins (The one will just build and want to start).

But lets describe the ‘socat‘ service. The ‘socat‘ service is used to make sure that our docker.sock file from our host can be used with Jenkins for starting the agents. If we do this from the Jenkins container itself and not using this ‘socat‘ service, we will get permission denied errors and Jenkins can not start any new Docker container (I am doing on a Mac, I don’t think people running it on Linux hosts will have issues ). 

The Jenkins service has several environment variables set. Before we start everything, we will need to create an environment variable first that contains the content of a private SSH key. I have used the following command for that:

export EXPORTED_PASSWORD=$(cat ~/.ssh/wd_id_rsa)

So this EXPORTED_PASSWORD contains the private SSH key and this one will be used in Jenkins as the SSH_GIT_KEY credential on multiple places. Also worth to mention is the JENKINS_ADMIN_PASSWORD environment variable, this is what is says: The password for the Admin user, so if you want to use something else here is the moment to change it.

We will start it with the following command:

docker compose up -d

I prefer starting it in the background, so that is why I added the -d argument. Once it is booted we open our favourite browser and go to http://localhost:8080 you will see something like the following:

So that is it for now. We started our newly build Docker image containing Jenkins, with the plugins we need in our environment and our own configuration!

We will go into the “Seed All DSL jobs” job with the next part of the blogpost. So stay tuned! 🙂

2nd blog post you can find here.

Finalizing the installation of Zabbix Agent with Ansible

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/finalizing-the-installation-of-zabbix-agent-with-ansible/13321/ (There are a lot of interesting posts on there, so please check the blogposts often!)

In the previous blog posts, we created a Zabbix Server with a new user, a media type, and an action. In the 2nd blog post, we continued with creating and configuring a Zabbix Proxy. In the last part of this series of blog posts, we will install the Zabbix Agent on all of the 3 nodes we have running.

This blog post is the 3rd part of a 3 part series of blog posts where Werner Dijkerman gives us an example of how to set up your Zabbix infrastructure by using Ansible.
You can find part 1 of the blog post by clicking here.

To summarize, so far we have a Zabbix Server and a Zabbix Proxy. The Zabbix Server has a MySQL instance running on a separate node, the MySQL instance for the Zabbix Proxy runs on the same node. But we are missing one component right now, and that is something we will install with the help of this blog post. We will install the Zabbix Agent on the 3 nodes.

A git repository containing the code used in these blog posts is available on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we run Ansible, we need to make sure we have opened a shell on the “bastion” node. We can do that with the following command:

$ vagrant ssh bastion

Once we have opened the shell, go to the “/ansible” directory where we have all of our Ansible files present.

$ cd /ansible

In the previous blog post, we executed the “zabbix-proxy.yml” playbook. Now we are going to use the “zabbix-agent.yml” playbook. The playbook will install the Zabbix Agent on all nodes (“node-1”, “node-2” and “node-3”). Next up, on both the “node-1” and “node-3”, we will add a user parameters file specifically for MySQL. With this user parameters file, we are able to monitor the MySQL instances.

$ ansible-playbook -i hosts zabbix-agent.yml

This playbook will run for a few minutes installing the Zabbix Agent on the nodes. It will install the zabbix-agent package and add the configuration file, but it will also make a connection to the Zabbix Server API. We will automatically create a host with the correct IP information and the correct templates! When the Ansible playbook has finished running, the hosts can immediately be found in the Frontend. And better yet, it is automatically correctly configured, so the hosts will be monitored immediately!

We have several configurations spread over multiple files to make this work. We first start with the “all” file.

The file “/ansible/group_vars/all” contains the properties that will apply to all hosts. Here we have the majority of essential properties configured that are overriding the default properties of the Ansible Roles. Each role has some default configuration, which will work out of the box. But in our case, we need to override these, and we will discuss some of these properties next.

zabbix_url

This is the URL on which the Zabbix Frontend is available and thus also the API. This property is for example used when we create the hosts via the API as part of the Proxy and Agent installation.

zabbix_proxy

The Zabbix Agents will be monitored by the Zabbix Proxy unless the Agent runs on the Zabbix Server or the host running the database for the Zabbix Server. Like with the previous blog post, we will also use some Ansible notation to get the IP address of the host running the Zabbix Proxy to configure the Zabbix Agent.

zabbix_proxy: node-3
zabbix_agent_server: "{{ hostvars[zabbix_proxy]['ansible_host'] }}"
zabbix_agent_serveractive: "{{ hostvars[zabbix_proxy]['ansible_host'] }}"

With the above configuration, we configure both the Server and ServerActive in the Zabbix Agent’s configuration file to use the IP address of the host running the Zabbix Proxy. If you look at the files “/ansible/group_vars/zabbix_database” and “/ansible/group_vars/zabbix_server/generic” you would see that these contain the following:

zabbix_agent_server: "{{ hostvars['node-1']['ansible_host'] }}"
zabbix_agent_serveractive: "{{ hostvars['node-1']['ansible_host'] }}"

The Zabbix Agent on the Zabbix Server and on its database is using the IP address of the Zabbix Server to be used as the value for both the “Server” and “ActiveServer” configuration settings for the Zabbix Agent.

zabbix_api_user & zabbix_api_pass

These are the default in the roles, but I have added them here so it is clear that they exist. When you change the Admin user password, don’t forget to change them here as well.

zabbix_api_create_hosts & zabbix_api_create_hostgroups 

Because we automatically want to create the Zabbix Frontend hosts via the API, we need to set both these properties to true. Firstly, we create the host groups that can be found with the property named “zabbix_host_groups”. After that, as part of the Zabbix Agent installation, the hosts will be created via the API because of the property zabbix_api_create_hosts.

Now we need to know what kind of information we want these hosts created with. Let’s go through some of them.

zabbix_agent_interfaces

This property contains a list of all interfaces that are used to monitoring the host. This is relatively simple in our case, as the hosts only have 1 interface available. You can find some more information about what to use when you have other interfaces like IPMI or SNMP: https://github.com/ansible-collections/community.zabbix/blob/main/docs/ZABBIX_AGENT_ROLE.md#other-interfaces We use the interface with the value from property “ansible_host” for port 10050.

zabbix_host_groups

This property was also discussed before – we automatically assign our new host to these host groups. Again, we have a fundamental setup, and thus it is an effortless property.

zabbix_link_templates

We provide a list of all Zabbix Templates we will want to assign to the hosts with this property. This property seems a bit complicated, but no worries – let’s dive in!

zabbix_link_templates:
  - "{{ zabbix_link_templates_append if zabbix_link_templates_append is defined else [] }}"
  - "{{ zabbix_link_templates_default }}"

With the first line, we add the property’s value “zabbix_link_templates_append”, but we only do that if that property exists. If Ansible can not find that property, then we basically add an empty list. So where can we find this property? We can check the files in the other directories in the group_vars directory. If we check, for example “/ansible/group_vars/database/generic”, we will find the property:

zabbix_link_templates_append:
  - 'MySQL by Zabbix agent'

So on all nodes that are part of the database group, we add the value to the property “zabbix_link_templates”. All of the database servers will get this template attached to the host. If we would check the file “/ansible/group_vars/zabbix_server/generic”, then we will find the following:

zabbix_link_templates_append:
  - 'Zabbix Server'

As you probably understand now, when we create the Zabbix Server host, we will add the “Zabbix Server” template to the host, because this file is only used for the hosts that are part of the zabbix_server group.

With this setup, we can configure specific templates for the specific groups, but there is also at least 1 template that we always want to add. We don’t want to add the template to each file as that is a lot of configuration, so we use a new property for this named “zabbix_link_templates_default”. In our case, we only have Linux hosts, so we always want to add the templates:

zabbix_link_templates_default:
  - "Linux by Zabbix agent active"

On the Zabbix Server, we both assign the “Zabbix Server” template and the template “Linux by Zabbix agent active” to the host.

But what if we have Macros?

zabbix_macros

As part of some extra tasks in this playbook execution, we also need to provide a macro for some hosts. This macro is needed to make the Zabbix Template we assign to the hosts work. For the hosts running a MySQL database, we need to add a macro, which can be found with the property zabbix_macros_append in the file “/ansible/group_vars/database/generic”.

zabbix_macros_append:
  - macro_key: "MYSQL.HOST"
    macro_value: "{{ ansible_host }}"

We will create 1 macro with the key name “MYSQL.HOST” and assign a value that will be equal to the contents of the property ansible_host (For the “node-2” host, the host running the database for the Zabbix Server), which is “10.10.1.12”.

User parameters

The “problem” with assigning the MySQL template is that it also requires some UserParameter entries set. The Zabbix Agent role can deploy files containing UserParameters to the given hosts. In “/ansible/group_vars/database/generic” we can find the following properties:

zabbix_agent_userparameters_templates_src: "{{ inventory_dir }}/files/zabbix/mysql"
zabbix_agent_userparameters:
  - name: template_db_mysql.conf

The first property “zabbix_agent_userparameters_templates_src” will let Ansible know where to find the files. The “{{ inventory_dir }}” will be translated to “/ansible” and here you will find a directory named “files” (and you will find the group_vars directory as well) and further drilling down the directories, you will find the file “template_db_mysql.conf”.

With the second property “zabbix_agent_userparameters” we let Ansible know which file we want to deploy to the host. In this case, the only file found in the directory named “template_db_mysql.conf”.

When the Zabbix agent role is fully executed, we have everything set to monitor all the hosts automatically. Open the dashboard, and you will see something like the following:

It provides an overview, and on the right side, you will notice we have a total of 3 nodes of which 3 are available. Maybe you will see a “Problem” like in the screenshot above, but it will go away.

If we go to “Configuration” and “Hosts,” we will see that we have the 3 nodes, and they have the status “Enabled” and the “ZBX” icon is green, so we have a proper connection.

We should verify that we have some data, so go to “Monitoring” and click on “Latest data.” We select in the Host form field the “Zabbix database,” and we select “MySQL” as Application and click on “Apply.” If everything is right, it should provide us with some information and values, just like the following screenshot. If not, please wait a few minutes and try again.

Summary

This is the end of a 3 part blog post in creating a fully working Zabbix environment with a Zabbix Server, Proxy, and Agent. With these 3 blogposts you were able to see how you can install and configure a complete Zabbix environment with Ansible. Keep in mind that the code shown was for demo purposes and it is not something you can immediately use for the Production environment. We also used some of the available functionality of the Ansible collection for Zabbix, there are many more possibilities like creating a maintenance period or a discovery rule. Not everything is possible, if you do miss a task or functionality of a role that Ansible should do or configure, please create an issue on Github so we can make it happen.

Don’t forget to execute the following command:

$ vagrant destroy -f

With this, we clean up our environment and delete our 4 nodes, thus finishing with the task at hand!

Installing and configuring the Zabbix Proxy

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/installing-and-configuring-the-zabbix-proxy/13319/ (There are a lot of interesting posts on there, so please check the blogposts often!)

In the previous blog post, we created a Zabbix Server setup, created several users, a media type, and an action. But today, we will install on a 3rd node the Zabbix Proxy. This Zabbix Proxy will have its database running on the same host, so the “node-3” host has both the MySQL and Zabbix Proxy running.

This blog post is the 2nd part of a 3 part series of blog posts where Werner Dijkerman gives us an example of how to set up your Zabbix infrastructure by using Ansible.
You can find part 1 of the blog post by clicking Here

A git repository containing the code of these blog posts is available, which can be found on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we run Ansible, we have opened a shell on the “bastion” node. We do that with the following command:

$ vagrant ssh bastion

Once we have opened the shell, go to the “/ansible” directory where we have all of our Ansible files present.

$ cd /ansible

With the previous blog post, we executed the “zabbix-server.yml” playbook. Now we use the “zabbix-proxy.yml” playbook. The playbook will deploy a MySQL database on “node-3” and also installs the Zabbix Proxy on the same host.

$ ansible-playbook -i hosts zabbix-proxy.yml

This playbook will run for a few minutes creating all services on the node. While it is running, we will explain some of the configuration options we have set.

The configuration which we will talk about can be found in “/ansible/group_vars/zabbix_proxy” directory. This is the directory that is only used when we deploy the Zabbix proxy and contains 2 files. 1 file called “secret”, and a file called “generic”. It doesn’t really matter what names the files have in this directory. I used a file called the “secret” for letting you know that this file contains secrets and should be encrypted with a tool like ansible-vault. As this is out of scope for this blog, I simply made sure the file is in plain text. So how do we know that this directory is used for the Zabbix Proxy node?

In the previous blog post, we mentioned that with the “-I” argument, we provided the location for the inventory file. This inventory file contains the hostnames and the groups that Ansible is using. If we open the inventory file “hosts”, we can see a group called “zabbix_proxy.” So Ansible uses the information in the “/ansible/group_vars/zabbix_proxy” directory as input for variables. But how does the “/ansible/zabbix-proxy.yml” file know which host or groups to use? At the beginning of this file, you will notice the following:

- hosts: zabbix_proxy
  become: true
  collections:
    - community.zabbix

Here you will see the that “hosts” key contains the value “zabbix_proxy”. All tasks and roles that we have configured in this play will be applied to all of the hosts that are part of the zabbix_proxy group. In our case, we have only 1 host part of the group. If you would have for example 4 different datacenters and within each datacenter you want to have a Zabbix Proxy running, executing this playbook will be done on these 4 hosts and at the end of the run you would have 4 Zabbix Proxy servers running.

Within the “/ansible/group_vars/zabbix_proxy/generic” the file, we have several options configured. Let’s discuss the following options:

  • zabbix_server_host
  • zabbix_proxy_name
  • zabbix_api_create_proxy
  • zabbix_proxy_configfrequency

zabbix_server_host

The first one, the “zabbix_server_host” property tells us where the Zabbix Proxy can find the Zabbix Server. This will allow the Zabbix Proxy and the Zabbix Server to communicate with each other. Normally you would have to configure the firewall (Iptables or Firewalld) as well to allow the traffic, but in this case, there is no need for that. Everything inside our environment which we have created with Vagrant has full access. When you are going to deploy a production-like environment, don’t forget to configure the firewall (Currently this configuration of the firewalls are not yet available as part of the Ansible Zabbix Collection for both the Zabbix Server and the Zabbix Proxy. So for now you should be creating a playbook in order to configure the local firewall to allow/deny traffic).

As you will notice, we didn’t configure the property with a value like an IP address or FQDN. We use some Ansible notation to do that for us, so we only have the Zabbix Server information in one place instead of multiple places. In this case, Ansible will get the information by reading the inventory file and looking for a host entry with the name “node-1” (Which is the hostname that is running the Zabbix Server), and we use the value found by the property named “ansible_host” (Which has a value “10.10.1.11”).

zabbix_proxy_name

This is the name of the Zabbix Proxy host, which will be shown in the Zabbix frontend. We will see this later in this blog when we will create a new host to be monitored. When you create a new host, you can configure if that new host should be monitored by a proxy and if so, you will see this name.

zabbix_api_create_proxy

When we deploy the Zabbix Proxy role, we will not only install the Zabbix Proxy package, the configuration file and start the service. We also perform an API call to the Zabbix Server to create a Zabbix Proxy entry. With this API call, we can configure hosts to be monitored via this new Zabbix Proxy.

zabbix_proxy_configfrequency

The last one is just for demonstration purposes. With a default installation/configuration of the Zabbix Proxy, it has a basic value of 3600. This means that the Zabbix Server sends the configuration every 3600 to the Zabbix Proxy. Because we are running a small demo here in this Vagrant setup, we have set this to 60 seconds.

Now the deployment of our Zabbix Proxy will be ready.

When we open the Zabbix Web interface again, we go to “Administration” and click on “Proxies”. Here we see the following:

We see an overview of all proxies available, and in our case, we only have 1. We have “node-3” configured, which has an “Active” mode. When you want to configure a “Passive” mode proxy, you’ll have to update the “/ansible/group_vars/zabbix_proxy” file and add somewhere in the file the following entry: “zabbix_proxy_status: passive”. Once you have updated and saved the file, you’ll have to rerun the “ansible-playbook -i hosts zabbix-proxy.yml” command. If you will then recheck the page, you will notice that it now has the “Passive” mode.

So let’s go to “Configuration” – “Hosts”. At the moment, you will only see 1 host, which is the “Zabbix server,” like in the following picture.

Let’s open the host creation page to demonstrate that you can now set the host to be monitored by a proxy. The actual creation of a host is something that we will do automatically when we deploy the Zabbix Agent with Ansible and not something we should do manually. 😉 As you will notice, you are able to click on the dropdown menu with the option “Monitored by proxy” and see the “node-3” appear. That is very good!

Summary

We have installed and configured both a Zabbix Server and a Zabbix Proxy, and we are all set now. With the Zabbix Proxy, we have installed both the MySQL database and the Zabbix Proxy on the same node. Whereas we did install them separately with the Zabbix Server. With the following blog post, we will go and install the Zabbix Agent on all nodes.

Installing the Zabbix Server with Ansible

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/installing-the-zabbix-server-with-ansible/13317/ (There are a lot of interesting posts on there, so please check the blogposts often!)

Today we are focusing more on the automation of installation and software configuration instead of using the manual approach. Installing and configuring software the manual way takes a lot more time, you can easily make more errors by forgetting steps or making typos, and it will probably be a bit boring when you need to do this for a large number of servers.

In this blog post, I will demonstrate how to install and configure a Zabbix environment with Ansible. Ansible has the potential to simplify many of your day-to-day tasks. As an alternative to Ansible, you may also opt in to use Puppet, Chef, and SaltStack to install and configure your Zabbix environment.

Ansible does not have any specific infrastructure requirements for it to do its job. We just need to make sure that the user exists on the target host, preferably configured with SSH keys. With tools like Puppet or Chef, you need to have a server running somewhere, and you will need to deploy an agent on your nodes. You can learn more about Ansible here:  https://docs.ansible.com/ansible/latest/index.html.

This post is the first in a series of three articles. We will set up a (MySQL) Database running on 1 node (“node-2”), Zabbix Server incl. Frontend, which will be running on a separate node (“node-1”). Once we have built this, we configure an action, media and we will create some users. In the following image you will see the environment we will create.

Our environment we will create.
The environment we will create.

In the 2nd blog post, we will set up a Zabbix Proxy and a MySQL database on a new but the same node (“node-3”). In the 3rd blog post, we will install the Zabbix Agent on all of the 3 nodes we were using so far and configure some user parameters. Where the Zabbix Agent on “node-3” is using the Zabbix Proxy, the Zabbix Agent on the nodes “node-1” and “node-2” will be monitored by the Zabbix Server.

Preparations

A git repository containing the code used in these blog posts is available, which can be found on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we can do anything, we have to install Vagrant (https://www.vagrantup.com/downloads.html) and Virtualbox (https://www.virtualbox.org/wiki/Downloads). Once you have done that, please clone the earlier mentioned git repository somewhere on your host. For this demo, we will not run the Zabbix Frontend with TLS certificates.

We have to update the hosts file. With the following line, we need to make sure that we can access the Zabbix Frontend.

10.10.1.11 zabbix.example.com

In the “ROOT” directory of the git repository which you cloned some moments ago, where you can also find the Vagrantfile, This Vagrantfile contains the configuration of the virtual machine of our setup. We will create 4 Virtual Machine’s running Ubuntu 20.04, each with 1 CPU and 1 GB of Ram which you can see in the first “config” block. In the 2nd config block, we configure our “bastion” host, which we discuss later. This node will get the ip 10.10.1.3 and we also mount the ansible directory in this Virtual Machine on location “/ansible”. For installing and configuring this node we will use a playbook bastion.yml to do this. With this playbook, we will install some packages like Python, git and Ansible inside this bastion virtual machine.

The 3rd config block is part of a loop that will configure and it will create 3 Virtual Machines. Each virtual machine is also an Ubuntu node, had its own ip (respectively 10.10.1.11 for the first node, 10.10.1.12 for the second and 10.10.1.13 for the 3rd node) and just like the “bastion” node, they have each 1 CPU and 1 GB of RAM.

You will have to execute the following command:

$ vagrant up

With this command, we will start our Virtual Machine’s. This might take a while, as it will download a VirtualBox image containing Ubuntu. The “vagrant up” command will start the “bastion” node and all other nodes as a part of this demo. Once that is done, we need to access a shell on the “bastion” node:

$ vagrant ssh bastion

This “bastion” node is a fundamental node on which we will execute Ansible, but we will not be installing anything on this host. We have opened a shell in the Virtual Machine we just created. You can compare it with creating an “ssh” connection. We have to go to the following directory before we can download the dependencies:

$ cd /ansible

As mentioned before, we have to download the Ansible dependencies. The installation depends on several Ansible Roles and an Ansible Collection. With the Ansible Roles and the Ansible Collection, we can install MySQL, Apache, and the Zabbix components. We have to execute the following command to download the dependencies:

$ ansible-galaxy install -r requirements.yml
Starting galaxy role install process
- downloading role 'mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/3.3.0.tar.gz
- extracting geerlingguy.mysql to /home/vagrant/.ansible/roles/geerlingguy.mysql
- geerlingguy.mysql (3.3.0) was installed successfully
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/3.1.4.tar.gz
- extracting geerlingguy.apache to /home/vagrant/.ansible/roles/geerlingguy.apache
- geerlingguy.apache (3.1.4) was installed successfully
- extracting wdijkerman.php to /home/vagrant/.ansible/roles/wdijkerman.php
- wdijkerman.php was installed successfully
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'community.zabbix:1.2.0' to '/home/vagrant/.ansible/collections/ansible_collections/community/zabbix'
Created collection for community.zabbix at /home/vagrant/.ansible/collections/ansible_collections/community/zabbix
community.zabbix (1.2.0) was installed successfully

Your output may vary because of versions that might have been updated already since writing this blog post. We now have downloaded the dependencies and are ready to install the rest of our environment. But why do we need to download a role for MySQL, Apache and php? A role contains all the neccecerry tasks and files to configure that specific service. So in the case for the MySQL Ansible role, it will install the MySQL-server and all other packages that MySQL requires on the host, it will configure that the mysqld service is created and is running, but it will also create the databases, create and configure MySQL users and configure the root password. Using a role will help us install our environment and we don’t have to figure out ourselves on installing and configuring a MySQL server manually.

So what about the collection, the Ansible Community Zabbix Collection? Ansible has introduced this concept with Ansible 2.10 and is basically a “collection” of plugins, modules and roles for a specific service. In our case, with the Zabbix Collection, the collection contains the roles for installing the Zabbix Server, Proxy, Agent, Javagateway and the Frond-end. But it also contains a plugin to use a Zabbix environment as our inventory and contains modules for creating resources in Zabbix. All of these modules will work with the Zabbix API to configure these resources, like actions, triggers, groups. templates, proxies etc. Basically, everything we want to create and use can be done with a role or a collection.

Installing Zabbix Server

Now we can execute the following command, which will install the MySQL database on “node-2” and installs the Zabbix Server on “node-1”:

$ ansible-playbook -i hosts zabbix-server.yml

This might take a while, a minute, or 10 depending on the performance of your host. We execute the “ansible-playbook” command, and then “-i” we provide the location of the inventory file. Here you see the contents of the inventory file:

[zabbix_server]
node-1 ansible_host=10.10.1.11

[zabbix_database]
node-2 ansible_host=10.10.1.12

[zabbix_proxy]
node-3 ansible_host=10.10.1.13

[database:children]
zabbix_database zabbix_proxy

This inventory file contains basically all of our nodes and to which group the hosts belong. We can see in that file that there is a group called “zabbix_server” (The value between [] square brackets is the name for the group) and contains the “node-1” host. Because we have a group called “zabbix_server,” we also have a directory containing some files. These are all the properties (or variables) that will be used for all hosts (in our case, only the “node-1”) in the “zabbix_server” group.

Web Interface

Now you can open your favorite browser and open “zabbix.example.com”, and you will see the Zabbix login screen. Please enter the default credentials:

Username: Admin
Password: zabbix

On the Dashboard, you will probably notice that it complains that it can not connect to the Zabbix Agent running on the Zabbix Server, which is fine as we haven’t  installed it yet. We will do this in a later blog post.

Dashboard overview

When we go to “Administration” and click on “Media types,” we will see a media type called “A: Ops email.” That is the one we have created. We can open the “/ansible/zabbix-server.yml” file and go to line 33, where we have configured the creation of the Mediatype. In this case, we have configured multiple templates for sending emails via the “mail.example.com” SMTP server.

Now we have seen the media type, we will look at the trigger we just created. This trigger makes use of the media type we just saw. The trigger can be found in the “/ansible/zabbix-server.yml” file on line 69. When you go to “Configuration” and “Actions,” you will see our created trigger “A: Send alerts to Admin”. But we don’t want to run this in Production, and for demonstrating purposes, we have selected to be triggered when the severity is Information or higher.

And lastly, we are going to see that we have also created new internal users. Navigate to “Administration” – “Users,” and you will see that we have created a user called “wdijkerman”, which can be found in the “/ansible/zabbix-server.yml” file on line 95. This user will be part of a group created earlier called “ops,”. The user type is Zabbix super admin and we have configured the email media type to be used 24×7.

We have defined a default password for this user – “password”. When you have changed the password in the Zabbix Frontend UI, executing the playbook would not change the password back again to “password.” So don’t worry about it. But if you would have removed – let’s say – the “ops” group, then, when you execute the playbook again, the group will be re-added to the user.

Summary

As you see, it is effortless to create and configure a Zabbix environment with Ansible. We didn’t have to do anything manually, and all installations and configurations were applied automatically when we executed the ansible-playbook command. You can find more information on either the Ansible page https://docs.ansible.com/ansible/latest/collections/community/zabbix/ or on the Github page https://github.com/ansible-collections/community.zabbix.

In the next post, we will install and configure the Zabbix Proxy.

The ethics of Pull Requests, improving the Pull Requests process

This is the 3rd and the last part blogpost series about Pull Request reviewing. I am writing this so I can give you my personal view on each side of the Pull Request saga, something I explain to every member joining my team. It contains about the following 3 parts:

We have discussed some aspects about being the author and reviewer when it is about Pull Requests. But are there any ways to improve the whole Pull Request process, so we can focus on what really matters. Like we want to merge quality code that brings functionality that someone wants to make use of it. But what ever we do, we still have to create an PR as being the Author and/or as a reviewer we still has to review it. But maybe we can do some things to make it easier for both parties even before we create a Pull Request.

Small changes

Making small changes in a code base are much easier to review. Not that you need to create lets say 5 Pull Requests with each 1 line changed, especially when these 5 lines together needs to work together for a specific funtionality. But when you are working on something big, try to think about splitting the work in smaller pieces. Think about if making something like a toggle to enable/disable the new functionality can work, so you can spread the work over various Pull Requests and knowing that your new functionality is not yet used. So eventually you have to enable it, but you can work more easily between all the pull requests and provide feedback to the rest of your code. And 5 smaller Pull Requests are much easier and faster reviewed than 1 big Pull Request.

precommit hooks

This is a bit technical, but it helps both you and the reviewer to focus on what matters. The functionality you want to add. Wehen the author is doing an “git commit” with precommit hook(s), several scripts and/or commands are executed on the files that have been changed. This can be for example linting commands that does some static analysis. If the linting fails, then the “git commit” also fails and then you can fix the issue and try again.

For an example, when you write Terraform code there is a tool called tfsec, which allows you to do static analysis for possible security related issues. If you want to create an Security Group in AWS and provides a cidr_range of “0.0.0.0/0” it will complain and fail the “git commit” execution, because it is widely open. When this happens, you can check and either update (Because it is always easy to just use “0.0.0.0/0” and not think about a smaller subnet) or validate again that this it is actually correct and add a line above with that it should be ignored (For example: “#tfsec:ignore:AWS009”). Once done and when you execute a “git commit” it will succeed.

This was just one example of possibilities with using pre-commit hooks, there are of course a lot more.

Pair programming

This is not for everyone, especially when everyone have to work from home during a pandamic and not everyone – especially in the beginning – likes that someone is looking “over the shoulders” to see you code. People will get nervous if they are “being watched” while coding and probably it adds another bit of extra pressure because you don’t want to make mistakes. But making mistakes are fine, you have a buddy that is that extra set of eyes to help you code, spot possible issues and you can discuss very easily together on the approache to take. As you are both working on it, you will both know what, how and why this new piece functionality is created.

When you create a Pull Request you can add either a comment or update the description that you both worked on it. This will let the reviewer also know that there was a 2nd pair of eyes on it and most of the actual review work was already done. So it then is a bit of a formality to approve the Pull Request. But if it is just a formality, why not commit into master|main?

Well, I think you should never merge into master. Always create a branch, create a Pull Request and merge that. When commiting to master|main directly the functionality is only known to both you and you both can still make errors. When you also have tests that are executed as part of the CI process with feature branches, you have an extra confirmation that you won’t break anything. And what I also think is very important with creating a Pull Request, it shows to the rest of your teammembers what you both have worked on. Teammember x knows that your are working on functionality y and is just be informed (I like knowing on a high level what my teammembers are working on, we are a team right?) or maybe (s)he has working on similar functionality that could affect him/her.

So this is the end of my 3 post of the ethics for Pull Requests. Are there any processes you missed on this page that I forgot? Please let me know in the comment and I will gladly update this page.