Finalizing the installation of Zabbix Agent with Ansible

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/finalizing-the-installation-of-zabbix-agent-with-ansible/13321/ (There are a lot of interesting posts on there, so please check the blogposts often!)

In the previous blog posts, we created a Zabbix Server with a new user, a media type, and an action. In the 2nd blog post, we continued with creating and configuring a Zabbix Proxy. In the last part of this series of blog posts, we will install the Zabbix Agent on all of the 3 nodes we have running.

This blog post is the 3rd part of a 3 part series of blog posts where Werner Dijkerman gives us an example of how to set up your Zabbix infrastructure by using Ansible.
You can find part 1 of the blog post by clicking here.

To summarize, so far we have a Zabbix Server and a Zabbix Proxy. The Zabbix Server has a MySQL instance running on a separate node, the MySQL instance for the Zabbix Proxy runs on the same node. But we are missing one component right now, and that is something we will install with the help of this blog post. We will install the Zabbix Agent on the 3 nodes.

A git repository containing the code used in these blog posts is available on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we run Ansible, we need to make sure we have opened a shell on the “bastion” node. We can do that with the following command:

$ vagrant ssh bastion

Once we have opened the shell, go to the “/ansible” directory where we have all of our Ansible files present.

$ cd /ansible

In the previous blog post, we executed the “zabbix-proxy.yml” playbook. Now we are going to use the “zabbix-agent.yml” playbook. The playbook will install the Zabbix Agent on all nodes (“node-1”, “node-2” and “node-3”). Next up, on both the “node-1” and “node-3”, we will add a user parameters file specifically for MySQL. With this user parameters file, we are able to monitor the MySQL instances.

$ ansible-playbook -i hosts zabbix-agent.yml

This playbook will run for a few minutes installing the Zabbix Agent on the nodes. It will install the zabbix-agent package and add the configuration file, but it will also make a connection to the Zabbix Server API. We will automatically create a host with the correct IP information and the correct templates! When the Ansible playbook has finished running, the hosts can immediately be found in the Frontend. And better yet, it is automatically correctly configured, so the hosts will be monitored immediately!

We have several configurations spread over multiple files to make this work. We first start with the “all” file.

The file “/ansible/group_vars/all” contains the properties that will apply to all hosts. Here we have the majority of essential properties configured that are overriding the default properties of the Ansible Roles. Each role has some default configuration, which will work out of the box. But in our case, we need to override these, and we will discuss some of these properties next.

zabbix_url

This is the URL on which the Zabbix Frontend is available and thus also the API. This property is for example used when we create the hosts via the API as part of the Proxy and Agent installation.

zabbix_proxy

The Zabbix Agents will be monitored by the Zabbix Proxy unless the Agent runs on the Zabbix Server or the host running the database for the Zabbix Server. Like with the previous blog post, we will also use some Ansible notation to get the IP address of the host running the Zabbix Proxy to configure the Zabbix Agent.

zabbix_proxy: node-3
zabbix_agent_server: "{{ hostvars[zabbix_proxy]['ansible_host'] }}"
zabbix_agent_serveractive: "{{ hostvars[zabbix_proxy]['ansible_host'] }}"

With the above configuration, we configure both the Server and ServerActive in the Zabbix Agent’s configuration file to use the IP address of the host running the Zabbix Proxy. If you look at the files “/ansible/group_vars/zabbix_database” and “/ansible/group_vars/zabbix_server/generic” you would see that these contain the following:

zabbix_agent_server: "{{ hostvars['node-1']['ansible_host'] }}"
zabbix_agent_serveractive: "{{ hostvars['node-1']['ansible_host'] }}"

The Zabbix Agent on the Zabbix Server and on its database is using the IP address of the Zabbix Server to be used as the value for both the “Server” and “ActiveServer” configuration settings for the Zabbix Agent.

zabbix_api_user & zabbix_api_pass

These are the default in the roles, but I have added them here so it is clear that they exist. When you change the Admin user password, don’t forget to change them here as well.

zabbix_api_create_hosts & zabbix_api_create_hostgroups 

Because we automatically want to create the Zabbix Frontend hosts via the API, we need to set both these properties to true. Firstly, we create the host groups that can be found with the property named “zabbix_host_groups”. After that, as part of the Zabbix Agent installation, the hosts will be created via the API because of the property zabbix_api_create_hosts.

Now we need to know what kind of information we want these hosts created with. Let’s go through some of them.

zabbix_agent_interfaces

This property contains a list of all interfaces that are used to monitoring the host. This is relatively simple in our case, as the hosts only have 1 interface available. You can find some more information about what to use when you have other interfaces like IPMI or SNMP: https://github.com/ansible-collections/community.zabbix/blob/main/docs/ZABBIX_AGENT_ROLE.md#other-interfaces We use the interface with the value from property “ansible_host” for port 10050.

zabbix_host_groups

This property was also discussed before – we automatically assign our new host to these host groups. Again, we have a fundamental setup, and thus it is an effortless property.

zabbix_link_templates

We provide a list of all Zabbix Templates we will want to assign to the hosts with this property. This property seems a bit complicated, but no worries – let’s dive in!

zabbix_link_templates:
  - "{{ zabbix_link_templates_append if zabbix_link_templates_append is defined else [] }}"
  - "{{ zabbix_link_templates_default }}"

With the first line, we add the property’s value “zabbix_link_templates_append”, but we only do that if that property exists. If Ansible can not find that property, then we basically add an empty list. So where can we find this property? We can check the files in the other directories in the group_vars directory. If we check, for example “/ansible/group_vars/database/generic”, we will find the property:

zabbix_link_templates_append:
  - 'MySQL by Zabbix agent'

So on all nodes that are part of the database group, we add the value to the property “zabbix_link_templates”. All of the database servers will get this template attached to the host. If we would check the file “/ansible/group_vars/zabbix_server/generic”, then we will find the following:

zabbix_link_templates_append:
  - 'Zabbix Server'

As you probably understand now, when we create the Zabbix Server host, we will add the “Zabbix Server” template to the host, because this file is only used for the hosts that are part of the zabbix_server group.

With this setup, we can configure specific templates for the specific groups, but there is also at least 1 template that we always want to add. We don’t want to add the template to each file as that is a lot of configuration, so we use a new property for this named “zabbix_link_templates_default”. In our case, we only have Linux hosts, so we always want to add the templates:

zabbix_link_templates_default:
  - "Linux by Zabbix agent active"

On the Zabbix Server, we both assign the “Zabbix Server” template and the template “Linux by Zabbix agent active” to the host.

But what if we have Macros?

zabbix_macros

As part of some extra tasks in this playbook execution, we also need to provide a macro for some hosts. This macro is needed to make the Zabbix Template we assign to the hosts work. For the hosts running a MySQL database, we need to add a macro, which can be found with the property zabbix_macros_append in the file “/ansible/group_vars/database/generic”.

zabbix_macros_append:
  - macro_key: "MYSQL.HOST"
    macro_value: "{{ ansible_host }}"

We will create 1 macro with the key name “MYSQL.HOST” and assign a value that will be equal to the contents of the property ansible_host (For the “node-2” host, the host running the database for the Zabbix Server), which is “10.10.1.12”.

User parameters

The “problem” with assigning the MySQL template is that it also requires some UserParameter entries set. The Zabbix Agent role can deploy files containing UserParameters to the given hosts. In “/ansible/group_vars/database/generic” we can find the following properties:

zabbix_agent_userparameters_templates_src: "{{ inventory_dir }}/files/zabbix/mysql"
zabbix_agent_userparameters:
  - name: template_db_mysql.conf

The first property “zabbix_agent_userparameters_templates_src” will let Ansible know where to find the files. The “{{ inventory_dir }}” will be translated to “/ansible” and here you will find a directory named “files” (and you will find the group_vars directory as well) and further drilling down the directories, you will find the file “template_db_mysql.conf”.

With the second property “zabbix_agent_userparameters” we let Ansible know which file we want to deploy to the host. In this case, the only file found in the directory named “template_db_mysql.conf”.

When the Zabbix agent role is fully executed, we have everything set to monitor all the hosts automatically. Open the dashboard, and you will see something like the following:

It provides an overview, and on the right side, you will notice we have a total of 3 nodes of which 3 are available. Maybe you will see a “Problem” like in the screenshot above, but it will go away.

If we go to “Configuration” and “Hosts,” we will see that we have the 3 nodes, and they have the status “Enabled” and the “ZBX” icon is green, so we have a proper connection.

We should verify that we have some data, so go to “Monitoring” and click on “Latest data.” We select in the Host form field the “Zabbix database,” and we select “MySQL” as Application and click on “Apply.” If everything is right, it should provide us with some information and values, just like the following screenshot. If not, please wait a few minutes and try again.

Summary

This is the end of a 3 part blog post in creating a fully working Zabbix environment with a Zabbix Server, Proxy, and Agent. With these 3 blogposts you were able to see how you can install and configure a complete Zabbix environment with Ansible. Keep in mind that the code shown was for demo purposes and it is not something you can immediately use for the Production environment. We also used some of the available functionality of the Ansible collection for Zabbix, there are many more possibilities like creating a maintenance period or a discovery rule. Not everything is possible, if you do miss a task or functionality of a role that Ansible should do or configure, please create an issue on Github so we can make it happen.

Don’t forget to execute the following command:

$ vagrant destroy -f

With this, we clean up our environment and delete our 4 nodes, thus finishing with the task at hand!

Advertisement

Installing and configuring the Zabbix Proxy

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/installing-and-configuring-the-zabbix-proxy/13319/ (There are a lot of interesting posts on there, so please check the blogposts often!)

In the previous blog post, we created a Zabbix Server setup, created several users, a media type, and an action. But today, we will install on a 3rd node the Zabbix Proxy. This Zabbix Proxy will have its database running on the same host, so the “node-3” host has both the MySQL and Zabbix Proxy running.

This blog post is the 2nd part of a 3 part series of blog posts where Werner Dijkerman gives us an example of how to set up your Zabbix infrastructure by using Ansible.
You can find part 1 of the blog post by clicking Here

A git repository containing the code of these blog posts is available, which can be found on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we run Ansible, we have opened a shell on the “bastion” node. We do that with the following command:

$ vagrant ssh bastion

Once we have opened the shell, go to the “/ansible” directory where we have all of our Ansible files present.

$ cd /ansible

With the previous blog post, we executed the “zabbix-server.yml” playbook. Now we use the “zabbix-proxy.yml” playbook. The playbook will deploy a MySQL database on “node-3” and also installs the Zabbix Proxy on the same host.

$ ansible-playbook -i hosts zabbix-proxy.yml

This playbook will run for a few minutes creating all services on the node. While it is running, we will explain some of the configuration options we have set.

The configuration which we will talk about can be found in “/ansible/group_vars/zabbix_proxy” directory. This is the directory that is only used when we deploy the Zabbix proxy and contains 2 files. 1 file called “secret”, and a file called “generic”. It doesn’t really matter what names the files have in this directory. I used a file called the “secret” for letting you know that this file contains secrets and should be encrypted with a tool like ansible-vault. As this is out of scope for this blog, I simply made sure the file is in plain text. So how do we know that this directory is used for the Zabbix Proxy node?

In the previous blog post, we mentioned that with the “-I” argument, we provided the location for the inventory file. This inventory file contains the hostnames and the groups that Ansible is using. If we open the inventory file “hosts”, we can see a group called “zabbix_proxy.” So Ansible uses the information in the “/ansible/group_vars/zabbix_proxy” directory as input for variables. But how does the “/ansible/zabbix-proxy.yml” file know which host or groups to use? At the beginning of this file, you will notice the following:

- hosts: zabbix_proxy
  become: true
  collections:
    - community.zabbix

Here you will see the that “hosts” key contains the value “zabbix_proxy”. All tasks and roles that we have configured in this play will be applied to all of the hosts that are part of the zabbix_proxy group. In our case, we have only 1 host part of the group. If you would have for example 4 different datacenters and within each datacenter you want to have a Zabbix Proxy running, executing this playbook will be done on these 4 hosts and at the end of the run you would have 4 Zabbix Proxy servers running.

Within the “/ansible/group_vars/zabbix_proxy/generic” the file, we have several options configured. Let’s discuss the following options:

  • zabbix_server_host
  • zabbix_proxy_name
  • zabbix_api_create_proxy
  • zabbix_proxy_configfrequency

zabbix_server_host

The first one, the “zabbix_server_host” property tells us where the Zabbix Proxy can find the Zabbix Server. This will allow the Zabbix Proxy and the Zabbix Server to communicate with each other. Normally you would have to configure the firewall (Iptables or Firewalld) as well to allow the traffic, but in this case, there is no need for that. Everything inside our environment which we have created with Vagrant has full access. When you are going to deploy a production-like environment, don’t forget to configure the firewall (Currently this configuration of the firewalls are not yet available as part of the Ansible Zabbix Collection for both the Zabbix Server and the Zabbix Proxy. So for now you should be creating a playbook in order to configure the local firewall to allow/deny traffic).

As you will notice, we didn’t configure the property with a value like an IP address or FQDN. We use some Ansible notation to do that for us, so we only have the Zabbix Server information in one place instead of multiple places. In this case, Ansible will get the information by reading the inventory file and looking for a host entry with the name “node-1” (Which is the hostname that is running the Zabbix Server), and we use the value found by the property named “ansible_host” (Which has a value “10.10.1.11”).

zabbix_proxy_name

This is the name of the Zabbix Proxy host, which will be shown in the Zabbix frontend. We will see this later in this blog when we will create a new host to be monitored. When you create a new host, you can configure if that new host should be monitored by a proxy and if so, you will see this name.

zabbix_api_create_proxy

When we deploy the Zabbix Proxy role, we will not only install the Zabbix Proxy package, the configuration file and start the service. We also perform an API call to the Zabbix Server to create a Zabbix Proxy entry. With this API call, we can configure hosts to be monitored via this new Zabbix Proxy.

zabbix_proxy_configfrequency

The last one is just for demonstration purposes. With a default installation/configuration of the Zabbix Proxy, it has a basic value of 3600. This means that the Zabbix Server sends the configuration every 3600 to the Zabbix Proxy. Because we are running a small demo here in this Vagrant setup, we have set this to 60 seconds.

Now the deployment of our Zabbix Proxy will be ready.

When we open the Zabbix Web interface again, we go to “Administration” and click on “Proxies”. Here we see the following:

We see an overview of all proxies available, and in our case, we only have 1. We have “node-3” configured, which has an “Active” mode. When you want to configure a “Passive” mode proxy, you’ll have to update the “/ansible/group_vars/zabbix_proxy” file and add somewhere in the file the following entry: “zabbix_proxy_status: passive”. Once you have updated and saved the file, you’ll have to rerun the “ansible-playbook -i hosts zabbix-proxy.yml” command. If you will then recheck the page, you will notice that it now has the “Passive” mode.

So let’s go to “Configuration” – “Hosts”. At the moment, you will only see 1 host, which is the “Zabbix server,” like in the following picture.

Let’s open the host creation page to demonstrate that you can now set the host to be monitored by a proxy. The actual creation of a host is something that we will do automatically when we deploy the Zabbix Agent with Ansible and not something we should do manually. 😉 As you will notice, you are able to click on the dropdown menu with the option “Monitored by proxy” and see the “node-3” appear. That is very good!

Summary

We have installed and configured both a Zabbix Server and a Zabbix Proxy, and we are all set now. With the Zabbix Proxy, we have installed both the MySQL database and the Zabbix Proxy on the same node. Whereas we did install them separately with the Zabbix Server. With the following blog post, we will go and install the Zabbix Agent on all nodes.

Installing the Zabbix Server with Ansible

I wrote this blog post I while back for Zabbix itself to be posted on the Blog site, see: https://blog.zabbix.com/installing-the-zabbix-server-with-ansible/13317/ (There are a lot of interesting posts on there, so please check the blogposts often!)

Today we are focusing more on the automation of installation and software configuration instead of using the manual approach. Installing and configuring software the manual way takes a lot more time, you can easily make more errors by forgetting steps or making typos, and it will probably be a bit boring when you need to do this for a large number of servers.

In this blog post, I will demonstrate how to install and configure a Zabbix environment with Ansible. Ansible has the potential to simplify many of your day-to-day tasks. As an alternative to Ansible, you may also opt in to use Puppet, Chef, and SaltStack to install and configure your Zabbix environment.

Ansible does not have any specific infrastructure requirements for it to do its job. We just need to make sure that the user exists on the target host, preferably configured with SSH keys. With tools like Puppet or Chef, you need to have a server running somewhere, and you will need to deploy an agent on your nodes. You can learn more about Ansible here:  https://docs.ansible.com/ansible/latest/index.html.

This post is the first in a series of three articles. We will set up a (MySQL) Database running on 1 node (“node-2”), Zabbix Server incl. Frontend, which will be running on a separate node (“node-1”). Once we have built this, we configure an action, media and we will create some users. In the following image you will see the environment we will create.

Our environment we will create.
The environment we will create.

In the 2nd blog post, we will set up a Zabbix Proxy and a MySQL database on a new but the same node (“node-3”). In the 3rd blog post, we will install the Zabbix Agent on all of the 3 nodes we were using so far and configure some user parameters. Where the Zabbix Agent on “node-3” is using the Zabbix Proxy, the Zabbix Agent on the nodes “node-1” and “node-2” will be monitored by the Zabbix Server.

Preparations

A git repository containing the code used in these blog posts is available, which can be found on https://github.com/dj-wasabi/blog-installing-zabbix-with-ansible. Before we can do anything, we have to install Vagrant (https://www.vagrantup.com/downloads.html) and Virtualbox (https://www.virtualbox.org/wiki/Downloads). Once you have done that, please clone the earlier mentioned git repository somewhere on your host. For this demo, we will not run the Zabbix Frontend with TLS certificates.

We have to update the hosts file. With the following line, we need to make sure that we can access the Zabbix Frontend.

10.10.1.11 zabbix.example.com

In the “ROOT” directory of the git repository which you cloned some moments ago, where you can also find the Vagrantfile, This Vagrantfile contains the configuration of the virtual machine of our setup. We will create 4 Virtual Machine’s running Ubuntu 20.04, each with 1 CPU and 1 GB of Ram which you can see in the first “config” block. In the 2nd config block, we configure our “bastion” host, which we discuss later. This node will get the ip 10.10.1.3 and we also mount the ansible directory in this Virtual Machine on location “/ansible”. For installing and configuring this node we will use a playbook bastion.yml to do this. With this playbook, we will install some packages like Python, git and Ansible inside this bastion virtual machine.

The 3rd config block is part of a loop that will configure and it will create 3 Virtual Machines. Each virtual machine is also an Ubuntu node, had its own ip (respectively 10.10.1.11 for the first node, 10.10.1.12 for the second and 10.10.1.13 for the 3rd node) and just like the “bastion” node, they have each 1 CPU and 1 GB of RAM.

You will have to execute the following command:

$ vagrant up

With this command, we will start our Virtual Machine’s. This might take a while, as it will download a VirtualBox image containing Ubuntu. The “vagrant up” command will start the “bastion” node and all other nodes as a part of this demo. Once that is done, we need to access a shell on the “bastion” node:

$ vagrant ssh bastion

This “bastion” node is a fundamental node on which we will execute Ansible, but we will not be installing anything on this host. We have opened a shell in the Virtual Machine we just created. You can compare it with creating an “ssh” connection. We have to go to the following directory before we can download the dependencies:

$ cd /ansible

As mentioned before, we have to download the Ansible dependencies. The installation depends on several Ansible Roles and an Ansible Collection. With the Ansible Roles and the Ansible Collection, we can install MySQL, Apache, and the Zabbix components. We have to execute the following command to download the dependencies:

$ ansible-galaxy install -r requirements.yml
Starting galaxy role install process
- downloading role 'mysql', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mysql/archive/3.3.0.tar.gz
- extracting geerlingguy.mysql to /home/vagrant/.ansible/roles/geerlingguy.mysql
- geerlingguy.mysql (3.3.0) was installed successfully
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/3.1.4.tar.gz
- extracting geerlingguy.apache to /home/vagrant/.ansible/roles/geerlingguy.apache
- geerlingguy.apache (3.1.4) was installed successfully
- extracting wdijkerman.php to /home/vagrant/.ansible/roles/wdijkerman.php
- wdijkerman.php was installed successfully
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'community.zabbix:1.2.0' to '/home/vagrant/.ansible/collections/ansible_collections/community/zabbix'
Created collection for community.zabbix at /home/vagrant/.ansible/collections/ansible_collections/community/zabbix
community.zabbix (1.2.0) was installed successfully

Your output may vary because of versions that might have been updated already since writing this blog post. We now have downloaded the dependencies and are ready to install the rest of our environment. But why do we need to download a role for MySQL, Apache and php? A role contains all the neccecerry tasks and files to configure that specific service. So in the case for the MySQL Ansible role, it will install the MySQL-server and all other packages that MySQL requires on the host, it will configure that the mysqld service is created and is running, but it will also create the databases, create and configure MySQL users and configure the root password. Using a role will help us install our environment and we don’t have to figure out ourselves on installing and configuring a MySQL server manually.

So what about the collection, the Ansible Community Zabbix Collection? Ansible has introduced this concept with Ansible 2.10 and is basically a “collection” of plugins, modules and roles for a specific service. In our case, with the Zabbix Collection, the collection contains the roles for installing the Zabbix Server, Proxy, Agent, Javagateway and the Frond-end. But it also contains a plugin to use a Zabbix environment as our inventory and contains modules for creating resources in Zabbix. All of these modules will work with the Zabbix API to configure these resources, like actions, triggers, groups. templates, proxies etc. Basically, everything we want to create and use can be done with a role or a collection.

Installing Zabbix Server

Now we can execute the following command, which will install the MySQL database on “node-2” and installs the Zabbix Server on “node-1”:

$ ansible-playbook -i hosts zabbix-server.yml

This might take a while, a minute, or 10 depending on the performance of your host. We execute the “ansible-playbook” command, and then “-i” we provide the location of the inventory file. Here you see the contents of the inventory file:

[zabbix_server]
node-1 ansible_host=10.10.1.11

[zabbix_database]
node-2 ansible_host=10.10.1.12

[zabbix_proxy]
node-3 ansible_host=10.10.1.13

[database:children]
zabbix_database zabbix_proxy

This inventory file contains basically all of our nodes and to which group the hosts belong. We can see in that file that there is a group called “zabbix_server” (The value between [] square brackets is the name for the group) and contains the “node-1” host. Because we have a group called “zabbix_server,” we also have a directory containing some files. These are all the properties (or variables) that will be used for all hosts (in our case, only the “node-1”) in the “zabbix_server” group.

Web Interface

Now you can open your favorite browser and open “zabbix.example.com”, and you will see the Zabbix login screen. Please enter the default credentials:

Username: Admin
Password: zabbix

On the Dashboard, you will probably notice that it complains that it can not connect to the Zabbix Agent running on the Zabbix Server, which is fine as we haven’t  installed it yet. We will do this in a later blog post.

Dashboard overview

When we go to “Administration” and click on “Media types,” we will see a media type called “A: Ops email.” That is the one we have created. We can open the “/ansible/zabbix-server.yml” file and go to line 33, where we have configured the creation of the Mediatype. In this case, we have configured multiple templates for sending emails via the “mail.example.com” SMTP server.

Now we have seen the media type, we will look at the trigger we just created. This trigger makes use of the media type we just saw. The trigger can be found in the “/ansible/zabbix-server.yml” file on line 69. When you go to “Configuration” and “Actions,” you will see our created trigger “A: Send alerts to Admin”. But we don’t want to run this in Production, and for demonstrating purposes, we have selected to be triggered when the severity is Information or higher.

And lastly, we are going to see that we have also created new internal users. Navigate to “Administration” – “Users,” and you will see that we have created a user called “wdijkerman”, which can be found in the “/ansible/zabbix-server.yml” file on line 95. This user will be part of a group created earlier called “ops,”. The user type is Zabbix super admin and we have configured the email media type to be used 24×7.

We have defined a default password for this user – “password”. When you have changed the password in the Zabbix Frontend UI, executing the playbook would not change the password back again to “password.” So don’t worry about it. But if you would have removed – let’s say – the “ops” group, then, when you execute the playbook again, the group will be re-added to the user.

Summary

As you see, it is effortless to create and configure a Zabbix environment with Ansible. We didn’t have to do anything manually, and all installations and configurations were applied automatically when we executed the ansible-playbook command. You can find more information on either the Ansible page https://docs.ansible.com/ansible/latest/collections/community/zabbix/ or on the Github page https://github.com/ansible-collections/community.zabbix.

In the next post, we will install and configure the Zabbix Proxy.

Installing zabbix-server with ansible

ansible_logo_black_squarezabbix_logo

Not only I have an puppet module which can be freely used from the forge, I also have some Ansible roles for Zabbix. This page will describe installing the zabbix-server with the dj-wasabi.zabbix-server role. If you want to know how you install the zabbix-agent, please check this page.

You can find the role and some information on this page: https://galaxy.ansible.com/list#/roles/2070

This role works on the 3 main Linux operating systems:

  • RedHat
  • Debian
  • Ubuntu

So, if your server has one of these operating system, you can continue. If you have however an other operating system and have some Ansible knowledge, please add some improvements and create an Pull Request on Github. I always accept Pull Requests related to the Ansible roles.

When you want to install this role, you only have to execute the following command:

ansible-galaxy install dj-wasabi.zabbix-server

Now we need to setup everything, but before we do anything we need to know what kind of database server is going to be used. Zabbix Server can work with several different databases as backend. This Ansible role only works with the following databases:

  • PostgreSQL
  • MySQL

Before we see the examples, there is one main parameter which is always needed: zabbix_url

This is the url on which the zabbix interface is available and should be an fqdn. Default it will create an Apache Virtual Host configuration file with this FQDN as ServerName. If you set this parameter as this:

zabbix_url: zabbix.example.com

the web interface will be available at: http://zabbix.example.com

PostgreSQL

Default the PostgreSQL is used as backend and before we can use this role, we need to find and download an Ansible role for PostgreSQL which can be used on your operating system. In this example we are using the following role: ‘galaxyprojectdotorg.postgresql’

The following is an example of an playbook for installing the ‘zabbix-server’ with an PostgreSQL database:

--- 
- hosts: zabbix-server
  roles: 
    - role: galaxyprojectdotorg.postgresql
      postgresql_pg_hba_conf: 
        - "host all all 127.0.0.1/32 trust"
        - "host all all ::1/128 trust"
      postgresql_pg_hba_local_ipv4: false
      postgresql_pg_hba_local_ipv6: false
    - role: dj-wasabi.zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server

This is the minimum configuration to use for this role with an PostgreSQL as database. What might help to secure everything is to use an more difficult to guess password for the ‘server_dbuser’ 😉

MySQL

Lets use MySQL as backend now. The following example is used with the following role: ‘geerlingguy.mysql’:

---
- hosts: localhost
  roles:
    - role: geerlingguy.mysql
    - role: ansible-zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      database_type: mysql
      database_type_long: mysql
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server

Same as for the example with PostgreSQL, use an different value for the server_dbpassword.

Other configurations

Don’t think that what you just saw with configuring this role is everything. There are a lot of other configuration parameters that can be set. Keep in mind, that all configuration options you’ll normally find in the ‘zabbix_server.conf’ configuration file, can also be set with this role.

Lets give an example:

When we need to set the StartPollers to value 10, we can update the MySQL playbook to look like this:

---
- hosts: localhost
  roles:
    - role: geerlingguy.mysql
    - role: ansible-zabbix-server
      zabbix_url: zabbix.example.com
      zabbix_version: 2.4
      database_type: mysql
      database_type_long: mysql
      server_dbuser: zabbix-server
      server_dbpassword: zabbix-server
      server_startpollers: 10

When the role is executed on the ‘zabbix-server’, we see the following in the configuration file:

### option: startpollers
#	number of pre-forked instances of pollers.
#
StartPollers=10

Keep in mind to lower the property setting and prefix it with ‘server_’ and you’ll have the property for this Ansible role.

As this Ansible role isn’t perfect, please let me know if you encounter any issues by creating an issue. Pull Request for bugs or new features are always welcome!

wdijkerman-zabbix version 1.7.0 puppet module released

Puppet LogoZabbix Logo

I just released an new version of the wdijkerman-zabbix puppet module: 1.7.0.

This is an “Pull requests” release, as this release only contains pull requests created by users of this module. 🙂

The change log for this release:

  • misspelled parameter path #116 (By pull request: karolisc (Thanks!))
  • Update template.pp #121 (By pull request: claflico (Thanks!))
  • add support for CloudLinux #122 (By pull request: bastelfreak (Thanks!))
  • Fping wrong path in debian #124 (By pull request: Oyabi (Thanks!))
  • refactoring of repo.pp #126 (By pull request: bastelfreak (Thanks!))
  • Added supporting new Zabbix params #128 (By pull request: akostetskiy (Thanks!))
  • Generalise the zabbix_url #129 (By pull request: DjxDeaf (Thanks!))

The “#<NUM>” is a reference to an Github issue. The release can now also be found on and downloaded from the forge.

If you find an bug or have an improvement for this Zabbix puppet module, please create pull requests!

Zabbix nodata trigger, really a lifesaver

Zabbix has a lot of triggers you can use for your environment. But when it comes to the most important checks, I’ll update the current trigger and create an “or” statement with the “nodata” trigger.

It happened for me a few times when Zabbix didn’t notify me about some checks that went into PROBLEM while the service/program wasn’t responding. In all of these cases, I had it configures with the “last” trigger. All of them said it was ok, but the latest update was at least 30 minutes ago (Some even longer!). I slept well though, but my start of the day wasn’t really good. 🙂

I did find that most of these kind of checks were items that used scripts which were created by myself or by my colleagues. These were simple scripts in bash or python, which had no proper way of exiting due to an timeout.

So the first and most easy way is to update the trigger with the “nodata” configuration. I use an “default” of 5 minutes, if I don’t have any data the trigger can be fired. An good example is the trigger for the zabbix-agent:

{Template App Zabbix Agent:agent.ping.nodata(5m)}=1

When there is no data retrieved in the last 5 minutes, it will be fired. So I’ve updated the trigger for my Apache template like this:

{Template Apache:apache.run.status[{HOST.NAME},'localhost'].last()}=0 or {Template Apache:apache.run.status[{HOST.NAME},'localhost'].nodata(5m)}=1

If the last value of the item apache.run.status is 0 or there is no data retrieved in the last 5 minutes, then we can assume Apache is down.

But these scripts needs to be able to exit within the TimeOut parameter in the configuration file. So each script should have an timeout and when this timeout occurs, it should print some information which Zabbix can handle. (Not an python stacktrace for example. 🙂 )

I know the “nodata” trigger has an few drawbacks. It do add some extra load on the server for these checks. I believe the “nodata” triggers are checked every 30 seconds, but I try to use the nodata only for those critical (Or Disaster in Zabbix terms 🙂 ) triggers.

Another one, but happens (hopefully) not that much, when the Zabbix-server is in some kind of maintenance or had some problems (Or when you have an Zabbix proxy which doesn’t send the data to the server) a lot of triggers are fired when the times has past. So you just updated the Zabbix-server with the latest (linux/kernel) patches and after say 10 minutes the Zabbix-server is up and running again, all of those nodata triggers are fired. 🙂

But anyways, it is an lifesaver! 🙂

Installing zabbix-agent with Ansible

ansible_logo_black_squarezabbix_logo

Not only I have an puppet module for installing Zabbix, I also have some Ansible roles for this. At the moment there are 4 roles:

In this blog item, we talk about the “zabbix-agent” role. The latest version is 0.2.0.

Defaults

Installing this role is very easy:


ansible-galaxy install dj-wasabi.zabbix-agent

It will be installed in your roles directory. Default is “/etc/ansible/roles” or whatever you have configured in the ansible.cfg file. After installation there is only 1 (or 2 when you make use of active items) parameters needed for making this role work:

agent_server: <IP_-_FQDN_OF_ZABBIX_SERVER>
agent_serveractive: <IP_-_FQDN_OF_ZABBIX_SERVER>

This will need the ip address or the FQDN of the “zabbix-server”.

OS?

This role works on several operating systems/families:

  • RedHat
  • Debian
  • Ubuntu
  • OpenSuse

If you have an operating system/family which isn’t in the list above, you can create an issue at the Github page and please fill in the request. I can’t make any guarantee that it will come, but I can try it. Or if you do have some Ansible skills, please create an Pull request and I would be happy to accept it. 🙂

Playbook

So, how does the playbook looks like? Like this:

- hosts: all
  sudo: yes
  roles:
   - role: dj-wasabi.zabbix-agent
     agent_server: <IP_-_FQDN_OF_ZABBIX_SERVER>
     agent_serveractive: <IP_-_FQDN_OF_ZABBIX_SERVER>

As you see it is very basic and does the job very good. This only installs the agent on the specific server and configures the configuration file. But we really want to automate everything right?

Cove

Few weeks ago I found this pull requests for the “ansible-modules-extra” repository. This pull requests had an few ansible modules which made sure that you can use the Zabbix API to create or update hosts configuration. In the pull requests there were something like 5 modules, but this Ansible role only use 3 of them. With this role, you can create the following:

  • host groups
  • Host itself.
  • Macros for the host

For now, when the host is created, it will only create the “zabbix interface”. Maybe with the next release I’ll make sure you can also create SNMP, JMX and IPMI interfaces.

How do we have to configure it? Something like this. You will have to change it to your environment.

- hosts: wdserver00
  roles:
     - role: zabbix-agent
       agent_server: 192.168.1.1
       agent_serveractive: 192.168.1.1
       zabbix_url: http://zabbix.example.com
       zabbix_api_use: true
       zabbix_api_user: Admin
       zabbix_api_pass: Zabbix
       zabbix_create_host: present
       zabbix_host_groups:
         - Linux servers
       zabbix_link_templates:
         - Template OS Linux
       zabbix_macros:
         - macro_key: apache_type
           macro_value: reverse_proxy

I’ll skip the first 2 parameters, as these are described earlier on this page.

zabbix_url: The url on which the Zabbix web interface is available.
zabbix_api_user: The username which will connect to the API.
zabbix_api_pass: The password for the “zabbix_api_user” user.
zabbix_create_host: present if we want to create the host, absent if we want to delete it.
zabbix_host_groups: List of hostgroup where this host belongs to.
zabbix_link_templates: List of templates which will be linked to the host.
zabbix_macros: key, value pair of macros that will be used by the host. 

When we run Ansible, we will see at the end of the run:

.. <skip> ..
TASK: [zabbix-agent | Create hostgroups] **************************************
ok: [wdserver00 -> 127.0.0.1]

TASK: [zabbix-agent | Create a new host or update an existing host's info] ****
changed: [wdserver00 -> 127.0.0.1]

TASK: [zabbix-agent | Updating host configuration with macros] ****************
changed: [wdserver00 -> 127.0.0.1] => (item={'macro_key': 'apache_type', 'macro_value': 'reverse_proxy'})

Nice! If you check the Web interface, you’ll see that the host is created with the correct host groups and templates. If not, you’ll see some error messages in the Ansible output which will say what went wrong.

This role isn’t perfect, so if you encounter an bug or found/have and enhancement, please create an Pull request at Github and I’ll accept it. We can all make this role beter. 🙂

Side note:

There are more parameters which can be overridden, please check the “defaults/main.yml” file or the README.

Zabbix agent installation and configuration with puppet exported resources

puppet zabbix_logo

Who doesn’t like to automate tasks when this is easy and can save a lot of time/troubles. When you are just starting with Zabbix, it seems that the only thing you can automate is installing the components with rpm/deb packages on the servers/clients. After installation of the packages, you’ll still need to create the host in the web interface which takes some time (And boring to if you have a lot of hosts to configure..). There is an possibility to use (auto) discovery rules, but this requires some configuration in the web interface and this only works once when the host isn’t configured in the web interface (Initial setup only).

Luckily, the Zabbix has an API which can be used for some tasks that can be automated, like creating hosts! The wdijkerman-zabbix module can use this API for the following tasks:

  • Create host
  • Update host with Templates

There is an requirement when you want to do this: Puppet Master needs the PuppetDB configured. If you want to configure an PuppetDB, please use Google for this as I won’t discuss this here.

Idea

The idea is that when an installation is done via the “zabbix::agent” class on the agent, it send some information via puppet to the PuppetDB. This is some basic information like ipaddress, name of the host and the hostgroups for this host. When the puppet agent on the “zabbix-server” runs, it will retrieve the information from the PuppetDB and will create via the Zabbix API the host(s). With this setup, everything is automated and you don’t have to worry for forgetting something.
So, how do you have to configure the zabbix module?
class { 'zabbix':
  zabbix_url       => 'zabbix.example.com',
  manage_resources => true,
}

This is an very basic configuration method mentioned above which will automatically configure the hosts in the web interface when a new “zabbix-agent” is installed. You don’t have to do anything else (yes ok, you’ll have to run some puppet runs on several machines. :-).)

This assumes that the password for the “Admin” user is still “zabbix”. You can override this of course, just like the database credentials which are default set to “zabbix-server”. (Don’t forget to change it to something non default 😉 )

Userparameters

There is an other nice feature in the “wdijkerman-zabbix” module which uses the Zabbix API. The “zabbix::userparameters” define can make use of the API. The goal for this define is installing an UserParameters file in the zabbix-agent includedir. This file consists of 1 or more UserParameter entries which explains what commands needs to be executed for an key.

An example:

zabbix::userparameters { 'Exim':
 content => 'UserParameter=exim.mailq,sudo /usr/sbin/exim -bpc',
}

There is an item in a template which executes the “sudo /usr/sbin/exim -pbc” command with the “exim.mailq” key. When the above 3 lines are in the exim puppet module and this module installs the UserParameter file, you still need to update the host in the web interface to make sure the “Template Exim” is assigned.

But stop, we can use it like the following now:

zabbix::userparameters { 'Exim':
 content  => 'UserParameter=exim.mailq,sudo /usr/sbin/exim -bpc',
 template => 'Template Exim',
}

We also supply the name of the Zabbix Template in this “user::parameters” define. When configured we first have to execute an puppet run on the agent, so it sends some data to the PuppetDB.

When the Puppet Agent runs on the “zabbix-server”, it first checks if the hosts exits and then it will assign the template “Template Exim” to the host automatically. Off course when the template does not exists, the Puppet run will fail.

Woot! 🙂

One node vs multi node setup zabbix module zabbix-proxy

zabbix_logo

With the previous post I showed you how you can use the zabbix-server on one or multiple machines with my wdijkerman-zabbix puppet module. At the end I said that you could also do this with the zabbix-proxy. So this blog item is showing how to setup the zabbix-proxy on 1 or 2 nodes. I don’t know if people are using it like this, but when I was busy for the zabbix-server, I thought that I also had to do this for the zabbix-proxy. 🙂

So sit back and watch the show. 🙂

Single node

PostgreSQL

So, we first show you have to do this with the PostgreSQL as backend.

node 'proxy.example.com'
 class { 'postgresql::server': }

 class { 'zabbix::proxy':
   zabbix_server_host => '192.168.20.11',
   zabbix_server_port => '10051',
 }
}

As you see in this manifests file, we have configured to use the postgresql-server class and the zabbix-proxy class. This zabbix-proxy class needs 1 parameter: ‘zabbix_server_host’. This is the ip address or fqdn of the zabbix-server host. The parameter ‘zabbix_server_port’ is optional, you only have to use this parameter when the zabbix-server is running on a different port than 10051.

MySQL

Okay, so you want to use MySQL as database backend? No worries, take a look at this:

node 'proxy.example.com'
 class { 'mysql::server': }

 class { 'zabbix::proxy':
   zabbix_server_host => '192.168.20.11',
   zabbix_server_port => '10051',
   database_type      => 'mysql',
 }
}

Like the zabbix-server, the zabbix-proxy uses the PostgreSQL as default database. So we have to use the ‘database_type’ parameter for specifying mysql.

Multi node

With the following example, we have 2 servers:

  • server11.example.com, which will be running the zabbix-proxy.
  • server12.example.com, which will be running the database.

MySQL

Lets start the manifests file again with the MySQL as ‘database_type’:

node 'server11.example.com' {
# My ip: 192.168.30.11
  class { 'mysql::client': }
  class { 'zabbix::proxy':
    zabbix_server_host => '192.168.20.11',
    manage_database    => false,
    database_host      => 'server12.example.com',
    database_type      => 'mysql',
  }
}

node 'server12.example.com' {
# My ip: 192.168.30.12
  class { 'mysql::server':
    override_options => {
      'mysqld'       => {
        'bind_address' => '192.168.30.12',
      },
    },
  }
  class { 'zabbix::database':
    database_type     => 'mysql',
    zabbix_type       => 'proxy',
    zabbix_proxy      => 'server11.example.com',
    database_name     => 'zabbix-proxy',
    database_user     => 'zabbix-proxy',
    database_password => 'zabbix-proxy',
  }
}

On the node of the zabbix-proxy, we had to add a new parameter: ‘manage_database’. We had to set it to false, the proxy class isn’t responsible for creating the database and loading the files. Thats what the ‘zabbix::database’ will do on the 2nd host.

PostgreSQL

Now with the PostgreSQL as ‘database_type’:

node 'server11.example.com' {
# My ip: 192.168.30.11
  class { 'postgresql::client': }
  class { 'zabbix::proxy':
    zabbix_server_host => '192.168.20.11',
    manage_database    => false,
    database_host      => 'server12.example.com',
  }
}

node 'server12.example.com' {
# My ip: 192.168.30.12
  class { 'postgresql::server':
    listen_addresses => '192.168.30.12'
  }
  class { 'zabbix::database':
    zabbix_type       => 'proxy',
    zabbix_proxy_ip   => '192.168.30.11',
    database_name     => 'zabbix-proxy',
    database_user     => 'zabbix-proxy',
    database_password => 'zabbix-proxy',
  }
}

With the above setup for both MySQL as PostgreSQL as ‘database_type’, we had to use the ‘database_name’ and ‘database_password’ parameters. The default values for the ‘zabbix::database’ class is for both parameters ‘zabbix-server’. Which is a kind of strange for the proxy. 🙂

Note

With the default installation on both the single as the multi node setup, the password for the database is ‘zabbix-server’. I would strongly recommend that you use the parameter: ‘database_password’ and give it a nice password.

One node vs multi node setup zabbix module zabbix-server

zabbix_logo
So it was quiet the last few months for posting stuff on the site, so I’ll try to do it now. I also moved the wordpress.com instead of hosting the site myself.
At the moment on writing this blog item, the wdijkerman-zabbix puppet module is at release 1.1.0. One of the biggest changes which came with release 1.0.0, is that you can choose whether your want everything one to run on a single host (Single Node, Like most of us) or running it on different servers (Multi node).

 

Up to release 1.0.0 you could only use the wdijkerman-zabbix puppet module if everything was running on the same host. Yes, from release > 0.5.0 you could even with some ‘hacking’ run the database on an other host, but that was not something I would document let alone it was a little bit confusing to do.

 

First, we start with the Single node setup. Apache, Zabbix-web, Zabbix-server and the database is installed on 1 system.

Single node

PostgreSQL

So, lets install the zabbix-server on one host:
node 'zabbix.example.com'
  class { 'apache':
    mpm_module => 'prefork',
  }
  include apache::mod::php
 
  class { 'postgresql::server': }
 
  class { 'zabbix':
    zabbix_url    => 'zabbix.example.com',
  }
}
With this manifest setup, you have configured an complete zabbix-server on one host.

 

It will install and configure Apache, postgresql (Default database for zabbix) and both zabbix components zabbix-web and zabbix-server. There is only one parameter needed for zabbix-server and it is the ‘zabbix_url’. This is the url on which the zabbix web interface is available, the rest of the parameters are optional.

MySQL

Oh, you want to run it on MySQL? No problem, this would be an very basic setup to begin with:
node 'zabbix.example.com'
  class { 'apache':
    mpm_module => 'prefork',
  }
  include apache::mod::php

  class { 'mysql::server': }

  class { 'zabbix':
    zabbix_url    => 'zabbix.example.com',
    database_type => 'mysql',
  }
}

multi node

Now, the multi node setup. This example consists of 3 nodes:
  • server01.example.com, which will be running Apache and the zabbix-web component.
  • server02.example.com, which will be running the zabbix-server component
  • server03.example.com, which will be running the database.

First we give an example of using this setup with the MySQL database.

MySQL

The following example will be given with the usage of MySQL as database backend. You can also choose to use the PostgreSQL database, but  than you have to change the setup a little bit. This will be described after the example:
node 'server01.example.com' {
  # My ip: 192.168.20.11
  class { 'apache':
    mpm_module => 'prefork',
  }
  class { 'apache::mod::php': }
  class { 'zabbix::web':
    zabbix_url => 'zabbix.example.com',
    zabbix_server => 'server02.example.com',
    database_host => 'server03.example.com',
    database_type => 'mysql',
  }
}

node 'server02.example.com' {
  # My ip: 192.168.20.12
  class { 'mysql::client': }
  class { 'zabbix::server':
    database_host => 'server03.example.com',
    database_type => 'mysql',
  }
}

node 'server03.example.com' {
  # My ip: 192.168.20.13
  class { 'mysql::server':
    override_options => {
      'mysqld' => {
      'bind_address' => '192.168.20.13',
      },
    },
  }
  class { 'zabbix::database':
  database_type => 'mysql',
    zabbix_server => 'server02.example.com',
    zabbix_web => 'server01.example.com',
  }
}
We have to fill in some ip’s or hostnames for this setup. Both the zabbix-web and -server needs to know where the database is and the zabbix-web needs to know where to find the zabbix-server.

PostgreSQL

When you want to use the PostgreSQL as backend in an multinode setup, you can use something like this:
node 'server01.example.com' {
# My ip: 192.168.20.11
 class { 'apache':
   mpm_module => 'prefork',
 }
 class { 'apache::mod::php': }
 class { 'zabbix::web':
   zabbix_url => 'zabbix.example.com',
   zabbix_server => 'server02.example.com',
   database_host => 'server03.example.com',
   database_type => ‘postgresql',
 }
}

node 'server02.example.com' {
# My ip: 192.168.20.12
 class { 'postgresql::client': }
 class { 'zabbix::server':
   database_host => 'server03.example.com',
   database_type => ‘postgresql',
 }
}

node 'server03.example.com' {
# My ip: 192.168.20.13
 class { 'postgresql::server':
   listen_addresses => '192.168.20.13'
 }
 class { 'zabbix::database':
   database_type => ‘postgresql',
   zabbix_web_ip => '192.168.20.12',
   zabbix_server_ip => '192.168.20.13',
 }
}
The PostgreSQL class accepts different values than MySQL: PostgreSQL accepts ipaddresses and MySQL accepts fqdn.

 

On a next item, I’ll go into the zabbix-proxy component. Like the zabbix-server, this can also be installed on a single node or multi node.

Note

With the default installation on both the single as the multi node setup, the password for the database is ‘zabbix-server’. I would strongly recommend that you use the parameter: ‘database_password’ and give it a nice password.