Setting up a secure Vault with a Consul backend

vault_logo

With this blogpost we continue working with a secure Consul environment: We are configuring a secure Vault setup with Consul as backend. YMMV, but this is what I needed to configure to make it work.

Environment

We should have an working Consul Cluster environment. If you don’t have one, please take a look at here for creating one. With this blogpost we expect a secure Consul cluster with SSL certificates and using ACL’s.

In this blogpost we make use of the wdijkerman/vault container. This container is created by myself and is running Vault (At moment of writing release 0.6.4) on Alpine (running on 3.5). Vault is running as user ‘vault’ and the container can be configured to use SSL certificates.

prerequisites

We have to create SSL certificates for the vault service. In this blogpost we use the domain ‘dj-wasabi.local’, as Consul is already running with this domain configuration so we have to create ssl certificates for the FQDN: ‘vault.service.dj-wasabi.local’.

On my host where my OpenSSL CA configuration is stored, I execute the following commands:

openssl genrsa -out private/vault.service.dj-wasabi.local.key 4096

Generate the key.

openssl req -new -extensions usr_cert -sha256 -subj "/C=NL/ST=Utrecht/L=Nieuwegin/O=dj-wasabi/CN=vault.service.dj-wasabi.local" -key private/vault.service.dj-wasabi.local.key -out csr/vault.service.dj-wasabi.local.csr

Create a signing request file and then sign it with the CA.

openssl ca -batch -config /etc/pki/tls/openssl.cnf -notext -in csr/vault.service.dj-wasabi.local.csr -out certs/vault.service.dj-wasabi.local.crt

We copy the ‘vault.service.dj-wasabi.local.key’, ‘vault.service.dj-wasabi.local.crt’ and the caroot certificate file to the hosts which will be running the Vault container into the directory /data/vault/ssl. Hashicorp advises to run vault on hosts where Consul Agents are running, not Consul Servers. This has probably todo with that for most use cases they see is that Consul is part of large networks and thus the servers will handle a lot of request (High load). As the Consul Servers will be very busy, it would then be wise to not run anything else on those servers.

But this is my own versy small environment (With 10 machines) so I will run Vault on the hosts running the Consul Server.

ACL

Before we do anything on these hosts, we create a ACL in Consul. We have to make sure that Vault can create keys in the key/value store and we have to allow that Vault may create a service in Consul named vault.

So our (Client) ACL will look like this:

key "vault/" {
  policy = "write"
}
service "vault" {
  policy = "write"
}

We use this in the ui on the Consul Server and create the ACL. In my case, the ACL is created with id ’94c507b4-6be8-9132-ea15-3fc5b196ea29′. This ID is needed later on when we configure Vault. Also check your ACL for the ‘Anonymous token’. Please make sure you have set the following rule if the Consul default policy is set to deny:

service "vault" {
  policy = "read"
}

With this, we make sure the service is resolvable via dns. In my case this is for ‘vault.service.dj-wasabi.local’.

Configuration

We have to configure the vault docker container. We have to create a directory that will be mounted in the container. First we have to create an user on the host and then we create the directory: /data/vault/config and own it to the just created user.

useradd -u 994 vault
mkdir /data/vault/config
chown vault:vault /data/vault/config

The container is using a user named vault and has UID 994 and we have to make sure that everything is in sync with names and id. Now we create a config.hcl file in the earlier mentioned directory:

backend "consul" {
  address = "vserver-202.node.dc1.dj-wasabi.local:8500"
  check_timeout = "5s"
  path = "vault/"
  token = "94c507b4-6be8-9132-ea15-3fc5b196ea29"
  scheme = "https"
  tls_skip_verify = 0
  tls_key_file = "/vault/ssl/vault.service.dj-wasabi.local.key"
  tls_cert_file = "/vault/ssl/vault.service.dj-wasabi.local.crt”
  tls_ca_file = "/vault/ssl/dj-wasabi.local.pem"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_key_file = "/vault/ssl/vault.service.dj-wasabi.local.key"
  tls_cert_file = "/vault/ssl/vault.service.dj-wasabi.local.crt"
  cluster_address = "0.0.0.0:8201"
}

disable_mlock = false

First we configure a backend for Vault. As we use Consul, we use the Consul backend. Because the Consul is running on https and is using certificates, we have to use the fqdn of the Consul node as the address (same as how we did in configuring Registratror in this post). We also have to configure the options ‘tls_key_file’, ‘tls_cert_file’ and ‘tls_ca_file’, these are the ssl certificates needed for accessing the secure Consul via SSL. Because of this, we have to set the ‘scheme’ to ‘https’ and we have to specify the token for the ACL we created earlier and add the value to the the token option.

Next we configure the listener for Vault. We configure the listener that it listens on all ips on port 8200. We also make sure we configure the earlier created SSL certificates by using them in the ‘tls_key_file’ and ‘tls_cert_file’ options.

The last option is to make sure that Vault can not swap data to the local disk.

Starting Vault

Now we are ready to start the docker container. We use the following command for this:

docker run -d -h vserver-202 --name vault \
--dns=172.17.0.2 --dns-search=service.dj-wasabi.local \
--cap-add IPC_LOCK -p 8200:8200 -p 8201:8201 \
-v /data/vault/ssl:/vault/ssl:ro \
-e VAULT_ADDR=https://vault.service.dj-wasabi.local:8200 \
-e VAULT_CLUSTER_ADDR=https://192.168.1.202:8200 \
-e VAULT_REDIRECT_ADDR=https://192.168.1.202:8200 \
-e VAULT_ADVERTISE_ADDR=https://192.168.1.202:8200 \
-e VAULT_CACERT=/vault/ssl/dj-wasabi.local.pem \
wdijkerman/vault

We have the SSL certificates stored in the /data/vault/ssl and we mount these as read only on /vault/ssl. With the VAULT_ADDR we specifiy on which url the vault service is available on, this is the url which Consul provides like any other server. With the VAULT_CACERT we specify on which location the CA Certificate file of our domain. The other 3 environment variables are needed for a High Available Vault environment and is to make sure how other vault instances can contact it.

When Vault is started, we will see something like this with the docker logs vault command:

==> Vault server configuration:

Backend: consul (HA available)
Cgo: disabled
Cluster Address: https://192.168.1.202:8200
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "enabled")
Log Level: info
Mlock: supported: true, enabled: true
Redirect Address: https://192.168.1.202:8200
Version: Vault v0.6.4
Version Sha: f4adc7fa960ed8e828f94bc6785bcdbae8d1b263

==> Vault server started! Log data will stream in below:

But where are not done yet. When Vault is started, it is in a sealed state and because this is the first vault in the cluster we have to initialise it to. Also when you check the ui of Consul, you’ll see that the vault is in an error state. Why? When Vault starts, it automatically creates a service in Consul and add health checks. These health checks will check if a vault instance is sealed or not.

Initialise

As vault is running in the container, we open a terminal to the container:

docker exec -it vault bash

Now we have a bash shell running and we going to initialise vault. First we have to make sure we set the ‘VAULT_ADDR’ to this container, by executing the following command:

export VAULT_ADDR='https://127.0.0.1:8200'

Every time we want to do something with the vault instance, we have to set the ‘VAULT_ADDR’ to localhost. If we won’t do that, we will send the commands directly against the cluster.

As this is the first vault instance in the environment, we have to initialise it and we do that by executing the following command:

vault init -tls-skip-verify
Unseal Key 1: hemsIyJD+KQSWtKp0fQ0r109fOv8TUBnugGUKVl5zjAB
Unseal Key 2: lIiIaKI1F6pJ11Jw/g1CiLyZurpfhCM9AYIylrG/SKUC
Unseal Key 3: 298bn4H8bLbJRsPASOl3R+RPuDKIt6i5fYzqxQ3wL4ED
Unseal Key 4: W4RUiOU3IzQSZ8GD2z8jBEg2wK/q17ldr3zJipFjzKQE
Unseal Key 5: FNPHf8b+WCiS9lAzbdsWyxDgwic95DLZ03IR2S0sq4AF
Initial Root Token: ed220674-24da-d446-375d-bbd0334bcb31

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.

As we set the ‘VAULT_ADDR’ to ‘https://127.0.0.1:8200’, we have to add the ‘-tls-skip-verify’ option to the vault command. If we don’t do that, it will complain the it can not validate the certificate that matches the configured url ‘vault.service.dj-wasabi.local.

After executing the command, we see some output appear. This output is very important and needs to be saved somewhere on a secure location. The output provides us 5 unseal keys and the root token. Every time a vault instance is (re)started, the instance will be in a sealed state and needs to be unsealed. 3 of the 5 tokens needs to be used when you need to unseal a vault instance.

bash-4.3$ vault unseal -tls-skip-verify
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
bash-4.3$ vault unseal -tls-skip-verify
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
bash-4.3$ vault unseal -tls-skip-verify
Key (will be hidden):
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

We have executed 3 times the unseal command and now this Vault instance is unsealed. You can see the ‘Unseal Progress’ changing after we enter an unseal key. We can verify that state of the vault instance by executing the vault status command:

bash-4.3$ vault status -tls-skip-verify
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Version: 0.6.4
Cluster Name: vault-cluster-7e01e371
Cluster ID: b9446acf-4551-e4c2-fa5f-03bd1bcf872f

High-Availability Enabled: true
Mode: active
Leader: https://192.168.1.202:8200

We see that this vault instance is not sealed and that the mode of this node is active. You can also see that the leader of the vault instance is in my case the current host. (Not strange as this is the first Vault instance of the environment.) If we want to add a 2nd and more, we have to execute the same commands as before. With the exception of the vault init command, as we already have an initialised environment.

As we are still logged in on the node, lets create a simple entry.

bash-4.3$ export VAULT_TOKEN=ed220674-24da-d446-375d-bbd0334bcb31
bash-4.3$ vault write secret/password value=secret
Success! Data written to: secret/password

We first set the ‘VAULT_TOKEN’ variable, this value of this variable is the value of the ‘Initial root token’. After that, we created a simple entry in the database. Key ‘secret/password’ is created and had the value ‘secret’.

It took some time to investigate how to setup a High Available Vault environment with Consul, not much information can be found on the internet. So maybe this page will help you setting one up yourself. If you do have improvements please let me know.

Configuring Access Control Lists in Consul

consul_logo

This is the 2nd post in securing Consul and this is about using ACLs in Consul. The first post (this one) we configured a Consul cluster by using gossip encryption and using SSL|TLS certificates. Now we cover the basics about Consul ACL’s (Access Control List) and configuring them in our cluster.

Master Token

First we have to create a master token. This is the token that has all rights (Thats why its called the master), sort of the ‘root’ token. We have to generate it first and we can use the uuidgen command in Linux (or Mac) for this. We use this output of the uuidgen command and place it in the following file: /data/consul/config/master-token.json

{
  "acl_master_token":"d9f1928e-1f84-407c-ab50-9579de563df5",
  "acl_datacenter":"dc1",
  "acl_default_policy":"deny",
  "acl_down_policy":"deny"
}

We have to store/configure this file on all Consul Servers. You’ll see that we set the default policy to “deny”, so we block everything and only enable the things we want. When we have created the file, we have to restart all Consul Servers to make the ACL’s active.

If you may recall what we did configure the Consul Server in the previous blogpost, we have configured the Consul Servers with this property:

"verify_incoming": true,

We have to open the ui on the Consul Server and because we have the property above configured, we need to load a SSL client certificate in our browser. (Or for now, you can also remove the property and restart Consul. But make sure you add it again when you are done!)

Now open the ui on the server and click on the right button (Settings). You’ll see something like this:

consul_settings

We enter the token we placed in the file in the field we see in our browser. Now we click on the button “ACL” (Token is saved automatically in your browser) and we see something like this:

consul_acl

This is an overview of all tokens available in Consul. You’ll see that 2 tokens exists in Consul right now:

  • Anonymous Token
  • Master Token

Anonymous Token

The anonymous token is used when you didn’t configure a Token in the settings page or didn’t supply it when using 3rd party software. You’ll only see the “consul” service, but won’t see anything else. If we would create a key in the key/value store, it will fail because the Anonymous token can’t do anything (Because of the property “acl_default_policy”:”deny”).

Master token

The master token is the token we just filled in the settings tab and the one configured in the json file in the beginning of this blogpost and is sort of the root token. The one token to rule them all.

So what do you need when you want to create an ACL? There are 3 types of policies that can be used:

  • read
  • write
  • deny

Might be obvious that the “read” policy is for reading data, “write” policy is for reading and writing data and “deny” is for NOT reading or writing data to Consul.

The ACL is written in the HCL language (HCL stands for HashiCorp Language) and we will create an ACL via the ui. You can also do that via the Consul API and automatically maintain them with for example Ansible, but that is out of the scope for this blogpost. In the ui we see on the right side of the page “New ACL”.

In the “name” field we enter for now “test” and select “client” as type. In the “Rules” field we enter the following:

key "" {
  policy = "read"
}
key "foo/" {
  policy = "write"
}

When we click on “create”, the ACL will be created. With this ACL, we choose the type “client” instead of the “management” type. When you have selected “management” as ACL type, the users/services which will use this ACL can also create/update/delete this and other ACL’s in the cluster. As we don’t want that, we select the “client” type.

We created 2 rules, both are for the key/value store. The first “key” rule specifies that all keys in the key/value store can be read with the ACL. With the 2nd “key” we specify that all keys in the “foo/” directory can be read and written. When we use this ACL, we can create the key “foo/bar”, but not the key “foobar”.

Next for using “key” in the rules, you can also configure “service”, “event” and “query” rules. It has the same format as the “key” example above and uses the same policies. With this you can easily give each application (or user) the correct rights.

Registrator

With registrator we can easily add docker containers as services into Consul. Now we have configured a default ACL policy to “deny” we have to update our configuration for the registrator. Registrator will attempt to sent the data to Consul for creating the services and registrator will think this is done, but Consul will deny because of the default policy. We can create a ACL specific to registrator.

Let’s create one via the UI. We enter the name “Registrator” and select “client” type. There are 2 possibilities to proceed regarding the “Rules”:

We can add a rule that will be used for all services the registry will add:

service "" {
  policy = "write"
}

Or we mention each service independently:

service "kibana" {
  policy = "write"
}
service "jenkins" {
  policy = "write"
}

Both have their pros and cons. With the first rule we allow that registrator can add all services into Consul and requires not much “maintenance”, it is a little bit to “open”. The 2nd rule requires more maintenance by adding all services but is more secure. With this, not all containers are added automatically and thus no rogue containers will be available in Consul.

We click on “create” to create the ACL. Now we have an token id and use that token in our docker run command. Our command to start registrator will look like this now:

docker run -h vserver-201 \
-v /var/run/docker.sock:/tmp/docker.sock \
-v /data/consul/config/ssl:/consul:ro \
-e CONSUL_CACERT=/consul/dj-wasabi.local.pem \
-e CONSUL_TLSCERT=/consul/vserver-201.node.dc1.dj-wasabi.local.crt \
-e CONSUL_TLSKEY=/consul/vserver-201.node.dc1.dj-wasabi.local.key \
-e CONSUL_HTTP_TOKEN=5c7d6559-cd90-d244-bbed-14d459a74bd2 \
gliderlabs/registrator:master \
-ip=192.168.1.201 consul-tls://vserver-201.node.dc1.dj-wasabi.local:8500

We had to add the -e CONSUL_HTTP_TOKEN variable with the token id as value. When I start the “kibana” container it will be added to Consul and we see the service is created.

We covered the basics for creating and using ACL’s in Consul. Using ACL’s in Consul will help securing Consul more by only allowing settings that is needed for the container purpose. Hopefully this will help you configuring ACLs in your environment.

Setting up a secure Consul cluster with docker

consul_logo

This post is the first of 2 blog items about setting up a secure Consul environment.

With the first post – which is this one – we will discuss how we setup a secure Consul environment. We will use a docker container and configure it with SSL certificates to secure the traffic from and to Consul. The 2nd post (This one), we will dive into ACLs and how we can make use of ACLs in Consul.

We will use the ‘wdijkerman/consul’ docker container to setup a secure environment. For now we create a Consul cluster with 2 hosts, named ‘vserver-201′ and ‘vserver-202′. ‘vserver-201′ will be the Consul Agent and ‘vserver-202′ will be the Consul Server. There is no specific need to use this container, you can also make this work with other Consul (containers) or installations.

Before we are going to setup the environment, we will briefly discuss the used docker container first.

wdijkerman/consul

This is a docker container created by myself which has Consul installed and configured. This container holds some basic Consul configuration and we can easily add some new configuration options by either supplying them to the command line or by creating a configuration json file. This container is running Consul 0.7.2 (Which is the latest version at moment of writing) and is running Alpine 3.5 (Also latest version at moment of writing). The most important thing is is that Consul isn’t running as user root, it is running as user ‘consul’ (with a fixed UID).

Before we start anything with the container, we going to add a user with that UID on the hosts running Consul.

useradd -u 995 consul

After this, we have to create 2 directories on the hosts running Consul. We use the following 2 directories:

mkdir -p /data/consul/data /data/consul/config
chown consul /data/consul/data /data/consul/config

The first directory is where Consul will store the Consul data and is only needed for the host running the Consul Server. The 2nd directory is where Consul will look for configuration files in which we create some files further in this post. On the host running the Consul Agent (In my case the host ‘vserver-201′) we only have to create the /data/consul/config directory. After the creation of the directories, we make sure these directories are owned by the earlier created user consul.

Before we are going to create some configuration files, take a look at the following json file. This json file is already present in the Consul docker container (So we don’t have to create it ourself) and is the default configuration of Consul:

{
  "data_dir": "/consul/data",
  "ui_dir": "/consul/ui",
  "log_level": "INFO",
  "client_addr": "0.0.0.0",
  "ports": {
    "dns": 53
  },
  "recursor": "8.8.8.8",
  "disable_update_check": true
}

As you see, this is a very basic configuration and we need to add some options to make it secure.

encrypt

We are going to expand our configuration by adding a new file in the /data/consul/config directory. With this file we are going to encrypt all of our internal Consul gossip traffic. This file should be placed on all of the hosts running Consul that will be/is part of this cluster.

Lets create a string with the following command:

docker run --rm --entrypoint consul wdijkerman/consul keygen

We use the output of this command and place it in the following file: /data/consul/config/encrypt.json

{
  "encrypt": "iuwMf/cScjTvKUKDC77kJA=="
}

We make sure that the rights of the file is set to 0400 and owned by the user consul.

chown consul:consul /data/consul/config/encrypt.json
chmod 0400 /data/consul/config/encrypt.json

All of the Consul nodes (Server and Agent) need this file, so make sure your Ansible (or Puppet, Chef or Saltstack) is configured to place this file on all of your nodes.

ssl

As all requests to and from Consul are done via http, we need to configure Consul that it listens on https instead of http. Before we do anything with Consul, we need access to a ssl crt, key and ca file first.

Before we execute a openssl command, we have to make sure that our CA SSL configuration is correct. Consul (Well, actually the go language: https://github.com/golang/go/issues/7423) requires some extra configuration specifically for using extentions in certificates. We have to add (or update) the property ‘extendedKeyUsage’ in the SSL CA configuration file so that the following values are added:

serverAuth,clientAuth

The usr_cert configuration in the CA openssl configuration file will look something like this:

[ usr_cert ]

basicConstraints=CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
extendedKeyUsage = critical,timeStamping,serverAuth,clientAuth

(I have no idea why critical and timeStamping are there, so I just keep them there. :-))

We have to create the certificates now, the FQDN for this is:

<name_of_node>.node.<datacenter>.<domain>

In my case, my nodes are ‘vserver-201′ and ‘vserver-201′ , my domain is ‘dj-wasabi.local’ and have the default ‘dc1′ as datacenter. I need to create a crt and key for the host ‘vserver-201.node.dc1.dj-wasabi.local’ and ‘vserver-202.node.dc1.dj-wasabi.local’.

So on the host where my ‘dj-wasabi.local’ CA is configured, I need to execute the following set of commands:

cd /etc/pki/CA
openssl genrsa -out private/vserver-202.node.dc1.dj-wasabi.local.key 4096

We first generate the SSL key.

openssl req -new -extensions usr_cert -sha256 -subj "/C=NL/ST=Utrecht/L=Nieuwegin/O=dj-wasabi/CN=vserver-202.node.dc1.dj-wasabi.local" -key private/vserver-202.node.dc1.dj-wasabi.local.key -out csr/vserver-202.node.dc1.dj-wasabi.local.csr

We generate the csr file from the earlier created key.

openssl ca -batch -config /etc/pki/tls/openssl.cnf -notext -in csr/vserver-202.node.dc1.dj-wasabi.local.csr -out certs/vserver-202.node.dc1.dj-wasabi.local.crt

And now we will create a crt by signing the csr via the OpenSSL CA.

(And I do the same for host vserver-201.node.dc1.dj-wasabi.local)

Now we have to copy these files (Including the CA certificate file) to the servers and make sure these files are stored in the /data/consul/config directory, owned and only available by user consul. I create a ssl directory and places all the ssl files in this directory.

Now we have to create a configuration file, so Consul knows that it has SSL certificates. First we configure the Consul Server, in my case it is running on the ‘vserver-202′ host. We create the file /data/consul/config/ssl.json with the following content:

{
  "ca_file": "/consul/config/ssl/dj-wasabi.local.pem",
  "cert_file": "/consul/config/ssl/vserver-202.node.dc1.dj-wasabi.local.crt",
  "key_file": "/consul/config/ssl/vserver-202.node.dc1.dj-wasabi.local.key",
  "verify_incoming": true,
  "verify_outgoing": true
}

(Keep in mind that /data/consul/config is mounted in the container as /consul/config).

With the ‘verify_incoming‘ and ‘verify_outgoing‘ we make sure that all traffic to and from the Server is encrypted. If we would start the container right now, you can only access the ui if you have have created client ssl certificates and loaded it in your browser.

For the Consul agent, we use the same ssl.conf configuration file as mentioned above, but without the ‘verify_incoming‘ option.

ports

Before we start the container, we have to do 1 small thing. With a default configuration which we currently have, port 8500 is used for http. We create a new configuration file and assign the http listener to a different port number, so we can configure port 8500 to be https.

We create the file: /data/consul/config/ports.json with the following content:

{
  "ports": {
    "http": 8501,
    "https": 8500
  }
}

We have to specifiy the http port and give this a port number, otherwise it will be set default to 8500. When we start the container with the next step, we only configure port 8500 to be opened and not port 8501 and thus we have a https enabled Consul container.

Start Consul

Now we are able to start the Consul Server on the Consul server ‘vserver-202‘. We execute the following command:

docker run -h vserver-202 --name consul \
-v /data/consul/cluster:/consul/data \
-v /data/consul/config:/consul/config \
-p 8300:8300 -p 8301:8301 -p 8301:8301/udp \
-p 8302:8302 -p 8302:8302/udp -p 8400:8400 \
-p 8500:8500 -p 8600:53/udp wdijkerman/consul \
-server -ui -ui-dir /consul/ui -bootstrap-expect=1 \
-advertise 192.168.1.202 -domain dj-wasabi.local \
-recursor=8.8.8.8 -recursor=8.8.4.4

The following output appears:

[root@vserver-202 config]# docker logs consul
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
Version: 'v0.7.2'
Node name: 'vserver-202'
Datacenter: 'dc1'
Server: true (bootstrap: false)
Client Addr: 0.0.0.0 (HTTP: 8501, HTTPS: 8500, DNS: 53, RPC: 8400)
Cluster Addr: 192.168.1.202 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true
Atlas: <disabled>

==> Log data will now stream in as it occurs:

Most important in this output are these 2 lines:

Client Addr: 0.0.0.0 (HTTP: 8501, HTTPS: 8500, DNS: 53, RPC: 8400)
Gossip encrypt: true, RPC-TLS: true, TLS-Incoming: true

First line we can see that port 8500 is used for HTTPS and port 8501 is used for HTTP.
2nd line we see that the parameter encrypt is active (Is set to true) and both the ‘verify_incoming’ and ‘verify_outgoing’ are also set to true.

Now we can start Consul on the ‘vserver-201′ (Consul Agent):

docker run -h vserver-201 --name consul \
-v /data/consul/config:/consul/config \
-p 8300:8300 -p 8301:8301 -p 8301:8301/udp \
-p 8302:8302 -p 8302:8302/udp -p 8400:8400 \
-p 8500:8500 -p 8600:53/udp wdijkerman/consul \
-join 192.168.1.202 -advertise 192.168.1.201 \
-domain dj-wasabi.local

The Consul Agent will connect to the Consul Server and we can open the ui on the Agent with url https://vserver-201.dc1.dj-wasabi.local:8500. In my case it complains that the certificate is not validated (I’m using a self-signed CA certficate), but I’m able to access the ui and see the service ‘consul’. I do have an issue with opening the ui on the Consul Server. Why?

We have added the following property in the file /data/consul/config/ssl.json

"verify_incoming": true,

This means that ALL traffic to the Consul Server should be done via SSL certificates. If we really want to access the ui on the Consul Server (And we do want that, ACL’s ;-)) we have to create a client SSL certificate, load it in the browser and try opening the ui again.

Registrator

I use registrator in my environment and have to make sure that it can work with SSL to. For registrator, we have to configure 3 environment variables which are used for the locations of the ssl crt, key and ca file. To do this, we also have to mount the ssl directory in the registrator container so it has access to theses files.

Next, we have to use the consul-tls:// option instead of the consul:// when starting registrator.
Our command looks like this now:

docker run -h vserver-201 \
-v /var/run/docker.sock:/tmp/docker.sock \
-v /data/consul/config/ssl:/consul:ro \
-e CONSUL_CACERT=/consul/dj-wasabi.local.pem \
-e CONSUL_TLSCERT=/consul/vserver-201.node.dc1.dj-wasabi.local.crt \
-e CONSUL_TLSKEY=/consul/vserver-201.node.dc1.dj-wasabi.local.key \
gliderlabs/registrator:master \
-ip=192.168.1.201 consul-tls://vserver-201.node.dc1.dj-wasabi.local:8500

After executing the above command, new docker containers will be added automatically in Consul as a service via tls.

We successfully created a secure Consul environment where all traffic from and to Consul are encrypted. Even with the registrator tool we add new services via TLS connections.

Next blog item we will discuss the ACLs in Consul to make sure that not everyone can create/update/delete keys in the k/v store and or create/add/delete services.