Signing Docker images with Notary server

This is the second of 2 blogs that we will do something with Docker and Security. In the first blogpost (Click), we will start Clair and use a tool called clair-scanner to scan Docker images that are on your host. In the 2nd blogpost (This one) we will start a Registry and Notary Server|Signer to sign Docker images. Notary allows use to sign images and we can configure the Docker daemon to only start containers from signed images.

For both blogposts, we will be using a sample configuration from the following Github repository:

This repository contains a docker-compose.yml file and some necessary files that are needed to run the applications. The docker-compose file contains the following applications:

  • Docker Registry v2
  • Notary (Server)
  • Notary (Signer)
  • Notary (DB)
  • Clair (DB)
  • Clair
  • Clair-scanner

The rest of the files are configuration files specific to these applications and I provided some self-signed certificates. These SSL certificates can only used for demo purposes.

Before we continue, lets do a clone of the Github repository and make sure you have Docker and docker-compose installed and running.

Why do we want to make use of a Notary server? Once you have Docker running, you are able to download all kinds of Docker images and run them. Some of them are the official ones, like Debian, CentOS and/or Hashicorp’s Consul, but you can also download and run Docker images from some one else. But you don’t know for sure what is installed and running in an image when you download one. With the previous blog-post we used Clair which can help to find if there are vulnerabilities in an Docker image, but you don’t know if an image is tampered with.

Notary will not fix this problem, it doesn’t scan the image to see if it is tampered with, but with Notary we will be able to sign our own Docker images. When a specific environment variable is set, we can only use these signed Docker images to run on our host(s). If we do want to download a Docker image from for example Docker hub, it will provide an error message.

We need to prepare some things before we can start the containers. First we need to make sure we add some entries in the hosts file, so we can resolve 2 FQDN’s which are used for the Registry Server and for the Notary Server.

Add following to hosts file:     notary-server.example.local registry-server.example.local

Once we have done that, we have to copy a configuration file and the ca-root certificate to a directory in our home-dir.

mkdir -p ~/.notary && cp files/config/config.json files/certs/ca-root.crt ~/.notary

Now we are done preparing and we can start de containers. We start the Registry server and the Notary server. When starting the Notary server, we will also automatically start the Notary signer and the database. So don’t be confused when you see extra containers running.

docker-compose up -d notary-server registry-server

You can verify if Notary server is correctly started by executing the following command:

openssl s_client -connect notary-server.example.local:4443 -CAfile files/certs/ca-root.crt -no_ssl3 -no_ssl2

This will return some information about the SSL certificate that is configured for the Notary server. Example:

$ openssl s_client -connect notary-server.example.local:4443 -CAfile files/certs/ca-root.crt -no_ssl3 -no_ssl2
depth=1 C = EU, ST = Example, L = Example, O = Example, OU = Example, CN = ca.example.local, emailAddress = root@ca.example.local
verify return:1
depth=0 C = EU, ST = Example, O = Example, CN = notary-server.example.local
verify return:1
Certificate chain
 0 s:/C=EU/ST=Example/O=Example/CN=notary-server.example.local

So lets pull an image and retag it so we can push it later on to our newly started Registry server. Lets make sure the image does have a tag like latest or 1.2.1.

docker pull wdijkerman/clair-scanner
docker tag wdijkerman/clair-scanner registry.example.local:5000/wdijkerman/clair-scanner:latest

But don’t push it yet, we first have to make sure that we have set some environment variables before doing so.

export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.example.local:4443

These 2 environment variables mean that we enable Docker Content Trust, so when we wants to do something with the image it will be checked with Notary server which is available on the provided URL.

The Registry server is configured with basic authentication, so we have to login first:

docker login registry.example.local:5000

Username: admin
Password: password

Now we are ready and we can now push our newly tagged image to the Docker Registry:

$ docker push registry.example.local:5000/wdijkerman/clair-scanner:latest
The push refers to repository [registry.example.local:5000/wdijkerman/clair-scanner]
4737f34f33f3: Pushed
5ff3301a32f4: Pushed
7bff100f35cb: Pushed
latest: digest: sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 size: 946
Signing and pushing trust metadata
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 98a3a53:
Repeat passphrase for new root key with ID 98a3a53:
Enter passphrase for new repository key with ID 3dd4fb6:
Repeat passphrase for new repository key with ID 3dd4fb6:
Finished initializing "registry.example.local:5000/wdijkerman/clair-scanner"
Successfully signed registry.example.local:5000/wdijkerman/clair-scanner:latest

Because this is the first time we have pushed an image, it asks us to enter a passphrase for the root key and for the repository. Generate a passphrase and enter these with the push command.

Ok, so now the Docker image is pushed in our Registry server and it is signed by the Notary server. We can verify this by executing the next command:

 $ notary -s ${DOCKER_CONTENT_TRUST_SERVER} -d ~/.docker/trust list registry.example.local:5000/wdijkerman/clair-scanner
NAME      DIGEST                                                              SIZE (BYTES)    ROLE
----      ------                                                              ------------    ----
latest    2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32    946             targets

Here you see that we have a single image pushed to the Notary server. The value in the DIGEST is the same as the Docker image ID.

Before we continue, in the home directory we have a hidden “.docker” directory. In one of the sub directories the keys that where generated with the first push a stored here. These are important, so make sure to backup these files. There is also a possibility to set some environment variables with the passphrase so you won’t have to backup these files, but couldn’t find them yet.

$ ls -l ~/.docker/trust/private/
total 16
-rw-------  1 wdijkerman  staff  477 Feb 23 20:00 3dd4fb64fbd1524884b02fefde0771d0708082c70201511f15580b42244f37cf.key
-rw-------  1 wdijkerman  staff  416 Feb 23 20:00 98a3a53715f98652478ab6cf0c58f56a720956cc405292a72fb7a97fb0fb4618.key

So lets remove the 2 images from the host so we can pull them later again.

docker image rm registry.example.local:5000/wdijkerman/clair-scanner:latest
docker image rm wdijkerman/clair-scanner

And now we will do 2 pulls, 1 from our Registry server to verify that it just works. After this, we download an image from Docker hub.

$ docker pull registry.example.local:5000/wdijkerman/clair-scanner:latest
Pull (1 of 1): registry.example.local:5000/wdijkerman/clair-scanner:latest@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32
sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32: Pulling from wdijkerman/clair-scanner
cd784148e348: Pull complete
8297cc41e539: Pull complete
ef2f20c2497d: Pull complete
Digest: sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32
Status: Downloaded newer image for registry.example.local:5000/wdijkerman/clair-scanner@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32
Tagging registry.example.local:5000/wdijkerman/clair-scanner@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 as registry.example.local:5000/wdijkerman/clair-scanner:latest

And now we download an image from the Docker hub:

$ docker pull wdijkerman/clair-scanner:latest
Error: error contacting notary server: x509: certificate signed by unknown authority

Where you just as me happy that it failed? 🙂

As you see, the pull failed from the Docker hub, as the image is not registered with Notary server so there is now no chance of running an container from any other source than our own Registry server. Mission accomplished.


Docker Content Trust

Docker Notary Server

Docker Registry Server


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s