This is the second of 2 blogs that we will do something with Docker and Security. In the first blogpost (Click), we will start Clair and use a tool called clair-scanner to scan Docker images that are on your host. In the 2nd blogpost (This one) we will start a Registry and Notary Server|Signer to sign Docker images. Notary allows use to sign images and we can configure the Docker daemon to only start containers from signed images.
For both blogposts, we will be using a sample configuration from the following Github repository: https://github.com/dj-wasabi/clair-and-docker-notary-example
This repository contains a docker-compose.yml file and some necessary files that are needed to run the applications. The docker-compose file contains the following applications:
- Docker Registry v2
- Notary (Server)
- Notary (Signer)
- Notary (DB)
- Clair (DB)
- Clair
- Clair-scanner
The rest of the files are configuration files specific to these applications and I provided some self-signed certificates. These SSL certificates can only used for demo purposes.
Before we continue, lets do a clone of the Github repository and make sure you have Docker and docker-compose installed and running.
Why do we want to make use of a Notary server? Once you have Docker running, you are able to download all kinds of Docker images and run them. Some of them are the official ones, like Debian, CentOS and/or Hashicorp’s Consul, but you can also download and run Docker images from some one else. But you don’t know for sure what is installed and running in an image when you download one. With the previous blog-post we used Clair which can help to find if there are vulnerabilities in an Docker image, but you don’t know if an image is tampered with.
Notary will not fix this problem, it doesn’t scan the image to see if it is tampered with, but with Notary we will be able to sign our own Docker images. When a specific environment variable is set, we can only use these signed Docker images to run on our host(s). If we do want to download a Docker image from for example Docker hub, it will provide an error message.
We need to prepare some things before we can start the containers. First we need to make sure we add some entries in the hosts file, so we can resolve 2 FQDN’s which are used for the Registry Server and for the Notary Server.
Add following to hosts file:
127.0.0.1 notary-server.example.local registry-server.example.local
Once we have done that, we have to copy a configuration file and the ca-root certificate to a directory in our home-dir.
mkdir -p ~/.notary && cp files/config/config.json files/certs/ca-root.crt ~/.notary
Now we are done preparing and we can start de containers. We start the Registry server and the Notary server. When starting the Notary server, we will also automatically start the Notary signer and the database. So don’t be confused when you see extra containers running.
docker-compose up -d notary-server registry-server
You can verify if Notary server is correctly started by executing the following command:
openssl s_client -connect notary-server.example.local:4443 -CAfile files/certs/ca-root.crt -no_ssl3 -no_ssl2
This will return some information about the SSL certificate that is configured for the Notary server. Example:
$ openssl s_client -connect notary-server.example.local:4443 -CAfile files/certs/ca-root.crt -no_ssl3 -no_ssl2 CONNECTED(00000005) depth=1 C = EU, ST = Example, L = Example, O = Example, OU = Example, CN = ca.example.local, emailAddress = root@ca.example.local verify return:1 depth=0 C = EU, ST = Example, O = Example, CN = notary-server.example.local verify return:1 --- Certificate chain 0 s:/C=EU/ST=Example/O=Example/CN=notary-server.example.local i:/C=EU/ST=Example/L=Example/O=Example/OU=Example/CN=ca.example.local/emailAddress=root@ca.example.local
So lets pull an image and retag it so we can push it later on to our newly started Registry server. Lets make sure the image does have a tag like latest or 1.2.1.
docker pull wdijkerman/clair-scanner docker tag wdijkerman/clair-scanner registry.example.local:5000/wdijkerman/clair-scanner:latest
But don’t push it yet, we first have to make sure that we have set some environment variables before doing so.
export DOCKER_CONTENT_TRUST_SERVER=https://notary-server.example.local:4443 export DOCKER_CONTENT_TRUST=1
These 2 environment variables mean that we enable Docker Content Trust, so when we wants to do something with the image it will be checked with Notary server which is available on the provided URL.
The Registry server is configured with basic authentication, so we have to login first:
docker login registry.example.local:5000
Username: admin
Password: password
Now we are ready and we can now push our newly tagged image to the Docker Registry:
$ docker push registry.example.local:5000/wdijkerman/clair-scanner:latest The push refers to repository [registry.example.local:5000/wdijkerman/clair-scanner] 4737f34f33f3: Pushed 5ff3301a32f4: Pushed 7bff100f35cb: Pushed latest: digest: sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 size: 946 Signing and pushing trust metadata You are about to create a new root signing key passphrase. This passphrase will be used to protect the most sensitive key in your signing system. Please choose a long, complex passphrase and be careful to keep the password and the key file itself secure and backed up. It is highly recommended that you use a password manager to generate the passphrase and keep it safe. There will be no way to recover this key. You can find the key in your config directory. Enter passphrase for new root key with ID 98a3a53: Repeat passphrase for new root key with ID 98a3a53: Enter passphrase for new repository key with ID 3dd4fb6: Repeat passphrase for new repository key with ID 3dd4fb6: Finished initializing "registry.example.local:5000/wdijkerman/clair-scanner" Successfully signed registry.example.local:5000/wdijkerman/clair-scanner:latest
Because this is the first time we have pushed an image, it asks us to enter a passphrase for the root key and for the repository. Generate a passphrase and enter these with the push command.
Ok, so now the Docker image is pushed in our Registry server and it is signed by the Notary server. We can verify this by executing the next command:
$ notary -s ${DOCKER_CONTENT_TRUST_SERVER} -d ~/.docker/trust list registry.example.local:5000/wdijkerman/clair-scanner
NAME DIGEST SIZE (BYTES) ROLE
---- ------ ------------ ----
latest 2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 946 targets
Here you see that we have a single image pushed to the Notary server. The value in the DIGEST is the same as the Docker image ID.
Before we continue, in the home directory we have a hidden “.docker” directory. In one of the sub directories the keys that where generated with the first push a stored here. These are important, so make sure to backup these files. There is also a possibility to set some environment variables with the passphrase so you won’t have to backup these files, but couldn’t find them yet.
$ ls -l ~/.docker/trust/private/ total 16 -rw------- 1 wdijkerman staff 477 Feb 23 20:00 3dd4fb64fbd1524884b02fefde0771d0708082c70201511f15580b42244f37cf.key -rw------- 1 wdijkerman staff 416 Feb 23 20:00 98a3a53715f98652478ab6cf0c58f56a720956cc405292a72fb7a97fb0fb4618.key
So lets remove the 2 images from the host so we can pull them later again.
docker image rm registry.example.local:5000/wdijkerman/clair-scanner:latest docker image rm wdijkerman/clair-scanner
And now we will do 2 pulls, 1 from our Registry server to verify that it just works. After this, we download an image from Docker hub.
$ docker pull registry.example.local:5000/wdijkerman/clair-scanner:latest Pull (1 of 1): registry.example.local:5000/wdijkerman/clair-scanner:latest@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32: Pulling from wdijkerman/clair-scanner cd784148e348: Pull complete 8297cc41e539: Pull complete ef2f20c2497d: Pull complete Digest: sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 Status: Downloaded newer image for registry.example.local:5000/wdijkerman/clair-scanner@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 Tagging registry.example.local:5000/wdijkerman/clair-scanner@sha256:2f876d115399b206181e8f185767f9d86a982780785f13eb62f982c958151a32 as registry.example.local:5000/wdijkerman/clair-scanner:latest
And now we download an image from the Docker hub:
$ docker pull wdijkerman/clair-scanner:latest Error: error contacting notary server: x509: certificate signed by unknown authority
Where you just as me happy that it failed? 🙂
As you see, the pull failed from the Docker hub, as the image is not registered with Notary server so there is now no chance of running an container from any other source than our own Registry server. Mission accomplished.
Links
werner-dijkerman.nl 