This is the first of 2 blogs that we will do something with Docker and Security. In the first blogpost (This one), we will start Clair and use a tool called clair-scanner to scan Docker images that are on your host. In the 2nd blogpost we will start a Registry and Notary Server|Signer to sign Docker images. Notary allows use to sign images and we can configure the Docker daemon to only start containers from signed images.

For both blogposts, we will be using a sample configuration from the following Github repository: https://github.com/dj-wasabi/clair-and-docker-notary-example

This repository contains a docker-compose.yml file and some necessary files that are needed to run the applications. The docker-compose file contains the following applications:

• Docker Registry v2
• Notary (Server)
• Notary (Signer)
• Notary (DB)
• Clair (DB)
• Clair
• Clair-scanner

The rest of the files are configuration files specific to these applications and I provided some self-signed certificates. These SSL certificates can only used for demo purposes.

Before we continue, lets do a clone of the Github repository and make sure you have Docker and docker-compose installed and running.

Clair

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker). Clair will analyze a layer to see if it finds any vulnerabilities. If vulnerabilities are found, Clair will provide information about the vulnerability. To let Clair scan these layers, we use a tool called “clair-scanner“. clair-scanner will get all layers from an Docker image on your host and provide these to Clair by uploading them 1-by-1. Once all layers have been scanned, the clair-scanner will provide the vulnerabilities (if there are any).

Lets start Clair by executing the following command:

docker-compose up -d clair

It will start a PostgreSQL container and the Clair container itself. Once Clair is started, it will fetch the vulnerabilities for the various operating systems that is configured in the file: files/config/clair-config.yaml in earlier mentioned repository. This might take a while (In my case it was 15 minutes).

The following is configured in earlier mentioned configuration file:

  updater:
    interval: 1m
    enabledupdaters:
      - debian
      - ubuntu
      - rhel
      - oracle
      - alpine
      - suse

As you see, Clair will download vulnerabilities information from the above mentioned operating systems.

Occasionally check the logfile of clair (docker logs -f clair) and see if you find the following log messages appear:

{"Event":"could not get NVD data feed hash","Level":"warning","Location":"nvd.go:137","Time":"2019-01-26 20:19:59.682956","data feed name":"2018","error":"invalid .meta file format"}
{"Event":"could not get NVD data feed hash","Level":"warning","Location":"nvd.go:137","Time":"2019-01-26 20:19:59.682956","data feed name":"2019","error":"invalid .meta file format"}

You’ll see them from year 2002 to 2019. Once these messages are logged, we are able to continue with scanning (a) Docker image(s).

Some basic Clair information

Some information about Clair while we are waiting.

When you want to scan an image, Clair expects to analyze each layer of a Docker image. This kind data needs to be POST’ed to the following endpoint: http://localhost:6060/v1/layers

Example of a POST data request (Can also be found on: https://coreos.com/clair/docs/latest/api_v1.html#layers)

{
  "Layer": {
    "Name": "523ef1d23f222195488575f52a39c729c76a8c5630c9a194139cb246fb212da6",
    "Path": "https://mystorage.com/layers/523ef1d23f222195488575f52a39c729c76a8c5630c9a194139cb246fb212da6/layer.tar",
    "Headers": {
      "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.EkN-DOsnsuRjRO6BxXemmJDm3HbxrbRzXglbN2S4sOkopdU4IsDxTI8jO19W_A4K8ZPJijNLis4EZsHeY559a4DFOd50_OqgHGuERTqYZyuhtF39yxJPAjUESwxk2J5k_4zM3O-vtd1Ghyo4IbqKKSy6J9mTniYJPenn5-HIirE"
    },
    "ParentName": "140f9bdfeb9784cf8730e9dab5dd12fbd704151cf555ac8cae650451794e5ac2",
    "Format": "Docker"
  }
}

The most important keys are the Name, Path and the ParentName.

When an image has 3 layers (Layer A, is the base image lets say debian:latest, B is installing a package and C is adding a file), Clair expects first a POST request to the /v1/layers endpoint with the layer information of layer A. The Name should have the SHA256 value of layer “A”, and the Path should contain an URL on which the layer can be downloaded. Clair will download the layer and run the actual analysis.

When layer A is analysed, layer B should be uploaded to the endpoint. But now the ParentName should contain the layer SHA256 value of layer A. When layer C is analysed, the ParentName should contain the SHA256 value of Layer B and etc.

Yes, you read that correctly. Clair will download the layer from for example a Docker Registry. This means that an image should for example already been pushed to a Docker Registry. But this is a bit to late, as you should only push an image to a Docker Registry if it doesn’t contain any vulnerabilities. Here comes the clair-scanner tool into play. This tool will basically start a web server where Clair can download the layers from when analysing.

Once Clair is completely started we can continue. Lets download an Docker image:

docker pull wdijkerman/consul

(You can of course also download an other Docker image)

This is a very basic Alpine image running Consul and Python. So lets check that Docker image.

 $ docker-compose run --rm clair-scanner wdijkerman/consul
2019/01/26 19:43:52 [INFO]  Start clair-scanner
2019/01/26 19:43:55 [INFO]  Server listening on port 9279
2019/01/26 19:43:55 [INFO]  Analyzing 5491dce778832e33c284cd8185100e76d6daa18f8cbc32458c706776894127fc
2019/01/26 19:43:55 [INFO]  Analyzing 28a9cc8dcad2060c54ae345db266ad00e4d84b1f7526e5186f93844eb3bb426e
2019/01/26 19:43:56 [INFO]  Analyzing 746b97d6fd172bacbe51699e383b5a47ceb3d779c3580b9dd35dfb7bd4a72a83
2019/01/26 19:43:56 [INFO]  Analyzing 1e04d30b4435c531eafe3d3b17155f3f3f4a9b4874ca1f1d3115ad273db43d1e
2019/01/26 19:43:56 [INFO]  Analyzing c3453aa5ff961a1d1710c2f110a788d796a5456241c664489e65fd269f0e1687
2019/01/26 19:43:56 [INFO]  Image [wdijkerman/consul] contains NO unapproved vulnerabilities

It shows us that there are 5 layers in this image and no vulnerabilities where detected! (This was during time of writing this blogpost, it can always be the case when new vulnerabilities are found!)

This blogpost is not really successful if we only show things that are ok, so lets check an image that contains 1 or more vulnerabilities.

Lets check the postgres:latest image (It is part of the Clair installation if you where wondering why you have downloaded the image). So lets check that one.

 $ docker-compose run --rm clair-scanner postgres:latest
2019/01/26 19:25:46 [INFO]  Start clair-scanner
2019/01/26 19:25:54 [INFO]  Server listening on port 9279
2019/01/26 19:25:54 [INFO]  Analyzing 08bf86d6624450c487db18071224c88003d970848fb8c5b2b07df27e3f6869b2
2019/01/26 19:25:54 [INFO]  Analyzing f419c5f6b63090e31755da12d65829dfd90ac42b90c70a725fb5dc7856395fc7
2019/01/26 19:25:54 [INFO]  Analyzing 906fb3014e147615f2219607d99604bdc53d0a6cdb0f4886ebf99548df918073
2019/01/26 19:25:54 [INFO]  Analyzing 1439e9b10c58144ac2acb85fa9aab36127201d1b2550b45216a341fa32957d17
2019/01/26 19:25:54 [INFO]  Analyzing 75d637800b713ea9c0bcd3a19eed8c144598ef8477da147a50d10cd6e85d2919
2019/01/26 19:25:54 [INFO]  Analyzing d91867cee1db8a866d638ae1d66c8078abfd236cda83c7ba72a5d214c5c8c4a3
2019/01/26 19:25:54 [INFO]  Analyzing b7d5ec0a0cb0939be115288b61e074a17359f8eb283e0deab13d30b4c0a060e8
2019/01/26 19:25:54 [INFO]  Analyzing 130e7676deae310571cbc46260a81adfc9f0de8a8684bbc33077b12c388594b7
2019/01/26 19:25:54 [INFO]  Analyzing 2bc990d0b93546a555b6abd28c365a1383f58fb64fd36142c5a9a0cbd26131e2
2019/01/26 19:25:54 [INFO]  Analyzing f4dfa6837911fd604bedc0d96126c12b2209a87421a7a0f56b0781d507b0aca8
2019/01/26 19:25:54 [INFO]  Analyzing a9598ee0b475f1cafbe8f63d6c7243ca37da704b9496e2d08c164238e8d0be3c
2019/01/26 19:25:54 [INFO]  Analyzing 1622bb5b03dc3ce4a5af7f2f89c443ce749b54be0703b92d7822f8789cf79281
2019/01/26 19:25:54 [INFO]  Analyzing fb4daa3b039b8e9889bc9f3c675c4811c85d34746c8862c673fe3da1998ae08b
2019/01/26 19:25:54 [INFO]  Analyzing 0b6857f87b6965b43ed41cc7a54591b3697b6049603cbd1b760030045915e3de
2019/01/26 19:25:54 [WARN]  Image [postgres:latest] contains 86 total vulnerabilities
2019/01/26 19:25:54 [ERRO]  Image [postgres:latest] contains 86 unapproved vulnerabilities
+------------+-----------------------------+--------------+------------------------+--------------------------------------------------------------+
| STATUS     | CVE SEVERITY                | PACKAGE NAME | PACKAGE VERSION        | CVE DESCRIPTION                                              |
+------------+-----------------------------+--------------+------------------------+--------------------------------------------------------------+
| Unapproved | High CVE-2017-16997         | glibc        | 2.24-11+deb9u3         | elf/dl-load.c in the GNU C Library (aka glibc or libc6)      |
|            |                             |              |                        | 2.19 through 2.26 mishandles RPATH and RUNPATH containing    |
|            |                             |              |                        | $ORIGIN for a privileged (setuid or AT_SECURE) program,      |
|            |                             |              |                        | which allows local users to gain privileges via a Trojan     |
|            |                             |              |                        | horse library in the current working directory, related      |
|            |                             |              |                        | to the fillin_rpath and decompose_rpath functions.           |
|            |                             |              |                        | This is associated with misinterpretion of an empty          |
|            |                             |              |                        | RPATH/RUNPATH token as the "./" directory. NOTE: this        |
|            |                             |              |                        | configuration of RPATH/RUNPATH for a privileged program      |
|            |                             |              |                        | is apparently very uncommon; most likely, no such            |
|            |                             |              |                        | program is shipped with any common Linux distribution.       |
|            |                             |              |                        | https://security-tracker.debian.org/tracker/CVE-2017-16997   |
+------------+-----------------------------+--------------+------------------------+--------------------------------------------------------------+
| Unapproved | High CVE-2017-12424         | shadow       | 1:4.4-4.1              | In shadow before 4.5, the newusers tool could be             |
|            |                             |              |                        | made to manipulate internal data structures in ways          |
|            |                             |              |                        | unintended by the authors. Malformed input may lead          |
|            |                             |              |                        | to crashes (with a buffer overflow or other memory           |
|            |                             |              |                        | corruption) or other unspecified behaviors. This             |
|            |                             |              |                        | crosses a privilege boundary in, for example, certain        |
|            |                             |              |                        | web-hosting environments in which a Control Panel allows     |
|            |                             |              |                        | an unprivileged user account to create subaccounts.          |
|            |                             |              |                        | https://security-tracker.debian.org/tracker/CVE-2017-12424   |
+------------+-----------------------------+--------------+------------------------+--------------------------------------------------------------+

Oops. 86 vulnerabilities! Well, this Docker image might not be really safe to use but in the end, that is all up 2 you.

As you might see in the postgres example, there is a column “STATUS” in the output and all of them are “Unapproved“. Why is that? clair-scanner allows you to whitelist specific vulnerabilities when scanning images.

The clair-scanner tool has an exit code of 0 when no vulnerabilities are found and has an exit code of !=0 when vulnerabilities are found. So if you would run the clair-scanner as part of your CI pipeline, this would fail. However, there could be a reason to whitelist a vulnerability and the clair-scanner will not provide an exit code of !=0 when whitelisted vulnerabilities are found.

Example of a whitelist file.

generalwhitelist: #Approve CVE for any image
  CVE-2017-6055: XML
  CVE-2017-5586: OpenText
images:
  ubuntu: #Apprive CVE only for ubuntu image, regardles of the version
    CVE-2017-5230: Java
    CVE-2017-5230: XSX
  alpine:
    CVE-2017-3261: SE

So we have 2 CVE vulnerabilities that we whitelist, no matter what base Docker OS image is used. For Ubuntu we whitelist 2 CVE’s and 1 for Alpine. I’m not sure, but I would say the XML, OpenText is just a basic description for what package the CVE belongs to.

Summary

So with this blogpost we where able to start Clair and do some Docker image analysing with the tool clair-scanner. It showed us that the postgresql image contains some vulnerabilities. So now you can update your CI pipeline by adding a check to scan for vulnerabilities, before pushing the image to a Docker Registry. Next blogpost, we will start a secure Docker Registry and we will sign Docker images with the Notary Server and Signer tool.

Links

Clair
Clair-scanner