Scanning Docker images with CoreOS Clair

This is the first of 2 blogs that we will do something with Docker and Security. In the first blogpost (This one), we will start Clair and use a tool called clair-scanner to scan Docker images that are on your host. In the 2nd blogpost we will start a Registry and Notary Server|Signer to sign Docker images. Notary allows use to sign images and we can configure the Docker daemon to only start containers from signed images.

For both blogposts, we will be using a sample configuration from the following Github repository:

This repository contains a docker-compose.yml file and some necessary files that are needed to run the applications. The docker-compose file contains the following applications:

  • Docker Registry v2
  • Notary (Server)
  • Notary (Signer)
  • Notary (DB)
  • Clair (DB)
  • Clair
  • Clair-scanner

The rest of the files are configuration files specific to these applications and I provided some self-signed certificates. These SSL certificates can only used for demo purposes.

Before we continue, lets do a clone of the Github repository and make sure you have Docker and docker-compose installed and running.


Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker). Clair will analyze a layer to see if it finds any vulnerabilities. If vulnerabilities are found, Clair will provide information about the vulnerability. To let Clair scan these layers, we use a tool called “clair-scanner“. clair-scanner will get all layers from an Docker image on your host and provide these to Clair by uploading them 1-by-1. Once all layers have been scanned, the clair-scanner will provide the vulnerabilities (if there are any).

Lets start Clair by executing the following command:

docker-compose up -d clair

It will start a PostgreSQL container and the Clair container itself. Once Clair is started, it will fetch the vulnerabilities for the various operating systems that is configured in the file: files/config/clair-config.yaml in earlier mentioned repository. This might take a while (In my case it was 15 minutes).

The following is configured in earlier mentioned configuration file:

    interval: 1m
      - debian
      - ubuntu
      - rhel
      - oracle
      - alpine
      - suse

As you see, Clair will download vulnerabilities information from the above mentioned operating systems.

Occasionally check the logfile of clair (docker logs -f clair) and see if you find the following log messages appear:

{"Event":"could not get NVD data feed hash","Level":"warning","Location":"nvd.go:137","Time":"2019-01-26 20:19:59.682956","data feed name":"2018","error":"invalid .meta file format"}
{"Event":"could not get NVD data feed hash","Level":"warning","Location":"nvd.go:137","Time":"2019-01-26 20:19:59.682956","data feed name":"2019","error":"invalid .meta file format"}

You’ll see them from year 2002 to 2019. Once these messages are logged, we are able to continue with scanning (a) Docker image(s).

Some basic Clair information

Some information about Clair while we are waiting.

When you want to scan an image, Clair expects to analyze each layer of a Docker image. This kind data needs to be POST’ed to the following endpoint: http://localhost:6060/v1/layers

Example of a POST data request (Can also be found on:

  "Layer": {
    "Name": "523ef1d23f222195488575f52a39c729c76a8c5630c9a194139cb246fb212da6",
    "Path": "",
    "Headers": {
      "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.EkN-DOsnsuRjRO6BxXemmJDm3HbxrbRzXglbN2S4sOkopdU4IsDxTI8jO19W_A4K8ZPJijNLis4EZsHeY559a4DFOd50_OqgHGuERTqYZyuhtF39yxJPAjUESwxk2J5k_4zM3O-vtd1Ghyo4IbqKKSy6J9mTniYJPenn5-HIirE"
    "ParentName": "140f9bdfeb9784cf8730e9dab5dd12fbd704151cf555ac8cae650451794e5ac2",
    "Format": "Docker"

The most important keys are the Name, Path and the ParentName.

When an image has 3 layers (Layer A, is the base image lets say debian:latest, B is installing a package and C is adding a file), Clair expects first a POST request to the /v1/layers endpoint with the layer information of layer A. The Name should have the SHA256 value of layer “A”, and the Path should contain an URL on which the layer can be downloaded. Clair will download the layer and run the actual analysis.

When layer A is analysed, layer B should be uploaded to the endpoint. But now the ParentName should contain the layer SHA256 value of layer A. When layer C is analysed, the ParentName should contain the SHA256 value of Layer B and etc.

Yes, you read that correctly. Clair will download the layer from for example a Docker Registry. This means that an image should for example already been pushed to a Docker Registry. But this is a bit to late, as you should only push an image to a Docker Registry if it doesn’t contain any vulnerabilities. Here comes the clair-scanner tool into play. This tool will basically start a web server where Clair can download the layers from when analysing.

Once Clair is completely started we can continue. Lets download an Docker image:

docker pull wdijkerman/consul

(You can of course also download an other Docker image)

This is a very basic Alpine image running Consul and Python. So lets check that Docker image.

 $ docker-compose run --rm clair-scanner wdijkerman/consul
2019/01/26 19:43:52 [INFO]  Start clair-scanner
2019/01/26 19:43:55 [INFO]  Server listening on port 9279
2019/01/26 19:43:55 [INFO]  Analyzing 5491dce778832e33c284cd8185100e76d6daa18f8cbc32458c706776894127fc
2019/01/26 19:43:55 [INFO]  Analyzing 28a9cc8dcad2060c54ae345db266ad00e4d84b1f7526e5186f93844eb3bb426e
2019/01/26 19:43:56 [INFO]  Analyzing 746b97d6fd172bacbe51699e383b5a47ceb3d779c3580b9dd35dfb7bd4a72a83
2019/01/26 19:43:56 [INFO]  Analyzing 1e04d30b4435c531eafe3d3b17155f3f3f4a9b4874ca1f1d3115ad273db43d1e
2019/01/26 19:43:56 [INFO]  Analyzing c3453aa5ff961a1d1710c2f110a788d796a5456241c664489e65fd269f0e1687
2019/01/26 19:43:56 [INFO]  Image [wdijkerman/consul] contains NO unapproved vulnerabilities

It shows us that there are 5 layers in this image and no vulnerabilities where detected! (This was during time of writing this blogpost, it can always be the case when new vulnerabilities are found!)

This blogpost is not really successful if we only show things that are ok, so lets check an image that contains 1 or more vulnerabilities.

Lets check the postgres:latest image (It is part of the Clair installation if you where wondering why you have downloaded the image). So lets check that one.

 $ docker-compose run --rm clair-scanner postgres:latest
2019/01/26 19:25:46 [INFO]  Start clair-scanner
2019/01/26 19:25:54 [INFO]  Server listening on port 9279
2019/01/26 19:25:54 [INFO]  Analyzing 08bf86d6624450c487db18071224c88003d970848fb8c5b2b07df27e3f6869b2
2019/01/26 19:25:54 [INFO]  Analyzing f419c5f6b63090e31755da12d65829dfd90ac42b90c70a725fb5dc7856395fc7
2019/01/26 19:25:54 [INFO]  Analyzing 906fb3014e147615f2219607d99604bdc53d0a6cdb0f4886ebf99548df918073
2019/01/26 19:25:54 [INFO]  Analyzing 1439e9b10c58144ac2acb85fa9aab36127201d1b2550b45216a341fa32957d17
2019/01/26 19:25:54 [INFO]  Analyzing 75d637800b713ea9c0bcd3a19eed8c144598ef8477da147a50d10cd6e85d2919
2019/01/26 19:25:54 [INFO]  Analyzing d91867cee1db8a866d638ae1d66c8078abfd236cda83c7ba72a5d214c5c8c4a3
2019/01/26 19:25:54 [INFO]  Analyzing b7d5ec0a0cb0939be115288b61e074a17359f8eb283e0deab13d30b4c0a060e8
2019/01/26 19:25:54 [INFO]  Analyzing 130e7676deae310571cbc46260a81adfc9f0de8a8684bbc33077b12c388594b7
2019/01/26 19:25:54 [INFO]  Analyzing 2bc990d0b93546a555b6abd28c365a1383f58fb64fd36142c5a9a0cbd26131e2
2019/01/26 19:25:54 [INFO]  Analyzing f4dfa6837911fd604bedc0d96126c12b2209a87421a7a0f56b0781d507b0aca8
2019/01/26 19:25:54 [INFO]  Analyzing a9598ee0b475f1cafbe8f63d6c7243ca37da704b9496e2d08c164238e8d0be3c
2019/01/26 19:25:54 [INFO]  Analyzing 1622bb5b03dc3ce4a5af7f2f89c443ce749b54be0703b92d7822f8789cf79281
2019/01/26 19:25:54 [INFO]  Analyzing fb4daa3b039b8e9889bc9f3c675c4811c85d34746c8862c673fe3da1998ae08b
2019/01/26 19:25:54 [INFO]  Analyzing 0b6857f87b6965b43ed41cc7a54591b3697b6049603cbd1b760030045915e3de
2019/01/26 19:25:54 [WARN]  Image [postgres:latest] contains 86 total vulnerabilities
2019/01/26 19:25:54 [ERRO]  Image [postgres:latest] contains 86 unapproved vulnerabilities
| STATUS     | CVE SEVERITY                | PACKAGE NAME | PACKAGE VERSION        | CVE DESCRIPTION                                              |
| Unapproved | High CVE-2017-16997         | glibc        | 2.24-11+deb9u3         | elf/dl-load.c in the GNU C Library (aka glibc or libc6)      |
|            |                             |              |                        | 2.19 through 2.26 mishandles RPATH and RUNPATH containing    |
|            |                             |              |                        | $ORIGIN for a privileged (setuid or AT_SECURE) program,      |
|            |                             |              |                        | which allows local users to gain privileges via a Trojan     |
|            |                             |              |                        | horse library in the current working directory, related      |
|            |                             |              |                        | to the fillin_rpath and decompose_rpath functions.           |
|            |                             |              |                        | This is associated with misinterpretion of an empty          |
|            |                             |              |                        | RPATH/RUNPATH token as the "./" directory. NOTE: this        |
|            |                             |              |                        | configuration of RPATH/RUNPATH for a privileged program      |
|            |                             |              |                        | is apparently very uncommon; most likely, no such            |
|            |                             |              |                        | program is shipped with any common Linux distribution.       |
|            |                             |              |                        |   |
| Unapproved | High CVE-2017-12424         | shadow       | 1:4.4-4.1              | In shadow before 4.5, the newusers tool could be             |
|            |                             |              |                        | made to manipulate internal data structures in ways          |
|            |                             |              |                        | unintended by the authors. Malformed input may lead          |
|            |                             |              |                        | to crashes (with a buffer overflow or other memory           |
|            |                             |              |                        | corruption) or other unspecified behaviors. This             |
|            |                             |              |                        | crosses a privilege boundary in, for example, certain        |
|            |                             |              |                        | web-hosting environments in which a Control Panel allows     |
|            |                             |              |                        | an unprivileged user account to create subaccounts.          |
|            |                             |              |                        |   |

Oops. 86 vulnerabilities! Well, this Docker image might not be really safe to use but in the end, that is all up 2 you.

As you might see in the postgres example, there is a column “STATUS” in the output and all of them are “Unapproved“. Why is that? clair-scanner allows you to whitelist specific vulnerabilities when scanning images.

The clair-scanner tool has an exit code of 0 when no vulnerabilities are found and has an exit code of !=0 when vulnerabilities are found. So if you would run the clair-scanner as part of your CI pipeline, this would fail. However, there could be a reason to whitelist a vulnerability and the clair-scanner will not provide an exit code of !=0 when whitelisted vulnerabilities are found.

Example of a whitelist file.

generalwhitelist: #Approve CVE for any image
  CVE-2017-6055: XML
  CVE-2017-5586: OpenText
  ubuntu: #Apprive CVE only for ubuntu image, regardles of the version
    CVE-2017-5230: Java
    CVE-2017-5230: XSX
    CVE-2017-3261: SE

So we have 2 CVE vulnerabilities that we whitelist, no matter what base Docker OS image is used. For Ubuntu we whitelist 2 CVE’s and 1 for Alpine. I’m not sure, but I would say the XML, OpenText is just a basic description for what package the CVE belongs to.


So with this blogpost we where able to start Clair and do some Docker image analysing with the tool clair-scanner. It showed us that the postgresql image contains some vulnerabilities. So now you can update your CI pipeline by adding a check to scan for vulnerabilities, before pushing the image to a Docker Registry. Next blogpost, we will start a secure Docker Registry and we will sign Docker images with the Notary Server and Signer tool.




6 thoughts on “Scanning Docker images with CoreOS Clair

  1. Donkey 2019-04-18 / 11:14

    hi I have some questions

    As I ran “docker logs -f clair”, I’ve seen a log is :

    {“Event”:”could not parse package version. skipping”,”Level”:”warning”,”Location”:”ubuntu.go:316″,”Time”:”2019-04-18 09:06:43.942348″,”error”:”invalid version”,”version”:”0.27-1+deb7u1build0.12.04.1, 0.28-1+deb8u1″}

    and didn’t saw the logs you post.

    And I ran “docker-compose up -d clair” before, it just took me a few seconds, not as your situation.

    Is my case worng?

    My environment is ubuntu 18.04.2 LTS, clair is cloned from your repo.

    thanks ^^


    • wdijkerman 2019-04-18 / 17:38

      I’ve seen that error before, but I ignored it as I didn’t had any Ubuntu Docker images to validate. But after a while I noticed the logs appear and then it worked fine. Is it still running? Did you already see the log messages appear?


      • Donkey 2019-04-19 / 15:30

        I still have seen

        {“Event”:”could not parse package version. skipping”,”Level”:”warning”,”Location”:”ubuntu.go:316″,”Time”:”2019-04-18 09:06:43.942348″,”error”:”invalid version”,”version”:”0.27-1+deb7u1build0.12.04.1, 0.28-1+deb8u1″}

        and not seen
        {“Event”:”could not get NVD data feed hash”,”Level”:”warning”,”Location”:”nvd.go:137″,”Time”:”2019-01-26 20:19:59.682956″,”data feed name”:”2018″,”error”:”invalid .meta file format”}
        {“Event”:”could not get NVD data feed hash”,”Level”:”warning”,”Location”:”nvd.go:137″,”Time”:”2019-01-26 20:19:59.682956″,”data feed name”:”2019″,”error”:”invalid .meta file format”}

        but it seems that no bother me to run your example.

        But in the second case, my result shows “NO unapproved vulnerabilities”

        here is the scan result log
        $ docker-compose run –rm clair-scanner postgres:latest
        2019/04/19 13:22:51 [INFO] ▶ Start clair-scanner
        2019/04/19 13:22:59 [INFO] ▶ Server listening on port 9279
        2019/04/19 13:22:59 [INFO] ▶ Analyzing 97f043d241d89a3fc54ba1f0dfb14fe804914ed7eadbef9dc7ceafafd7d75b72
        2019/04/19 13:22:59 [INFO] ▶ Analyzing 5d8998c0d781eaa2a79c9a5da2c1562faaf4e0ff2e8c6007614183cc801eb511
        2019/04/19 13:22:59 [INFO] ▶ Analyzing 5d74e20af921ae73f143fad11430cfd2338603b6f7e3d8f1fcea9e549e30f2a2
        2019/04/19 13:22:59 [INFO] ▶ Analyzing 8320da1da804ca15ef525752f04df6c181dc2283707a60ce8f422762f4f3723e
        2019/04/19 13:22:59 [INFO] ▶ Analyzing 88eda149d2c2417666a4d7309991efdc364ceafdc88f142ec12e95b30952d521
        2019/04/19 13:23:00 [INFO] ▶ Analyzing 256a8982d15d79c634bd965f4e6a2a599d09a53918298879d65194fc7e28dd1c
        2019/04/19 13:23:00 [INFO] ▶ Analyzing ee114db71d26a73dcd903e63c02cf4d360770d6c98f03ed67fb7eadef97bd28e
        2019/04/19 13:23:00 [INFO] ▶ Analyzing 21a6e7caee1d408f61c604f9c6fff6cf3ee85a0baed3a9c39c1b6be4770c1146
        2019/04/19 13:23:00 [INFO] ▶ Analyzing d94bd3ac61b1dced99d80f334386aea23a0a98820bd1be8308076e9ad31e5a33
        2019/04/19 13:23:00 [INFO] ▶ Analyzing 57a1daf7f79acce02197953d625848213e712dcb25e9d3bebdeb357815249f2d
        2019/04/19 13:23:00 [INFO] ▶ Analyzing 83a57ecb0c30de218b7806ae816bb65581f6e29c02d6c47baf304d783adb0899
        2019/04/19 13:23:00 [INFO] ▶ Analyzing a0708efd0a3b02971074850c468cde3658d28aad5432179e2bd3d7ee8d1d0af5
        2019/04/19 13:23:00 [INFO] ▶ Analyzing 0b5d884e940f65bea5d2967ae50d43037d13dbace1d44a87a84eae138ef8adec
        2019/04/19 13:23:00 [INFO] ▶ Analyzing da2393fb06ee7cb554ac538530a14c84fd49756fa8f4cfad07f64f1530c68f52
        2019/04/19 13:23:00 [INFO] ▶ Image [postgres:latest] contains NO unapproved vulnerabilities

        is this right?

        thank you again :”)


        • wdijkerman 2019-04-23 / 16:46

          Hmm. That is indeed strange, as I would have expected that it would show you a lot of errors (Or they fixed all the errors in the meantime 😉 ).

          The host running the containers does have access to the internet?


          • Donkey 2019-05-02 / 11:59

            My host running the containers probably had access to the internet, I’m not sure…

            I guessed that it might be fixed those errors so that my result not as same as yours.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s