Gitlab CI/CD and Container Scanning

Here we will learn how to automate container image scanning with Gitlab CI/CD

Gitlab CI/CD and Container Scanning

Have you ever found a docker container online, but it is missing one little thing you need? So you go ahead and you make some changes, push it to docker hub. You then use it once and it gets forgotten about?

Well you might not but I do, I guess thats the simplicity of containers. So what if we could set up a Gitlab CI/CD pipeline that does everything for you along with telling you vulnerabilities? Sound like a good plan?, then lets have a look at how we do that.

Teeny Tiny Back Story

I was trying to deploy a container via a helm chart to my kubernetes cluster at home, it was an influxdb image but it needed one extra file, the types.db file for collectd. So I found the Dockerfile and just added one line.

Pre-Requisites

Lets Go

The dockerfile above is your starting point, after you have this you need to create a .gitlab-ci.yml in the root of the project. This gitlab will contain all your stages and define what each stage does.

Lets start with the first bit:

stages:
  - build
  - test
  - check_results
  - push

variables:
  DOCKER_DRIVER: overlay2

build:
  image: docker:stable
  stage: build
  services:
    - docker:19.03.12-dind
  script:
    - docker info
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
include:
  - template: Container-Scanning.gitlab-ci.yml

container_scanning:
  variables:
    GIT_STRATEGY: fetch
    CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE
    CI_APPLICATION_TAG: $CI_COMMIT_SHA
    CLAIR_OUTPUT: High
  artifacts:
    paths:
      - gl-container-scanning-report.json

So above what we are declaring is:

Stages: Here we are just declaring what stages will be part of the job.

Build: Simple set up:

  • We state the docker image we want to use
  • We declare the stage we want this step to be part of
  • We declare a script which logs us in to the gitlab container registry, builds the image and pushes it back to the container registry with the commit sha as the tag.
  • The next part is something that can be found in the Gitlab documentation here which is the part that triggers the container scanning - https://docs.gitlab.com/ee/user/application_security/container_scanning/#configuration
  • We then define some variables for the job (these are also in the above documentation link) and it stores the results in - gl-container-scanning-report.json

And that is the initial stage done.

Lets see how we can get a stage to pipe out the output so that we have a report stored for each build and you can view the vulnerabilities within Gitlab Pipelines build job. It will also fail the job if there is vulnerabilities.

scanning_results:
  stage: check_results
  image: ubuntu:focal
  services:
    - docker:dind
  script:
    - apt-get update 
    - apt-get install -y jq
    - jq -e "( .vulnerabilities | length ) == 0" ./gl-container-scanning-report.json
  dependencies:
    - container_scanning
  allow_failure: false

In the step we are just pulling a docker image, installing jq. We then proceed to run a query which checks if there is 0 records, 0 records means 0 vulnerabilities have been found. If it is greater than 0 then the job will fail.

The last flag essentially allows the stage to pass even if there is vulnerabilites in the report. I would suggest leaving it as it is and try to remediate the vulnerabilities that are found.

It is all coming along nicely, we have a job that builds the docker container and scans it. We then set up a seperate step which reads out the results of the scan and passes/fails the job depending on the result. This is good so far but if it passes with 0 vulnerabilities we can then tag this as latest and push it back up. So that is our next step, lets finish this off.

push_latest:
  image: docker:stable
  services:
    - docker:19.03.12-dind
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    refs:
      - master
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest

push_tag:
  image: docker:stable
  services:
    - docker:19.03.12-dind
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    - tags
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

I have left some comments in the yaml itself but essentially we are only running this step when it has reached master. This is what we are doing:

  • We pull the last image, and change the tag from the commit sha to latest and push this new tag up.
  • The last step runs the same styled steps if we are working with git tags only.

I will add below what the end goal looks like within the Gitlab UI. This is what the pipeline should resemble.

Pipieline View

Below i will give you the complete .gitlab-ci.yaml file for you to play around with, it is probably not perfect but it gets you started and in the right direction. The gitlab ci/cd pipeline is capable of a hell of a lot, this is just touching the iceberg.

stages:
  - build
  - test
  - check_results
  - push

variables:
  DOCKER_DRIVER: overlay2

build:
  image: docker:stable
  stage: build
  services:
    - docker:19.03.12-dind
  script:
    - docker info
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

include:
  - template: Container-Scanning.gitlab-ci.yml

container_scanning:
  variables:
    GIT_STRATEGY: fetch
    CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE
    CI_APPLICATION_TAG: $CI_COMMIT_SHA
    CLAIR_OUTPUT: High
  artifacts:
    paths:
      - gl-container-scanning-report.json

scanning_results:
  stage: check_results
  image: ubuntu:focal
  services:
    - docker:dind
  script:
    - apt-get update 
    - apt-get install -y jq
    - jq -e "( .vulnerabilities | length ) == 0" ./gl-container-scanning-report.json
  dependencies:
    - container_scanning
  allow_failure: false

push_latest:
  image: docker:stable
  services:
    - docker:19.03.12-dind
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    refs:
      - master
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest

push_tag:
  image: docker:stable
  services:
    - docker:19.03.12-dind
  variables:
    GIT_STRATEGY: none
  stage: push
  only:
    - tags
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME