Publishing Docker images is a common CI/CD task, and the flexibility CircleCI offers makes it a great tool for the job.
Motivation
Automated builds, when combined with tests, are a great tool to increase iteration speed on projects through time savings.
Imagine that you maintain a public Docker image of an application that has frequent version changes - every time a new version is released, you'll have to spend time building, tagging, testing, and publishing a new version. Now imagine you had a tool to update the application version for you automatically in your Dockerfile, and a CI/CD pipeline that would take care of the building, tagging, testing, and publishing of the image - it could maintain itself indefinitely without your intervention.
We'll touch on the second part here, the CI/CD pipeline for Docker images.
About CircleCI
CircleCI is a generalized CI/CD tool similar to Jenkins , Travis CI , and GitHub Actions . As of writing, it is the CI/CD tool that I have most of my personal projects using due to familiarity.
CircleCI Terminology
In case you haven't used CircleCI before, here are some definitions for their terminology:
Steps are a collection of executable commands which are run during a job. (source )
Jobs are collections of steps. All the steps in the job are executed in a single unit, either within a fresh container or VM. (source )
A workflow is a set of rules for defining a collection of jobs and their run order. Workflows support complex job orchestration using a simple set of configuration keys to help you resolve failures sooner. (source )
Setup
For this project, you will need a few things:
- CircleCI installed to your version control of choice
- A CircleCI project set up from your selected repository
- A Docker Hub account to push your image to
Building the image
Let's make a simple hello world Dockerfile:
1FROM alpine:3.13.2
2CMD ["echo", "Hello world!"]
This can be tested with a command such as:
$ docker build --tag helloworld . && docker run helloworld
Hello world!Then we'll create the beginnings of our CircleCI config at the default location .circleci/config.yml:
1version: 2.1
2
3jobs:
4 build:
5 # Use `docker:stable` as the Docker container to run this job in
6 docker:
7 - image: docker:stable
8
9 steps:
10 # Checkout the repository files
11 - checkout
12
13 # Set up a separate Docker environment to run `docker` commands in
14 - setup_remote_docker
15
16 # Build the hello world image
17 - run:
18 name: Build Docker image
19 command: docker build --tag helloworld .
By default, CircleCI will execute the pipeline on every code push, so once these files are committed and pushed, CircleCI will run docker build and continue to on every subsequent push. The pipeline should finish in under 30 seconds, and if we see that succeed in the CircleCI UI, then we know the image built successfully.
Publishing the image
Building the Docker image is great, but that image artifact disappeared when the job finished. In order to save our work, we'll want to publish the image - what this post is all about!
First, we'll want to set an image name to publish under. To keep things easier to read, I will collapse some parts of the Dockerfile that haven't changed.
1version: 2.1
2
3# Define a common Docker container and environment for jobs
4executors:
5 docker-publisher:
6 # Define the image tag
7 environment:
8 IMAGE_TAG: <username>/helloworld:latest
9 # Use `docker:stable` as the Docker image for this executor
10 docker:
11 - image: docker:stable
12
13jobs:
14 build:
15 # Use docker-publisher from above as the Docker container to run this job in
16 executor: docker-publisher
17
18 steps:
19 - checkout
20 - setup_remote_docker
21 - run:
22 name: Build Docker image
23 # Tag using the image tag above
24 command: docker build --tag "${IMAGE_TAG}" .
Replace <username> with your own Docker Hub username.
To save the output of the build to be used later, we'll save it to the workflow's temporary "workspace":
1version: 2.1
2
3# executors: (collapsed)
4
5jobs:
6 build:
7 executor: docker-publisher
8 steps:
9 - checkout
10 - setup_remote_docker
11 - run:
12 name: Build Docker image
13 command: docker build --tag helloworld .
14
15 # Archive and persist the Docker image
16 - run:
17 name: Archive Docker image
18 command: docker save --output image.tar "${IMAGE_TAG}"
19 - persist_to_workspace:
20 root: .
21 paths:
22 - ./image.tar
Because the image is very small, even archiving and saving the output should still keep our build time under 30 seconds.
In order to push images to Docker Hub, we'll need to supply CircleCI with our credentials. In your CircleCI project's settings, there is a page to configure environment variables. We'll need a variable named DOCKERHUB_USERNAME with the value of your username, and one named DOCKERHUB_PASS with the value of a personal access token created for CircleCI. I would recommend against saving your password in plaintext.
Now we're ready to add a job to log in to Docker Hub and push the image:
1version: 2.1
2
3# executors: (collapsed)
4
5jobs:
6 # build: (collapsed)
7
8 push:
9 # Use docker-publisher from above as the Docker container to run this job in
10 executor: docker-publisher
11
12 steps:
13 # Set up a separate Docker environment to run `docker` commands in
14 - setup_remote_docker
15
16 # Load and un-archive the Docker image
17 - attach_workspace:
18 at: /tmp/workspace
19 - run:
20 name: Load Docker image
21 command: docker load --input /tmp/workspace/image.tar
22
23 # Log in to Docker Hub and push the image
24 - run:
25 name: Publish Docker image
26 command: |
27 echo "${DOCKERHUB_PASS}" | docker login --username "${DOCKERHUB_USERNAME}" --password-stdin
28 docker push "${IMAGE_TAG}"
29
30# Run the two different jobs as a sequenced workflow
31workflows:
32 version: 2
33 build-push:
34 jobs:
35 # Run the build first
36 - build
37 # Push the image second
38 - push:
39 # Build needs to finish first
40 requires:
41 - build
42 # Only push images from the main branch
43 filters:
44 branches:
45 only: main
The final .circleci/config.yml will look like this:
1version: 2.1
2
3executors:
4 docker-publisher:
5 environment:
6 IMAGE_TAG: <username>/helloworld:latest
7 docker:
8 - image: docker:stable
9
10jobs:
11 build:
12 executor: docker-publisher
13 steps:
14 - checkout
15 - setup_remote_docker
16 - run:
17 name: Build Docker image
18 command: docker build --tag "${IMAGE_TAG}" .
19 - run:
20 name: Archive Docker image
21 command: docker save --output image.tar "${IMAGE_TAG}"
22 - persist_to_workspace:
23 root: .
24 paths:
25 - ./image.tar
26
27 push:
28 executor: docker-publisher
29 steps:
30 - setup_remote_docker
31 - attach_workspace:
32 at: /tmp/workspace
33 - run:
34 name: Load Docker image
35 command: docker load --input /tmp/workspace/image.tar
36 - run:
37 name: Publish Docker image
38 command: |
39 echo "${DOCKERHUB_PASS}" | docker login --username "${DOCKERHUB_USERNAME}" --password-stdin
40 docker push "${IMAGE_TAG}"
41
42workflows:
43 version: 2
44 build-push:
45 jobs:
46 - build
47 - push:
48 requires:
49 - build
50 filters:
51 branches:
52 only: main
Upon pushing that change, CircleCI will publish a public image named helloworld with a single tag latest under your Docker Hub account. The whole pipeline should take less than 60 seconds.
Bonus: testing the image
I've previously talked about testing Docker images with Container Structure Test, a great tool from Google to test your built docker images - and it's very easy to add to our CircleCI pipeline.
Testing Docker Images with Container Structure Test
Jul 18, 2020 · 5 min read
Just because a Docker image builds successfully doesn't mean it will perform as expected. Google's container Structure Test tool helps you check images to make sure they're working as intended.
First, make a container-structure-test.yml:
1schemaVersion: 2.0.0
2commandTests:
3 - name: "docker run"
4 # `command` is required, so we'll have it match the Dockerfile
5 command: "echo"
6 args: ["Hello world!"]
7 expectedOutput: ["Hello world!"]
8 excludedError: [".+"]
9 exitCode: 0
This isn't a very valuable test because it replaces all the functionality of the Dockerfile, allowing for the two get out of sync - but it will work well enough to show the CircleCI config.
To run Container Structure Test in our CircleCI pipeline, add a step just after docker build:
1version: 2.1
2
3# executors: (collapsed)
4jobs:
5 build:
6 executor: docker-publisher
7 steps:
8 - checkout
9 - setup_remote_docker
10 - run:
11 name: Build Docker image
12 command: docker build --tag "${IMAGE_TAG}" .
13 # Run Container Structure Test against the built image
14 - run:
15 name: Test Docker image
16 command: |
17 apk add --no-cache curl > /dev/null
18 curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
19 container-structure-test test --config container-structure-test.yml --image "${IMAGE_TAG}"
20 # - run (collapsed)
21 # - persist_to_workspace (collapsed)
22
23 # push: (collapsed)
24
25# workflows: (collapsed)
If Container Structure Test doesn't pass, it will exit with a non-zero exit code, which will fail the step and the entire job.
Bonus: linting the Dockerfile
I've also previously talked about linting Dockerfiles with Hadolint, to both help enforce style guidelines and help catch potential issues. It's also a simple addition to our pipeline.
Linting Dockerfiles with Hadolint
Aug 10, 2020 · 3 min read
Linters don't just enforce style guidelines, they also catch potential issues. hadolint (Haskell Dockerfile Linter) is the most popular linter for Dockerfiles, and it's incredibly easy to use.
To run Hadolint, add a step just before docker build:
1version: 2.1
2
3# executors: (collapsed)
4
5jobs:
6 build:
7 executor: docker-publisher
8 steps:
9 - checkout
10 - setup_remote_docker
11 # Lint the Dockerfile
12 - run:
13 name: Lint Dockerfile
14 command: docker run --rm --interactive hadolint/hadolint < Dockerfile
15 - run:
16 name: Build Docker image
17 command: docker build --tag "${IMAGE_TAG}" .
18 # - run (collapsed)
19 # - persist_to_workspace (collapsed)
20
21 # push: (collapsed)
22
23# workflows: (collapsed)
If Hadolint doesn't pass, it will exit with a non-zero exit code, which will fail the step and the entire job.
Other CI/CD tools
If CircleCI isn't the right tool for you, check out how to accomplish this same pipeline using GitHub Actions.
Publishing Docker Images with GitHub Actions
Feb 20, 2021 · 8 min read
Publishing Docker images is a common CI/CD task, and the tight integration GitHub Actions has with GitHub repositories makes it a great tool for the job.
