Automated solution for updating running Docker containers - watchtower
source link: https://blog.wu-boy.com/2023/02/automating-docker-container-base-image-updates-eng/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Automated solution for updating running Docker containers - watchtower
Posted on February 28, 2023 | 5 minutes | 1046 words | appleboyNowadays, most of us have containerized our services, and effectively managing and upgrading containers without affecting existing services is a critical issue. Two steps are required in the CI/CD process: first, packaging the environment as a Docker image and uploading it to the company’s private Docker registry; and second, after the upload is complete, possibly connecting to the machine via SSH, pulling the new image, and restarting the running service via the Graceful Shutdown mechanism. You can learn more about Graceful Shutdown in this article. I am going to introduces Watchtower, a brand new tool that automatically upgrades and updates running containers, allowing for further CD process streamlining. Developers only need to upload the Docker image, and the remote servers can update the running container automatically.
The infrastructure diagram will become as below:
What’s Watchtower
Watchtower is an application developed in Go language that monitors running Docker containers and observes whether the Docker images used when these containers were initially started have been changed. If Watchtower detects a change in the Docker image, it will automatically restart the container using the new image.
Through Watchtower, developers can easily update the running version of containerized applications by pushing new Docker images to Docker Hub or their own Docker registry. Watchtower will download your new image, gracefully shut down the existing container, and then restart it using the same options used during the initial deployment.
For example, suppose you are running Watchtower and an instance of an image called ghcr.io/go-training/example53
:
Every few minutes, Watchtower will download the latest ghcr.io/go-training/example53
image and compare it to the image used to run the “example53” container. If it finds that the image has been changed, it will stop/delete the “example53” container and then restart it using the new image and the same docker run options used when the container was initially started.
How to use?
Watchtower is packaged as a Docker container itself, so installation is very simple; just pull the containrrr/watchtower
image. If you’re using an ARM-based architecture, pull the appropriate containrrr/watchtower:armhf-tag
image from Docker Hub.
Since Watchtower’s code needs to interact with the Docker API to monitor running containers, the /var/run/docker.sock
needs to be mounted into the container using the -v flag when running the container.
Use the following command to run the Watchtower container:
docker run -d \
--name watchtower \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower
If pulling images from a private Docker registry, use the REPO_USER
and REPO_PASS
environment variables, or mount the host’s Docker configuration file into the container (located at the root of the container filesystem /).
docker run -d \
--name watchtower \
-e REPO_USER=username \
-e REPO_PASS=password \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower container_to_watch --debug
Also, if you have 2FA authentication set up on Docker Hub, providing only your account and password won’t be enough. Instead, you can run the docker login command to store the credentials in the $HOME/.docker/config.json
file, and then mount this configuration file to make it available to the Watchtower container:
docker run -d \
--name watchtower \
-v $HOME/.docker/config.json:/config.json \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower container_to_watch --debug
Example
Here we use the docker-compose method to test the running container.
version: "3"
services:
example53:
image: ghcr.io/go-training/example53:latest
restart: always
labels:
- "com.centurylinklabs.watchtower.enable=true"
ports:
- "8080:8080"
watchtower:
image: containrrr/watchtower
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: --interval 5
After starting, you can see the following log messages:
example53_1 | [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
example53_1 |
example53_1 | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
example53_1 | - using env: export GIN_MODE=release
example53_1 | - using code: gin.SetMode(gin.ReleaseMode)
example53_1 |
example53_1 | [GIN-debug] GET /ping --> main.main.func1 (3 handlers)
example53_1 | [GIN-debug] GET / --> main.main.func2 (3 handlers)
example53_1 | [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
example53_1 | Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
example53_1 | [GIN-debug] Environment variable PORT is undefined. Using port :8080 by default
example53_1 | [GIN-debug] Listening and serving HTTP on :8080
watchtower_1 | time="2023-03-02T01:13:07Z" level=info msg="Watchtower 1.5.3"
watchtower_1 | time="2023-03-02T01:13:07Z" level=info msg="Using no notifications"
watchtower_1 | time="2023-03-02T01:13:07Z" level=info msg="Checking all containers (except explicitly disabled with label)"
watchtower_1 | time="2023-03-02T01:13:07Z" level=info msg="Scheduling first run: 2023-03-02 01:13:12 +0000 UTC"
watchtower_1 | time="2023-03-02T01:13:07Z" level=info msg="Note that the first check will be performed in 4 seconds"
watchtower_1 | time="2023-03-02T01:13:14Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower_1 | time="2023-03-02T01:13:19Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
watchtower_1 | time="2023-03-02T01:13:24Z" level=info msg="Session done" Failed=0 Scanned=2 Updated=0 notify=no
You can adjust the --interval 5
parameter based on the time interval you want to monitor. Here, we set it to 5 seconds. Watchtower monitors all containers on the host by default. If you don’t want certain containers to be updated, you can set the label
in docker-compose file.
labels:
- "com.centurylinklabs.watchtower.enable=false"
After upgrading, the old containers or image files still exist on the host and take up some space. You can use the --cleanup
parameter to let Watchtower delete the old image files after using the new ones to restart the containers.
watchtower:
image: containrrr/watchtower
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: --interval 5 --cleanup
During the process, if a new image is pulled, you will see an error message below, and Watchtower will send a SIGTERM signal to the container for graceful shutdown.
watchtower_1 | time="2023-03-02T01:35:15Z" level=info msg="Found new ghcr.io/go-training/example53:latest image (040d01951ee2)"
watchtower_1 | time="2023-03-02T01:35:17Z" level=info msg="Stopping /root_example53_1 (57fc95adf8cd) with SIGTERM"
If you want to change the signals, you can use the label. Please modify the Dockerfile
.
LABEL com.centurylinklabs.watchtower.stop-signal="SIGHUP"
Or add them when starting the container.
docker run -d --label=com.centurylinklabs.watchtower.stop-signal=SIGHUP someimage
Experience Sharing
Our team can focus on packaging the image and uploading it to the Docker Registry in the CI/CD process in the future. Watchtower monitors all services on the machine, and uploaded images follow the semver principles. This reduces a lot of the work involved in writing shell scripts.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK