Tool for managing the build and publication of docker images. Mirror from https://gerrit.wikimedia.org/g/operations/docker-images/docker-pkg - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing)
GPL-3.0 License
Docker-pkg
is a tool that allows writing and managing docker images in a
unified way, providing out of the box some useful functions:
docker-pkg
can perform three fundamental actions:
build
[BUILD_OPTS] which builds the images in the directoryprune
[PRUNE_OPTS] which prunes old images from theupdate
[UPDATE_OPTS] will create a changelogAs can be seen, in all cases we provide docker-pkg
with a
directory where our image definitions are located. docker-pkg
will scan that directory recursively for directories containing:
Dockerfile.template
filechangelog
Based on information in the changelog, on configuration options, and on what is found in the Dockerfile.template, a Dockerfile is generated.
If you chose to build images, that dockerfile (and the content of the directory) are used to build an image. The name and tag of an image will be derived by the last entry its changelog. Finally, the built images will be pushed to the configured registry, when authentication credentials are provided.
You can provide any variable via a configuration file (in YAML format) that you will then be able to reference in your templates.
A user configuration file $XDG_CONFIG_HOME/docker-pkg.yaml
or
~/.config/docker-pkg.yaml
will be loaded first when it exists. It can be used
to share common settings between multiple docker-pkg definition repositories
(for example apt_only_proxy
). Settings defined in --configfile
take
precedence over user settings.
In addition to those, there are some builtin configuration variables that will
not only be usable in your templates, but also affect how docker-pkg
works:
registry
: The address of the docker registry to use for all docker-relatednamespace
: The namespace under which all the images to build should live.username
and password
: if set, they allow publishing the images you builthttp_proxy
: will set the http proxy to be used for all docker-related operations.apt_only_proxy
: will set up an http proxy to be used exclusively by apt. Pleasehttp_proxy
and not this variable, the value of http_proxy
apt_options
: a string containing additional apt options you want to add to the aptbase_images
: images that are not built by docker-pkg but should be presentscan_workers
: maximum number of threads to use when scanning localdocker-pkg
queries the localknown_uid_mappings
is a dictionary of username:uid mappings that can be used with theuid
template helper.verify_command
and verify_args
specify which command to run, with which arguments, to verifytest.sh
file in the.. code-block:: YAML
verify_command: pytest
# This assumes your tests will accept a `--image` parameter.
verify_args: ["{path}/test_image.py", "--image", "{image}"]
This is as simple as launching docker-pkg build
, indicating as next argument
the directory to scan for dockerfiles.
The build script will search the directory you provided it for Docker image templates, which will be identified by four files:
Dockerfile.template
which is the template for the final Dockerfilechangelog
which is the changelog for the image and adheres to the Debiancontrol
(optional) which has the format of a Debian control file, and can beDockerfile.build.template
(optional) a dockerfile template for a buildThe name and tag of the image will be determined from the changelog file, so it's mandatory that you add your changelog entry there. For most containers, a debian-like versioning is a good idea to keep into account the security updates that might happen.
In case you run docker-pkg build
with the command-line switch
--nightly
, a nightly build will be performed, appending the
current date and time to the tag defined in the changelog of each image, and
thus triggering a rebuild of all images.
Instead of writing plain dockerfiles, we think using jinja2 templates gives us an edge: there are a lot of common constructs that we don't want to replicate, and they are exposed to the templates as variables and filters:
Variables '''''''''
registry
: the address of the docker registryFilters '''''''
image_tag
: This filter allows to retrieve the current image tag for a.. code-block:: dockerfile
FROM {{ registry }}/{{ "nodejs-dev" | image_tag }}
# Will render to e.g. 'FROM my-registry/nodejs-dev:0.3.1'
apt_install
: this filter will get the string you pass it as a list of
packages to install with apt, and add the correct stanza to your dockerfile.
It will also manage the setup of a proxy for apt if one is provided in the
configuration via a http_proxy
key
apt_remove
: this filter will remove the packages listed in the string you
pass to it, acting pretty much the same way apt_install
does.
uid
: this filter will take a username as input, and output the corresponding
UID if a corresponding mapping is saved in known_uid_mappings
in the configuration.
add_user
: this filter will take a username as input, and output the instructions
to create an user with that username, as long as an uid mapping is provided in the
configuration.
Every build is performed in a temporary directory, and any leftovers of the build (so the build image, any container spawned out of it, etc) will be taken care of by the program.
When you build images often, you'll end up with a sizable amount of
wasted disk space by hosting old image builds on your
system. docker-pkg prune
will remove from the local docker daemon
all those images that are contained in <directory>
at version
different than the most recent entry in the changelog file.
It's pretty common we need to rebuild a base image and having to
rebuild all the images that depend upon it. docker-pkg update
partially automates the process creating a changelog entry with a
pre-baked message for each of those images. this will trigger a
rebuild of those images next time docker-pkg build
is launched.
.. code-block:: console
$ docker-pkg update --reason 'CVE-XYZ isArrayish RCE' nodejs images
$ docker-pkg build
When building images on macOS, you may see an error like this:
.. code-block:: console
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt
To work around this, open Keychain Access, navigate to System Roots ->
Certificates, select all certificates and go to File -> Export Items. Select
the export format as Certificate (.cer). Save the file to a temporary
location, then mv
it to /etc/ssl/certs/ca-certificates.crt
.