A solution for implementing efficient and consistent software delivery to Kubernetes facilitating best practices.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by flant-team-sysdev over 4 years ago
werf build --stages-storage=REPO
command on the single host for now.Published by flant-team-sysdev over 4 years ago
Added ability to cleanup images published into dockerhub.
Specify following params for werf build-and-publish
or werf publish
commands:
--images-repo-docker-hub-password='':
Docker Hub password for images repo implementation (default $WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD or
$WERF_REPO_DOCKER_HUB_PASSWORD).
--images-repo-docker-hub-username='':
Docker Hub username for images repo implementation (default $WERF_IMAGES_REPO_DOCKER_HUB_USERNAME or
$WERF_REPO_DOCKER_HUB_USERNAME).
--images-repo-implementation='':
Choose images repo implementation.
The following docker registry implementations are supported: ecr, default, dockerhub, gcr, github, gitlab, quay.
Default $WERF_IMAGES_REPO_IMPLEMENTATION, $WERF_REPO_IMPLEMENTATION or auto mode (detect implementation by a registry).
For example:
source <(multiwerf use 1.1 alpha)
...
export WERF_IMAGES_REPO_DOCKER_HUB_USERNAME=myuser
export WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD=password
export WERF_IMAGES_REPO_IMPLEMENTATION=dockerhub
source <(werf ci-env gitlab)
werf build-and-publish -s :local
It is better to define WERF_IMAGES_REPO_DOCKER_HUB_USERNAME
, WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD
and WERF_IMAGES_REPO_IMPLEMENTATION
variables as gitlab secret environment variables.
Published by flant-team-sysdev over 4 years ago
When running werf in ci with werf ci-env
command, then either:
create cleanup config for the host:
mkdir ~/.werf/config
echo "stagesSignatureStrategyExpiryDays: 30" >> ~/.werf/config/cleanup.yaml
echo "stagesSignatureStrategyLimit: 50" >> ~/.werf/config/cleanup.yaml
export environment variables in your CI/CD system: WERF_STAGES_SIGNATURE_STRATEGY_LIMIT=50
and WERF_STAGES_SIGNATURE_STRATEGY_EXPIRY_DAYS=30
.
By default werf will not cleanup images published with stages-signature tagging strategy and related stages.
Published by flant-team-sysdev over 4 years ago
Change gopkg.in/yaml.v2
to github.com/ghodss/yaml
in custom werf templates engine,
as github.com/ghodss/yaml
is used everywhere in the helm 2.
In the helm 3 sigs.k8s.io/yaml
is used, but it is not possible for now to switch to sigs.k8s.io/yaml
before switching to helm 3 codebase.
https://github.com/helm/helm/blob/master/pkg/engine/funcs.go#L27
Published by flant-team-sysdev over 4 years ago
Set WERF_DISABLE_GIT_MIN_VERSION_CONSTRAINT=1 to disable min git version constraint.
Published by flant-team-sysdev over 4 years ago
Added rss feeds for werf releases.
[dockerfile] Fail build if dockerfile COPY instruction refers to nonexistent stage.
Published by flant-team-sysdev over 4 years ago
Docker parser coverts dockerfile stage name (AS) to lowercase but in places of the name usage (COPY --from) does not.
This affected dockerfile stage signature calculation and occurred false warning messages.
Published by flant-team-sysdev over 4 years ago
[dockerfile] Fix calculating context files checksum when submodule is used
Published by flant-team-sysdev over 4 years ago
Fix creating an archive fails when nested submodules is used
Published by flant-team-sysdev over 4 years ago
Fix creating an archive fails when submodule dir/file is specified in git.add
Published by flant-team-sysdev over 4 years ago
Implemented content based tagging and new stages storage architecture as a big step toward distributed and concurrent builds.
New stage name generation rule. Every build of the stage generates uniq stage-cache image name, which consists of 2 parts: signature (as in v1.0) plus unique identifier.
For example, full stage image name will look like:
werf-stages-storage/myproject:d2c5ad3d2c9fcd9e57b50edd9cb26c32d156165eb355318cebc3412b-1582656767835
werf-stages-storage/PROJECT:SIGNATURE-TIMESTAMP_MILLISEC
Signature identifier of the stage represents content of the stage and depends on git history which lead to this content.
Werf stage selection algorithm is based on the git commits ancestry detection:
There may be multiple built images for a single signature.
Stage for different git branches can have the same signature, but werf will prevent cache of different git branches from
being reused for totally different branch.
For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/stages_and_images.html#stage-naming.
If suitable stage has not been found during stage selection, werf starts building a new image for stage.
Note that multiple processes (on a single or multiple hosts) may start building the same stage at the same time. Werf uses optimistic locking when saving newly built image into the stages storage: when a new stage has been built werf locks stages storage and saves newly built stage image into storage stages cache only if there are no suitable already existing stages exists. Newly saved image will have a guaranteed unique identifier
by TIMESTAMP_MILLISEC
. In the case when already existing stage has been found in the stages storage werf will discard newly built image and
use already existing one as a cache.
In other words: the first process which finishes the build (the fastest one) will have a chance to save newly built stage into the stages storage. The slow build process will not block faster processes from saving build results and building next stages.
For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/stages_and_images.html#stage-building-and-saving.
This new stages storage architecture opens up an opportunities to build stages concurrently and even distributely
from multiple build machines. For now there are no theoretical limitations of current werf architecture to implement
distributed or concurrent builds. So we consider this release as a big step towards distributed and concurrent builds.
Werf have an --stages-storage
and --synchronization
parameters which specify addresses of the stages storage and
synchronization lock manager. For now there is only :local
implementations of both primitives. By changing these implementations to docker-registry for the stages storage and Redis or Kubernetes server based synchronization lock manager, werf will implement distributed builds out of the box.
Werf v1.1 supports so called content based tagging. Tags of resulting docker images depend on the content of these images.
When using werf publish --tags-by-stages-signature
or werf ci-env --tagging-strategy=stages-signature
werf will tag result images by so called image stages signature. Each image tagged by own stages signature which calculated by the same rules as regular signature of image stage.
Image stages signature depends on content of the image and depends on the git history which lead to this content.
There may be dummy commits into the git repo that do not change resulting images. For example empty commits, merge commits or commits which change files that are not imported into the resulting image.
When using tagging by git-commits these dummy commits will cause werf to create new images names even if content of these images is the same. New images names in turn will cause restarts of application Pods in Kubernetes which is totally not a desired behaviour. All in all this is the reason preventing storing multiple application services in the single git repo.
Stages signature on the countrary will not change on dummy commits, so these commits will not cause restarts of application Pods in kubernetes, yet it similarly to commit-id relates to the git history of edits and depends on the content of the files.
Also tagging by stages signatures is more realiable tagging method than tagging by a git-branch for example, because resulting images content does not depend on order of pipelines execution. Stages signature leads to stable immutable images names which represent the address of the certain image content.
NOTE From now and on stages-signature is the default tagging strategy and the only recommended one for usage.
For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/publish_process.html#content-based-tagging.
Idle builds when all stages are exist in the stages storage are really fast now (in the most cases build retry will run under 1sec).
Stages verification performance during werf deploy
and werf run
commands also improved a lot.
Werf Dockerfile builder signature calculation performance is improved due to using git ls-tree
checksums
for docker context files. Signature calculation does not depend on the docker context size.
Werf uses rsync server to import files from artifacts and images.
MacOS imports performance does not depend on docker implementation of volumes now. MacOS imports performance is the same as in Linux and Windows.
--log-debug
, --log-verbose
and --log-quiet
modes.--insecure-repo
legacy option removed, only --skip-tls-verify-registry is available).werf.io/recreate
annotation, werf uses helm.sh/hook-delete-policy=before-hook-creation
mode by default (as in helm 3).werf managed-images ls|add|rm
.Published by flant-team-sysdev over 4 years ago
herebyIAdmitThatBranchMightBreakReproducibility
and herebyIAdmitThatFromLatestMightBreakReproducibility
params;werf images managed
subcommand renamed to werf managed-images
;Published by flant-team-sysdev over 4 years ago
Remove 'Switch work tree to commit ...' messages in default mode (will be printed in --log-verbose or --log-debug).
Published by flant-team-sysdev over 4 years ago
Stages signatures was changed, pipelines that were already build will use new signatures on retry, so rebuild needed.
Calculate docker build context files checksum with git ls-tree and status.
import:
- artifact: myartifact
add: /mydir
to: /my/existing/dir
after: install
Werf will automatically merge content of /mydir
into /my/existing/dir
.
This case was broken in v1.1.0-beta.4
due to imports optimization rework: the specified config will create my/existing/dir/mydir
instead of merging directories.
v1.1.0-beta.3
was not affected by this bug.
Fixed logboek library logging performance when processing large output, e.g. from make build
command. Logging performance affects build time.
Stapel image updated to version 0.6.1.
--log-debug
and --log-verbose
options are avaiable.--log-debug
mode.Images that were built by werf will be remained in stages storage. For each such image managed image record created as werf-managed-images/PROJECT:IMAGE_NAME
.
During cleanup procedure werf will preserve stages cache of images that are defined in the current werf.yaml config and images from managed images list. This prevents images defined only in custom git branches from being cleaned up, when running werf cleanup
procedure main (master) branch.
Werf automatically creates records for managed images during build procedure.
Managed images list could be manually viewed and changed using commands:
werf images managed -s :local ls
werf images managed -s :local add IMAGE_NAME
werf images managed -s :local rm [IMAGE_NAME, ...]
--stages-storage-lock
param to --synchronization
: address of synchronizer for multiple werf processes to work with a single stages storage (:local by default).Published by flant-team-sysdev over 4 years ago
Published by flant-team-sysdev over 4 years ago
Published by flant-team-sysdev over 4 years ago
The import stage image has the specific label value that is used during stages cleanup procedure. In this commit, related image/artifact signature which stored in the label is replaced on image ID.
The signature does not identify the stage since we started to use not only signature in an image tag. From v1.1, werf is building stages consistently so related image/artifact image ID is available when werf is preparing import stage.
INCOMPATIBLE change, because signature of import stage has been changed. Already built images should be rebuilt with this version of werf.
Also prevented possible bug, which may occur when artifacts are used and there are empty stages.
Published by flant-team-sysdev over 4 years ago
Published by flant-team-sysdev over 4 years ago
Published by flant-team-sysdev over 4 years ago
Main changes since v1.0:
werf publish --tag-by-stages-signature
option or werf ci-env --tagging-strategy=stages-signature
tagging strategy.stages-signature
, werf ci-env parameter --tagging-strategy
may be omitted to use default strategy.IMPORTANT Following feature are not supported in the current beta version and will be repaired in the following releases very soon!
werf cleanup
command will not actually clean stages, only images, due to the bug in the cleaning procedure. werf cleanup
command can and should be used already now and will be fixed soon by automatic releases updater for 1.1 beta channel.Incompatibilities with v1.0:
It is OK to use v1.0 and v1.1 simultaneously:
Change multiwerf use command:
type multiwerf && source <(multiwerf use 1.1 beta)
(Optional) To use content based tagging change ci-env command:
type werf && source <(werf ci-env gitlab)