Remote storages support: first working build with --stages-storage=REPO + fixes

• fixed panic 'should_be_built_phase.go:47 +0x45';
• working werf build --stages-storage=REPO command on the single host for now.

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support (1): full dockerhub support

Added ability to cleanup images published into dockerhub.

Specify following params for werf build-and-publish or werf publish commands:

      --images-repo-docker-hub-password='':
            Docker Hub password for images repo implementation (default $WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD or                      
            $WERF_REPO_DOCKER_HUB_PASSWORD).
      --images-repo-docker-hub-username='':
            Docker Hub username for images repo implementation (default $WERF_IMAGES_REPO_DOCKER_HUB_USERNAME or                      
            $WERF_REPO_DOCKER_HUB_USERNAME).
      --images-repo-implementation='':
            Choose images repo implementation.
            The following docker registry implementations are supported: ecr, default, dockerhub, gcr, github, gitlab, quay.
            Default $WERF_IMAGES_REPO_IMPLEMENTATION, $WERF_REPO_IMPLEMENTATION or auto mode (detect implementation by a registry).

For example:

source <(multiwerf use 1.1 alpha)
...
export WERF_IMAGES_REPO_DOCKER_HUB_USERNAME=myuser
export WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD=password
export WERF_IMAGES_REPO_IMPLEMENTATION=dockerhub
source <(werf ci-env gitlab)
werf build-and-publish -s :local

It is better to define WERF_IMAGES_REPO_DOCKER_HUB_USERNAME, WERF_IMAGES_REPO_DOCKER_HUB_PASSWORD and WERF_IMAGES_REPO_IMPLEMENTATION variables as gitlab secret environment variables.

Installation

Linux amd64

Darwin amd64

Windows amd64

An option to enable cleanup for stages-signature tagging strategy

When running werf in ci with werf ci-env command, then either:

• create cleanup config for the host:

  mkdir ~/.werf/config
  echo "stagesSignatureStrategyExpiryDays: 30" >> ~/.werf/config/cleanup.yaml
  echo "stagesSignatureStrategyLimit: 50" >> ~/.werf/config/cleanup.yaml
  

• export environment variables in your CI/CD system: WERF_STAGES_SIGNATURE_STRATEGY_LIMIT=50 and WERF_STAGES_SIGNATURE_STRATEGY_EXPIRY_DAYS=30.

By default werf will not cleanup images published with stages-signature tagging strategy and related stages.

Installation

Linux amd64

Darwin amd64

Windows amd64

[deploy] Hotfix werf helm render: fix problem with '|' rendered as '|-'

Change gopkg.in/yaml.v2 to github.com/ghodss/yaml in custom werf templates engine,
as github.com/ghodss/yaml is used everywhere in the helm 2.

In the helm 3 sigs.k8s.io/yaml is used, but it is not possible for now to switch to sigs.k8s.io/yaml before switching to helm 3 codebase.
https://github.com/helm/helm/blob/master/pkg/engine/funcs.go#L27

Installation

Linux amd64

Darwin amd64

Windows amd64

Secret unsafe option to disable git minimum version constraint

Set WERF_DISABLE_GIT_MIN_VERSION_CONSTRAINT=1 to disable min git version constraint.

Installation

Linux amd64

Darwin amd64

Windows amd64

Docs

Added rss feeds for werf releases.

Fixes

[dockerfile] Fail build if dockerfile COPY instruction refers to nonexistent stage.

Installation

Linux amd64

Darwin amd64

Windows amd64

[dockerfile] Fix dockerfile stage name from "COPY --from" is not detected

Docker parser coverts dockerfile stage name (AS) to lowercase but in places of the name usage (COPY --from) does not.
This affected dockerfile stage signature calculation and occurred false warning messages.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix release

[dockerfile] Fix calculating context files checksum when submodule is used

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix git problems release

Fix creating an archive fails when nested submodules is used

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix git problem release

Fix creating an archive fails when submodule dir/file is specified in git.add

Installation

Linux amd64

Darwin amd64

Windows amd64

Official v1.1 beta

Implemented content based tagging and new stages storage architecture as a big step toward distributed and concurrent builds.

New stages format and cache selection algorithm

New stage name generation rule. Every build of the stage generates uniq stage-cache image name, which consists of 2 parts: signature (as in v1.0) plus unique identifier.

For example, full stage image name will look like:

werf-stages-storage/myproject:d2c5ad3d2c9fcd9e57b50edd9cb26c32d156165eb355318cebc3412b-1582656767835
werf-stages-storage/PROJECT:SIGNATURE-TIMESTAMP_MILLISEC

Signature identifier of the stage represents content of the stage and depends on git history which lead to this content.

Werf stage selection algorithm is based on the git commits ancestry detection:

1. Werf calculates a stage signature for some stage.
2. There may be multiple stages in the stages storage by this signature, werf selects all suitable stages by the signature.
3. If current stage is related to git (git-archive, user stage with git patch or git latest patch), then werf selects only
   those stages which are relaed to the commit that is ancestor of current git commit.
4. Select from the remaining stages the oldest by the creation timestamp.

There may be multiple built images for a single signature.

Stage for different git branches can have the same signature, but werf will prevent cache of different git branches from
being reused for totally different branch.

For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/stages_and_images.html#stage-naming.

New stage building and saving algorithm

If suitable stage has not been found during stage selection, werf starts building a new image for stage.

Note that multiple processes (on a single or multiple hosts) may start building the same stage at the same time. Werf uses optimistic locking when saving newly built image into the stages storage: when a new stage has been built werf locks stages storage and saves newly built stage image into storage stages cache only if there are no suitable already existing stages exists. Newly saved image will have a guaranteed unique identifier
by TIMESTAMP_MILLISEC. In the case when already existing stage has been found in the stages storage werf will discard newly built image and
use already existing one as a cache.

In other words: the first process which finishes the build (the fastest one) will have a chance to save newly built stage into the stages storage. The slow build process will not block faster processes from saving build results and building next stages.

For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/stages_and_images.html#stage-building-and-saving.

Benefits of new stages storage architecture

This new stages storage architecture opens up an opportunities to build stages concurrently and even distributely
from multiple build machines. For now there are no theoretical limitations of current werf architecture to implement
distributed or concurrent builds. So we consider this release as a big step towards distributed and concurrent builds.

Werf have an --stages-storage and --synchronization parameters which specify addresses of the stages storage and
synchronization lock manager. For now there is only :local implementations of both primitives. By changing these implementations to docker-registry for the stages storage and Redis or Kubernetes server based synchronization lock manager, werf will implement distributed builds out of the box.

Content based tagging

Werf v1.1 supports so called content based tagging. Tags of resulting docker images depend on the content of these images.

When using werf publish --tags-by-stages-signature or werf ci-env --tagging-strategy=stages-signature werf will tag result images by so called image stages signature. Each image tagged by own stages signature which calculated by the same rules as regular signature of image stage.

Image stages signature depends on content of the image and depends on the git history which lead to this content.

There may be dummy commits into the git repo that do not change resulting images. For example empty commits, merge commits or commits which change files that are not imported into the resulting image.

When using tagging by git-commits these dummy commits will cause werf to create new images names even if content of these images is the same. New images names in turn will cause restarts of application Pods in Kubernetes which is totally not a desired behaviour. All in all this is the reason preventing storing multiple application services in the single git repo.

Stages signature on the countrary will not change on dummy commits, so these commits will not cause restarts of application Pods in kubernetes, yet it similarly to commit-id relates to the git history of edits and depends on the content of the files.

Also tagging by stages signatures is more realiable tagging method than tagging by a git-branch for example, because resulting images content does not depend on order of pipelines execution. Stages signature leads to stable immutable images names which represent the address of the certain image content.

NOTE From now and on stages-signature is the default tagging strategy and the only recommended one for usage.

For more info see documentation: https://werf.io/v1.1-beta/documentation/reference/publish_process.html#content-based-tagging.

Improved performance of stages selection and verification

Idle builds when all stages are exist in the stages storage are really fast now (in the most cases build retry will run under 1sec).
Stages verification performance during werf deploy and werf run commands also improved a lot.

Improved performance of Dockerfile builder

Werf Dockerfile builder signature calculation performance is improved due to using git ls-tree checksums
for docker context files. Signature calculation does not depend on the docker context size.

Improved performance of artifact/image imports

Werf uses rsync server to import files from artifacts and images.

MacOS imports performance does not depend on docker implementation of volumes now. MacOS imports performance is the same as in Linux and Windows.

Other changes

• Added --log-debug, --log-verbose and --log-quiet modes.
• Removed legacy params and code (dapp secret key legacy support removed; --insecure-repo legacy option removed, only --skip-tls-verify-registry is available).
• Eliminated werf.io/recreate annotation, werf uses helm.sh/hook-delete-policy=before-hook-creation mode by default (as in helm 3).
• Improved werf cleanup procedure not to clean images from werf.yaml defined in the different git branches,
  those are not exist in the werf.yaml config of the master branch. Added new commands to control managed images: werf managed-images ls|add|rm.

Installation

Linux amd64

Darwin amd64

Windows amd64

Docs

• New stages storage architecture documented.
• Content based tagging documented.

Other fixes

• fromLatest/git.Branch params could be used only with herebyIAdmitThatBranchMightBreakReproducibility and herebyIAdmitThatFromLatestMightBreakReproducibility params;
• werf images managed subcommand renamed to werf managed-images;
• tiny logging improvements:
  • hide docker container output in silent mode;
  • initialize stapel container before preparing of stage run args.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix logs in default mode

Remove 'Switch work tree to commit ...' messages in default mode (will be printed in --log-verbose or --log-debug).

Installation

Linux amd64

Darwin amd64

Windows amd64

WARNING: incompatible release

Stages signatures was changed, pipelines that were already build will use new signatures on retry, so rebuild needed.

Dockerfile builder support for v1.1!

Calculate docker build context files checksum with git ls-tree and status.

Fixed imports: allow merging of directories

import:
- artifact: myartifact
  add: /mydir
  to: /my/existing/dir
  after: install

Werf will automatically merge content of /mydir into /my/existing/dir.

This case was broken in v1.1.0-beta.4 due to imports optimization rework: the specified config will create my/existing/dir/mydir instead of merging directories.

v1.1.0-beta.3 was not affected by this bug.

Fixed logging performance

Fixed logboek library logging performance when processing large output, e.g. from make build command. Logging performance affects build time.

Stapel image updated to version 0.6.1.

Logging improvement

• Disabled debug logging by default.
• Default logging includes minimum viable info for commands.
• --log-debug and --log-verbose options are avaiable.
• More info in --log-debug mode.

Store managed config images in stages storage

Images that were built by werf will be remained in stages storage. For each such image managed image record created as werf-managed-images/PROJECT:IMAGE_NAME.

During cleanup procedure werf will preserve stages cache of images that are defined in the current werf.yaml config and images from managed images list. This prevents images defined only in custom git branches from being cleaned up, when running werf cleanup procedure main (master) branch.

Werf automatically creates records for managed images during build procedure.

Managed images list could be manually viewed and changed using commands:

• werf images managed -s :local ls
• werf images managed -s :local add IMAGE_NAME
• werf images managed -s :local rm [IMAGE_NAME, ...]

Small fixes and changes

• Fix publishing of only specified in cli images by names.
• Renamed --stages-storage-lock param to --synchronization: address of synchronizer for multiple werf processes to work with a single stages storage (:local by default).
• Update kubedog: load more kubernetes client-go auth plugins: azure, exec, oidc, openstack.

Installation

Linux amd64

Darwin amd64

Windows amd64

Load more kubernetes client auth plugins

• azure;
• exec;
• oidc;
• openstack.

Installation

Linux amd64

Darwin amd64

Windows amd64

Optimize imports: rsync import server

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes release (INCOMPATIBLE — more info below)

Small fixes and additions

• Add --log-verbose, --log-quiet options, use --log-debug=true by default for werf v1.1 for now.
• Fixed and enabled cleanup procedure.
• Do not publish artifacts fix.
• Fix: cleanup dockerfile builder tmp img on conveyor termination.
• Fix helm release name slugify: should be a valid DNS-1123 subdomain and be maximum 53 chars.

Fix import stage logic and stages cleanup

The import stage image has the specific label value that is used during stages cleanup procedure. In this commit, related image/artifact signature which stored in the label is replaced on image ID.

The signature does not identify the stage since we started to use not only signature in an image tag. From v1.1, werf is building stages consistently so related image/artifact image ID is available when werf is preparing import stage.

INCOMPATIBLE change, because signature of import stage has been changed. Already built images should be rebuilt with this version of werf.

Also prevented possible bug, which may occur when artifacts are used and there are empty stages.

Installation

Linux amd64

Darwin amd64

Windows amd64

[1.0] Fixes related to the "Resource already exists" deploy error

• fix "resource already exists" error on redeploy when new resource with an error has been added to the chart,
  refs https://github.com/flant/helm/pull/40
• fix "resource already exists" error message when creating a helm.sh/hook that already exists,
  refs https://github.com/flant/helm/pull/41
• do not allow resource adoption during initial release installation, fix error messages,
  refs https://github.com/flant/helm/pull/42

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes related to the "Resource already exists" deploy error

• fix "resource already exists" error on redeploy when new resource with an error has been added to the chart,
  refs https://github.com/flant/helm/pull/40
• fix "resource already exists" error message when creating a helm.sh/hook that already exists,
  refs https://github.com/flant/helm/pull/41
• do not allow resource adoption during initial release installation, fix error messages,
  refs https://github.com/flant/helm/pull/42

Installation

Linux amd64

Darwin amd64

Windows amd64

First Werf v1.1 beta release!

Main changes since v1.0:

1. Fully reworked build process, new incompatible stages storage format (new build cache).

• Improved performance of idle builds (from cache) nearly by one order of magnitude when git repo of the project is big.
• Build cache isolation based on the git history. Build cache of different branches without same base is not overlapped. No more "rebase" related problems.
• Also improved performance of stage-dependencies calculation (using git ls-tree mechanix) when large folders are specified as stage dependencies.

2. Content based tagging using werf publish --tag-by-stages-signature option or werf ci-env --tagging-strategy=stages-signature tagging strategy.
   The default tagging strategy is stages-signature, werf ci-env parameter --tagging-strategy may be omitted to use default strategy.

Not supported in this version

IMPORTANT Following feature are not supported in the current beta version and will be repaired in the following releases very soon!

• Building Dockerfile-based images. This version of werf v1.1 beta should not be used to build dockerfile-based images.
• werf cleanup command will not actually clean stages, only images, due to the bug in the cleaning procedure. werf cleanup command can and should be used already now and will be fixed soon by automatic releases updater for 1.1 beta channel.

Fully reworked build process

• Stage image name format changed: werf-stages-storage/PROJECT_NAME:SIGNATURE-UNIX_TIMESTAMP_MILLISECS.
• UNIX_TIMESTAMP_MILLISECS is the unique part of stage image name.
• There may be multiple stages with the same SIGNATURE part. Signature is based on the content of the image, while timestamp is added to keep separately built images separate.
• IMPORTANT! Multiple processes may build the same stage by the same signature simultaneously at the same time. The first builder that builds the stage will store the stage in the stages storage. Other builders will discard build results when saving newly built image into stages storage if stages storage already contains suitable image built by the other process.
• There may be multiple stages with the same SIGNATURE for different git branches. Stage selection algorithm will make sure that stage is suitable for your branch.

Compatibility notes

Incompatibilities with v1.0:

• Werf v1.1 will generate new stages for stages storage.
• Removed legacy ruby dapp secret-key format support.
• Removed --insecure-repo option, use --skip-tls-verify-registry option instead.
• "helm.sh/hook-delete-policy=before-hook-creation" by default for all kinds of hooks (not only Jobs as in v1.0).
• Strict yaml parsing which does not allow duplicate keys in yaml docs (affects werf.yaml, .helm/templates/**, .helm/values.yaml etc.).

It is OK to use v1.0 and v1.1 simultaneously:

• v1.1 and v1.0 may exists on the same host system, without any restrictions;
• v1.1 and v1.0 may be used in the different branches of the same project without any restrictions;
• cleanup procedure of v1.0 will not affect v1.1 and vice versa;
• build cache (stages in the stages-storage) of v1.0 and v1.0 is fully separated.

How to enable v1.1

1. Change multiwerf use command:

   type multiwerf && source <(multiwerf use 1.1 beta)
   

2. (Optional) To use content based tagging change ci-env command:

   type werf && source <(werf ci-env gitlab)
   

Installation

Linux amd64

Darwin amd64

Windows amd64