Fix cleanup deletes stages that are related to deployed images

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix cleanup fails to delete stages from repo stages storage

Automatically set gitlab repo implementation option (--stages-storage-repo-implementation=gitlab) in werf ci-env gitlab.

Other

[docs] Added three new publications about werf.
[docs] Publications on the site updated. Added three new publications about werf.

[docs] Fix README coming soon and features list

Installation

Linux amd64

Darwin amd64

Windows amd64

Introducing distributed builds

Werf now can store stages in the docker registry by specifying --stages-storage REPO instead of --stages-storage :local.

There is no more restriction that a single host should be used to run all werf commands for a project:

• All werf commands that need an access to the stages should specify the same stages storage.
• It is irrelevant on which host werf command is running as long as the same stages storage is used for the commands like: build, publish, cleanup and deploy.
• Also all hosts which run werf should have a connection to the Kubernetes cluster, because werf uses Kubernetes to synchronize multiple build/publish/deploy processes running from different machines.

"werf ci-env gitlab" default mode now is to store stages by path CI_REGISTRY_IMAGE/stages. All existing werf installations will continue to use :local stages storage because --stages-storage :local param has explicitly been specified in all CI-configs. So to use distributed builds user needs to remove this param, see guided steps below about migrating to the distributed stages storage.

Synchronization, caches and cleanup

Introduced a new parameter named --synchronization, which controls how multiple werf processes are synchronized between each other. There is a stages-storage cache and locks components that werf needs to run. Stages storage cache is an internal component used to boost werf performance for idle builds when calculating stages. Locks are needed to organize correct publishing of new stages into stages-storage, publishing images into images-repo and for deploy process also.

When --synchronization :local used (which is factually by default in the previous versions < v1.1.10), then stages-storage cache reside in the local host file system (~/.werf/shared_context/storage/stages_storage_cache/1/PROJECT) and local file locks are used.

When --synchronization kubernetes://NAMESPACE is used, then werf creates a ConfigMap for project in this namespace, stores stages storage cache in this ConfigMap and uses distributed locking over this ConfigMap to synchronize multiple werf processes running from multiple hosts.

By default, when non local stages-storage is used werf automatically use kubernetes://werf-synchronization namespace and create a ConfigMap named werf-PROJECT in this namespace. User can specify arbitrary non-standard namespace on own needs.

What about local docker images cache on hosts where werf is running? — There is such a cache, which is cleared either by the werf itself or can freely be removed by other tools (docker rmi).

New commands to sync and switch stages

1. werf stages sync --from=:local|REPO --to=:local|REPO.

   • Command will copy stages from one stages-storage to another.
   • Command will copy only difference of stages from one stages-storage to another.
   • Furthermore command will copy multiple stages in parallel.
   • Command run result is idempotent: sync can be called multiple times, interrupted, then called again — the result will be the same.
   • There are delete options: --remove-source and --cleanup-local-cache, which controls whether werf will delete synced stages from source stages-storage and wheter werf will cleanup localhost from temporary docker images created during sync process.
   • This command can be used to download project stages-storage to the localhost for development purpose as well as backuping and migrating purposes.

2. werf stages switch-from-local --to=REPO

   • Command will automatically sync existing stages from :local stages storage to the specified REPO.
   • Command will block project from being used with :local stages-storage.
     • This means after werf stages switch-from-local is done, any werf command that specifies :local stages-storage for the project will fail preventing storing and using build results from different stages-storages.
     • Note that project is blocked after all existing stages has been synced.

How to migrate to distributed mode

Let's say you have some project in Gitlab CI, that uses werf. Steps:

1. Go to the directory where your project reside on the runner host.
2. Create an MR in the project, that removes --stages-storage :local param from all werf invocations. There is no need to set this param explicitly anymore, because werf ci-env will automatically set it to CI_REGISTRY_IMAGE/stages.
3. Run werf stages switch-from-local --to=CI_REGISTRY_IMAGE/stages where CI_REGISTRY_IMAGE is the address of container registry for your gitlab project. After this step all werf invocations from different git branches of the project that uses --stages-storage :local will refuse to work, because switch-from-local command sets a block to use :local.
4. Merge your MR with correct stages-storage param.
5. All git branches that need to run werf should rebase then onto the new changes to use new stages-storage, otherwise werf will refuse to use :local stages storage.

Other changes since last release v1.1.10-alpha.7

[docs] Remove --stages-storage :local from Gitlab CI docs
[ci] Cache GO modules
[tests] New tests
[tests] K8S/Cleanup Suite: test cleanup with deployed image
[tests] K8S/Guides Suite: actualize after each
Error: --stages-storage=ADDRESS param required
[tests] Helm/Render Suite: add extra annotations and labels check
[tests] Cleanup Suite: test stages storage repo

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support: distributed builds

Use kubertes locker for deploy and dismiss commands.

Fixes

[stapel] Fix apply patch on windows
~/gitLatestPatch error: /app/dir\test2: No such file or directory
[stapel] Fix custom mounts on windows
Error: daemon: Duplicate mount point: /path

Other

[tests] Imports suite: disable auto gc
[tests] Build/Dockerfile Suite: fix checksum without git test for darwin
git system darwin symlink mode differs from linux
[tests] CI-env Suite: take into account Github Actions environment
[tests] Build/Dockerfile Image Suite: fix typo
[tests] Force deleting test AWS ECR environment

[ci] Disable GCR test repositories cleanup

[docs] Regenerate reference cli docs

Installation

Linux amd64

Darwin amd64

Windows amd64

Github Actions support (alpha)

[ci-env] GitHub support.
[testing] CI-env Suite: add github.

werf ci-env github --as-file

Remote storages support: fix lockgate ConfigMap data key invalid

https://github.com/flant/lockgate/pull/12

Refactor and fix kubernetes cm key name

• Renamed LockHandle.ID to LockHandle.UUID to be more clear.
• Use "lockgate.io/SHA3_224(lock-name)" for ConfigMap key names due to ConfigMap names restrictions.

Other

Fix provided shell 'WERF_SHELL' not supported.
Retry docker pull/push when specific errors occur.

[logging] Turn off Docker Registry API debug messages by default.

[ci] Fix Cleanup Docker Hub test repositories.
[ci] Split test workflow into two independent workflows.
[ci] Disable GitHub Packages in cleanup test suite.

[docs] Regen commands cli documentation reference.

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support: fix kubernetes locks

Distributed builds can be used now by specifying --stages-storage=REPO_ADDRESS.

• Fixed bugs in kubernetes locks subsystem.
• Fixed panic in 'werf images purge' due to missed synchronization param setup

Werf ci-env refactor

Added --as-file option, which is more portable way to use werf ci-env. Example:

. $(werf ci-env gitlab --as-file)

Changed all docs and guides to use --as-file.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes

Fix gitLatestPatch signature not stable against index-line changes in git patches.

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support

Auto reset stages-storage-cache and retry action when docker-image has been deleted from stages-storage, but exists in the stages-storage-cache.

Fixes

Fix gitLatestPatch signature not stable to index line in patches.

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support

1. Refactor --synchronization param:

• -S shorthand;
• synchronization=:local by default when --stages-storage=:local;
• synchronization=kubernetes://werf-synchronization by default when stages-storage != :local;
• ability to specify arbitrary namespace with --synchronization kubernetes://mynamespace.

2. Auto reset stages-storage-cache and retry action when docker-image has been deleted from stages-storage, but exists in the stages-storage-cache.

Fixes

Fix gitLatestPatch signature not stable to index line in patches.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix release

Fix problems related to changes introduced in v1.1.9+fix4: signatures has been accidentally changed in v1.1.9+fix4.

Fixes that were made in v1.1.9+fix4 are also available in this v1.1.9+fix5 release:

[cleanup] Skip images with MANIFEST_UNKNOWN error

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix release

Fix problems related to changes introduced in v1.1.10-alpha.2: signatures has been accidentally changed in v1.1.10-alpha.2.

Fixes that were made in v1.1.10-alpha.2 are also available in this v1.1.10-alpha.3 release:

[cleanup] Skip images with MANIFEST_UNKNOWN error

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix build panics and errors when newly built image has been discarded

Example of error:

Error: phase build on image frontend stage gitCache handler failed: error checking stage      ↵
 gitCache is empty: invalid stage image: can not find git commit in stage image labels:        ↵
 delete stage image fdf27403-2c05-4966-850f-dbb0c79ff577 manually and retry the build

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix release

Fix problems related to changes introduced in v1.1.8+fix13: signatures has been accidentally changed in v1.1.8+fix13.

Fixes that were made in v1.1.8+fix13 are also available in this v1.1.8+fix14 release:

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should
  calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes

[cleanup] Skip images with MANIFEST_UNKNOWN error

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fixes

[cleanup] Skip images with MANIFEST_UNKNOWN error

[build][git] Handle "commits not present" line in git diffs

[build][git] Fix gitLatestPatch not built when previous stage has been discarded during build

• remove git-building-stage map from Conveyor;
• remove AfterSignatureCalculated Stage hook.

[dockerfile] Fix ONBUILD instruction processing

• dockerfile stage signature should take into account ONBUILD value;
• dockerfile stage that uses another one with ONBUILD instructions should calculate files checksum for COPY/ADD trigger instruction in addition to ONBUILD value.

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix dockerfile builder

[dockerfile] Fix git status checksum algorithm does not count symlinks properly
[dockerfile] Skip outside the build context wildcard till fail on docker build
[dockerfile] Take into account ignored files by .gitignore files
[dockerfile] Change checksum calculation without git logic

• Add relative file path and mode to checksum
• Add symlink link to checksum instead of linked data
  [dockerfile] Fix git status with not initialized submodules

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix dockerfile builder

[dockerfile] Fix git status checksum algorithm does not count symlinks properly
[dockerfile] Skip outside the build context wildcard till fail on docker build
[dockerfile] Take into account ignored files by .gitignore files
[dockerfile] Change checksum calculation without git logic

• Add relative file path and mode to checksum
• Add symlink link to checksum instead of linked data
  [dockerfile] Fix git status with not initialized submodules

Installation

Linux amd64

Darwin amd64

Windows amd64

Fix dockerfile builder

[dockerfile] Fix checksum calculation algorithm does not count ignored files (.gitignore).

https://github.com/flant/werf/issues/2315

Remote storages support: distributed builds support

'werf switch-from-local' command implemented

werf switch-from-local --to registry.mycompany.com/mygroup/myproject

This command will:

1. Sync existing local stages to the specified stages storage.
2. Globally lock stages and images after sync.
3. Set block to prevent any werf-command with --stages-storage=:local:
   user should change this param to the new stages-storage repo.
4. Sync existing local stages to the specified stages storage second time.

Implemented stages-storage-cache in ConfigMap

• Store stages-storage-cache in ns/werf-synchronization cm/werf-PROJECT_NAME
  when --synchronization=:kubernetes has been specified.
  • The same ConfigMap also used for kubernetes-based locks in annotations
    when --synchronization=:kubernetes has been specified.

Kubernetes based locks support

When --synchronization=:kubernetes werf will use kubernetes-locks in
ConfigMap for project: cm/werf-PROJECT_NAME.

Installation

Linux amd64

Darwin amd64

Windows amd64

Remote storages support: safe switch between alpha and beta werf versions

Invalidate stages-storage-cache directory when old cache directory exists. Remove old and current stages-storage-cache directories in this case.

Installation

Linux amd64

Darwin amd64

Windows amd64