[gardener]

🐛 Bug Fixes

• [OPERATOR] Deletion of Shoot is no longer wrongly blocked because of Bastion in the same Project that is not related to this Shoot. (gardener/gardener#5369, @ialidzhikov)
• [OPERATOR] When the ReversedVPN feature gate is disabled, the kube-apiserver-http-proxy secret is properly removed from the ShootState and the shoot's control plane. (gardener/gardener#5367, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.39.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.39.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.39.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.39.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.39.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.39.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.39.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.39.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which caused clusters which are being hibernated from succeeding because of a Gardener-Resource-Manager deployment issue. (gardener/gardener#5333, @timuthy)
• [DEVELOPER] Fix logging integration test to remove the IPs from loki.Spec.ClusterIPs (gardener/gardener#5335, @timuthy)
• [DEPENDENCY] A bug regarding the usage of the token requestor in the generic ControlPlane actuator package has been fixed. (gardener/gardener#5339, @rfranzke)

📰 Noteworthy

• [USER] Shoot clusters using Kubernetes 1.23 or above will continue to use the cgroupfs cgroup driver (the change to use systemd was reverted). (gardener/gardener#5334, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.39.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.39.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.39.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.39.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.39.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.39.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.39.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.39.1

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue causing the reconciliation of existing Shoot to be marked as Failed when the Secrets quota is exhausted is now fixed. (gardener/gardener#5230, @ialidzhikov)
• [OPERATOR] An issue preventing the seed-nginx-ingress-controller to pick the right ingress class for Seeds with K8s version >=1.22.0 is now fixed. (gardener/gardener#5226, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.36.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.36.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.36.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.36.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.36.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.36.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.36.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.36.2

[gardener]

⚠️ Breaking Changes

• [OPERATOR] Before upgrading to this Gardener version make sure that your existing Gardener runs on at least v1.37. (gardener/gardener#5121, @rfranzke)
• [DEVELOPER] The NewActuator function of the generic Worker actuator now takes two additional parameters: useTokenRequestor (set this to true only when running with Gardener >= 1.36), and useProjectedTokenMount (set this to true only when running with Gardener >= 1.37). They allow switching to the token requestor and projected ServiceAccount tokens instead of relying on static credentials for the machine-controller-manager. Caution: Make sure to adapt your Deployments similar to https://github.com/gardener/gardener/pull/5008/commits/e3cb8d84b9217667aaf5c5ce0ba60204ed4a4db3#diff-4ecb783d75e20fae3586a525a59b334f42474f9465af8defaac8e3da965cff3a when set to true. (gardener/gardener#5163, @rfranzke)
• [DEVELOPER] The NewActuator function of the generic ControlPlane actuator now takes four additional parameters: shootAccessSecrets and legacySecretNamesToCleanup, and exposureShootAccessSecrets and legacyExposureSecretNamesToCleanup (use them only when running with Gardener >= 1.36). They allow switching to the token requestor instead of relying on static client certificates for the control plane components like cloud-controller-manager. Caution: Make sure to adapt your Deployments similar to https://github.com/gardener/gardener/pull/5008/commits/e3cb8d84b9217667aaf5c5ce0ba60204ed4a4db3#diff-4ecb783d75e20fae3586a525a59b334f42474f9465af8defaac8e3da965cff3a when set to true. (gardener/gardener#5163, @rfranzke)
• [DEPENDENCY] The genericmutator.Ensurer interface methods concerning kubelets were extended with an additional parameter for the effective kubelet versions. Implementors of this interface should make use of this parameter instead of parsing the version from the Cluster object. (gardener/gardener#5256, @rfranzke)
• [DEPENDENCY] The generate-controller-registration.sh script does no longer take a path to a VERSION file but instead the VERSION directly. (gardener/gardener#5202, @rfranzke)
• [DEPENDENCY] The deprecated functions pkg/controllerutils.Try{Patch,Update} and friends have been removed. Please switch to the usual client methods for updating/patching without RetryOnConflict-semantics. See https://github.com/gardener/gardener/blob/master/docs/development/kubernetes-clients.md#dont-retry-on-conflict for more details on why their usage was discouraged. (gardener/gardener#4799, @timebertt)
• [DEPENDENCY] The extensions/pkg/controller.DeleteAllFinalizers function has been removed. You can use pkg/controllerutils.RemoveAllFinalizers instead. (gardener/gardener#4799, @timebertt)

✨ New Features

• [USER] New shoot clusters created with this version will have the TechnicalID (.status.technicalID) in the SAN list of the Kube-Apiserver server certificate. (gardener/gardener#5177, @timuthy)
• [USER] If the ForceRestore feature gate is enabled, the shoot's restoration to the destination seed during control plane migration will be forced if the preparation for migration in the source seed is not finished after a certain grace period and is considered unlikely to succeed ("bad case" scenario). (gardener/gardener#5123, @stoyanr)
• [OPERATOR] Gardener can now support shoot and seed clusters with Kubernetes version 1.23. In order to allow creation/update of 1.23 clusters you will have to update the version of your provider extension(s) to a version that supports 1.23 as well. Please consult the respective releases and notes in the provider extension's repository. (gardener/gardener#5255, @rfranzke)
• [OPERATOR] Add priority for default-domains to select which default-domain is used for new shoots (gardener/gardener#5213, @dergeberl)
• [OPERATOR] Gardener now cleans up VolumeSnapshots and VolumeSnapshotContents during cluster deletion. These resources are forcefully deleted after a 1h grace period which eventually lead to leaked snapshots on the cloud provider side. Hence, if the CSI-Snapshotter cannot delete affected snapshots successfully for 1h, operators/shoot-owners have to purge them manually. (gardener/gardener#5104, @timuthy)
• [DEVELOPER] Gardener can now support seed and shoot clusters with Kubernetes version 1.23. Extension developers have to prepare individual extensions as well to work with 1.23. (gardener/gardener#5255, @rfranzke)
• [DEVELOPER] A developer guideline on logging in Gardener components has been added. Please consult this document as a developer or reviewer to ensure consistency in our logs across the codebase. You can find the document here. (gardener/gardener#5175, @timebertt)
• [DEVELOPER] The Terraformer interface does now support a new UseProjectedTokenMount method for switching the terraformer pods to a projected ServiceAccount token. Set this to true only when running with Gardener >= 1.37. (gardener/gardener#5163, @rfranzke)
• [DEVELOPER] The ControllerInstallation controller in gardenlet is now populating the .gardener.version field when rendering Helm charts. Extension controllers can use this information to turn on or off certain features. The new General{Options,Config} structures introduced in the extensions/pkg/controller/cmd package can be used for exposing the --gardener-version flag. This allows to read the Gardener version information from the Helm chart values and use it. (gardener/gardener#5162, @rfranzke)
• [DEVELOPER] Two new functions UseTokenRequestor and UseServiceAccountTokenVolumeProjection have been introduced in the extensions/pkg/controller package. They can be used to decide (based on the used Gardener version) whether the respective features should be enabled. (gardener/gardener#5162, @rfranzke)

🐛 Bug Fixes

• [USER] An issue preventing kube-proxy Pods to be created when Shoot .spec.kubernetes.allowPrivilegedContainers=false is now fixed. (gardener/gardener#5263, @DockToFuture)
• [USER] An issue has been fixed which prevented the configuration of unmanaged primary DNS providers. (gardener/gardener#5241, @timuthy)
• [OPERATOR] Fixes bugs which could cause Seed conditions not being properly copied over to the corresponding Shoot due to a stale cache. (gardener/gardener#5305, @BeckerMax)
• [OPERATOR] A bug has been fixed which might cause extension controllers from not picking up resources which are in erroneous deletion states. (gardener/gardener#5297, @rfranzke)
• [OPERATOR] The shoot access Kubeconfig was reverted to a client cert based one in order to prevent the Dependency Watchdog scaling down the Kube-Controller-Manager accidentally. (gardener/gardener#5289, @timuthy)
• [OPERATOR] hvpa-controller component is reverted back to use static ServiceAccount tokens as currently the component cannot properly handle projected ServiceAccount tokens. (gardener/gardener#5228, @ialidzhikov)
• [OPERATOR] An issue preventing the seed-nginx-ingress-controller to pick the right ingress class for Seeds with K8s version >=1.22.0 is now fixed. (gardener/gardener#5223, @BeckerMax)
• [OPERATOR] Fix the 90% latency metric for the API Server dashboard (gardener/gardener#5215, @istvanballok)
• [OPERATOR] Fix the "CPU $recommendation vs CPU Usage" panel on the VPA dashboard (gardener/gardener#5211, @istvanballok)
• [OPERATOR] A bug has been fixed that caused etcd pods still to be active in the control plane even though the cluster was hibernated successfully. (gardener/gardener#5209, @timuthy)
• [OPERATOR] An issue has been fixed that caused the etcd-main pod to constantly crash on seed clusters that their Kube-Apiservers in the same cluster (usually not managed by Gardener). (gardener/gardener#5198, @timuthy)
• [OPERATOR] Finalizers are now properly added to the Worker resource at the start of a restore operation. (gardener/gardener#5187, @plkokanov)
• [OPERATOR] Adds network policy label to blackbox reporter so it can talk to the K8s APIServer (gardener/gardener#5186, @christianhuening)
• [DEPENDENCY] MachineClass CRD has been updated with nodeTemplate field. This is essential for scale-from-zero feature of CA to work. (gardener/gardener#5266, @himanshu-kun)

🏃 Others

• [USER] Cluster Autoscaler flag --ignore-taint is exposed at Shoot API level with which user can specify a taint to ignore in node templates when considering to scale a node group. (gardener/gardener#5169, @AxiomSamarth)
• [OPERATOR] The following image is updated: (gardener/gardener#5271, @wyb1)
  • grafana/grafana: 7.5.11 -> 7.5.12
• [OPERATOR] The create-shoot TestDefinition does now specify the --networking-type flag that can be configured via the $NETWORKING_TYPE env var. (gardener/gardener#5257, @hendrikKahl)
• [OPERATOR] The portforward and exec subresource latency is excluded from the KubeApiServerLatency alert (gardener/gardener#5236, @istvanballok)
• [OPERATOR] The AdminKubeconfigRequest feature gate in the apiserver has been promoted to beta and is now enabled by default. (gardener/gardener#5220, @shafeeqes)
• [OPERATOR] The UseDNSRecords feature gate in the apiserver, controllermanager and gardenlet has been promoted to beta and is now enabled by default. Please see the support forDNSRecords resources in provider extensions documentation. (gardener/gardener#5219, @shafeeqes)
• [OPERATOR] Added the landscaper component to deploy the Gardener Control Plane. For more information, please read the documentation: landscaper/pkg/controlplane/README.md (gardener/gardener#5130, @danielfoehrKn)
• [DEVELOPER] The Golang version was bumped to 1.17.6. (gardener/gardener#5231, @ialidzhikov)
• [DEPENDENCY] Add Created condition to DNSRecord to allow deletion on unsuccessful creation in infrastructure (gardener/gardener#5234, @MartinWeindel)

📰 Noteworthy

• [USER] Shoot clusters using Kubernetes 1.23 or above will use the systemd cgroup driver as recommended by the community. (gardener/gardener#5255, @rfranzke)
• [USER] It is now possible to deploy and run Gardener entirely locally on your machine (i.e., without any real infrastructure/cloud provider involved). Make sure to check out this document if you are interested! (gardener/gardener#5115, @rfranzke)
• [OPERATOR] gardener-resource-manager does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 24h. (gardener/gardener#5138, @rfranzke)
• [OPERATOR] cloud-config-downloader does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 90d. (gardener/gardener#5121, @rfranzke)
• [DEVELOPER] A new provider-local Gardener extension has been introduced. It allows to run Gardener entirely locally on your machine (i.e., without any real infrastructure/cloud provider involved). This document describes how it works and which limitations exist. (gardener/gardener#5115, @rfranzke)
• [DEVELOPER] It is now possible to develop Gardener entirely locally on your machine (i.e., no existing seed cluster on a real infrastructure/cloud provider required). Make sure to check out this document if you are interested! (gardener/gardener#5115, @rfranzke)
• [DEPENDENCY] In order to prepare provider extensions with CSI migration for supporting different Kubernetes versions per worker pools, the validation for Shoot resources should be extended according to this document. (gardener/gardener#5256, @rfranzke)
• [DEPENDENCY] The GetOrCreateShootKubeconfig function in the extensions/pkg/util package is deprecated since it generates a kubeconfig with a static client certificate. Switch to the token requestor instead. (gardener/gardener#5162, @rfranzke)

[autoscaler]

🏃 Others

• [USER] Gardener Autoscaler now supports generic scale from zero. Traditionally, nodes are scaled by Autoscaler by generating node template for a new node that would accomodate the pending pods. This node template is generated either by referring to the node spec of the already existing node in the nodeGroup or by referring to the cloud provider specific catalogue which provides the node details like cpu, gpu, memory, region, zone, instanceType etc. The latter scenario is the scale from zero scenario where there are zero nodes in the nodeGroup that is expected to scale up and so far the existing logic limited this feature to AWS and Azure only. (gardener/autoscaler#102, @AxiomSamarth)
  • Now, with the introduction of nodeTemplate property in the MachineClass, the scale from zero feature has been extended generically across all providers.

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.39.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.39.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.39.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.39.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.39.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.39.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.39.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.39.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] Fixes bugs which could cause Seed conditions not being properly copied over to the corresponding Shoot due to a stale cache. (gardener/gardener#5309, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.6
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.6
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.6
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.6
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.6
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.6
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.6
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.6

[gardener]

🐛 Bug Fixes

• [OPERATOR] The shoot access Kubeconfig was reverted to a client cert based one in order to prevent the Dependency Watchdog scaling down the Kube-Controller-Manager accidentally. (gardener/gardener#5287, @timuthy)
• [OPERATOR] Fixes bugs which could cause Seed conditions not being properly copied over to the corresponding Shoot due to a stale cache. (gardener/gardener#5308, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.38.4
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.38.4
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.38.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.38.4
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.38.4
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.38.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.38.4
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.38.4

[gardener]

✨ New Features

• [DEVELOPER] Support for the resources.gardener.cloud/preserve-resources annotation was added for Jobs, CronJobs, and DaemonSets. (gardener/gardener#5264, @rfranzke)

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that caused etcd pods still to be active in the control plane even though the cluster was hibernated successfully. (gardener/gardener#5238, @timuthy)
• [DEVELOPER] The resources.gardener.cloud/preserve-resources annotation does now work properly for StatefulSets. (gardener/gardener#5264, @rfranzke)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.5
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.5
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.5
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.5
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.5
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.5
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.5
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.5

[gardener]

🐛 Bug Fixes

• [USER] An issue preventing kube-proxy Pods to be created when Shoot .spec.kubernetes.allowPrivilegedContainers=false is now fixed. (gardener/gardener#5265, @ialidzhikov)
• [OPERATOR] A bug has been fixed that caused etcd pods still to be active in the control plane even though the cluster was hibernated successfully. (gardener/gardener#5237, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.38.3
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.38.3
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.38.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.38.3
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.38.3
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.38.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.38.3
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.38.3

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue causing the reconciliation of existing Shoot to be marked as Failed when the Secrets quota is exhausted is now fixed. (gardener/gardener#5229, @ialidzhikov)
• [OPERATOR] An issue preventing the seed-nginx-ingress-controller to pick the right ingress class for Seeds with K8s version >=1.22.0 is now fixed. (gardener/gardener#5225, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.4
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.4
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.4
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.4
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.4
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.4

[gardener]

🐛 Bug Fixes

• [OPERATOR] hvpa-controller component is reverted back to use static ServiceAccount tokens as currently the component cannot properly handle projected ServiceAccount tokens. (gardener/gardener#5232, @ialidzhikov)
• [OPERATOR] An issue preventing the seed-nginx-ingress-controller to pick the right ingress class for Seeds with K8s version >=1.22.0 is now fixed. (gardener/gardener#5224, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.38.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.38.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.38.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.38.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.38.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.38.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.38.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.38.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] Finalizers are now properly added to the Worker resource at the start of a restore operation. (gardener/gardener#5196, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.36.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.36.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.36.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.36.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.36.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.36.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.36.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.36.1

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue has been fixed that caused the etcd-main pod to constantly crash on seed clusters that their Kube-Apiservers in the same cluster (usually not managed by Gardener). (gardener/gardener#5199, @timuthy)
• [OPERATOR] Finalizers are now properly added to the Worker resource at the start of a restore operation. (gardener/gardener#5195, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.3
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.3
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.3
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.3
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.3
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.3

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue has been fixed that caused the etcd-main pod to constantly crash on seed clusters that their Kube-Apiservers in the same cluster (usually not managed by Gardener). (gardener/gardener#5200, @timuthy)
• [OPERATOR] Finalizers are now properly added to the Worker resource at the start of a restore operation. (gardener/gardener#5194, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.38.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.38.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.38.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.38.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.38.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.38.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.38.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.38.1

[gardener]

⚠️ Breaking Changes

• [USER] New Shoots can no longer specify __internal for the apiVersion in their InfrastructureConfig. For compatibility reasons, existing Shoots with this configuration can still be updated. (gardener/gardener#4927, @voelzmo)
• [OPERATOR] The etcd backup compaction feature has been disabled by default. It can by enabled anytime via the etcdConfig section in the Gardenlet-Componentconfig. (gardener/gardener#5144, @timuthy)
• [OPERATOR] Before upgrading to this Gardener version make sure that your existing Gardener runs on at least v1.37. (gardener/gardener#5128, @rfranzke)
• [OPERATOR] gardener-apiserver's DisallowKubeconfigRotationForShootInDeletion feature gate that is GA since v1.36 is unconditionally enabled, and can no longer be specified in the gardener-apiserver's configuration. (gardener/gardener#5124, @acumino)
• [OPERATOR] The SeedAuthorizer and SeedRestriction features do no longer support "ambiguous" gardenlets (i.e., gardenlets responsible for multiple seed clusters) since this feature was dropped already with Gardener v1.27. In case you have activated these features then you have to make sure that you deploy a dedicated gardenlet per seed cluster and that they don't use a client certificate with the (now removed) gardener.cloud:system:seeds:<ambiguous> common name before updating to this Gardener version. This document describes how to make the gardenlet regenerate its client certificate after you have reconfigured it. (gardener/gardener#5093, @rfranzke)

✨ New Features

• [OPERATOR] The error code detection has been enhanced for ManagedResource objects that are stuck when a shoot is deleted. This enables Gardener to assign the corresponding error code(s) to the shoot object. (gardener/gardener#5111, @timuthy)
• [OPERATOR] Two fields LeaseDurationSeconds and LeaseResyncSeconds added under SeedControllerConfiguration to make Seed lease and duration configurable. Both field have default value of 2 seconds. (gardener/gardener#5092, @ary1992)
• [DEVELOPER] Support for the resources.gardener.cloud/preserve-resources annotation was added for Jobs, CronJobs, and DaemonSets. (gardener/gardener#5131, @rfranzke)
• [DEVELOPER] The TokenRequestor controller (part of gardener-resource-manager) can now optionally sync the tokens into a Secret in the target cluster (see this document for more information). (gardener/gardener#5084, @rfranzke)

🐛 Bug Fixes

• [OPERATOR] Gardener-Resource-Manager does not try overwrite the immutable field .spec.selector of Job objects anymore. (gardener/gardener#5167, @timuthy)
• [OPERATOR] An issue has been fixed that prevented etcd worker counts from being set correctly in the GardenletConfiguration. (gardener/gardener#5151, @timuthy)
• [OPERATOR] Fixed a bug that prevented shoot deletion with UseDNSRecords enabled if it was never previously reconciled with this feature gate enabled. (gardener/gardener#5135, @stoyanr)
• [OPERATOR] An issue causing the reconciliation of existing Shoot to be marked as Failed when the Secrets quota is exhausted is now fixed. (gardener/gardener#5126, @ialidzhikov)
• [OPERATOR] Endless waits are now avoided when deleting DNSProvider, DNSEntry, and DNSOwner resources. (gardener/gardener#5119, @stoyanr)
• [OPERATOR] A bug has been fixed which caused the gardener-controller-manager to hang forever in case the internal domain secret got deleted before the last ControllerRegistration. (gardener/gardener#5105, @rfranzke)
• [OPERATOR] Fix failing health check if loki is disabled in gardenlet configuration (gardener/gardener#5103, @dergeberl)
• [OPERATOR] A bug in the SeedAuthorizer has been fixed which allowed gardenlets to unconditionally delete BackupBuckets. (gardener/gardener#5091, @rfranzke)
• [DEVELOPER] The ManagedResource CRD in the /example dir was fixed. (gardener/gardener#5168, @timuthy)
• [DEVELOPER] The resources.gardener.cloud/preserve-resources annotation does now work properly for StatefulSets. (gardener/gardener#5131, @rfranzke)

📖 Documentation

• [USER] The usage document for NodeLocalDNS is now enhanced with explanation on effects during shoot reconciliation and recommendation for the same. (gardener/gardener#5070, @ashwani2k)
• [OPERATOR] Added figure to Gardener concept docs depicting how the more relevant resources like cloudprofile, seed, project, shoot`, etc. relate to each other. (gardener/gardener#5137, @vlerenc)
• [OPERATOR] A Gardener landscape operator can now consult to the SecretBinding ProviderController on how to enable the controller that helps on automatically setting the provider type of existing SecretBindings and on how to enable validation checks (required/immutable field) for the new provider type field. (gardener/gardener#5058, @ialidzhikov)

🏃 Others

• [USER] The SecretBinding resource does now contain a new field that denotes its provider type - provider.type. (gardener/gardener#5058, @ialidzhikov)
• [OPERATOR] Add rateLimit fields to CRD dnsproviders.dns.gardener.cloud (gardener/gardener#5165, @MartinWeindel)
• [OPERATOR] The namespace and pod labels are kept for the metrics in the seed-prometheus. (gardener/gardener#5148, @istvanballok)
• [OPERATOR] The gardener-resource-manager VPA does now specify minAllowed values to prevent too low resource recommendations from VPA that lead to OOM. (gardener/gardener#5116, @ialidzhikov)
• [OPERATOR] A final full snapshot is created and waited upon while copying backups for control plane migration "good case" and "bad case" scenarios. (gardener/gardener#5095, @stoyanr)
• [OPERATOR] Update istio to version 1.12.0. (gardener/gardener#5080, @DockToFuture)
• [OPERATOR] Added heatmaps for dns request latency to dns dashboards. (gardener/gardener#5078, @ScheererJ)
• [OPERATOR] The ManagedSeed controller does no longer add the Shoot cloud provider credentials into the Seed kubeconfig Secret (Seed .spec.secretRef). (gardener/gardener#5073, @ialidzhikov)
• [OPERATOR] Add access logs to istio ingress-gateway. (gardener/gardener#5071, @DockToFuture)
• [OPERATOR] gardener-controller-manager and gardenlet have started switching from logrus to zap. Make sure to use the json log format to have harmonized logging during the migration period. (gardener/gardener#5057, @timebertt)
• [DEVELOPER] The Golang version was bumped to 1.17.5. (gardener/gardener#5152, @ialidzhikov)
• [DEVELOPER] New check-docforge step will be executed on each PR in the CI/CD (gardener/gardener#5108, @Kristian-ZH)
• [DEVELOPER] The SeedAuthorizer does now allow DELETE requests if the resource does not exist in the system. (gardener/gardener#5091, @rfranzke)
• [DEVELOPER] When the gardenlet is locally started against the nodeless setup then a seed-specific client certificate is generated. This improves the accuracy of the SeedAuthorizer feature when it is activated. (gardener/gardener#5086, @rfranzke)

📰 Noteworthy

• [USER] In order to reduce the validity of client certificates used by kubelets running on the worker nodes of shoot clusters, the expiration duration for certificates issued via CertificateSigningRequests has been reduced from 1y to 30d. A custom expiration duration per CertificateSigningRequest can be set via the .spec.expirationSeconds fields (available from Kubernetes v1.22). (gardener/gardener#5096, @rfranzke)
• [OPERATOR] An official version skew policy document was added. You can take a look here. (gardener/gardener#5129, @rfranzke)
• [OPERATOR] All seed system components deployed by Gardener have been switched to projected ServiceAccount tokens (instead of continued usage of static tokens). (gardener/gardener#5128, @rfranzke)
• [OPERATOR] All shoot system components deployed by Gardener have been switched to projected ServiceAccount tokens (instead of continued usage of static tokens). (gardener/gardener#5099, @rfranzke)
• [OPERATOR] gardenlet and extension controllers do no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5012, @rfranzke)
• [OPERATOR] dependency-watchdog-probe does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5011, @rfranzke)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] Took care of a bug where contexts were created multiple times without being cancelled. This caused a significant higher memory and CPU footprint. (gardener/etcd-backup-restore#409, @aaronfern)

[etcd-druid]

🏃 Others

• [OPERATOR] Updated labels used in compaction job to differentiate them from etcd pods. This allows for pod scheduling policies to schedule compaction jobs on predetermined nodes (gardener/etcd-druid#270, @aaronfern)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.38.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.38.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.38.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.38.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.38.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.38.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.38.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.38.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue has been fixed that prevented etcd worker counts from being set correctly in the GardenletConfiguration. (gardener/gardener#5155, @timuthy)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] Took care of a bug where contexts were created multiple times without being cancelled. This caused a significant higher memory and CPU footprint. (gardener/etcd-backup-restore#409, @aaronfern)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.2
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.2

[gardener]

🏃 Others

• [OPERATOR] The gardener-resource-manager VPA does now specify minAllowed values to prevent too low resource recommendations from VPA that lead to OOM. (gardener/gardener#5140, @timebertt)
• [OPERATOR] Fixed a bug that prevented shoot deletion with UseDNSRecords enabled if it was never previously reconciled with this feature gate enabled. (gardener/gardener#5139, @timebertt)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.1

[gardener]

⚠️ Breaking Changes

• [DEVELOPER] The monitoring scrape configurations (particularly the tls_config and authorization section) of extension controllers must be adapted such that they match the example in https://github.com/gardener/gardener/blob/master/docs/extensions/logging-and-monitoring.md#extensions-monitoring-integration. (gardener/gardener#5008, @rfranzke)

✨ New Features

• [USER] It's now possible to override the grace periods for the infrastructure cleanup wait step in the shoot deletion by specifying the shoot.gardener.cloud/cleanup-infrastructure-resources-grace-period-seconds annotation on the Shoot (default behaviour: "300"). Please be aware that overriding this value might lead to orphaned infrastructure artifacts. (gardener/gardener#5044, @rfranzke)
• [USER] If the WorkerPoolKubernetesVersion feature gate is enabled then it is possible to specify the Kubernetes version per worker pool for Shoots by setting .spec.provider.workers[].kubernetes.version. Please consult this document for more information. (gardener/gardener#4971, @rfranzke)
• [OPERATOR] It is now possible to disable the dependency-watchdogs in the seed cluster by configuring the .spec.settings.dependencyWatchdog section. Please consult the documentation for more information. (gardener/gardener#5075, @rfranzke)
• [OPERATOR] A new shoot status label value unknown that corresponds to Unknown conditions has been introduced. (gardener/gardener#5041, @stoyanr)
• [OPERATOR] Updating the shoot status label has been moved to GCM. (gardener/gardener#5022, @stoyanr)
• [OPERATOR] The copying of seed conditions to shoots registered as seeds has been moved to GCM. (gardener/gardener#5013, @stoyanr)
• [OPERATOR] Node resource capacity (cpu, gpu, memory) are now filled in the extensionsv1alpha1.Worker resource under spec.pools[].nodeTemplate.capacity from the CloudProfile for the corresponding machine type. These resources will be carried to the MachineClass and will later be used by cluster-autoscaler for scale-from-zero. (gardener/gardener#4980, @himanshu-kun)
• [OPERATOR] The WorkerPoolKubernetesVersion feature gate must only be enabled when all provider extensions explicitly announce in their release notes that they support specific Kubernetes versions per worker pool. Otherwise, worker nodes of shoot clusters might be misconfigured or rolled out unexpectedly. (gardener/gardener#4971, @rfranzke)
• [DEVELOPER] It is now possible to provide an ObjectSelector when registering an extension webhook by configuring github.com/gardener/gardener/extensions/pkg/webhook.Webhook. (gardener/gardener#5043, @rfranzke)

🐛 Bug Fixes

• [USER] The node bootstrapping mechanism has been enhanced to retry failed hyperkube preload operations. (gardener/gardener#5019, @timebertt)
• [OPERATOR] The validation whether referenced extension types are actually registered in the system is now only performed when a resource is newly created or when its spec section has changed. (gardener/gardener#5049, @rfranzke)
• [OPERATOR] A bug has been fixed which could prevent proper deletion of ControllerInstallations when a Seed was marked for deletion. (gardener/gardener#5047, @rfranzke)
• [DEVELOPER] A bug has been fixed which caused issues when hack/generate-seed-crds.sh was called with an empty <file-name-prefix>. (gardener/gardener#5053, @timuthy)
• [DEVELOPER] Parts of make-generate are no longer executed in parallel. (gardener/gardener#5020, @BeckerMax)

🏃 Others

• [OPERATOR] Removed log output of readiness/liveness probes from vpn-seed-server log. (gardener/gardener#5087, @ScheererJ)
• [OPERATOR] Fixed several panels and corresponding metric ingestion in the istio mesh dashboard. (gardener/gardener#5077, @ScheererJ)
• [OPERATOR] It's now possible to configure the SerializeImagePulls fields for the kubelet configuration (defaults: true) in the Shoot API via .spec.{provider.workers[]}.kubernetes.kubelet.SerializeImagePulls. (gardener/gardener#5074, @shafeeqes)
• [OPERATOR] Provide access logs for workload requests going through apiserver-proxy. (gardener/gardener#5065, @ScheererJ)
• [OPERATOR] Access logging in api-server-proxy now works. (gardener/gardener#5060, @ScheererJ)
• [OPERATOR] Enable metrics collection of and introduce dashboard for envoy proxy side car used in reversed vpn (gardener/gardener#5055, @ScheererJ)
• [OPERATOR] Expose a metric on frequent container restarts in the seed (gardener/gardener#5048, @istvanballok)
• [OPERATOR] Expose a metric for inconsistent persistent volume sizes (gardener/gardener#5040, @istvanballok)
• [OPERATOR] ⚠️ Due to the github.com/gardener/etcd-druid update, etcd pods of shoot clusters will be restarted during their next reconciliation (e.g. within next maintenance time window, manual reconciliation, spec updates). (gardener/gardener#5037, @abdasgupta)
• [OPERATOR] Enable access log of envoy proxy side car of reversed vpn server for better visibility. (gardener/gardener#5035, @ScheererJ)
• [OPERATOR] The condition handling was improved in Gardener which sometimes resulted in conditions having outdated reasons or messages. (gardener/gardener#5021, @timuthy)
• [OPERATOR] During the restoration phase of control plane migration ETCD backups will be copied from the backup bucket of the source seed to the backup bucket of the destination seed. (gardener/gardener#4894, @plkokanov)
• [OPERATOR] The Gardenlet wait timeout for infrastructure reconciliation has been increased from 30 seconds to 3 minutes. This should reduce unnecessary reconciliations and improve the UX when updating the infrastructure of failed Shoots to a valid configuration. (gardener/gardener#4881, @danielfoehrKn)
• [OPERATOR] The gardenlet shoot controller will now set the owner check configuration parameters in the etcd-main Etcd resource. This will cause etcd-backup-restore to disable the cluster if the owner domain name no longer resolves to the specified owner ID. The creation and checking of owner DNS record can be disabled via the spec.settings.ownerChecks seed setting. (gardener/gardener#4813, @stoyanr)

📰 Noteworthy

• [OPERATOR] kube-rbac-proxy does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5010, @rfranzke)
• [OPERATOR] vpa-{admission-controller,recommender,updater} do no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5009, @rfranzke)
• [OPERATOR] kube-state-metrics does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. prometheus has such a token as well, but for backwards-compatibility it also still has access to its client certificate (this will be dropped in the future). (gardener/gardener#5008, @rfranzke)
• [OPERATOR] kube-controller-manager and cluster-autoscaler do no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5007, @rfranzke)
• [OPERATOR] The gardener-resource-manager's TokenInvalidator and the ProjectedTokenMount webhooks are now enabled for the seed and shoot clusters. (gardener/gardener#5002, @rfranzke)
• [DEVELOPER] The ReversedVPN feature gate is now activated by default for local development. (gardener/gardener#5045, @rfranzke)
• [DEVELOPER] In order to make provider extensions ready for properly handling Shoot clusters with overridden Kubernetes versions per worker pool, you need to revendor the extensions library. (gardener/gardener#4971, @rfranzke)

[etcd-backup-restore]

🏃 Others

• [USER] Compaction job will now throw warning instead of error if revisions are already compacted (gardener/etcd-backup-restore#358, @shreyas-s-rao)
• [OPERATOR] The backup-restore sidecar will now check if the owner domain name resolves to the specified owner ID and if not, take a final full snapshot and disable the cluster. (gardener/etcd-backup-restore#383, @stoyanr)
• [OPERATOR] Added '--etcd-snapshot-timeout' and '--etcd-defrag-timeout' CLI flags in 'server' and 'compact' subcommands (gardener/etcd-backup-restore#361, @ishan16696)
• [OPERATOR] Updated number of chunks while uploading to never exceed the cloud provider limits. (gardener/etcd-backup-restore#334, @abdasgupta)

📰 Noteworthy

• [OPERATOR] Added health package to allow backup-restore to renew member leases to indicate member health and snapshot leases to indicate snapshots being taken successfully (gardener/etcd-backup-restore#382, @aaronfern)
  • Added flags to the server and compact subcommand. --enable-snapshot-lease-renewal to enable snapshot lease renewal, --enable-member-lease-renewal to enable member lease updates, full-snapshot-lease-name to specify the full snapshot lease name, and delta-snapshot-lease-name to specify the delta snapshot lease name
  • Pod name and namespace has to be passed via environment variables (POD_NAME and POD_NAMESPACE) when running the server subcommand when --enable-member-lease-renewal flag is set to true
  • Pod namespace needs to be passed via an environment variable (POD_NAMESPACE) when running the server subcommand when --enable-snapshot-lease-renewal flag is set to true
  • Pod namespace needs to be passed via an environment variable (POD_NAMESPACE) when running the compact subcommand when --enable-snapshot-lease-renewal flag is set to true
• [OPERATOR] Added compaction command support to ETCD Backup Restore. Users can now use compaction sub command with etcdbrctl tool to compact basesnapshot and all it's subsequent deltasnapshot to one single compacted snapshot. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] Snapshots are not stored under separate prefix (Backup-XXX) in backup storage. Instead, all of the snapshots are stored under v2 backup version prefix in a flat structure. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] Support for v1 backup version is still there but only for restoration and snapshot garbage collection. So backup storages that had v1 backup version and where snapshots were stored under separate prefix (Backup-XXX), can still be used with compaction sub command (as well as for restoration). However, the new snapshots will be stored under v2 backup version prefix. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] The support for the v1 backup structure with separate prefix (Backup-XXX) will be dropped in a subsequent release. (gardener/etcd-backup-restore#301, @abdasgupta)

[etcd-custom-image]

📰 Noteworthy

• [OPERATOR] Update etcd version from v3.4.13-bootstrap to v3.4.13-bootstrap-1. ⚠️ This will cause a restart of the etcd. (gardener/etcd-custom-image@10c534ecbfe7)

[etcd-druid]

⚠️ Breaking Changes

• [OPERATOR] CronJob is no more used to schedule compaction job at regular interval. Instead, we are using Job. (gardener/etcd-druid#235, @abdasgupta)
  • Therefore, BackupCompactionSchedule field is removed from ETCD backup spec, as it was only necessary for scheduling CronJob.
• [DEVELOPER] The sub-module github.com/gardener/etcd-druid/api has been removed. Please use github.com/gardener/etcd-druid instead if your module(s) depend on etcd-druid. (gardener/etcd-druid#244, @timuthy)
• [DEVELOPER] Switch to github.com/gardener/etcd-druid-api if you vendor only the API of etcd-druid. (gardener/etcd-druid#169, @amshuman-kr)

✨ New Features

• [USER] druid will now also reconcile a serviceaccount, a role, and a rolebinding as part of it's etcd reconcile flow and associate it with the etcd pod (gardener/etcd-druid#233, @aaronfern)
• [OPERATOR] A new flag --enable-backup-compation has been introduced which globally enables automatic compaction of backups. (gardener/etcd-druid#258, @timuthy)
• [OPERATOR] A new controller named lease controller has been introduced. Lease controller will be responsible for creating compaction job based on the delta event lease. (gardener/etcd-druid#235, @abdasgupta)
  • For this, two new Leases are introduced: One to hold the value of the latest full snapshot revision and one for the last delta revision.
• [OPERATOR] Added support for etcd druid to regularly schedule backup compactions via configurable etcd spec spec.backup.compactionSchedule (gardener/etcd-druid#197, @aaronfern)
• [OPERATOR] Various condition and etcd member checks have been added to Etcd-Druid. The results of those checks will be reflected in the etcd.status sub-resource. (gardener/etcd-druid#188, @timuthy)
  • Conditions:
  • Ready check: Checks if resource has enough Ready members in status.members to fulfill the quorum.
  • AllMembersReady check: Checks if all members in status.members are Ready.
  • Members:
  • Ready check: Treats the LastUpdateTime as a heartbeat and checks if it is within the expected time range (configurable via --etcd-member-threshold).
• [OPERATOR] A re-sync mechanism has been added for the Custodian controller. The new flag --custodian-sync-period (default 30s) controls the duration after which the Custodian controller re-enqueues etcd resources for reconciliation. This can be considered as a health check interval. (gardener/etcd-druid#188, @timuthy)
• [OPERATOR] It is now possible to configure the count of custodian controller workers by --custodian-workers. (gardener/etcd-druid#180, @timuthy)

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that led to multiple update conflicts when the etcd resource was reconciled. (gardener/etcd-druid#264, @timuthy)
• [OPERATOR] The operation annotation is now removed before reconciling the Etcd resource. (gardener/etcd-druid#205, @shreyas-s-rao)
• [OPERATOR] If operation annotation is to be honoured, the reconciliation predicates now match only if either operation annotation is present, or last operation didn't succeed or if the resource is undergoing deletion. No other change (if not accompanied by any of these conditions) will trigger reconciliation. (gardener/etcd-druid#202, @amshuman-kr)

📖 Documentation

• [OPERATOR] The multi-node etcd proposal has been updated and now considers ephemeral volumes not being an integral part of the multi-node feature set. (gardener/etcd-druid#256, @timuthy)

🏃 Others

• [OPERATOR] It is now possible to specify owner check parameters in the Etcd resource. (gardener/etcd-druid#239, @stoyanr)
• [OPERATOR] Remove finalizers from Etcd resources after waiting for statefulset. (gardener/etcd-druid#222, @amshuman-kr)
• [OPERATOR] Added a new CLI flag --enable-compaction-tempfs to etcd druid to enable tempfs in the compaction job volumeMount (defaults to false) (gardener/etcd-druid#220, @aaronfern)
• [OPERATOR] Two new fields added to etcd spec, etcdSnapshotTimeout and etcdDefragTimeout which configure the snapshotter timeout and defragmentation timeout respectively of etcd-backup-restore (gardener/etcd-druid#216, @aaronfern)
• [OPERATOR] Druid now fetches Lease resources in order to derive the readiness state of an etcd cluster member. This serves as a preparation for the etcd multi-node feature. (gardener/etcd-druid#214, @timuthy)
• [OPERATOR] Updated number of chunks while uploading to never exceed the cloud provider limits. (gardener/etcd-druid#182, @amshuman-kr)
• [OPERATOR] The reconciliation loop count of the custodian controller has been improved. Now, reconciliation only happens at relevant create/update/delete events. (gardener/etcd-druid#180, @timuthy)
• [OPERATOR] Removed synchronisation before updating ETCD status. (gardener/etcd-druid#174, @abdasgupta)
• [DEVELOPER] A new Make target check-generate has been added to check if generated code and the vendor dir are up-to-date. (gardener/etcd-druid#177, @timuthy)

📰 Noteworthy

• [OPERATOR] A new module github.com/gardener/etcd-druid/api can be used to get the API definitions. (gardener/etcd-druid#169, @amshuman-kr)

• Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.0

[gardener]

⚠️ Breaking Changes

• [DEPENDENCY] The default terraformer version in package github.com/gardener/gardener/extensions/pkg/terraformer is now changed to v2. Consumers of this package can still configure the terraformer version using the funcs UseV1 and UseV2(both of these functions are deprecated). (gardener/gardener#4930, @acumino)
• [DEPENDENCY] hack/install-requirements.sh is deprecated and will be removed in a future version. Other repos can reuse newly added make recipes as a replacement to build/install tool binaries by including hack/tools.mk in their top-level make file. (gardener/gardener#4879, @timebertt)

✨ New Features

• [OPERATOR] Add setting to disable Loki deployments in Garden and Shoot namespace, if fluent-bit is used to log to a central instance. (gardener/gardener#4949, @dergeberl)
• [OPERATOR] Allow configuration of remote write endpoint for Shoot Prometheus to collect all Shoot metrics on a central instance (gardener/gardener#4935, @dergeberl)
• [OPERATOR] gardener-resource-manager can now auto-mount projected ServiceAccount tokens into Pods in case their referenced ServiceAccount sets .automountServiceAccountToken=false. Please consult this document for more information. (gardener/gardener#4873, @rfranzke)
• [DEVELOPER] Developers don't need to run make install-requirements anymore. Instead, the needed binaries are built/installed on the fly to a local directory as needed. (gardener/gardener#4879, @timebertt)

🐛 Bug Fixes

• [USER] It should now be possible to delete shoots with invalid DNS provider credentials if there are no DNSEntry resources with non-empty Status.Targets for the corresponding provider in the shoot namespace. (gardener/gardener#4962, @stoyanr)
• [USER] Fixed acquiring and releasing the machine class credentials secret causing the MCM finalizer not to be properly removed from the secret in some circumstances. (gardener/gardener#4961, @stoyanr)
• [OPERATOR] Fix indentation for extensions Grafana dashboards (gardener/gardener#5023, @MartinWeindel)
• [OPERATOR] The bootstrap-token placeholder will no longer be present in the original operatingsystemconfig (gardener/gardener#4994, @BeckerMax)
• [OPERATOR] An issue causing seed bootstrap to fail when the ManagedIstio feature gate is disabled is now fixed. (gardener/gardener#4991, @ScheererJ)
• [OPERATOR] When a Shoot is deleted and the corresponding extensions.gardener.cloud/v1alpha1.ControlPlane resource already has a .metadata.deletionTimestamp it will not be redeployed if the deletion flow is restarted. (gardener/gardener#4976, @plkokanov)
• [OPERATOR] Fixed a bug which prevented Shoot deletion on Seeds with non-tolerated taints. (gardener/gardener#4911, @voelzmo)

🏃 Others

• [OPERATOR] github.com/gardener/gardener/extensions/pkg/terraformer does now log by default the termination message of the Terraformer Pod (or its logs) when the Terraformer Pod finishes with error. (gardener/gardener#5016, @ialidzhikov)
• [OPERATOR] Update envoy proxy to version 1.18.4 (gardener/gardener#5006, @ScheererJ)
• [OPERATOR] Coredns in the shoot cluster is updated to version 1.8.6. (gardener/gardener#5004, @ScheererJ)
• [OPERATOR] HA-settings in the Gardener Control Plane chart have been enhanced. (gardener/gardener#5000, @timebertt)
• [OPERATOR] The ShootCoreAddonRestarter is now disabled by default in the gardener controlplane chart. (gardener/gardener#4997, @vpnachev)
• [OPERATOR] The external authentication server (required for reversed VPN tunnel) is now being cleaned-up if the ManagedIstio feature gate is turned off. (gardener/gardener#4996, @timuthy)
• [OPERATOR] Allow users of node-local-dns to switch between udp and tcp for upstream dns queries. (gardener/gardener#4988, @ScheererJ)
• [OPERATOR] The DisallowKubeconfigRotationForShootInDeletion feature gate in the gardener-apiserver has been promoted to GA. (gardener/gardener#4984, @acumino)
• [OPERATOR] Fixed a bug in the node bootstrapping for shoots without DNS records. (gardener/gardener#4982, @timebertt)
• [OPERATOR] Add automaxprocs to the controller manager (gardener/gardener#4979, @wyb1)
• [OPERATOR] Resources blocking the deletion of a ManagedResource are now listed in the error returned when waiting for the deletion of the ManagedResource fails. (gardener/gardener#4965, @plkokanov)
• [OPERATOR] The shoot.status.advertisedAddresses are now set as early as possible in the shoot reconciliation flow. (gardener/gardener#4963, @vpnachev)
• [OPERATOR] Extend DNSOwner CRD with DNSActivation fields (gardener/gardener#4960, @MartinWeindel)
• [OPERATOR] gardener-resource-manager is now deployed in a highly-available way (3 replicas, pod anti-affinity and a permitted pod disruption of 1). (gardener/gardener#4941, @rfranzke)
• [OPERATOR] The Cluster resource is now synced at the start of a migrate operation, similar to what is done for reconciliation and deletion operations. (gardener/gardener#4883, @plkokanov)
• [OPERATOR] The Request Latency panel in the API Server dashboard uses a better query (gardener/gardener#4792, @wyb1)

📰 Noteworthy

• [OPERATOR] The kube-proxy version does now exactly match the kubelet version on all shoot worker nodes as required by https://kubernetes.io/releases/version-skew-policy/#kube-proxy. (gardener/gardener#4969, @rfranzke)
• [OPERATOR] There is now a new, dedicated certificate authority (ca-vpn) for the components related to the reversed VPN feature for shoot clusters. (gardener/gardener#4942, @rfranzke)
• [OPERATOR] kube-scheduler does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#4931, @rfranzke)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.36.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.36.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.36.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.36.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.36.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.36.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.36.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.36.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] Fix indentation for extensions Grafana dashboards (gardener/gardener#5029, @timebertt)

🏃 Others

• [OPERATOR] github.com/gardener/gardener/extensions/pkg/terraformer does now log by default the termination message of the Terraformer Pod (or its logs) when the Terraformer Pod finishes with error. (gardener/gardener#5032, @ialidzhikov)

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue causing seed bootstrap to fail when the ManagedIstio feature gate is disabled is now fixed. (gardener/gardener#5017, @timebertt)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.35.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.35.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.35.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.35.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.35.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.35.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.35.1
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.35.1