[gardener]

⚠️ Breaking Changes

• [OPERATOR] The custom controller metrics (garden*) exposed by gardener-controller-manager and gardenlet are removed, as they will soon be replaced by controller-runtime built-in metrics. (gardener/gardener#4913, @timebertt)
• [OPERATOR] The deprecated .spec.{type,providerConfig} fields in the ControllerRegistration resource have now been dropped. Make sure to migrate to ControllerDeployments before updating to this Gardener version. (gardener/gardener#4864, @rfranzke)
• [OPERATOR] Remove creation of shared and long-valid node bootstrapTokens. Instead use short-lived tokens unique for each node. You are required to update to a version of the operatingsystem-extension and the infrastructure extension that support creating bootstrap-tokens. Please see the compatibility matrix before updating gardener. (gardener/gardener#4824, @BeckerMax)
• [OPERATOR] gardener-resource-manager now requires operators to provide a TLS certificate and key for its webhook server. The respective directory can be configured with --tls-cert-dir, bind address and port can be configured with --bind-address and --port, respectively. (gardener/gardener#4817, @rfranzke)
• [OPERATOR] gardenlet does no longer maintain the deprecated garden.sapcloud.io/role label key in the control plane Pod labels. Before upgrading this this version of Gardener, make sure that you first upgraded to at least Gardener v1.31.0. (gardener/gardener#4783, @ialidzhikov)
• [DEVELOPER] Cloud extension providers that have the name of the provider hardcoded when invoking the hack/hook-me script should change that to the whole service name. (gardener/gardener#4887, @dimityrmirchev)
• [DEVELOPER] The github.com/gardener/gardener/extensions/pkg/controller.{ReconcileErr,ReconcileErrCause,ReconcileErrCauseOrErr} functions have been moved to github.com/gardener/gardener/pkg/controllerutils/reconciler`. (gardener/gardener#4880, @rfranzke)
• [DEVELOPER] The package pkg/resourcemanager/manager was moved to pkg/utils/managedresources/builder. (gardener/gardener#4862, @rfranzke)
• [DEVELOPER] Some functions were moved from extensions/... to other packages. You might need to adapt your import paths. (gardener/gardener#4860, @rfranzke)

✨ New Features

• [OPERATOR] gardener-resource-manager can now make request and auto-rotate short-lived ServiceAccount tokens via the TokenRequest API for components running in the source cluster and communicating with the target cluster. Please consult this document for more information. (gardener/gardener#4867, @BeckerMax)
• [OPERATOR] druid.gardener.cloud/v1alpha1.Etcd resources are now protected from unintentional deletion, i.e. they must be annotated with confirmation.gardener.cloud/deletion=true before any DELETE call can succeed. (gardener/gardener#4861, @vanjiii)
• [OPERATOR] gardener-resource-manager can now make sure static ServiceAccount tokens are invalidated. This might be helpful to enforce usage of Kubernetes' upstream ServiceAccount Token Volume Projection feature. Please consult this document for more information. (gardener/gardener#4817, @rfranzke)

🐛 Bug Fixes

• [OPERATOR] Fixes a nil pointer exception during shoot creation that can occur when deploying the etcd-main and etcd-events Etcd resources if their etcd.Status.Etcd fields are not set by the etcd-druid fast enough. (gardener/gardener#4975, @plkokanov)
• [OPERATOR] A bug has been fixed which caused some of no longer referenced immutable ConfigMaps/Secrets in shoot namespaces in seed clusters not to be deleted. (gardener/gardener#4904, @rfranzke)
• [OPERATOR] fix metrics-server for scenarios where address resolution via hostname does not work. (gardener/gardener#4884, @vasu1124)
• [DEVELOPER] Script hack/hook-me.sh is now successfully establishing connection to the remote tunnel server. Also the script now supports different kind of services and not only cloud provider extensions. (gardener/gardener#4887, @dimityrmirchev)

📖 Documentation

• [USER] Documentation about the Shoot info ConfigMap is now added. (gardener/gardener#4918, @poelzi)
• [OPERATOR] The allowed version skew between gardenlet and the Gardener control plane is now documented (at most one minor version behind, never ahead). (gardener/gardener#4917, @vlerenc)

🏃 Others

• [OPERATOR] Markdown files under /hack/api-reference are moved to /docs/api-reference directory. (gardener/gardener#4959, @Kristian-ZH)
• [OPERATOR] Set dns.gardener.cloud/include-zones annotation for the default domain secret in the Gardener controlplane chart. (gardener/gardener#4950, @MartinWeindel)
• [OPERATOR] Kube-proxy can now be enabled/disabled for clusters by setting the Shoot.Spec.Kubernetes.KubeProxy.Enabled flag accordingly. The change might be rejected by the used networking provider depending on the cluster configuration. (gardener/gardener#4903, @ScheererJ)
  • Please ensure that the networking provider extensions you use have at least v1.20.1 (calico) or v1.6.0 (cilium).
• [OPERATOR] Upgrade Grafana to 7.5.11 (gardener/gardener#4875, @wyb1)
• [DEVELOPER] The Shoot networking test does no longer fail against Shoots that do not allow privileged containers (.spec.kubernetes.allowPrivilegedContainers=false). (gardener/gardener#4899, @ialidzhikov)
• [DEPENDENCY] extensions.NewGardenDecoder has been removed in favor of kubernetes.GardenCodec. (gardener/gardener#4912, @timebertt)

📰 Noteworthy

• [USER] Shoot clusters with ReversedVPN enabled will have to be reconciled once in case their existing VPN connection gets reset, which happened in the old setup during shoot cluster creation/deletion. (gardener/gardener#4434, @ScheererJ)

[ext-authz-server]

✨ New Features

• [USER] improve log messages (gardener/ext-authz-server#2, @DockToFuture)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.35.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.35.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.35.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.35.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.35.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.35.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.35.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.35.0

[gardener]

🏃 Others

• [OPERATOR] Set dns.gardener.cloud/include-zones annotation for the default domain secret in the Gardener controlplane chart. (gardener/gardener#4954, @timuthy)
• [DEVELOPER] The Shoot networking test does no longer fail against Shoots that do not allow privileged containers (.spec.kubernetes.allowPrivilegedContainers=false). (gardener/gardener#4905, @ialidzhikov)

[gardener]

🐛 Bug Fixes

• [USER] Fixed a bug that caused owner DNSRecord resources to be never reconciled again after they are in an Error state. (gardener/gardener#4858, @stoyanr)

🏃 Others

• [OPERATOR] Set dns.gardener.cloud/include-zones annotation for the default domain secret in the Gardener controlplane chart. (gardener/gardener#4952, @timuthy)

[gardener]

🏃 Others

• [OPERATOR] Set dns.gardener.cloud/include-zones annotation for the default domain secret in the Gardener controlplane chart. (gardener/gardener#4953, @timuthy)

[gardener]

⚠️ Breaking Changes

• [USER] Since go1.17 both net.ParseIP and net.ParseCIDR reject leading zeros in the dot-decimal notation of IPv4 addresses. With the update to go1.17, gardener-apiserver now rejects Shoot objects with CIDR ranges that have such leading zeros in the dot-decimal notation. Before updating to this version of gardener-apiserver, make sure that there are no Shoot objects with leading zeros in the dot-decimal notation of an IPv4 address. For reference: https://nvd.nist.gov/vuln/detail/CVE-2021-29923 (gardener/gardener#4822, @ialidzhikov)
• [DEVELOPER] pkg/utils/validation/cidr package has been changed to properly detect CIDR overlaps. Please make sure to adapt your use of the library when revendoring. (gardener/gardener#4829, @kon-angelo)
  • CIDR.ValidateNotSubset have been replaced by CIDR.ValidateNotOverlap. CIDR.ValidateNotOverlap is stricter as it does not allow its subject to be a superset or subset of the CIDRs it tests against (previously only the superset condition was checked).
• [DEPENDENCY] hack/{generate,generate-parallel.sh} don't set GO111MODULE=off anymore as they used to. This was done to speed up generation with k8s.io/code-generator. If your repo reuses these scripts to generate code using k8s.io/code-generator you might want to consider setting GO111MODULE=off explicitly in hack/update-codegen.sh. (gardener/gardener#4854, @timebertt)
• [DEPENDENCY] The extensions/pkg/controller.Try* functions have been removed, as the usage was discouraged anyways (see this document on Kubernetes clients). (gardener/gardener#4757, @rfranzke)

✨ New Features

• [USER] The CoreDNS health plugin is now configured with lameduck of 15 seconds. This way, when a coredns replica is being shut down, it will keep serving the currently established clients for up to 15 seconds so that they can reconnect to some of the other replicas. More info about the lameduck can be found here. (gardener/gardener#4839, @vpnachev)
• [USER] It is now possible to control the --event-ttl kube-apiserver flag by configuring .spec.kubernetes.kubeAPIServer.eventTTL in the Shoot resource. (gardener/gardener#4758, @rfranzke)
• [OPERATOR] Certificates for Alertmanager, Grafana, Loki and Prometheus are now automatically renewed in a time windows of 30 days before they expire. (gardener/gardener#4836, @timuthy)
• [OPERATOR] The CachedRuntimeClients feature gate is promoted to beta and now enabled by default. (gardener/gardener#4831, @timebertt)
• [OPERATOR] The apiserver_crd_webhook_conversion_duration_seconds metric is now kept in the shoot monitoring stack. (gardener/gardener#4795, @timebertt)
• [OPERATOR] gardenlets now report the expiration date of their client certificates in the .status.clientCertificateExpirationTimestamp field of their respective Seed resources. If they are managed by ManagedSeeds then this enables to re-bootstrap gardenlets in case their certificate expired and they weren't able to refresh it themselves. gardener-controller-manager will automatically trigger this process if necessary. (gardener/gardener#4740, @rfranzke)
• [DEVELOPER] CRDs that are installed by Gardener on a Seed cluster are now generated to example/seed-crds. This allows to quickly apply all Seed CRDs for development purposes. (gardener/gardener#4854, @timebertt)
• [DEPENDENCY] A new hack script generate-seed-crds.sh was added, that can generate all of Gardener's Seed CRDs using controller-gen. See this file for an example usage. Make sure to add controller-gen to the list of requirements and example to the list of generated paths. (gardener/gardener#4854, @timebertt)

🐛 Bug Fixes

• [USER] Fixed a bug that caused owner DNSRecord resources to be never reconciled again after they are in an Error state. (gardener/gardener#4815, @stoyanr)
• [OPERATOR] Several issues have been fixed in the Gardener-Seed-Admission-Controller when DELETECOLLECTION requests are sent to the admission webhook. (gardener/gardener#4869, @timuthy)
• [OPERATOR] Fixes an issue that could cause the cluster-autoscaler to be started without --nodes during the restore phase of control plane migration. (gardener/gardener#4820, @plkokanov)
• [OPERATOR] Fix a bug where the CIDR subset validation did not check if the whole range overlaps. (gardener/gardener#4810, @kon-angelo)
• [OPERATOR] OperatingSystemConfig spec.file[] entries will no longer be added or modified by provider controlplane mutating webhooks if the new file content is empty. (gardener/gardener#4782, @plkokanov)
• [OPERATOR] During the restore phase of control plane migration, the kube-controller-manager deployment will be properly created with 1 replica if the shoot is not hibernated. (gardener/gardener#4781, @plkokanov)
• [DEVELOPER] Error codes are now removed from gardencorev1beta1.Condition created with the ConditionBuilder if ConditionBuilder.WithOldCondition(oldCondition) is used to initialize the condition, but error codes are not provided with ConditionBuilder.WithCodes(codes...) (gardener/gardener#4885, @plkokanov)

🏃 Others

• [USER] The following image is updated: (gardener/gardener#4759, @ialidzhikov)
  • k8s.gcr.io/metrics-server/metrics-server: v0.5.0 -> v0.5.1 (see the release notes)
• [OPERATOR] Creating owner check watchdogs will no longer fail if the Shoot field in the Cluster resource is nil or the dnsrecords CRD is not present. (gardener/gardener#4876, @stoyanr)
• [OPERATOR] Gardenlet only patches the Shoot's status label if it actually changed to save network traffic. (gardener/gardener#4870, @timebertt)
• [DEVELOPER] Envtest binaries have been upgraded to version v1.22. By this, we disable etcd fsync in envtests which speeds up test execution by not writing etcd data to disk. (gardener/gardener#4851, @timebertt)
• [DEVELOPER] Logging load integration test checks also for tenants logs. (gardener/gardener#4850, @vlvasilev)
• [DEVELOPER] The golang version is now updated to 1.17.2. (gardener/gardener#4822, @ialidzhikov)

📰 Noteworthy

• [DEVELOPER] The https://github.com/gardener/gardener-resource-manager/ project is now integrated into this repository. Hence, when gardener/gardener is released then a new image eu.gcr.io/gardener-project/gardener/resource-manager:<tag> will be released as well. (gardener/gardener#4757, @rfranzke)

[logging]

🏃 Others

• [OPERATOR] Remove the redundant runtime.Extension deserilizations in the gardener fluent-bit-to-loki output plugin. (gardener/logging#115, @vlvasilev)
• [OPERATOR] The gardener fluent-bit-to-loki concurrent map interaction and write issue is fixed. (gardener/logging#114, @vlvasilev)
• [DEVELOPER] The gardener fluent-bit-to-loki output plugin has a health probe. (gardener/logging#116, @vlvasilev)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.34.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.34.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.34.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.34.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.34.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.34.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.34.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.34.0

[gardener]

🐛 Bug Fixes

• [USER] Fixed a bug that caused owner DNSRecord resources to be never reconciled again after they are in an Error state. (gardener/gardener#4859, @stoyanr)
• [OPERATOR] Creating owner check watchdogs will no longer fail if the Shoot field in the Cluster resource is nil or the dnsrecords CRD is not present. (gardener/gardener#4877, @stoyanr)
• [OPERATOR] During the restore phase of control plane migration, the kube-controller-manager deployment will be properly created with 1 replica if the shoot is not hibernated. (gardener/gardener#4811, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.33.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.33.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.33.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.33.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.33.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.33.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.33.1

[gardener]

✨ New Features

• [USER] It is now possible to control the --service-account-{extend-token-expiration,max-token-expiration} kube-apiserver flags by configuring .spec.kubernetes.kubeAPIServer.serviceAccountConfig.{extendTokenExpiration,maxTokenExpiration} in the Shoot resource. (gardener/gardener#4753, @rfranzke)
• [USER] Allows configuration of MaxGracefulTerminationSeconds flag on ClusterAutoscaler. This allows end-users to configure maximum graceful termination (drain) seconds beyond which the node is force deleted during scale-down of cluster nodes. The default value is 600 seconds. (gardener/gardener#4697, @prashanth26)
• [OPERATOR] Logs can be spread across Shoot control-plane Loki and central Loki base on the Shoot state. (gardener/gardener#4751, @vlvasilev)
• [OPERATOR] A reserved Loki stream label __gardener_multitenant_id__ is introduced to specify multiple tenants separated by semicolon. (gardener/gardener#4751, @vlvasilev)
• [DEVELOPER] Extension controller reconciliations will now not be started or aborted if already running if it is detected that the shoot owner has changed compared to the one found in the owner DNSRecord resource in the seed cluster. (gardener/gardener#4638, @stoyanr)

🐛 Bug Fixes

• [USER] A bug was fixed that caused shoot creations to fail at the Deploying owner domain DNS record step. (gardener/gardener#4756, @timebertt)
• [OPERATOR] Fixed an issue that was preventing hibernated shoots from being migrated with the following error: missing information for required secret EtcdEncryptionConfig (gardener/gardener#4780, @plkokanov)
• [OPERATOR] gardener-controller-manager does no longer try to update needed ControllerInstallation that is being deleted. Previously gardener-controller-manager was trying to update such ControllerInstallation and the update calls were failing with reason field is immutable. With this change gardener-controller-manager properly indicates that it is waiting until the deletion of the needed ControllerInstallation is completed. (gardener/gardener#4718, @ialidzhikov)
• [OPERATOR] Fix a nil pointer exception during control plane migration restore phase that can happen when the UseDNSRecords feature gate is enabled. (gardener/gardener#4709, @plkokanov)
• [OPERATOR] Added a missing sideEffects field to the ValidatingWebhookConfiguration template in the Gardener control plane helm chart. (gardener/gardener#4707, @Diaphteiros)
• [OPERATOR] Potential deadlock in gardenlet on sequential ControllerInstallation deletion and creation is now fixed. (gardener/gardener#4704, @ialidzhikov)
• [OPERATOR] kubelet flags which only have an effect when using the docker container runtime are no longer set when using containerd (gardener/gardener#4693, @voelzmo)
• [OPERATOR] A bug in the seed bootstrap flow which was failing due to attempts to deploy VPA resource before the VPA CRD to be deployed has been fixed. (gardener/gardener#4537, @vpnachev)
• [DEVELOPER] make test-integration works with bash version 3 now. (gardener/gardener#4715, @timebertt)

🏃 Others

• [OPERATOR] Upgrade prometheus version to v2.30.2 (gardener/gardener#4761, @wyb1)
• [OPERATOR] The Gardener-Controller-Manager now also caches Secret objects if the CachedRuntimeClients feature gate is enabled. (gardener/gardener#4754, @timuthy)
• [OPERATOR] Fixed a deadlock issue in the owner check watchdogs implementation. (gardener/gardener#4752, @stoyanr)
• [OPERATOR] Adapt the logging integration tests to use the gardener fluent-bit-to-loki plugin version 0.37.1 (gardener/gardener#4751, @vlvasilev)
• [OPERATOR] The health checks executed on the shoot cluster nodes to detect if the cloud-config is up-to-date now skip the nodes that are about to be deleted during node rollout. (gardener/gardener#4746, @vpnachev)
• [OPERATOR] Increased the kube-rback-proxy and telegraf container resources in Loki pod to withstand higher resource usage spikes. (gardener/gardener#4714, @vlvasilev)
• [OPERATOR] Fixes an issue when scraping the gardenlet. Https is now used instead of http. (gardener/gardener#4706, @wyb1)
• [OPERATOR] Add kube-state-metrics metrics to the seed-prometheus (gardener/gardener#4706, @wyb1)
• [OPERATOR] Seed prometheus scrapes itself for meta information (gardener/gardener#4706, @wyb1)
• [OPERATOR] The fluent-bit version is upgraded from 1.7.8 to 1.8.7. (gardener/gardener#4699, @vlvasilev)
• [OPERATOR] The Promtail version is upgraded from 2.1.0 to 2.2.1. (gardener/gardener#4699, @vlvasilev)
• [OPERATOR] Stop Promtail server which runs on the shoot nodes. (gardener/gardener#4699, @vlvasilev)
• [OPERATOR] botanist.WaitUntilTunnelConnectionExists step is now enhanced to do not fail right away when the tunnel connection cannot be established. Instead, the step will retry tunnel connection check until the configured timeout is exceeded. (gardener/gardener#4691, @ialidzhikov)

[logging]

🐛 Bug Fixes

• [OPERATOR] Fix bug in gardener fluent-bit-to-loki plugin where __gardener_multitenant_id__ is removed from a shared label set. (gardener/logging#113, @vlvasilev)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.33.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.33.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.33.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.33.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.33.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.33.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.33.0

[gardener]

🐛 Bug Fixes

• [USER] An issue has been fixed which caused a worker node with a configured data disk not to get ready. This issue only happened if the data disk was of the same size as the boot disk. (gardener/gardener#4632, @timuthy)
• [OPERATOR] Fix an issue where the gardenlet no longer exposed metrics (gardener/gardener#4623, @wyb1)

🏃 Others

• [OPERATOR] Increased the kube-rback-proxy and telegraf container resources in Loki pod to withstand higher resource usage spikes. (gardener/gardener#4771, @vpnachev)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.30.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.30.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.30.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.30.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.30.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.30.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.30.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] Added a missing sideEffects field to the ValidatingWebhookConfiguration template in the Gardener control plane helm chart. (gardener/gardener#4711, @ialidzhikov)
• [OPERATOR] A bug has been fixed, which caused the ManagedSeed.spec.gardenlet.config.debugging.* fields to be wrongly set to false. (gardener/gardener#4686, @timebertt)
  • Note: if you have been running gardener version v1.31.[0-3] already, upgrading to this version will not remove the wrongly added fields. If you want to remove the fields in order to rely on the defaults (settings from the parent gardenlet) please use the following kubectl command: kubectl -n garden patch ms my-seed -p '{"spec":{"gardenlet":{"config":{"debugging":null}}}}'.
• [DEPENDENCY] Fixed an issue during the restoration of the worker resource where the status of the restored Machine resource cannot be updated with the name of the corresponding Node. (gardener/gardener#4702, @plkokanov)

🏃 Others

• [OPERATOR] Increased the kube-rback-proxy and telegraf container resources in Loki pod to withstand higher resource usage spikes. (gardener/gardener#4770, @vpnachev)
• [DEPENDENCY] If a resource has been successfully migrated, following reconcile operations are properly skipped, whereas migrate operations can be performed. (gardener/gardener#4701, @plkokanov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.31.4
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.31.4
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.31.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.31.4
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.31.4
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.31.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.31.4

[gardener]

🐛 Bug Fixes

• [USER] A bug was fixed that caused shoot creations to fail at the Deploying owner domain DNS record step. (gardener/gardener#4760, @timebertt)

🏃 Others

• [OPERATOR] Increased the kube-rback-proxy and telegraf container resources in Loki pod to withstand higher resource usage spikes. (gardener/gardener#4769, @vpnachev)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.32.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.32.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.32.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.32.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.32.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.32.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.32.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] Fix a nil pointer exception during control plane migration restore phase that can happen when the UseDNSRecords feature gate is enabled. (#4728, @plkokanov)
• [OPERATOR] Added a missing sideEffects field to the ValidatingWebhookConfiguration template in the Gardener control plane helm chart. (#4710, @ialidzhikov)

🏃 Others

• [OPERATOR] Fixes an issue when scraping the gardenlet. Https is now used instead of http. (#4716, @wyb1)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.32.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.32.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.32.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.32.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.32.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.32.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.32.1

[gardener]

⚠️ Breaking Changes

• [USER] Gardener now enforces the Seeds' capacity limits for shoot clusters when manually scheduling Shoots (i.e., when setting Shoot.spec.seedName). (#4521, @timebertt)
  • If your requests are denied by gardener-apiserver because of this, unset spec.seedName and let gardener-scheduler pick the best Seed with available capacity for you.
• [DEVELOPER] The e2e test network policy generator has been removed from gardener/gardener because it has been out of maintenance for a longer period of time. Please remove any references to github.com/gardener/gardener/extensions/test/e2e/framework/networkpolicies/generators. (#4643, @timuthy)

✨ New Features

• [OPERATOR] The gardener-{admission-controller,controller-manager} components now prefer using the certificates.k8s.io/v1 API if available. (#4671, @rfranzke)
• [OPERATOR] Operators can now explicitly trigger a client certificate renewal for the gardenlets. Please consult this document for more information. (#4667, @rfranzke)
• [OPERATOR] Users assigned to the cluster roles gardener.cloud:viewer or gardener.cloud:admin now have access to the customresourcedefinitions resources. (#4636, @vpnachev)
• [OPERATOR] Add a new Prometheus that operators can use to monitor extensions and components running in the garden namespace (#4560, @wyb1)

🐛 Bug Fixes

• [USER] Field .spec.resourcePolicy.containerPolicies[].controlledResources is now available for VerticalPodAutoscaler v1beta2 objects. (#4656, @timuthy)
• [USER] An issue has been fixed which caused a worker node with a configured data disk not to get ready. This issue only happened if the data disk was of the same size as the boot disk. (#4629, @timuthy)
• [OPERATOR] A bug has been fixed that did not add the required ingress class in some seed clusters that are not Gardener managed. (#4676, @timuthy)
• [OPERATOR] A race in gardenlet has been fixed, that caused gardenlet not to wait for Extension and other resources to be deleted if the CachedRuntimeClients feature gate is enabled. (#4674, @timebertt)
• [OPERATOR] An issue has been fix that prevented the HVPA to scale target resources adequately. (#4668, @timuthy)
• [OPERATOR] A bug has been fixed which prevented renewing gardenlet's client certificates for Garden clusters using Kubernetes v1.19 or higher. (#4665, @rfranzke)
• [OPERATOR] Another memory leak in gardenlet has been fixed. (#4641, @timebertt)
• [OPERATOR] Fix a memory leak in the gardenlet. (#4628, @BeckerMax)
• [DEPENDENCY] Fixed an issue during the restoration of the worker resource where the status of the restored Machine resource cannot be updated with the name of the corresponding Node. (#4681, @plkokanov)
• [OPERATOR] A bug has been fixed, which caused the ManagedSeed.spec.gardenlet.config.debugging.* fields to be wrongly set to false.
  • Note: if you have been running gardener version v1.31.[0-3] already, upgrading to this version will not remove the wrongly added fields. If you want to remove the fields in order to rely on the defaults (settings from the parent gardenlet) please use the following kubectl command: kubectl -n garden patch ms my-seed -p '{"spec":{"gardenlet":{"config":{"debugging":null}}}}'.
    (#4684, @timebertt)

🏃 Others

• [USER] VPA validation has been relaxed further through the corresponding CRD. Unknown fields are now kept instead of resulting in validation errors. (#4675, @timuthy)
• [OPERATOR] gardenlet now generates unique OSC resources per machine image name in order to avoid validation errors. (#4666, @stoyanr)
• [OPERATOR] add documentation on rotation of cloud provider secret (#4631, @BeckerMax)
• [OPERATOR] Reads and writes to the ShootState resource are now performed in a concurrency safe way. (#4620, @plkokanov)
• [OPERATOR] The ShootState.Spec.Gardener is now patched via a json merge patch with optimistic lock. (#4620, @plkokanov)
• [OPERATOR] If the etcd encryption secret and its corresponding data in the ShootState exist and the secret is annotated with `shoot.gardener.cloud/etcd-encryption-force-plaintext-secrets=true", the change will be properly reflected in the ShootState as well. (#4620, @plkokanov)
• [OPERATOR] Add the metric coredns_kubernetes_dns_programming_duration_seconds and provide a panel for it in grafana. (#4618, @wyb1)
• [OPERATOR] Gardener now considers Shoots, that are currently in migration between Seeds, when calculating the Seed usage for adhering to Seed capacity settings (Seed.status.{capacity,allocatable}). (#4604, @timebertt)
• [OPERATOR] Add API types for the landscaper control plane component. (#3981, @danielfoehrKn)
• [DEVELOPER] A few functions now accept a context.Context, please adapt your usages accordingly and pass a proper context. (#4644, @timebertt)
• [DEVELOPER] Missing or wrong doc comments and a few other common style errors will now be reported by the linter. (#4627, @stoyanr)
• [DEPENDENCY] If a resource has been successfully migrated, following reconcile operations are properly skipped, whereas migrate operations can be performed. (#4663, @plkokanov)

📰 Noteworthy

• [OPERATOR] The DisallowKubeconfigRotationForShootInDeletion feature gate in the gardener-apiserver has been promoted to beta and is now enabled by default. (#4645, @ialidzhikov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.32.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.32.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.32.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.32.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.32.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.32.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.32.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that did not add the required ingress class in some seed clusters that are not Gardener managed. (#4679, @timuthy)

🏃 Others

• [USER] VPA validation has been relaxed further through the corresponding CRD. Unknown fields are now kept instead of resulting in validation errors. (#4677, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.31.3
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.31.3
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.31.3
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.31.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.31.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.31.3

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue has been fix that prevented the HVPA to scale target resources adequately. (#4669, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.31.2
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.31.2
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.31.2
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.31.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.31.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.31.2
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.31.2

[gardener]

🐛 Bug Fixes

• [USER] Field .spec.resourcePolicy.containerPolicies[].controlledResources is now available for VerticalPodAutoscaler v1beta2 objects. (#4658, @timuthy)
• [USER] An issue has been fixed which caused a worker node with a configured data disk not to get ready. This issue only happened if the data disk was of the same size as the boot disk. (#4634, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.31.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.31.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.31.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.31.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.31.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.31.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.31.1

[gardener]

✨ New Features

• [USER] The NGINX Ingress Controller addon has been updated to v0.49.0 for shoot clusters running Kubernetes 1.20, 1.21. For shoot clusters running Kubernetes >= 1.22 NGINX Ingress Controller v1.0.0 is used. Please have a detailed look at this FAQ document which explains the most important ingress changes when updating to NGINX Ingress Controller v1.0.0. (#4614, @timuthy)
• [OPERATOR] The NGINX Ingress Controller has been updated to v0.49.0 for seed clusters running Kubernetes 1.18, 1.19, 1.20, 1.21. For seed clusters running Kubernetes >= 1.22 NGINX Ingress Controller v1.0.0 is used. (#4614, @timuthy)
• [OPERATOR] Gardener components now support enabling profiling handlers. See this document for more details. (#4568, @timebertt)
• [OPERATOR] Gardener can now support shoot and seed clusters with Kubernetes version 1.22. In order to allow creation/update of 1.22 clusters you will have to update the version of your provider extension(s) to a version that supports 1.22 as well. Please consult the respective releases and notes in the provider extension's repository. (#4562, @timuthy)
• [DEVELOPER] Gardener can now support shoot clusters with Kubernetes version 1.22. Extension developers have to prepare individual extensions as well to work with 1.22. Please take extra care to check the built-in APIs that have been removed with Kubernetes v1.22 (ref). Those need to be replaced by a corresponding newer API version. (#4562, @timuthy)
• [DEVELOPER] A guideline on the usage of kubernetes clients in Gardener's components and controllers has been added for developers, new contributors, new team members and maintainers. You can consult this document, if you want to get a better understanding of different client types, common gotchas and caveats of cached clients, conflict management, optimistic locking, different patch types, etc.. Find the full document here. (#4425, @timebertt)

🐛 Bug Fixes

• [USER] Grafana is no longer deployed for shoot clusters with testing purpose. (#4594, @vlvasilev)
• [USER] Fixed a bug that caused the shoot reconciliation to fail if gardenlet was restarted just before removing the operation annotation from a BackupEntry. (#4545, @stoyanr)
• [OPERATOR] Fix an issue where the gardenlet no longer exposed metrics (#4619, @wyb1)
• [OPERATOR] A bug has been fixed which caused Gardener's internal clients to use the old Kubernetes version instead the new one after a shoot cluster upgrade had been triggered. This rarely led to situations where two reconciliations in a row were necessary to get an upgraded shoot into a healthy state. (#4554, @timuthy)
• [OPERATOR] A bug has been fixed which allowed users to enter a value in the .spec.region field of a Shoot which was not part of the .spec.regions[] list of the respective CloudProfile. (#4553, @rfranzke)
• [DEVELOPER] Logging integration tests are adapted to the logging stack changing config maps and secrets (#4603, @vlvasilev)

🏃 Others

• [OPERATOR] Extensions that have been successfully migrated can now be restored. (#4608, @plkokanov)
• [OPERATOR] The default revisionHistoryLimit for gardenlets deployed by ManagedSeeds was increased to 10. (#4580, @timebertt)
• [OPERATOR] Upgrade Prometheus to v2.29.1 (#4546, @wyb1)
• [OPERATOR] Upgrade Alertmanager to v0.22.2 (#4546, @wyb1)
• [OPERATOR] Extensions enabled through the Shoot spec will be added as labels in the namespace for that specific Shoot in the Seed cluster. Labels will be as follow extensions.gardener.cloud/my-service: "true" where my-service is the type of the extension. (#4476, @dimityrmirchev)
• [OPERATOR] Improved handling of the seed resource in the seed controller to ensure that potential data races are avoided in the future. (#4465, @stoyanr)
• [OPERATOR] Reads and writes to the ShootState resource are now performed in a concurrency safe way. (#4411, @plkokanov)
• [OPERATOR] The ShootState.Spec.Gardener is now patched via a json merge patch with optimistic lock. (#4411, @plkokanov)
• [OPERATOR] If the etcd encryption secret and its corresponding data in the ShootState exist and the secret is annotated with `shoot.gardener.cloud/etcd-encryption-force-plaintext-secrets=true", the change will be properly reflected in the ShootState as well. (#4411, @plkokanov)
• [DEVELOPER] All of the fluent-bit embeded re-emitters have their own unique name for debugging purposes. (#4583, @vlvasilev)

[autoscaler]

🐛 Bug Fixes

• [DEVELOPER] Avoids panics when VM type isn't found during scale from zero (gardener/autoscaler#77, @prashanth26)
• [DEVELOPER] Fetches the VM from the correct map for MCM provider Azure and hence doesn't panic anymore (gardener/autoscaler#77, @prashanth26)

🏃 Others

• [USER] Added support for 12 new AWS instance types and 1 new Azure Instance Types (gardener/autoscaler#85, @AxiomSamarth)
• [USER] Do not return errors while fetching details for an unmanaged node. (gardener/autoscaler#79, @jsravn)
• [OPERATOR] Added support for m6i type AWS machine types (gardener/autoscaler#97, @patrickhuy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.31.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.31.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.31.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.31.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.31.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.31.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.31.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which can cause the Gardenlet to panic when VPA is enabled for shoot clusters. (#4523, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.29.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.29.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.29.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.29.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.29.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.29.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.29.1

[gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which can cause the Gardenlet to panic when VPA is enabled for shoot clusters. (#4524, @timuthy)

🏃 Others

• [OPERATOR] A bug has been fixed which prevented the CSR auto-approval process for Gardenlet certificates when the SeedAuthorizer is enabled. Hence, the user certificate used by Gardenlet to connect to the Garden cluster was not renewed successfully. (#4505, @timuthy)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.28.3
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.28.3
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.28.3
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.28.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.28.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.28.3
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.28.3

[gardener]

🐛 Bug Fixes

• [USER] Grafana is no longer deployed for shoot clusters with testing purpose. (#4596, @ialidzhikov)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.30.1
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.30.1
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.30.1
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.30.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.30.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.30.1
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.30.1

[gardener]

⚠️ Breaking Changes

• [USER] Shoots using kubernetes version >= 1.23 can no longer specify .spec.provider.workers[].cri.name: docker, as the in-tree dockershim will be removed by upstream kubernetes. See our documentation for more details on the dockershim removal. (#4529, @voelzmo)
• [OPERATOR] gardenlet's MountHostCADirectories feature gate that is GA since v1.27 is unconditionally enabled, and can no longer be specified in the gardenlet's configuration. (#4526, @ialidzhikov)

✨ New Features

• [USER] Cluster Autoscaler is now more configurable at the shoot level with options like expander and maxNodeProvisionTime. The default value for expander is least-waste and for maxNodeProvisionTime it is 20m. This can be overridden by flags on the shoot.spec.kubernetes.clusterAutoscaler section. (#4508, @prashanth26)
• [USER] The metrics-server has been updated to version v0.5.0. (#4496, @vpnachev)
  • Please see the release notes for more details.
• [OPERATOR] Gardener will add docker explicitly to the list of supported container runtimes for all MachineImageVersions in your Cloud Profile. This is not a functional change: Previously, docker support was implicitly assumed for all MachineImageVersions. This is now changed in the context of the dockershim removal. (#4500, @voelzmo)
• [DEVELOPER] Extensions using the generic controlplane mutator webhook can now easily mutate the vpn-seed-server deployment by implementing the EnsureVPNSeedServerDeployment function. (#4544, @rfranzke)

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed preventing gardenlet from properly reconciling the NetworkPolicys in the shoot namespaces in the seed for shoots without static node CIDRs. (#4543, @rfranzke)
• [OPERATOR] A bug has been fixed which can cause the Gardenlet to panic when VPA is enabled for shoot clusters. (#4509, @timuthy)
• [OPERATOR] Fixes a bug that prevented dashboards with node information to be shown in Grafana when a shoot was stuck in worker creation. (#4507, @vlvasilev)
• [OPERATOR] Gardenlet is now provided with the required permissions in the seed cluster to properly deploy or destroy resources related to the APIServerSNI feature. (#4494, @vpnachev)

🏃 Others

• [OPERATOR] The Kubernetes version requirement for Seeds, that was raised to >= 1.18 in [email protected], is now technically enforced as well. (#4503, @timebertt)
• [OPERATOR] The initial delay of the CoreDNS readiness probe is increased from 5 to 30 seconds to give more time to the newly started pods to initialize their cache. (#4498, @vpnachev)
• [OPERATOR] New Gardener role gardener.cloud:system:viewers for managing viewer permissions (without access to view secrets) for all Gardener and Kubernetes resources across all Gardener projects (#4497, @donistz)
• [OPERATOR] Ensure proper istio RBAC permissions when ManagedIstio feature gate is enabled by default. (#4481, @amshuman-kr)
• [OPERATOR] Deletion of shoots that are in Migrate or Restore phase is now forbidden. (#4316, @kris94)
• [DEVELOPER] The telegraf component is added into hack/.ci/set_dependency_version script (#4492, @vlvasilev)

📰 Noteworthy

• [OPERATOR] The .spec.deployment.{type,providerConfig} fields in the ControllerRegistration resource (deprecated since v1.23) will be removed from the API starting with v1.32. Please consider adapting to ControllerDeployments now (see https://github.com/gardener/gardener/blob/master/docs/extensions/controllerregistration.md and https://github.com/gardener/gardener/pull/3995). (#4532, @rfranzke)
• [OPERATOR] All MachineImageVersions in the Cloud Profile must have set an accurate set of supported Container Runtimes! Previously, support for the docker runtime was implicit, it now needs to be set explicitly. See our dockershim removal document for more information. To ease this transition this release adds adds docker to the list of supported container runtimes for all MachineImageVersions in your Cloud Profile see #4500. (#4438, @BeckerMax)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.30.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.30.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.30.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.30.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.30.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.30.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.30.0