[gardener]

Action Required

• [USER] As the Kubernetes community is in the process of deprecating basic authentication for the kube-apiserver newly created 1.16 shoot clusters don't enable it by default anymore. Similarly, the default authentication mode for the kubernetes-dashboard addon is token instead of basic. You can still enable it manually, but you should consider migrating away from it. We will drop basic authentication support in a future release when Kubernetes doesn't support it anymore. (#1443, @rfranzke)
• [USER] The subdomain for the the user monitoring has changed from g-users to gu. (#1417, @wyb1)
• [OPERATOR] In order to support Kubernetes 1.16 you must use at least v0.13.0 of the provider extension controllers. (#1443, @rfranzke)
• [OPERATOR] The old garden.sapcloud.io/v1beta1.BackupInfrastructure resources are deprecated and will be removed in the next release. The core.gardener.cloud/v1alpha1.BackupBucket and core.gardener.cloud/v1alpha1 are the replacement. (#1427, @swapnilgm)
• [OPERATOR] The subdomain for the the operator monitoring has changed from g-operators to go. (#1417, @wyb1)
• [OPERATOR] The false positive KubeEtcdFullBackupFailed alert has been fixed. This requires at least version 0.12.0 of the gardener-extensions (containing 0.7.3 of etcd-backup-restore). (#1381, @wyb1)

Most notable changes

• [USER] Gardener does now support shoot clusters with Kubernetes version 1.16. You should consider the Kubernetes release notes before upgrading to 1.16. (#1443, @rfranzke)
• [USER] The CoreDNS version for all shoots has been upgraded from 1.4.0 to 1.6.3. (#1443, @rfranzke)
• [USER] The deprecated .spec.backup section in the garden.sapcloud.io/v1beta1.Shoot resource has been removed. (#1430, @rfranzke)
• [USER] It is now possible to specify a CA certificate bundle per worker pool. Gardener will automatically install the CA certificates on every worker node of this pool. (#1430, @rfranzke)
• [USER] Gardener does now feature a new Shoot resource which is part of the core.gardener.cloud/v1alpha1 API group. It is fully forwards and backwards compatible to the old Shoot resource in the garden.sapcloud.io/v1beta1 API group. It will be the new default, and the old garden.sapcloud.io/v1beta1.Shoot resource are deprecated now. It will be removed in a future version. Consider switching to using the new core.gardener.cloud/v1alpha1.Shoot resource. The example directory contains proper example manifests. (#1430, @rfranzke)
• [USER] Gardener does now feature a new CloudProfile resource which is part of the core.gardener.cloud/v1alpha1 API group. It is fully forwards and backwards compatible to the old CloudProfile resource in the garden.sapcloud.io/v1beta1 API group. It will be the new default, and the old garden.sapcloud.io/v1beta1.CloudProfile resource are deprecated now. It will be removed in a future version. Consider switching to using the new core.gardener.cloud/v1alpha1.CloudProfile resource. The example directory contains proper example manifests. (#1403, @rfranzke)
• [USER] Cluster Autoscaler now balances similar worker-groups while scaling-up. Similar worker-groups are defined as those having nodes with the same resource capacities and exactly the same labels. Refer doc for more info. (#1401, @hardikdr)
• [USER] The new OpenIDConnectPreset resource allows for specifying OpenID Connect configurations which are applied to Shoot namespace-wide. (#1394, @mvladev)
• [USER] A new optional field in Shoot specification has been added spec.kubernetes.kubeAPIServer.oidcConfig.clientAuthentication. It can specify OpenID Connect settings used for kubeconfig generation. (#1394, @mvladev)
• [OPERATOR] It is now possible to configure which domains shall be included or excluded for DNS providers. If you are using a custom domain for your shoot then you can use .spec.dns.includeDomains, .spec.dns.excludeDomains. (#1430, @rfranzke)
• [OPERATOR] The orphaned BackupInfrastructure resources post reconciliation of shoots using Gardener 0.29+ are now getting deleted. (#1427, @swapnilgm)
  • ⚠️ For hibernated cluster it now guarantees only latest snapshot will be there on shared bucket. It doesn't migrate the old snapshots and hence break the policy of the configured deletion grace period. If you want to keep the old data you would need to download it manually from the buckets before upgrading to this Gardener version.
• [OPERATOR] The new ClusterOpenIDConnectPreset resource allows for specifying OpenID Connect configurations which are applied to Projects and Shoot cluster-wide. (#1394, @mvladev)
• [OPERATOR] Two new mutation admission controllers are introduced - ClusterOpenIDConnectPreset and OpenIDConnectPreset. Those controllers can are enabled by default and can be disabled with --disable-admission-plugins flag on gardener-apiserver. (#1394, @mvladev)
• [OPERATOR] The client configuration for the Kubernetes clients for the garden cluster, seed clusters, and shoot clusters has been separated. You can configure them individually in the gardener-controller-manager's componentconfig. (ddfdc74622a37893040a83718c361615790af37e)
• [OPERATOR] The memory limits for the kube-controller-manager have been significantly increased. Also, the kube-apiserver CPU and memory requests and limits have been increased for shooted seeds. The same happened for the nginx-ingress-controller. (5d5a89017ba2e36f5a05dc4e4e300c4a42e28d05)
• [DEVELOPER] The gardener-scheduler is now working only with core.gardener.cloud/v1alpha1 instead of garden.sapcloud.io/v1beta1 resources. (#1435, @rfranzke)
• [DEVELOPER] The garden.sapcloud.io.CIDR and garden.sapcloud.io/v1beta1.CIDR type has been replaced with string type. (#1430, @rfranzke)
• [DEVELOPER] The .spec.kubernetes.kubeAPIServer.admissionPlugins[].config type has been changed from *string to *ProviderConfig (which effectively is a *runtime.RawExtension). (#1430, @rfranzke)
• [DEVELOPER] ./hack/dev-setup-register-gardener must be ran to register the new settings.gardener.cloud API group. (#1394, @mvladev)

Improvements

• [USER] Users can now use an Alertmanager if they configured their shoot to receive alerts. (#1417, @wyb1)
• [OPERATOR] Increase shoot prometheus retention to 30d. Retention is guaranteed if prometheus exceeds 15GB of storage. (#1444, @wyb1)
• [OPERATOR] Vertical pod autoscaling can be enabled for the gardener-apiserver, gardener-controller-manager and gardener-scheduler. To enable VPA for each component global.apiserver.vpa, global.controller.vpa and global.scheduler.vpa must be set to true respectively. (#1440, @wyb1)
• [OPERATOR] Fix bug in maintenance-controller not respecting spec.maintenance.autoUpdate.kubernetesVersion: false in the Shoot when the Kubernetes version does not exist in the CloudProfile. (#1423, @danielfoehrKn)
• [OPERATOR] Volume plugin directory on the kubelet is now statically configured to /var/lib/kubelet/volumeplugins, this is to support Calico versions >= 3.8 on CoreOS which does not have write access to the defaule volume-plugin-dir. (#1414, @zanetworker)
• [OPERATOR] Prometheus image is upgraded to 2.12.0 (#1410, @wyb1)
• [OPERATOR] Upgraded monitoring images. Some metrics have changed or were removed. (#1406, @wyb1)
• [OPERATOR] Operators can now access all aggregate monitoring components using one set of basic auth credentials. The basic auth credentials are stored in the garden cluster in the secret garden/seed-monitoring-ingress-credentials. (#1405, @wyb1)
• [OPERATOR] A bug that prevented managed resources from getting properly labeled has been fixed. (ea72650865eaa0ed22e81d6fcec943cbc316d01e)
• [OPERATOR] The admission plugins inside the gardener-apiserver now also react for resources migrated to the new core.gardener.cloud/v1alpha1 API. (9b73a905e3136d840a1ee6ffe5c47a0494d118a1)
• [OPERATOR] Shooted seeds default backup provider is now correctly set to the shoot provider (instead of the seed provider). (e8218523c2e78e8934ad39e0134fdfe05ed0e968)
• [OPERATOR] The Seed object for shooted seeds does no longer have an owner reference to the respective Shoot because Seed is cluster-scoped while Shoot is namespaced, and according to https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ this is not supported. (0dc122cc110759b255fa3f59ec7e8c7a345bed8d)
• [DEVELOPER] Add alert rule for failed etcd restorations. (#1426, @shreyas-s-rao)
• [DEVELOPER] The golang version has been updated to 1.13. Please upgrade your local go installation to 1.13. (#1425, @ialidzhikov)

[gardener-resource-manager]

Improvements

• [OPERATOR] The resource manager now observes version updates for objects and supports configurable api group migrations. (gardener/gardener-resource-manager#16, @mandelsoft)
• [OPERATOR] An issue has been resolved which caused the Gardener-Resource-Manage to throw errors for Kinds belonging to a recently deployed CRD. (gardener/gardener-resource-manager#14, @timuthy)
• [OPERATOR] ManagedResource status is now enriched with two conditions - ResourcesApplied and ResourcesHealthy. (gardener/gardener-resource-manager#11, @ialidzhikov)
• [OPERATOR] Fixes an issue which left resources in the target cluster even though they were supposed to be deleted through a change or removal of the ManagedResource. (gardener/gardener-resource-manager#8, @timuthy)
• [DEVELOPER] The golang version has been updated to 1.13. Please upgrade your local go installation to 1.13. (gardener/gardener-resource-manager#12, @ialidzhikov)

[terraformer]

Most notable changes

• [OPERATOR] Provider versions are upgraded: (gardener/terraformer#23, @mvladev)
  • aws 1.60.0 -> 2.26.0
  • google 1.20.0 -> 2.14.0
  • azurerm 1.22.1 -> 1.33.1
  • openstack 1.16.0 -> 1.21.1
  • alicloud 1.31.0 -> 1.55.2
  • packet 1.7.2 -> 2.3.0

Improvements

• [OPERATOR] Provider versions are upgraded: (gardener/terraformer#26, @ialidzhikov)
  • template 1.0.0 -> 2.1.2
  • null 1.0.0 -> 2.1.2
• [OPERATOR] Added google beta provider (gardener/terraformer#25, @DockToFuture)
• [OPERATOR] tzdata package is now used instead of assets/zoneinfo.zip to make all timezones available. (gardener/terraformer#24, @ialidzhikov)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.30.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.30.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.30.0

[gardener]

Action Required

• [OPERATOR] The false positive KubeEtcdFullBackupFailed alert has been fixed. This requires at least version 0.12.0 of the gardener-extensions (containing 0.7.3 of etcd-backup-restore). (9f1bb8d510ca6c40680d1b0406583a63004597af)

Most notable changes

• [OPERATOR] The client configuration for the Kubernetes clients for the garden cluster, seed clusters, and shoot clusters has been separated. You can configure them individually in the gardener-controller-manager's componentconfig. (b7b3a14bbbcf799d7af6665c8f569273bdeb8999)
• [OPERATOR] The memory limits for the kube-controller-manager have been significantly increased. Also, the kube-apiserver CPU and memory requests and limits have been increased for shooted seeds. The same happened for the nginx-ingress-controller. (1a5848cd96320833abf6060b55ccbb24f2ee40ca)

Improvements

• [OPERATOR] A bug that prevented managed resources from getting properly labeled has been fixed. (fd3125db0431cc4d1bcac1ec021ce0a3f5b990e9)
• [OPERATOR] The admission plugins inside the gardener-apiserver now also react for resources migrated to the new core.gardener.cloud/v1alpha1 API. (31d150efca5441c29fb4b5424619fa1c5f05f9f7)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.29.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.29.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.29.2

[gardener]

Improvements

• [OPERATOR] Shooted seeds default backup provider is now correctly set to the shoot provider (instead of the seed provider). (e6b835cd13495bda176d23306338b3de7623e9ba)
• [OPERATOR] The Seed object for shooted seeds does no longer have an owner reference to the respective Shoot because Seed is cluster-scoped while Shoot is namespaced, and according to https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ this is not supported. (3fc63b22ecb04d35fcff6babc560ad6f4e52ba20)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.29.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.29.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.29.1

[gardener]

Action Required

• [USER] Creation of multi-zone OpenStack clusters now require exactly one CIDR in spec.cloud.openstack.networks.workers[]. (#1370, @mvladev)
  • Gardener will automatically patch existing multi-zone OpenStack Shoots which are failing by removing the additional CIDRs (if they exist). Please update your Shoot manifests for future compatibility.
• [OPERATOR] ⚠️ Gardener will now rely on extension controller for managing the backup infrastructure for the etcd backups of shoots. The in-tree supported will be removed in the next release. Also, together with the extension controllers a shared bucket approach is implemented now (previously, every shoot had its own bucket). For more details see GEP-1 on extensibility and GEP-2 on the shared bucket approach. We have prepared the implementation of the Alicloud provider, AWS provider, Azure provider, GCP provider, and OpenStack provider. After updating Gardener you need to create corresponding ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). After updating Gardener you need to update your existing ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). You should already have ControllerRegistration resources for various providers. Add .spec.resources[].kind="BackupBucket" and .spec.resources[].type="<provider-name>", and also .spec.resources[].kind="BackupEntry" and .spec.resources[].type="<provider-name>" to it. Please find example ControllerRegistration resources here (Alicloud), here (AWS), here (Azure), here (GCP), and here (OpenStack). To get information about more details please walk through these documents. (#1128, @swapnilgm)
  • ⚠️ Seed resources have to be updated with backup configuration, i.e. you have to specify spec.backup with backup provider details before updating gardener-controller-manager version to this release.
  • ⚠️ The concrete steps for deploying this Gardener version are:
  • 1. Scale down the gardener-controller-manager deployment: kubectl -n garden scale deployment/gardener-controller-manager --replicas=0
  • 2. Deploy the Gardener 0.29.0 Helm chart.
  • 3. Update all your Seeds to contain the correct backup configuration.
  • 4. Deploy the needed ControllerRegistration resources for the provider extensions you want to use.
  • 5. Scale up the gardener-controller-manager deployment: kubectl -n garden scale deployment/gardener-controller-manager --replicas=1
• [OPERATOR] If you use have a shooted seed of provider OpenStack then you have to make sure that the referenced provider secret contains a key named authURL with the keystone URL. This is due to https://github.com/gardener/gardener/pull/1128. (1a91fed670be1969f2281b268bfa214ffa3885ea)

Most notable changes

• [USER] Gardener does now feature a new Project resource which is part of the core.gardener.cloud/v1alpha1 API group. It is fully forwards and backwards compatible to the old Project resource in the garden.sapcloud.io/v1beta1 API group. It will be the new default, and the old garden.sapcloud.io/v1beta1.Project resource are deprecated now. It will be removed in a future version. Consider switching to using the new core.gardener.cloud/v1alpha1.Project resource. The example directory contains proper example manifests. (#1382, @rfranzke)
• [USER] Gardener does now feature a new SecretBinding and Quota resources which are part of the core.gardener.cloud/v1alpha1 API group. They are fully forwards and backwards compatible to the old SecretBinding and Quota resources in the garden.sapcloud.io/v1beta1 API group. They will be the new defaults, and the old garden.sapcloud.io/v1beta1.SecretBinding and garden.sapcloud.io/v1beta1.Quota resources are deprecated now. They will be removed in a future version. Consider switching to using the new core.gardener.cloud/v1alpha1.SecretBinding and core.gardener.cloud/v1alpha1.Quota resources. The example directory contains proper example manifests. (#1377, @rfranzke)
• [USER] The deprectated .spec.dns.hostedZoneID has been removed from the Shoot API. (#1372, @rfranzke)
• [OPERATOR] Shoot's that use non-canonicalized CIDRs are now rejected by the Gardener API server. Always use CIDRs in their canonical form. (#1373, @danielfoehrKn)
• [USER] If you opted-out of the automatic Kubernetes patch version update the shoot Kubernetes version won't be updated unless the expiration date specified by the administrator for the used version expires. The expiration date for the respective Kubernetes versions can be found in the referenced CloudProfile in the .spec.<cloud>.constraints.kubernetes[].offeredVersions[].expirationDate fields. Example: The CloudProfile specifies (1.14.2, 2019-08-08T14:00:00Z) and (1.14.5) as possible versions for 1.14. A shoot that is opted-out of automatic updates uses (1.14.2). In its maintenance time window it won't be updated to 1.14.5 before 2019-08-08T14:00:00Z. (#1363, @danielfoehrKn)
• [USER] Gardener does now feature a new Seed resource which is part of the core.gardener.cloud/v1alpha1 API group. It is fully forwards and backwards compatible to the old Seed resource in the garden.sapcloud.io/v1beta1 API group. It will be the new default, and the old garden.sapcloud.io/v1beta1.Seed resource is deprecated now. It will be removed in a future version. Consider switching to using the new core.gardener.cloud/v1alpha1.Seed resource. The example directory contains proper example manifests. (#1308, @rfranzke)
• [OPERATOR] It is now possible to specify shoot default pod/service CIDRs per seed. Please configure spec.networks.shootDefaults on your Seed resources if you want to use this. (#1386, @rfranzke)
• [OPERATOR] Extension resources are now only deleted after the shoot Kubernetes resources have been cleaned up during the shoot deletion flow. (#1374, @rfranzke)
• [OPERATOR] It is now possible to configure which hosted zone IDs shall be included or excluded for DNS providers. For the internal and the default domains you can use the dns.gardener.cloud/include-zones and dns.gardener.cloud/exclude-zones annotations on the secrets (see example resource). If you are using a custom domain for your shoot then you can use .spec.dns.includeZones, .spec.dns.excludeZones. (#1372, @rfranzke)
• [OPERATOR] Due to faulty validation, multi-zone OpenStack shoot clusters containing more than one CIDR in spec.cloud.gcp.networks.workers[] could be created (which is not a valid behavior). To ensure that those clusters are going to be safely reconciled with this update, on the next Shoot UPDATE request only the first spec.cloud.openstack.networks.workers[0] is going to be persisted. (#1370, @mvladev)
  • Rollback to previous Gardener versions is not supported.
• [OPERATOR] The .spec.<cloud>.constraints.kubernetes[].versions field in the CloudProfile is deprecated and will be removed in the future. It is now replaced with a list of offered versions (.spec.<cloud>.constraints.kubernetes[].offeredVersions). Each version can have an expirationDate which specifies the time after all shoots that are using this version and opted-out of automatic Kubernetes patch version updates will get forcefully updated to the latest Kubernetes patch version for the used <major>.<minor> version. Example: The CloudProfile specifies (1.14.2, 2019-08-08T14:00:00Z) and (1.14.5) as possible versions for 1.14. A shoot that is opted-out of automatic updates uses (1.14.2). In its maintenance time window it won't be updated to 1.14.5 before 2019-08-08T14:00:00Z. (#1363, @danielfoehrKn)
• [OPERATOR] Gardener will have two new resources BackupBucket and BackupEntry in its core.gardener.cloud/v1alpha1 API group. The backup infrastructure for new and existing shoots will be provisioned as per GEP-2. (#1128, @swapnilgm)
• [OPERATOR] Gardener now supports having backup provider independent of seed cloud provider. Please take a look at the example Seed manifest to see how this can be configured. (#1128, @swapnilgm)
• [DEVELOPER] The flow package now has a LimitSubmitter which can be used to restrict the amount of operations being executed in parallel. (#1391, @timuthy)
• [DEVELOPER] Gardener does now read ConfigMaps labelled with extensions.gardener.cloud/configuration=logging and injects their data into the fluent-bit configuration. This allows extension controllers to define their provider-specific logging configuration for the components they deploy. (#1371, @svetlinas)

Improvements

• [USER] It is now possible to delete Shoots that were created with a wrong .spec.dns.domain field. (#1378, @rfranzke)
• [USER] The shoot flow does now propagate errors provided by extension controllers faster to the end-user (via the Shoot's .status fields). (#1364, @rfranzke)
• [OPERATOR] Gardener now rewrites shoot secrets in parallel if the etcd encryption has been changed to improve the overall reconciliation performance. (#1391, @timuthy)
• [OPERATOR] Client connection settings for the gardener-controller-manager are now being propagated to every Kubernetes client used by Gardener. This improves the overall performance of the system, especially for ControllerInstallation, Seed, and Shoot reconciliations. (#1379, @timuthy)
• [OPERATOR] Gardener does now annotate DNSProvider objects for the external cluster domain(s) with dns.gardener.cloud/realms="<shoot-namespace>,". (#1368, @rfranzke)
• [OPERATOR] The DNSProviders for a shoot do now only include the shoot's base domain (without api. prefix). (#1368, @rfranzke)
• [OPERATOR] It is now possible to configure the backup profile via the shoot.garden.sapcloud.io/use-as-seed annotation. Please set backup.provider, backup.region, backup.secretRef.name, backup.SecretRef.namespace if you want to configure it. If no such configuration is found the default behaviour is that the same provider will be used for backup. In order to explicitly disable backup for a seed configure backup.provider=none in the annotation. (1a91fed670be1969f2281b268bfa214ffa3885ea)
• [OPERATOR] It is now possible to configure the default pod CIDR for shoots via the shoot.garden.sapcloud.io/use-as-seed annotation. Please set shootDefaults.pods=<cidr> and shootDefaults.services=<cidr> if you want to configure it. (018214b04bdb2336089515ff4a80d25daeb28268)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.29.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.29.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.29.0

[gardener]

Action Required

• [USER] Creation of multi-zone GCP clusters now require exactly one CIDR in spec.cloud.gcp.networks.workers[]. (#1346, @mvladev)
  • Gardener will automatically patch existing multi-zone GCP Shoots which are failing by removing the additional CIDRs (if they exist). Please update your Shoot manifests for future compatibility.
• [USER] The kubeconfigs for newly created shoot clusters do no longer contain client certificates. Instead, they contain a token that can be rotated on your demand. When annotation the shoot with shoot.garden.sapcloud.io/operation=rotate-kubeconfig-credentials this token will be rotated. (#1339, @rfranzke)
• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing provider-specific control plane configuration and deployments of additional provider-specific components relevant for exposing the control plane (like aws-lb-readvertiser). Instead, it now relies on extension controllers to deploy the respective components for the specific provider they have been developed for, e.g. AWS, Azure, GCP, etc. (see GEP-1 on extensibility). We have prepared the implementation of the Alicloud provider, AWS provider, Azure provider, GCP provider, OpenStack provider, and Packet provider. After updating Gardener you need to update your existing ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). Please find example ControllerRegistration resources here (Alicloud), here (AWS), here (Azure), here (GCP), here (OpenStack), and here (Packet). To get information about more details please walk through these and these documents. (#1313, @svetlinas)
  • ⚠️ Please make sure you upgrade your extension controllers to at least version [0.10.0] before upgrading Gardener.
• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing the shoot's network plugin. Instead, it now relies on extension controllers to deploy the network plugin for the specific provider they have been developed for, e.g. Calico, Flannel, etc. (see GEP-3 on network extensibility). We have prepared the implementation of the Calico networking extension. After updating Gardener you need to create a corresponding ControllerRegistration resource to make this extension controller known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). Please find example ControllerRegistration resources here (Calico). To get information about more details please walk through these documents. (#1293, @zanetworker)

Most notable changes

• [USER] The shoot deletion flow now forcefully deletes all namespaces and doesn't block the flow any longer. Please note that this may leave orphaned resources in the infrastructure if you delete the shoot without deleting these infrastructure resources beforehand. (#1353, @ialidzhikov)
• [USER] It is now possible to create or update Shoot resources by only specifying the <major>.<minor> parts of the desired Kubernetes version. Gardener will automatically try to pick the latest patch version offered by the referenced CloudProfile. (#1350, @danielfoehrKn)
• [USER] It is now possible to disable basic authentication for shoot clusters by setting .spec.kubernetes.kubeAPIServer.enableBasicAuthentication=false. By default, basic authentication is still enabled for all shoots. (#1341, @rfranzke)
• [USER] You may now rotate the basic authentication password of your shoot cluster by annotating the Shoot resource with shoot.garden.sapcloud.io/operation=rotate-kubeconfig-credentials. ⚠️ Please be aware that if this cluster was created before Gardener version 0.28.0 then your new kubeconfig will no longer contain a client certificate (however, it the previously issued one will remain valid). (#1339, @rfranzke)
• [USER] When deleting a shoot, registered CustomResourceDefinitions and APIServices now have 1h time to cleanup before getting forcefully finalized. (#1326, @adracus)
• [USER] It is now possible to override the kubelet settings for shoots in .spec.kubernetes.kubelet per worker pool in .spec.cloud.<name>.workers[].kubelet. If no worker-specific kubelet settings are provided then the defaults in .spec.kubernetes.kubelet apply. (#1299, @danielfoehrKn)
• [USER] It is now possible to configure various eviction settings for the kubelet. Please see this example for further details. (#1299, @danielfoehrKn)
• [USER] It is now possible to configure the maximum number per pods setting for the kubelet. The default value is 110 (like in Kubernetes), and you can override it based on your node mask CIDR. Set .spec.kubernetes.kubelet.maxPods and .spec.kubernetes.kubeControllerManager.nodeMaskCIDR. Please be aware that the node mask CIDR is immutable, so you cannot increase or shrink it without recreating your shoot. The maximum number of pods can be changed as long as it is in the boundaries of the node mask CIDR. (#1299, @danielfoehrKn)
• [USER] During shoot deletion, all Kubernetes resources will now be finalized and deleted after a grace period of 5 minutes. This helps getting rid of shoot clusters stuck in deletion. Please note that resources protected by the finalizers may leak when being finalized, so make sure that all resources can be properly deleted before deleting a shoot. (#1286, @adracus)
• [USER] It's now possible to configure the node CIDR mask size for the kube-controller-manager in the .spec.kubernetes.kubeControllerManager.nodeCIDRMaskSize flag in the Shoot resource. (#1280, @afritzler)
• [OPERATOR] Due to faulty validation, multi-zone GCP shoot clusters containing more than one CIDR in spec.cloud.gcp.networks.workers[] could be created (which is not a valid behavior). To ensure that those clusters are going to be safely reconciled with this update, on the next Shoot UPDATE request only the first spec.cloud.gcp.networks.workers[0] is going to be persisted. (#1346, @mvladev)
  • Rollback to previous Gardener versions is not supported.
• [OPERATOR] Operators now have access to dashboards that collect information about the seed. The dashboards also have aggregated information about the shoots running on the seed. ⚠️ This feature is temporary and will be replaced with global view where all data from all seeds is aggregated. (#1343, @wyb1)
• [OPERATOR] Due to lack of possibilities to check whether all infrastructure resources have been cleaned up Gardener now waits on a best-effort base before continuing the shoot deletion flow. (#1317, @rfranzke)
• [DEVELOPER] Every seed cluster now gets a gardener-resource-manager deployment in its garden namespace. This instance manages ManagedResources with .spec.class=seed for which contained manifests are applied to or removed from the very same seed cluster. Extension controllers might be interested in using this feature to let the gardener-resource-manager handle resources that need to be deployed to the seed. (#1331, @timuthy)
• [DEVELOPER] The golang version has been updated to 1.12.8 (fixing CVE-2019-9512 and CVE-2019-9514). (#1328, @mvladev)
• [DEVELOPER] The ControlPlane extension CRD does now have an optional .spec.purpose field. The default purpose will be normal. Another purpose is exposure which can be used to trigger the deployment of components that are required for exposing a shoot control plane. As the shoot control plane is running in the seed cluster this is specific to the seed provider. (#1303, @svetlinas)

Improvements

• [USER] If a ConfigMap that contains an audit policy which is referenced by a Shoot is changed then the Shoot gets reconciled immediately again (even if the Shoot is not within its maintenance time window (in case reconciliation should only happen in the maintenance time window)). (#1320, @rfranzke)
• [USER] Service account projection can now be configured via a Shoot's spec.kubernetes.kubeAPIServer.serviceAccountConfig and spec.kubernetes.kubeAPIServer.apiAudiences as described in https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection. (#1288, @adracus)
• [OPERATOR] Defined alert rules for missed etcd backups [both delta and full backups] (#1355, @PadmaB)
• [OPERATOR] Added gardener state dump to the integration test suite that gets logged if a test fails. (#1354, @schrodit)
• [OPERATOR] Extensions that are removed from the Shoot or through the ControllerRegistration are now correctly deleted. (#1342, @timuthy)
• [OPERATOR] OperatingSystemConfigs that are obsolete (no longer used) are now cleaned up during reconciliation. (#1337, @ialidzhikov)
• [OPERATOR] When a ControllerRegistration for Extension resources is deleted then the ControllerInstallation controller now only waits for the relevant Extension objects to be deleted. Previously it was waiting for all Extension objects which resulted in an end-less wait loop. (#1329, @rfranzke)
• [OPERATOR] Fixes an issue for which led to an abortion of the shoot deletion flow in case Gardener tried to clean up namespaces in the shoot. (#1323, @timuthy)
• [OPERATOR] Shoot infrastructure provider is now included in alerts. (#1322, @wyb1)
• [OPERATOR] The etcd encryption Secret that is synced to the project namespace in the garden cluster does now have a owner reference to the Shoot (allows proper garbage collection after shoot deletion). (#1319, @rfranzke)
• [OPERATOR] The resource limits for the nginx-ingress controller on shooted seeds have been increased to 1 CPU, 2Gi memory. (#1318, @rfranzke)
• [OPERATOR] TLS ciphers suites TLS_RSA_WITH_AES_128_GCM_SHA256 and TLS_RSA_WITH_AES_256_GCM_SHA384 are now removed from kube-apiserver and kube-controller-manager. (#1312, @ialidzhikov)
• [OPERATOR] Shooted seed cluster is deleted before the backing shoot, in case the shoot is scheduled for deletion. (#1309, @vpnachev)
• [OPERATOR] The prometheus stateful set does now have the requirement environment variables for the latest VPN version. (#1307, @zanetworker)
• [OPERATOR] When deleting a hibernated shoot then the ControlPlane extension CRD is now deployed before the sleeping components are woken up. This is to ensure that possible provider specific configuration exists before the wake-up. (#1305, @rfranzke)
• [OPERATOR] The shoot deletion flow does now redeploy the ControlPlane CRD if the kube-apiserver deployment does still exist. The reason for this is that the kube-apiserver depends on some resources that are managed via the ControlPlane CRD for some providers. (#1304, @rfranzke)
• [OPERATOR] The shoot deletion flow does now trigger a redeployment of the ControlPlane and Worker extension resource only if they still exist. (#1301, @rfranzke)
• [OPERATOR] The shoot secret rewriting process (after the etcd encryption config has been changed) has been improved. It is now retried, and errors are collected and returned at the end (instead of immediately returned). (#1296, @rfranzke)
• [OPERATOR] An issue that accidentally reconciled shoots outside of their maintenance time windows (although, the controller-manager componentconfig stated that it should only happen within the time windows) has been fixed. (#1289, @adracus)
• [OPERATOR] The kube-apiserver deployment does now have the requirement environment variables for the latest VPN version. (#1287, @DockToFuture)
• [OPERATOR] gihtub.com/gardener/gardener can now properly be vendored with go modules. (#1283, @ialidzhikov)
• [OPERATOR] TODO: You need to install at least v0.10.0 of gardener-extensions because in-tree provider-specific customization of Gardener core addons has been removed. (#1282, @rfranzke)
• [OPERATOR] The monitoring dashboards have been overhauled. The structure of the dashboards has been harmonised and aligned. New dashboards for CoreDNS, VPN and Machine-Controller-Manager have been added. (#1279, @dkistner)
• [OPERATOR] Components running in the garden namespace can now be monitored using the seed-prometheus. (#1179, @wyb1)
• [OPERATOR] Shoot metrics are now aggregated in each seed in the seed-prometheus. (#1179, @wyb1)
• [OPERATOR] Prometheus will now scrape metrics for fluent-bit, fluentd, and elasticsearch in the garden namespace. (#1006, @KristianZH)
• [DEVELOPER] The cloud-config-downloader running on every shoot worker nodes is now enabling all the systemd units before starting them. (#1334, @vlvasilev)
• [DEVELOPER] The shoot validator admission plugin exits early in case of update operations that did not change the .spec field. (#1292, @timuthy)

[autoscaler]

Improvements

• [OPERATOR] Updated APIs of MCM to include NodeTemplate in machine objects spec. This fixes issues while syncing annotations/labels/taints maintained on worker-pools (machine-deployments). (gardener/autoscaler#21, @hardikdr)

[gardener-resource-manager]

Most notable changes

• [USER] It is now possible to configure whether objects managed by a ManagedResource should be kept in the system although the ManagedResource is deleted. The .spec.keepObjects field defaults to false. (gardener/gardener-resource-manager@6b2029430332753e158c1611b8cf4b1dcd68f0c0)
• [USER] By default, any resource managed by the gardener-resource-manager does no longer overwrite existing labels and annotations. If you want to force the overwriting you can set .spec.forceOverwriteLabels=true or .spec.forceOverwriteAnnotations=true. (gardener/gardener-resource-manager@3eff3a859192584a07ded5f174b403de4aeea1db)

Improvements

• [OPERATOR] The resource manager can now be started with a dedicated resource class. If specified, it only handles resource object annotated to be intended for this class. The resource class in the ManagedResource is specified in .spec.class. (gardener/gardener-resource-manager#5, @mandelsoft)
• [OPERATOR] dep is replaced by go mod (gardener/gardener-resource-manager#4, @ialidzhikov)

[vpn]

Most notable changes

• [OPERATOR] Prometheus checks to the VPN tun interface should now work as expected. (gardener/vpn#46, @DockToFuture)
• [OPERATOR] The vpn-seed container now allows authentication against the kube-apiserver with a client certificate. The new environment variables APISERVER_AUTH_MODE can be either basic-auth or client-cert. (gardener/vpn@02eb33cad28ad214d1631219021569a5126d14c0)
  • In case of basic-auth the APISERVER_AUTH_MODE_BASIC_AUTH_CSV environment variable tells the path to the basic auth CSV file, and APISERVER_AUTH_MODE_BASIC_AUTH_USERNAME tells the user name (only the password will be read out of the CSV).
  • In case of client-cert the APISERVER_AUTH_MODE_CLIENT_CERT_CA, APISERVER_AUTH_MODE_CLIENT_CERT_CRT, APISERVER_AUTH_MODE_CLIENT_CERT_KEY variables tell the paths to the CA, client cert, and client key.
• [OPERATOR] Add pull-filter to vpn-seed (gardener/vpn#45, @DockToFuture)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.28.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.28.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.28.0

[gardener]

Most notable changes

• [USER] When deleting a shoot, registered CustomResourceDefinitions and APIServices now have 1h time to cleanup before getting forcefully finalized. (c50eb1fb8c8cd41453e5651da5fef83d2cbf109d)
• [DEVELOPER] The golang version has been updated to 1.12.8 (fixing CVE-2019-9512 and CVE-2019-9514). (a9904b5d84f999612913c4f7eb5bba0e4da8bf64)

Improvements

• [OPERATOR] When a ControllerRegistration for Extension resources is deleted then the ControllerInstallation controller now only waits for the relevant Extension objects to be deleted. Previously it was waiting for all Extension objects which resulted in an end-less wait loop. (a0d815430fc219c3f4649d9eaace6a09ed25ec94)

[autoscaler]

Improvements

• [OPERATOR] Updated APIs of MCM to include NodeTemplate in machine objects spec. This fixes issues while syncing annotations/labels/taints maintained on worker-pools (machine-deployments). (gardener/autoscaler#21, @hardikdr)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.6
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.6
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.6

[gardener]

Most notable changes

• [USER] During shoot deletion, all Kubernetes resources will now be finalized and deleted after a grace period of 5 minutes. This helps getting rid of shoot clusters stuck in deletion. Please note that resources protected by the finalizers may leak when being finalized, so make sure that all resources can be properly deleted before deleting a shoot. (797db0f3a64190112ecb4ac3d7a3f7d0be98dced)
• [OPERATOR] Due to lack of possibilities to check whether all infrastructure resources have been cleaned up Gardener now waits on a best-effort base before continuing the shoot deletion flow. (8aab605528bb9f9bbd2afab0a0bf7703746e8053)

Improvements

• [USER] If a ConfigMap that contains an audit policy which is referenced by a Shoot is changed then the Shoot gets reconciled immediately again (even if the Shoot is not within its maintenance time window (in case reconciliation should only happen in the maintenance time window)). (2657b7c1283c6ac94c395ee2a0f08bae4d0eabbd)
• [OPERATOR] Shooted seed cluster is deleted before the backing shoot, in case the shoot is scheduled for deletion. (b778ac695e0c55dd5102f2bce2bbff918f354d6d)
• [OPERATOR] Fixes an issue for which led to an abortion of the shoot deletion flow in case Gardener tried to clean up namespaces in the shoot. (632b75f521f85bd612440f96cfc32474d3474ee1)
• [OPERATOR] Shoot infrastructure provider is now included in alerts. (9daa5b06dd65eae61288b60e74526ce95ce1527c)
• [OPERATOR] The resource limits for the nginx-ingress controller on shooted seeds have been increased to 1 CPU, 2Gi memory. (8cdcf4469c4a79be616b0edc025bd81f63c12fc7)
• [OPERATOR] The etcd encryption Secret that is synced to the project namespace in the garden cluster does now have a owner reference to the Shoot (allows proper garbage collection after shoot deletion). (35260e440a3931eee4af5d4b5afa935348bf97ee)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.5
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.5

[gardener]

Improvements

• [OPERATOR] The prometheus stateful set does now have the requirement environment variables for the latest VPN version. (c04d33ffc7e3eb2f483fec14e8eabc5a19ea57d4)
• [OPERATOR] When deleting a hibernated shoot then the ControlPlane extension CRD is now deployed before the sleeping components are woken up. This is to ensure that possible provider specific configuration exists before the wake-up. (8048af0a06d10163e61f41bb9d958810a1d656d3)
• [OPERATOR] The kube-apiserver deployment does now have the requirement environment variables for the latest VPN version. (8967d71271fa8399de5333a502a54a09ab6500ed)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.4

[gardener]

Improvements

• [OPERATOR] The shoot deletion flow does now redeploy the ControlPlane CRD if the kube-apiserver deployment does still exist. The reason for this is that the kube-apiserver depends on some resources that are managed via the ControlPlane CRD for some providers. (f4a28f592fbd55a79dc9f7820ecbbbeeb1b4ee2e)

[vpn]

Most notable changes

• [OPERATOR] Add pull-filter to vpn-seed (gardener/vpn#45, @DockToFuture)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.3

[gardener]

Improvements

• [OPERATOR] The shoot deletion flow does now trigger a redeployment of the ControlPlane and Worker extension resource only if they still exist. (8bc6083554a518e4724a6d7923eaf395593415c8)
• [OPERATOR] The shoot secret rewriting process (after the etcd encryption config has been changed) has been improved. It is now retried, and errors are collected and returned at the end (instead of immediately returned). (d8a0f509b5abaf8b7c50f08601b1a8e5d70cca5d)
• [DEVELOPER] The shoot validator admission plugin for exits early in case of update operations that did not change the .spec field. (a883e63cf345b9d1e0480996dfe06e4304c7a50a)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.2

[gardener]

Improvements

• [OPERATOR] An issue that accidentally reconciled shoots outside of their maintenance time windows (although, the controller-manager componentconfig stated that it should only happen within the time windows) has been fixed. (9e98723975b14a125c97b40b566fe14ff6784030)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.1

[gardener]

Action Required

• [OPERATOR] Some left-overs after the merge of the control plane extensibility PR have now been cleaned up. You need to deploy at least version 0.9.0 of the provider extension controllers when using this Gardener version. (#1237, @rfranzke)
• [OPERATOR] The VPA feature gate has been entirely removed. VPA components will now always be deployed. You have to remove the VPA feature gate from your gardener-controller-manager configuration after upgrading Gardener. (#1173, @wyb1)
• [DEVELOPER] In order to give Gardener control over when to reconcile shoot clusters it now annotates ControlPlane, Extension, Infrastructure, OperatingSystemConfig and Worker extension resources with gardener.cloud/operation=reconcile. Extension controllers must only react/reconcile their resources if this annotation is set. After they have picked up the event they should update the status to Reconcile Processing and remove the annotation. This only applies for extension resources that have to be reconciled. Those that are newly created or marked for deletion shall be operated on independent of the annotation set by Gardener. Also, you might want to take a look at gardener/gardener-extensions#178 or this document. (#1165, @timuthy)
• [DEVELOPER] Deprecated methods from pkg/client/kubernetes are now removed. Consider to switch your invocations to kubernetes-sigs/controller-runtime client (.Client()) or kubernetes/client-go (.Kubernetes()). (#1140, @ialidzhikov)

Most notable changes

• [USER] The .spec.hibernated.enabled field in the Shoot resource is now optional. (#1277, @danielfoehrKn)
• [USER] It is now possible to specify machine image specific configuration to customize the operating system. However, this is only possible if the respective operating system controller supports this (or, more generally, only what the controller supports can be customized). The machineImage section in the Shoot resource does now have a providerConfig field that can be filled with operating system specific configuration. Please consult the documentation of the respective OS-controller you are using. (#1261, @pablochacin)
• [USER] Prior releases of Gardener only allowed exactly one machine image/version per Shoot. It is now possible to configure different machine images per worker pool. The .spec.cloud.<provider-name>.machineImage section contains the default image that should be used for worker pools which don't explicitly configure a machine image (which is possible via the .spec.cloud.<provider-name>.workers[*].machineImage field). (#1250, @KristianZH)
• [USER] It is now possible to opt-out of automated machine image version updates for shoot clusters. If opted-out the shoot machine image won't be updated unless the expiration date specified by the administrator for the used version expires. The expiration date for the respective versions of the machine image can be found in the referenced CloudProfile in the .spec.<cloud>.constraints.machineImages[].versions.expirationDate fields. If you want to opt-out set .spec.maintenance.machineImageVersion=false in your Shoot resource (default: true). Example: The CloudProfile specifies (coreos, 1967.5.0, 2019-08-08T14:00:00Z) and (coreos, 2023.4.0) as possible versions for coreos. A shoot that is opted-out of automatic updates uses (coreos, 1967.5.0). In its maintenance time window it won't be updated to 2023.4.0 before 2019-08-08T14:00:00Z. (#1177, @danielfoehrKn)
• [OPERATOR] The pods of all gardener-apiserver, gardener-controller-manager, and gardener-scheduler replicas are now deployed with anti-affinity such that they run on different worker nodes. Additionally, for each of these deployments, a pod disruption budget configuration will be created if replicas > 1. It will allow max. replicas-1 unavailable pods. (#1231, @rfranzke)
• [OPERATOR] The field deletionGracePeriodHoursByPurpose is now introduced for the backupInfrastructure controller to specify different deletion grace period values per shoot purpose. (#1204, @shreyas-s-rao)
• [OPERATOR] The .spec.<cloud>.constraints.machineImages[].version field in the CloudProfile is deprecated and will be removed in the future. It is now replaced with a list of versions (.spec.<cloud>.constraints.machineImages[].versions). Each version can have an expirationDate which specifies the time after all shoots that are using this version and opted-out of automatic machine image version updates will get forcefully updated to the latest machine image version for the used machine image name. Example: The CloudProfile specifies (coreos, 1967.5.0, 2019-08-08T14:00:00Z) and (coreos, 2023.4.0) as possible versions for coreos. A shoot that is opted-out of automatic updates uses (coreos, 1967.5.0). In its maintenance time window it won't be updated to 2023.4.0 before 2019-08-08T14:00:00Z. (#1177, @danielfoehrKn)
• [OPERATOR] Gardener now instructs extension controllers when they are supposed to reconcile ControlPlane, Extension, Infrastructure, OperatingSystemConfig and Worker resources by annotating them with gardener.cloud/operation=reconcile. (#1165, @timuthy)
• [OPERATOR] All secrets for shoot clusters of Kubernetes version v1.13 and higher are now encrypted before stored in etcd. It is possible to force Gardener to not encrypt secrets by annotating the etcd encryption secret in the shoot namespace in the seed with shoot.gardener.cloud/etcd-encryption-force-plaintext-secrets=true. (#1066, @michael-engler)
• [DEVELOPER] dep is replaced by go mod. github.com/gardener/gardener now publishes go module files containing dependency version information. (#1185, @ialidzhikov)

Improvements

• [USER] The Shoot status does now contain a new .status.hibernated field which indicates whether the shoot is hibernated or not. The .spec.hibernated.enabled field just indicates the desired state of the user. (#1277, @danielfoehrKn)
• [USER] The gardener now validates whether the k8s version of the shoot is compatible with the apiVersion of the provided audit policy. (#1208, @vpnachev)
• [USER] An issue that prevented shoot clusters with non-finalized CRDs or APIServices from being deleted has been fixed. (#1171, @adracus)
• [USER] An issue that prevented deleting hibernated shoot clusters has now been fixed. (c77bb749b845b824bb9d51a5d311064d6937efc4)
• [USER] An issue that sometimes prevented shoot addons from being provisioned after the addon configuration had changed has been fixed. (23aaeced9eb71b82e9d6641fdb61f0329f420a60)
• [OPERATOR] Added integration for hibernation (#1276, @schrodit)
• [OPERATOR] Improvement of the availability of kube-apiserver during maintenance activities such as a rolling update or scaling (say, by HPA) of the kube-apiserver Deployment. (#1275, @amshuman-kr)
• [OPERATOR] The HPA for the shoot's kube-apiserver is now deleted only after the control plane gets scaled down when the shoot shall be hibernated, preventing that it accidentally scales up the kube-apiserver deployment again (although the shoot should be hibernated). (#1260, @rfranzke)
• [OPERATOR] Added gardener integration tests to test a full gardener lifecycle with a specific commit. (#1257, @schrodit)
• [OPERATOR] Shooted seed clusters that get deleted will now also cause automatic deletion of the corresponding Seed object and seed secret. (#1242, @rfranzke)
• [OPERATOR] The newly introduced gardener-resource-manager component is now always deployed (even if the shoot is hibernated (with replicas=0)). This allows waking it up again when a hibernated shoot shall be deleted. (#1239, @rfranzke)
• [OPERATOR] Gardener does now support support shoot clusters with Kubernetes version 1.15 on Alicloud. (#1227, @EmoinLanyu)
• [OPERATOR] Added vpa for all controlplane components and set it to Auto except for apiserver and etcd. That would be done by HVPA eventually. (#1222, @georgekuruvillak)
• [OPERATOR] An issue with CRDs not being cleaned up is fixed. (#1221, @adracus)
• [OPERATOR] Adds detailed API server usage grafana dashboard for operators and users. (#1212, @plkokanov)
• [OPERATOR] For kubernetes versions below 1.14 the metric apiserver_request_count is relabled to apiserver_request_total (#1212, @plkokanov)
• [OPERATOR] Fixed the template for backupInfrastructure deletionGracePeriodHoursByPurpose in gardener chart (#1210, @swapnilgm)
• [OPERATOR] Added an integration test to validate the successful reconciliation of a shoot after its kubernetes version is updated (#1203, @schrodit)
• [OPERATOR] Added a test that checks if all shoots are successfully reconciled after an gardener update (#1202, @schrodit)
• [OPERATOR] The garden.sapcloud.io:admin cluster role now includes the dashboard.gardener.cloud API Group (#1197, @petersutter)
• [OPERATOR] The ControllerInstallation controller does now inject the volume provider name into the Helm charts for the respective seed if it was annotated with persistentvolume.garden.sapcloud.io/provider=<name>. The path of the value is .gardener.seed.volumeProvider. (#1162, @jia-jerry)
• [OPERATOR] The bootstrap script for the etcd container now sleeps for 1s in each iteration polling for the etcd backup sidecar to be ready. Earlier, the wait before the next poll used to happen only if etcd backup sidecar was already started and a tight polling loop used to happen if the etcd container starts before the backup sidecar. (#1160, @amshuman-kr)
• [OPERATOR] vpa added for vpa-exporter to prevent OOMKill of vpa-exporter pod. (#1159, @georgekuruvillak)
• [OPERATOR] Workarounds for the new NetworkPolicy migrations are now removed. (#1144, @mvladev)
• [OPERATOR] Monitoring and alerts for apiserver audit health (#1131, @shturec)
• [DEVELOPER] The image vector does now allow to add runtime version constraints. This is especially helpful if components that are running in the seed cluster can only target specific shoot cluster versions. (#1243, @rfranzke)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.27.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.27.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.27.0

[gardener]

Improvements

• [OPERATOR] Fixed the template for backupInfrastructure deletionGracePeriodHoursByPurpose in gardener chart (6f2b1669f7482d6c1cf0d6d2f8ddf91bae27cd23)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.26.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.26.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.26.4

[gardener]

Action Required

• [USER] The field deletionGracePeriodDays for the backupInfrastructure controller under controller manager config is now modified deletionGracePeriodHours to provide more fine-grained control over backup infrastructure retention period post shoot deletion. ⚠️ Please update your component-config.yaml to reflect the same. (#1204, @shreyas-s-rao)

Most notable changes

• [USER] The field deletionGracePeriodHoursByPurpose is now introduced for the backupInfrastructure controller to specify different deletion grace period values per shoot purpose. (#1204, @shreyas-s-rao)

Improvements

• [USER] An issue that prevented shoot clusters with non-finalized CRDs or APIServices from being deleted has been fixed. (#1171, @adracus)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.26.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.26.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.26.3

[gardener]

Improvements

• [USER] An issue that prevented deleting hibernated shoot clusters has now been fixed. (ce29f8f785ae1bb0201749ac07a78de4cf824899)
• [OPERATOR] The ControllerInstallation controller does now inject the volume provider name into the Helm charts for the respective seed if it was annotated with persistentvolume.garden.sapcloud.io/provider=<name>. The path of the value is .gardener.seed.volumeProvider. (#1163, @jia-jerry)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.26.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.26.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.26.2

[gardener]

Improvements

• [USER] An issue that sometimes prevented shoot addons from being provisioned after the addon configuration had changed has been fixed. (919c65a66ac123fbf0b8ea2f5452287004b097a7)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.26.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.26.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.26.1

[gardener]

Least backwards compatible version

• ⚠️ The least Gardener version that you can upgrade from is 0.25.0.

Action Required

• [OPERATOR] In order to support Kubernetes 1.15 you must use at least v0.7.1 of the provider extension controllers. (#1127, @rfranzke)
• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing provider-specific control plane configuration and deployments of additional provider-specific components (like cloud-controller-manager or CSI controllers). Instead, it now relies on extension controllers to inject the provider-specific configuration via mutating webhooks and to deploy the respective cloud-controller-manager (and cloud-provider-config) for the specific provider they have been developed for, e.g. AWS, Azure, GCP, etc. (see GEP-1 on extensibility). We have prepared the implementation of the Alicloud provider, AWS provider, Azure provider, GCP provider, OpenStack provider, and Packet provider. After updating Gardener you need to update your existing ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). You should already have ControllerRegistration resources for various providers. Add .spec.resources[].kind="ControlPlane" and .spec.resources[].type="<provider-name>" to it. Please find example ControllerRegistration resources here (Alicloud), here (AWS), here (Azure), here (GCP), here (OpenStack), and here (Packet). It is recommended to use more than one replica as the control plane extensions use webhooks to inject configuration into the standard control plane, and you want the webhook to be highly available. To get information about more details please walk through these documents. (#1076, @stoyanr)

Most notable changes

• [USER] Gardener does now support shoot clusters with Kubernetes version 1.15. You should consider the Kubernetes release notes before upgrading to 1.15. (#1127, @rfranzke)
• [OPERATOR] When cleaning up a shoot cluster in the deletion flow Gardener does now exclude all resources labeled with shoot.gardener.cloud/no-cleanup="true". This might be useful for extension controllers that are deploying resources into the shoot. (#1114, @rfranzke)
• [OPERATOR] Gardener does now use the gardener-resource-manager, a component that manages all kind of resources in the shoot cluster with the help of CRDs. Take a look at these example CRDs to get started. (#1114, @rfranzke)
• [OPERATOR] It is now possible to configure Gardener to only reconcile shoots during their respective maintenance time windows by setting .controllers.shoot.reconcileInMaintenanceOnly=true in the controller-manager's component configuration. If the shoot's spec changes outside of their maintenance time windows then they will be reconciled immediately. Also, they will be reconciled immediately if they didn't complete their last operation successfully. With this in place users are now able to have a HA setup of shoots if their maintenance time windows do not overlap. (#1094, @adracus)
• [OPERATOR] Due to the introduction of the control plane extension controllers the etcds, kube-apiservers, cloud-controller-managers, and csi-{*} controllers of existing shoots will be restarted. The reason for this is that the extension controllers use another mechanism to compute the checksum of mounted secrets and configmaps. (#1076, @stoyanr)

Improvements

• [USER] The .status.lastError section in the shoot status does now contain a new field lastUpdateTime that indicates when the section has been updated the last time. (#1125, @vpnachev)
• [USER] Add PodSecurityPolicy for calico-kube-controller. (#1120, @mvladev)
• [OPERATOR] Added integration test to check if RBAC is enabled and if the garden namespace is protected. (#1139, @schrodit)
• [OPERATOR] The dependency-watchdog controller does now check for service dependency hierarchy (e.g., etcd -> kube-apiserver, kube-apiserver -> other control plane components). This will lead to even faster restarts of the control plane components after CrashLoopBackoffs. (#1124, @georgekuruvillak)

[dependency-watchdog]

Improvements

• [OPERATOR] Enabled the dependency-watchdog to have hierarchy of service-pod dependency. (gardener/dependency-watchdog#6, @georgekuruvillak)

[gardener-resource-manager]

Most notable changes

• [OPERATOR] First version of Gardener Resource Manager (gardener/gardener-resource-manager@06037ec840ebd955a6feda32122eee2d903aab2f)
  • The gardener-resource-manager is a project similar to the kube-addon-manager.
  • It manages Kubernetes resources in a target cluster which means that it creates, updates, and deletes them.
  • Also, it makes sure that manual modifications to these resources are reconciled back to the desired state.
  • Currently, it is doing this in a loop, however, the project might evolve to use smarter techniques like watches, etc.
  • In the Gardener project we were using the kube-addon-manager since more than two years.
  • While we have progressed with our extensibility story (moving cloud providers out-of-tree) we had decided that the kube-addon-manager is no longer suitable for this use-case.
  • The problem with it is that it needs to have its managed resources on its file system.
  • This requires storing the resources in ConfigMaps or Secrets and mounting them to the kube-addon-manager pod during deployment time.
  • The gardener-resource-manager uses CustomResourceDefinitions which allows to dynamically add, change, and remove resources with immediate action and without the need to reconfigure the volume mounts/restarting the pod.

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.26.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.26.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.26.0

[gardener]

Action Required

• [OPERATOR] The Seed resource does now contain a new .spec.blockCIDRs field. Every CIDR that is added here will be blocked for communcation of any control plane component of shoots in this seed. Typically, you should add the (cloud) provider's metadata service CIDR(s) here. For shooted seed clusters you might want to extend your shoot.garden.sapcloud.io/use-as-seed annotation with blockCIDRs, e.g. shoot.garden.sapcloud.io/use-as-seed="true,blockCIDRs=1.2.3.4/5;6.7.8.9/10. (#1081, @rfranzke)
• [OPERATOR] ⚠️ This is optional step. Before bumping the gardener version to latest, make sure that all the shoots are reconciled with gardener version 0.22.0+. Use the helping script to perform migration of etcd PVC for hibernated cluster. This will reduce the time for etcd migration. (#1038, @swapnilgm)
• [OPERATOR] The Gardener does now feature a third component: the gardener-scheduler. In previous releases, the Gardener API server contained an admission plugin called ShootSeedManager that was responsible for finding an adequate seed cluster when creating a shoot. Now, this admission plugin has been removed and its logic has been moved to the gardener-scheduler. Similar to how the kube-scheduler finds an adequate node when for scheduling a pod, the gardener-scheduler finds an adequate seed when scheduling a shoot. The Gardener Helm chart does now contain configuration for the scheduler that you can set when deploying this and further versions. (#981, @danielfoehrKn)
• [OPERATOR] Additional NetworkPolicies must be created, allowing Egress or/and Ingress traffic if custom components are deployed next to the Shoot control plane in the Seed cluster. (#904, @mvladev)
  • Use the new networking.gardener.cloud/to- labels on these control plane components to enable Egress traffic to various other components or endpoints.
• [OPERATOR] It's recommended to install a NetworkPolicy controller (such as Calico) to all Seed clusters, so that NetworkPolicies are enforced. (#904, @mvladev)

Most notable changes

• [USER] spec.cloud.{PROVIDER}.networks.nodes is now defaulted to the first worker CIDR, only if there is one worker in spec.cloud.{PROVIDER}.networks.workers. (#1082, @mvladev)
• [USER] End users can now have access to monitoring dashboards for a few of their control plane components. This change is effective for all shoots. The required credentials are stored with name <shootName>.monitoring in the project namespace in the garden cluster. (#1000, @wyb1)
• [USER] It is no longer possible to switch the kube-proxy mode for Kubernetes clusters >= 1.14.1 because Kube-Proxy's cleanup logic is broken (https://github.com/kubernetes/kubernetes/issues/78735). (#961, @DockToFuture)
• [USER] End users can now have access to logging dashboards for a few of their control plane components. This change is effective for all shoots. The required credentials are stored with name <shootName>.logging in the project namespace in the garden cluster. (This requires that the Logging feature gate is enabled in the gardener-controller-manager configuration.) (#851, @KristianZH)
• [OPERATOR] When deleting an extension controller that manages Extension resources then, before Gardener deletes the controller, first all Extension resources in the respective seed cluster(s) are deleted. Only after that the extension controller is deleted. This is to allow proper clean up. (#1095, @timuthy)
• [OPERATOR] etcd-main now emits extensive (histogram) metrics. The cardinality of these metrics could be similar to the kube-apiserver's latency metrics. (#1069, @amshuman-kr)
• [OPERATOR] Gardener will now cleanup the unused PVC etcd-main-etcd-main-0 which was previously backing etcd-main pod before gardener version 0.22.0. This will result in a restart of all etcd pods. (#1038, @swapnilgm)
• [OPERATOR] All components part of a shoot control plane running in Seed clusters are now completely isolated from each other. By default, all Egress and Ingress traffic is disabled. (#904, @mvladev)

Improvements

• [OPERATOR] Add support for different machine image and version in tm tests (#1102, @schrodit)
• [OPERATOR] Improved end-user facing dashboards. (#1099, @wyb1)
• [OPERATOR] Exposed etcd-backup-restore metrics to Prometheus (#1087, @PadmaB)
• [OPERATOR] Fixed an issue with Waiting for Infrastructure on failed Shoots that never had an infrastructure. (#1080, @adracus)
• [OPERATOR] The limits of kube-apiserver of shooted seed clusters have been increased to 2000m CPU and 7Gi memory, based on consistent recommendation from VPA. The requests are left untouched. (#1078, @amshuman-kr)
• [OPERATOR] kubelet.service now reads from /etc/hostname for hostname override. (#1077, @mvladev)
• [OPERATOR] Fix an issue with new Shoots sometimes not having health conditions. (#1072, @adracus)
• [OPERATOR] An issue has been fixed which prevented some shoot clusters from being removed because depending Extension resources hung in deletion. (#1071, @timuthy)
• [OPERATOR] Shoots running on 'shooted Seeds' are now reconciled simultaneously with their hosting Seed. This results in shorter reconciliation and thus also faster hibernation wakeup times. (#1068, @adracus)
• [OPERATOR] Fix kube-proxy Init container crash, when IPVS is enabled on systems without loaded IPv6 kernel module. (#1067, @mvladev)

[etcd-backup-restore]

Improvements

• [USER] In the case that initial delta snapshot fails, a full snapshot is tried instead. (gardener/etcd-backup-restore#166, @swapnilgm)

[terraformer]

Most notable changes

• [OPERATOR] The Terraform version has been upgraded to 0.11.14. (gardener/terraformer@157bce76c33d1d2ab95b28822be61f0edf300e63)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.25.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.25.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:0.25.0

[gardener]

Action Required

• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing certificate-services. Instead, it now relies on extension controllers which handle extensions.extensions.gardener.cloud of type: certificate-service. (see GEP-1 on extensibility). This change also implies the removal of CertificateManagement feature gate from gardener-controller-manager configuration (please remove when deploying this Gardener version, otherwise the gardener-controller-manager won't start). Also, if you previously have enabled the CertificateManagement feature gate you have to create an adequate ControllerRegistration resource now (e.g., see this). You also have to set .spec.resources.globallyEnabled=true so that every shoot gets the certificate service extension. Previously, the configuration had been provided as a Secret, but now it's part of the extension itself. See this for information how to configure ACME and the providers (note that clusterIssuerName has been renamed to issuerName). (#1029, @timuthy)
• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing infrastructure. Instead, it now relies on extension controllers to manage the infrastructure (networks, route tables, etc.) for the specific provider they have been developed for, e.g. AWS, Azure, GCP, etc. (see GEP-1 on extensibility). We have prepared the implementation of the Alicloud provider, AWS provider, Azure provider, GCP provider, OpenStack provider, and Packet provider. After updating Gardener you need to create corresponding ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). Please find example ControllerRegistration resources here (Alicloud), here (AWS), here (Azure), here (GCP), and here (OpenStack). To get information about more details please walk through these documents. (#930, @rfranzke)
• [OPERATOR] ⚠️ Gardener does no longer have in-tree supported for managing machines/workers. Instead, it now relies on extension controllers to manage the workers (deployment of machine-controller-manager, machine classes, machien deployments, etc.) for the specific provider they have been developed for, e.g. AWS, Azure, GCP, etc. (see GEP-1 on extensibility). We have prepared the implementation of the Alicloud provider, AWS provider, Azure provider, GCP provider, OpenStack provider, and Packet provider. After updating Gardener you need to create corresponding ControllerRegistration resources to make these extension controllers known to Gardener (otherwise none of your shoots will be able to get reconciled anymore). Please find example ControllerRegistration resources here (Alicloud), here (AWS), here (Azure), here (GCP), and here (OpenStack). To get information about more details please walk through these documents. (#930, @rfranzke)
• [OPERATOR] ⚠️ Together with the externalisation of the worker/machine management we have changed the way how supported machine image versions are declared in the CloudProfiles, and Shoots. Now, the machine image for every cloud provider is just defined by a (name,version)-tuple, e.g. {machineImage: {name: coreos, version: 1234.0}} (see example CloudProfile resources, e.g. this). The respective extension controller has to map this information to the cloud specific data, e.g., the AWS controller would map this to an AMI, the Azure controller to SKU, Offer, Publisher, etc. The extension controllers have a configuration (e.g., see this) that contains this mapping (basically, everything that previously was part of the CloudProfile is now part of this configuration). You can configure this also via their Helm charts, see e.g. this. Please make sure, that you use exactly the same versions of the machine images that you used previously in the CloudProfiles, otherwise, Gardener will trigger a rolling update of the worker machines of the shoot clusters. (#930, @rfranzke)
  • ⚠️ The concrete steps for deploying this Gardener version are:
  • 1. Scale down the gardener-controller-manager deployment: kubectl -n garden scale deployment/gardener-controller-manager --replicas=0
  • 2. Deploy the Gardener 0.24.0 Helm chart.
  • 3. Update all your CloudProfiles to correctly reflect the machine images (name/version) tuples instead of provider-specific fields/information.
  • 4. Deploy the needed ControllerRegistration resources for the provider extensions you want to use (with correct machine image mapping configuration).
  • 5. Scale up the gardener-controller-manager deployment: kubectl -n garden scale deployment/gardener-controller-manager --replicas=1
• [OPERATOR] In the previous release, all main etcd StatefulSets for the shoot clusters have been updated to use a fast storage volume. However, the migration of etcd data directory from the old volume to the new volume happens only when the etcd pod runs post migration. For hibernated clusters, this is not the case. Hence, migration does not happen for these clusters. To force migration of etcd for these clusters, we should use this script to scale up the etcd StatefulSets from hibernation and subsequently scale it back down post migration. This script is meant to be run once before the next release of Gardener which removes the slow volumes of etcd. (#1043, @georgekuruvillak)
• [USER] PodSecurityPolicy gardener.unprivileged now uses the default docker seccomp policy (https://docs.docker.com/engine/security/seccomp/). (#1048, @mvladev)
  • If privileged containers are disabled via .spec.kubernetes.allowPrivilegedContainers=false and blocked system calls are used by containers, then a new PodSecurityPolicy needs to be created, allowing unconfined seccomp access. See https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp.
• [OPERATOR] ⚠️ Gardener does no longer have support for the local (VirtualBox/Vagrant-based) provider. You have to delete all your local clusters before upgrading Gardener. (#930, @rfranzke)

Most notable changes

• [USER] It is now possible to configure CPU manager policy in the .spec.kubernetes.kubelet configuration for shoot clusters. (#1064, @afritzler)
• [USER] It is now possible to configure various settings (scaleDownUnneededTime, scaleDownDelayAfterAdd, scaleDownDelayAfterFailure, scaleDownDelayAfterDelete, scanInterval) for the cluster-autoscaler in the shoot. Please consult the example Shoot manifests for details. (#1049, @afritzler)
• [USER] It is now possible to configure CPU throttling in the .spec.kubernetes.kubelet configuration for shoot clusters. (#1040, @afritzler)
• [USER] The Project resource is extended with the optional field viewers which is a list of subjects with read-only access (except Secrets) to the Gardener API. (#1004, @vpnachev)
• [OPERATOR] It is now possible to force the deletion of pending BackupInfrastructure resources prior to globally configured deletionGracePeriodDays by annotating the resource with backupinfrastructure.garden.sapcloud.io/force-deletion=true. (#1058, @swapnilgm)
• [OPERATOR] When multiple machine images are specified in the CloudProfile and a Shoot is created without specifying a machine image then the first one in the CloudProfile is always chosen and considered to be the default. Previous versions of Gardener prohibited creating Shoot resources without specifying a machine image when the referenced CloudProfile did contain more than one machine image. (8969ff71e9549a5b2fb4255f408afaad097ab488)

Improvements

• [OPERATOR] Added KUBELET_EXTRA_ARGS to kubelet service (#1061, @afritzler)
• [OPERATOR] The BackupInfrastructure controller does now delete pending buckets faster (based on the configured grace period and the actual deletion and independent of the controller's sync period). (#1053, @swapnilgm)
• [OPERATOR] PodSecurityPolicy gardener.unprivileged now has a default seccomp policy runtime/default for clusters >= 1.11 and docker/default for clusters < 1.11. (#1048, @mvladev)
• [OPERATOR] Update controlplane webhooks documentation (#1047, @stoyanr)
• [OPERATOR] Gardener now reconciles project resources on restart. (#1037, @swilen-iwanow)
• [OPERATOR] It is now possible to configure several command line flags that influence the cluster autoscaler behaviour in the .spec.kubernetes.clusterAutoscaler configuration for shoot clusters. (#1033, @afritzler)
• [OPERATOR] The etcd-backup secret in the shoot namespaces does now also contain the bucket name. This is only a temporary solution until the backup infrastructure extension controllers have been implemented. By then, this information can be read out of the backup infrastructure extension CRD. (8fe1ce87d7ec89f45f7d71472c77272c7b545050)
• [OPERATOR] All extension namespaces in the seed clusters are now labeled with gardener.cloud/role=extension as well as controllerregistration.core.gardener.cloud/name=<name>. (2c0cb6f88fb612730fca7d05e6500540e6d8bd56)
• [OPERATOR] All shoot namespaces in the seed are now labeled with gardener.cloud/role=shoot. The shoot.garden.sapcloud.io/hibernated label still exists but is deprecated now. To get the information whether the shoot is hibernated or not the Cluster extension resource should be consulted. (2c0cb6f88fb612730fca7d05e6500540e6d8bd56)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.24.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.24.0

[gardener]

Most notable changes

• [OPERATOR] Calico has been upgraded to v3.7.2. (#1018, @rfranzke)

Improvements

• [OPERATOR] Increased the failedProposal count in the alert KubeEtcd3HighNumberOfFailedProposals to 80 in order to raise an etcd alert (#1036, @PadmaB)
• [OPERATOR] Gardener can now make use of Flexvolumes for selected Seeds on Alicloud. (#1025, @jia-jerry)

[machine-controller-manager]

Most notable changes

• [USER] Deletion of annotations, labels and taints are now supported (gardener/machine-controller-manager#268, @prashanth26)

Improvements

• [OPERATOR] The machine-controller-manager does now correctly add its finalizer to Alicloud machine class secrets. (gardener/machine-controller-manager#271, @rfranzke)
• [OPERATOR] Multiple taints with the same key but different values are now supported (gardener/machine-controller-manager#268, @prashanth26)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.23.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.23.0