[gardener]

Improvements

• [OPERATOR] A bug that was preventing the ShootState resource to be updated with newly generated secrets is now fixed. (6b7fc47d5b0348cfdf42ef588e8456ff8e99f6a6)

[gardener]

Improvements

• [USER] gardenlet no longer tries to deploy new resources in the Shoot namespace in the Seed when the corresponding namespace is marked for deletion (no new resources can be created in such namespace). (5967c29f4a3b780a9851232de4762aa08ca70ea0)
• [OPERATOR] A cache issue preventing Shoot reference controller of gardener-controller-manager to do not reconcile sometimes references for newly created Shoots when the CachedRuntimeClients feature gate is enabled is now fixed. (2a320c81e85550a7ef41e7e7179c9daddea076e3)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.3
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.3
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.3

[gardener]

Most notable changes

• [OPERATOR] apiserver-proxy overload's manager is removed. (#3063, @mvladev)

Improvements

• [USER] gardener-controller-manager's Shoot reference controller now also handles audit policy ConfigMap references. (17084191c752c206537b9506b54828f4d723d9b7)
• [OPERATOR] Gardener now deletes (Cluster)RoleBindings of system components or addons, that were changed to an invalid state by endusers to be able to reconcile them back to the desired state. (f5bad77a249c57a06d7bdc586073b3ff40386573)
• [OPERATOR] A bug that was preventing the ShootState resource to be updated with newly generated secrets is now fixed. (17ac770ddaba554b512463c7b578534ac148c64f)
• [OPERATOR] Remove egress restrictions for vpn-shoot pod as it was incompatible with the cilium network plugin. (cd5584d0944bd53e9964b420c23730adfda3e394)
• [OPERATOR] The kube-controller-manager VPA now has minAllowed values to prevent VPA from scaling it down too much. (9de318e3e03fbc056418eab8e224feb92c1bfbb2)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.2
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.2
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.2

[gardener]

Improvements

• [USER] An issue causing spec.kubernetes.kubelet.kubeReserved.pid field of the Shoot to be set for Kubernetes versions that don't support the corresponding feature is now fixed. (0e4a3cfea10df856c25fd61c7d983c1cfdbbca93)
• [USER] Node exporter properly reports filesystem size for operating systems that use an xfs filesystem (1e49ce36ebc5d972a62524ecce8875de9b6f1fea)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.1
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.1

[gardener]

Action Required

• [USER] apiserver-proxy now listens on port 16910 on all Nodes if APIServerSNI feature gate is enabled. It causes a conflict with workloads with hostNetwork: true and listening on 0.0.0.0:16910, so it's required to change your workload's bind port. (#3044, @mvladev)
• [USER] DNS providers not used for the shoot's .spec.dns.domain, also known as additional, non-primary providers, must now be specified with .type and .secretName. Such providers were previously removed automatically from the shoot specification during create or update requests. (#3036, @timuthy)
• [OPERATOR] The seed taints seed.gardener.cloud/{disable-capacity-reservation,disable-dns,invisible} are no longer respected and are disallowed in gardener version v1.12 (will be enabled again in the next minor version without any special semantic like previously). They are automatically removed from the seed resources , please use the respective seed.spec.settings fields from now on. (#2955, @vpnachev)
• [OPERATOR] global.apiserver.admissionConfig configuration of charts/gardener/controlplane is no longer used. Existing plugin configuration must be migrated to use the new global.apiserver.admission.plugins list. ValidatingAdmissionWebhook or MutatingAdmissionWebhook plugins must not be used. (#2673, @mvladev)

Most notable changes

• [USER] Defaults reserved PIDs for kubelet and container runtime to 20k. (#3042, @BeckerMax)
• [USER] When a Shoot load balancer (vpn-shoot or addons-nginx-ingress-controller) cannot be ensured, gardenlet now fetches the involved object events (with type Warning) and adds them to the error message which is shown in the Shoot status. In this way users and operators will be able to identify better issues in which load balancer cannot be ensured because of invalid cloud provider credentials or another cloud provider issue. (#3028, @ialidzhikov)
• [USER] Failed conditions on Shoots do now transition from False status to Progressing in case the reason or message changes (if thresholds are defined in the gardenlet component config only, otherwise Progressing is not used anyways). (#3013, @rfranzke)
• [USER] The KonnectivityTunnel feature can now be configured for a single shoot cluster via the alpha.featuregates.shoot.gardener.cloud/konnectivity-tunnel annotation on the Shoot (true to enable it, false to disable it). (#3007, @DockToFuture)
• [USER] The EveryNodeReady condition on Shoot resources does now reflect kubelet version mismatches for shoot worker nodes (e.g., its status will be False if a kubelet wasn't updated after a patch version change, for example). (#3002, @rfranzke)
• [USER] Gardener now protects DNS provider secrets from deletion requests as long as they are still being referenced by one or multiple shoot clusters. More information can be found in the documentation. (#2771, @timuthy)
• [OPERATOR] It is now possible to submit provider configuration (for the .spec.provider.providerConfig field in Seed resources) when creating shooted seeds. Please consult the documentation for more information (#3035, @MartinWeindel)
• [OPERATOR] gardener-apiserver now has a new feature gate SeedChange. If set, this feature gate enables updating the spec.seedName field during shoot validation from a non-empty value in order to trigger shoot control plane migration. (#3024, @stoyanr)
• [OPERATOR] The vpa-exporter is now deployed again into the garden namespace of seed clusters (also for shooted seeds). (#3022, @wyb1)
• [OPERATOR] gardenlet is now deleting all VolumeAttachments on shoot hibernation. As during hibernation machine-controller-manager performs a "force" deletion of machines and does not wait for volumes to detach, kube-controller-manager is not able to delete the corresponding VolumeAttachments (and also the external-attacher to notice this deletion and remove its finalizer from the VolumeAttachment). Deleting VolumeAttachments on hibernation should prevent VolumeAttachments to be orphaned. Currently in the upstream kube-controller-manager, there is no garbage collection for VolumeAttachments (see kubernetes/kubernetes#77324). (#2963, @ialidzhikov)
• [OPERATOR] Change the shoot namespace matching regex of fluent-bit to shoot- (#2933, @vlvasilev)
• [OPERATOR] Change the promtail batch wait to 30 seconds (#2933, @vlvasilev)
• [OPERATOR] Deletion of a Seed is now possible with an existing Backup Bucket (but having no Shoots deployed!). The Bucket is deleted automatically during the Seed reconciliation flow. (#2931, @danielfoehrKn)
• [OPERATOR] global.apiserver.admission.plugins can now be used to configure admission plugins of the Gardener API Server. ValidatingAdmissionWebhookorMutatingAdmissionWebhook` plugins must not be used. (#2673, @mvladev)
• [OPERATOR] global.apiserver.admission.validatingWebhook and global.apiserver.admission.mutatingWebhook can now be used to configure validating/mutating admission plugins of the Gardener API Server. If enabled, Service Account Token Volume Projection is could be used to generate tokens which are used for authentication against webhooks. (#2673, @mvladev)
• [OPERATOR] The Gardener API server now supports the usage of ResourceQuotas for Gardener API groups and resources like Shoots, Seeds, SecretBindings, etc.. At the moment the quota supports object counts only, e.g. count/shoots.core.gardener.cloud. (#2627, @timuthy)
  • ⚠️ Please make sure the kube-controller-manager in your Garden cluster runs with --controllers=...,resourcequota,... if you want to use them.
• [DEVELOPER] The terraformer library can now be used to deploy terraformer@v2 Pods. To enable this, you have to set UseV2(true) on the Terraformer instance, otherwise the old PodSpec, that is compatible with v1, will be used. (#3034, @timebertt)
• [DEVELOPER] The newly added client.Client implementation under pkg/client/kubernetes/utils can be used to enable/disable the controller-runtime cache only for a given set of object kinds. (#2940, @timebertt)
  • e.g. you can pass utils.NewClientFuncWithDisabledCacheFor(&corev1.Secret{}) to manager.Options.NewClient to use a client, that always reads Secrets directly from the API server instead of caching all Secrets in the cluster.

Improvements

• [USER] New dashboard called API Server Proxy showing data for apiserver-proxy is now available in the Grafana dashboard of the cluster, if APIServerSNI feature gate is enabled. (#3051, @mvladev)
• [USER] Improved custom DNS usage documentation (#3043, @afritzler)
• [USER] New logging dashboards are added, prefixed with Logging for each exposed component. (#2945, @Kristian-ZH)
  • Logging panels in Kubernetes Controlplane Status are removed
• [OPERATOR] Prometheus now scrapes apiserver-proxy, if APIServerSNI feature gate is enabled. (#3051, @mvladev)
• [OPERATOR] New dashboard called API Server Proxy showing data for apiserver-proxy is now available in the Grafana dashboard of the cluster, if APIServerSNI feature gate is enabled. (#3051, @mvladev)
• [OPERATOR] nginx is replaced with envoy for apiserver-proxy. This allows to have the same proxy for both the client and the server. (#3041, @mvladev)
• [OPERATOR] apiserver-proxy now uses PROXY Protocol v2 when talking to the upstream kube-apiserver. (#3041, @mvladev)
• [OPERATOR] Fix old integration test to set tenant Id and to use pod_name for search (#3037, @vlvasilev)
• [OPERATOR] A bug has been fixed that can cause an unwanted Shoot reconciliation when changing the worker specification of a hibernated Shoot. (#3032, @danielfoehrKn)
• [OPERATOR] Fixes a bug in the Shoot reconciliation that could lead to failed Shoot deletions when the Hibernation spec has been changed during an active reconciliation. (#3029, @danielfoehrKn)
• [OPERATOR] If APIServerSNI featuregate is enabled, system components in the kube-system namespace are talking to their kube-apiserver via its FQDN. (#3026, @mvladev)
• [OPERATOR] Loki vpa is replaced with hvpa (#3025, @Kristian-ZH)
• [OPERATOR] A bug has been fixed that prevented the correct cluster-autoscaler seed bootstrapping (concretely, for new seeds the central cluster-autoscaler RBAC ClusterRole was not created), which, as a consequence, led to broken/non-working cluster-autoscaler pods for all shoots on that seed. (#3015, @rfranzke)
• [OPERATOR] Upgrade grafana to version 7.2.1 (#3008, @wyb1)
• [OPERATOR] Grafana dashboards now use $__rate_interval instead of hard coded intervals. This should improve dashboard performance when selecting large time ranges. (#3008, @wyb1)
• [OPERATOR] istiod and istio-ingress-gateway now have PodDisruptionBudget. (#3005, @mvladev)
• [OPERATOR] Fix a bug where gardener.cloud--allow-to-dns network policy do not allow traffic to DNS when NodeLocalDNS feature gate is enabled. (#3003, @mvladev)
• [OPERATOR] Liveness probe added to konnectivity-server (#3000, @DockToFuture)
• [OPERATOR] Several improvements for extension health checks have been implemented which increase the runtime performance and reduce network traffic. (#2995, @timuthy)
• [OPERATOR] Loki labels which come from the metadata are removed (#2994, @Kristian-ZH)
• [OPERATOR] Separate the fluent-bit-to-loki plugin build from a specific fluent-bit image. (#2994, @Kristian-ZH)
• [OPERATOR] Extract docker_id from the log tag to the kubernetes metadata when FallbackToTagWhenMetadataIsMissing flag is set. (#2994, @Kristian-ZH)
• [OPERATOR] port-forwarding of the operator's and user's grafana will not work anymore for accessing logs. (#2992, @Kristian-ZH)
  • The only way to access logs is by using the Ingress, because it attaches a new header with the tenantID in the request.
• [OPERATOR] Istio Dashboards can now be accessed in the aggregate Grafana, if ManagedIstio feature gate is enabled. (#2989, @mvladev)
• [OPERATOR] For newly created machines, cloud-config-downloader systemd service is now configured with maximum allowed time of 20 minutes to do its job. This change is needed, to mitigate issues where the execution of the service got stuck and in-place changes might not be applied, for example kubelet patch update. (#2987, @vpnachev)
• [OPERATOR] A bug has been fixed which prevented CloudProfiles with a SeedSelector from being created/applied/updated/deleted. (#2985, @timebertt)
• [OPERATOR] An issue causing CoreDNS dashboard to show always 'No Data' is now fixed. (#2984, @wyb1)
• [OPERATOR] MCM provider now also returns unregistered nodes to Autoscaler. This change enables autoscaler to pick up an alternate worker-pool if the chosen one can't be scaled-up. (#2974, @hardikdr)
• [OPERATOR] Autoscaler avoids scaling down the machines which are already being terminated. (#2974, @hardikdr)
• [OPERATOR] When migrating hibernated Shoots, the kube-apiserver is now properly scaled down to 0 if it was previously scaled up. (#2973, @plkokanov)
• [OPERATOR] Fix a bug which was causing the seed.spec.settings fields to be flapping between true and false when the settings and the taints in the sent request are not synced. (#2955, @vpnachev)
• [OPERATOR] Add image_id label to seed:images:count recording rule (#2951, @wyb1)
• [OPERATOR] An issue causing Seed bootrap to fail for v1.19 clusters is now fixed. (#2949, @ialidzhikov)
• [OPERATOR] VPA minAllowed configuration for etcd-druid to avoid crashloops if VPA scales down the resources too low from which automatic recovery doesn't happen. (#2941, @amshuman-kr)
• [OPERATOR] An issue causing premature deletion of the stale VPA webhook before the creation of the new VPA webhook for ShootedSeeds is now fixed. (#2937, @ialidzhikov)
• [OPERATOR] An issue causing the stale VPA webhook to do not be cleaned up is now fixed. (#2937, @ialidzhikov)
• [OPERATOR] A bug has been fixed that may cause the shoot deletion to stuck. (#2923, @rfranzke)
• [OPERATOR] Add network policies for Core DNS and VPN Shoot (#2859, @wyb1)
• [OPERATOR] Fluent-bit configurations can be overwritten in the Gardenlet's configurations. (#2796, @Kristian-ZH)
• [DEVELOPER] Gardener components now support --version flag that prints the component version information and useful metadata. (#3040, @ialidzhikov)
• [DEVELOPER] The Golang version has been upgraded to 1.15.3. (#3038, @ialidzhikov)
• [DEVELOPER] The hack/check-generate.sh script is now able to detect manual changes in the vendor folder by running the revendor Make rule. (#2999, @rfranzke)
• [DEVELOPER] Several improvements for extension health checks (lib github.com/gardener/gardener/extensions) have been implemented which increase the runtime performance and reduce network traffic. Please update your github.com/gardener/gardener to the latest version to profit from these changes. (#2995, @timuthy)
• [DEVELOPER] Two hack scripts have been added (hack/local-development/local-garden/etcdctl-{gardener,kube}-etcd.sh) for talking to the etcds running in the local-garden. (#2980, @timebertt)
• [DEVELOPER] Restore the working logging configuration (#2952, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.0

[gardener]

Improvements

• [OPERATOR] A bug has been fixed which prevented CloudProfiles with a SeedSelector from being created/applied/updated/deleted. (28cb7e15c9340c90413bb507591b5d73adb35be6)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.11.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.11.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.11.3
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.11.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.11.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.11.3

[gardener]

Improvements

• [OPERATOR] gardenlet is now deleting all VolumeAttachments on hibernation. As during hibernation machine-controller-manager performs a "force" deletion of machines and does not wait for volumes to detach, kube-controller-manager is not able to delete the corresponding VolumeAttachments (and also the external-attacher to notice this deletion and remove its finalizer from the VolumeAttachment). Deleting VolumeAttachments on hibernation should prevent VolumeAttachments to be orphaned. Currently in the upstream kube-controller-manager, there is no garbage collection for VolumeAttachments (see kubernetes/kubernetes#77324). (fe90914ebb16ae677717fa2f870a1313a28bebca)
• [OPERATOR] An issue causing Seed bootrap to fail for v1.19 clusters is now fixed. (a37532e70837063aaded69484e8863599415ec36)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.11.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.11.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.11.2
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.11.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.11.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.11.2

[gardener]

Improvements

• [OPERATOR] An issue causing premature deletion of the stale VPA webhook before the creation of the new VPA webhook for ShootedSeeds is now fixed. (34778022c16ff2d50d438dd2e6d75424e41fa739)
• [OPERATOR] An issue causing the stale VPA webhook to do not be cleaned up is now fixed. (34778022c16ff2d50d438dd2e6d75424e41fa739)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.11.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.11.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.11.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.11.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.11.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.11.1

[gardener]

Action Required

• [OPERATOR] Kubernetes 1.16+ is required for Seed clusters which have ManagedIstio feature gate. (#2866, @mvladev)
• [DEVELOPER] Some commonly used test utils and gomega matchers have been moved to pkg/utils/test(/matchers) and isolated from unneeded dependencies to make them easier to reuse. (#2912, @timebertt)
• [DEVELOPER] The flow package's progress reporter option has been changed to return the new ProgressReporter interface. You can call flow.NewImmediateProgressReporter with your reporter function as a replacement. (#2908, @rfranzke)
• [DEVELOPER] ./hack/test.sh now executes tests via go test instead of ginkgo. Please adapt your extensions' Makefiles, if you use the vendored hack scripts. (#2809, @timebertt)
• [DEVELOPER] Machine dependency hook methods DeployMachineDependencies and DeployMachineDependencies need to be implemented in the worker controllers of the provider extensions. (#2806, @dkistner)
• [DEVELOPER] The provider extension need to transform the existing GetMachineImages() in the implementation of the WorkerDelegate interface to the new UpdateMachineImageStatus() method. The provider extensions need now to update the provider status on their own in the new UpdateMachineImageStatus() method instead of returning it. (#2806, @dkistner)

Most notable changes

• [OPERATOR] Machines without .spec.providerID or .status.node will no longer be persisted in the Worker' .status.state field. This is to prevent unnecessary updates to the ShootState resources. (#2909, @rfranzke)
• [OPERATOR] It is now possible to delay the progress reporting for shoot operations by setting the .controllers.shoot.progressReportPeriod field in the gardenlet component config. This might helpful in large landscape where a lot of shoots exist to limit the number of updates to the Shoots' status sections. (#2908, @rfranzke)
• [OPERATOR] The ControllerRegistration resource does now support the new AlwaysExceptNoShoots deployment policy. Respective extension controllers using this policy are only being deployed to seeds if there is at least one shoot. (#2896, @rfranzke)
• [OPERATOR] Gardener validating webhooks have been moved from the Gardener-Controller-Manager to a dedicated component Gardener-Admission-Controller. Therefore, new values have been added to the Gardener Helm chart (charts/gardener/controlplane). Please consult the documentation (docs/concepts/admission-controller.md) for more information about the Gardener-Admission-Controller. (#2832, @timuthy)
• [OPERATOR] The Gardener Controller Manager is now equipped with a validation handler which checks incoming resource requests against configured quota configurations. It especially enables operators to restrict the maximum size of a single resource (e.g. shoot, plant, secret, ...) users apply to the Garden cluster and is at the same time a measure against DoS attacks. Please consult the documentation docs/concepts/controller-manager.md#Resource-Size-Validator for more details. (#2781, @timuthy)
• [DEVELOPER] Gardener managed secrets, like kubeconfig, ssh-keypair, monitoring, are now labelled with gardener.cloud/role which enables clients to filter these secret types via label matchers. (#2844, @timuthy)
• [DEVELOPER] Gardener validating webhooks have been moved from the Gardener-Controller-Manager to a dedicated component Gardener-Admission-Controller. Please run make dev-setup to create the necessary configuration for this component for local development use-cases. make start-admission-controller starts the new component locally. (#2832, @timuthy)
• [DEVELOPER] make test now makes use of cached test results to reduce turn-around times when writing unit tests. (#2809, @timebertt)

Improvements

• [USER] A bug has been fixed that reported inappropriate codes for certain error situations. (#2918, @rfranzke)
• [USER] The version of the Kubernetes Dashboard addon has been bumped from v2.0.0 to v2.0.3. For 1.19 shoot clusters the v2.0.4 version will be used. (#2813, @rfranzke)
• [OPERATOR] kube-state-metrics is now updated to v1.9.7. (#2913, @ialidzhikov)
• [OPERATOR] Fix secrets.Interface not matching the implementing struct Secrets (#2903, @kon-angelo)
• [OPERATOR] Istio is updated to 1.7.2 (#2902, @mvladev)
• [OPERATOR] A bug has been fixed that prevented BackupBucket and BackupEntry resources from being deleted despite the respective infrastructure provider credentials have been updated to the correct data. (#2898, @rfranzke)
• [OPERATOR] A bug has been fixed which prevented the deletion of ControllerInstallation resources in case the Seed has a deletion timestamp. (#2896, @rfranzke)
• [OPERATOR] The BackupBucket and BackupEntry resources in the garden cluster are now only reconciled once per 24h by the gardenlet. (#2889, @rfranzke)
• [OPERATOR] The shoot's .status.conditions[].lastUpdateTime is no longer continuously updated. Note that if the gardenlet does not renew its lease anymore then the gardener-controller-manager will, after a certain threshold, set the Shoot's conditions to Unknown to indicate that the displayed information is most likely outdated. (#2888, @rfranzke)
• [OPERATOR] Istio is updated to 1.7.1 (#2866, @mvladev)
• [OPERATOR] Defrag shooted seed etcd-main every day. This is to temporarily mitigate the high load on seed apiserver from recent changes. (#2862, @amshuman-kr)
• [OPERATOR] An issue preventing an admission plugin config (.spec.kubernetes.kubeAPIServer.admissionPlugins[*].config) to be specified for the Kubernetes apiserver is now fixed. (#2861, @ialidzhikov)
• [OPERATOR] An issue where the custom admission plugin settings might of one shoot cluster can be applied to other shoots reconciled by the same seed cluster has been fixed. (#2860, @vpnachev)
• [OPERATOR] Add API Resources panel to the API Server Details dashboard (#2857, @wyb1)
• [OPERATOR] Fix the query for the API Resources panel (#2857, @wyb1)
• [OPERATOR] The metrics-server now has a readiness and liveness probes. (#2855, @vpnachev)
• [OPERATOR] Increase default apiservers limits-requests gap to deal with sluggish CPU recommendations from VPA. (#2853, @amshuman-kr)
• [OPERATOR] Add the feature flag MountHostCADirectories that enables mounting common CA certificate directories in the Shoot API server pod that might be required for webhooks or OIDC. (#2852, @danielfoehrKn)
  • Enabling this feature gate removes mounting common CA files directly - as this can lead to problems.
  • The reason for adding this as a feature flag is that there are known compatibility issues with the openstack extension provider (can lead to failed kube-apiserver deployments for certain Kubernetes versions) and the provider-aws/gcp/azure (can lead to missing /etc/ssl/ mounts in the Shoot API Server).
  • Please consult the compatibility docs under /docs in the extension repository.
• [OPERATOR] The konnectivity-server is now logging to the stdout. (#2850, @vpnachev)
• [OPERATOR] The VPA for the nginx addon is now set to guarantee at least 100m cpu and 128Mi memory for the controller pod. (#2848, @vpnachev)
• [OPERATOR] Enabling SNI feature gate now changes the type of kube-apiserver Service to ClusterIP only after DNSEntries are updated to point to the istio ingress gateway. (#2847, @mvladev)
• [OPERATOR] Upgrade fluent-bit-to-loki plugin to version 0.26.0 (#2846, @vlvasilev)
• [OPERATOR] Gardener managed secrets, like kubeconfig, ssh-keypair, monitoring, are now labelled with gardener.cloud/role which enables clients to filter these secret types via label matchers. (#2844, @timuthy)
• [OPERATOR] Make VPA scale modes for gardener-apiserver's hvpa configurable. (#2843, @ggaurav10)
• [OPERATOR] The default rolling update settings of the gardener-apiserver (maxSurge=1, maxUnavailable=0, minReadySeconds=30) can now be controlled in the gardener/controlplane Helm chart. (#2841, @rfranzke)
• [OPERATOR] An issue causing the 'VPN Connection' tile in Grafana to show always 'No data' is now fixed. (#2835, @wyb1)
• [OPERATOR] The generic worker actuator does now wait for undesired MachineDeployment deletion only when the machine-controller-manager is scaled up. (#2834, @ialidzhikov)
• [OPERATOR] The generic worker actuator does no longer consider machine class secrets with the label garden.sapcloud.io/purpose. (#2826, @ialidzhikov)
• [OPERATOR] Upgrade Prometheus to v2.20.1 (#2820, @wyb1)
• [OPERATOR] node-problem-detector priority class is now set to system-cluster-critical. (#2819, @ialidzhikov)
• [OPERATOR] The gardenlet's ControllerInstallationRequired controller was optimized to execute less API calls to improve stability on startup of the gardenlet. (#2816, @timebertt)
• [OPERATOR] Logs in the worker controller and generic actuator have been improved to make them more consistent and useful. (#2807, @timebertt)
• [OPERATOR] Disable self-registration of vpa-webhook by vpa-admission-controller. Now Gardener manages the vpa-webhook (#2802, @ggaurav10)
• [OPERATOR] Fixed a caching issue when updating the ShootState resource. Additionally PATCH is now used to update the ShootState.Spec.Gardener field. (#2798, @plkokanov)
• [OPERATOR] The step which loads existing secrets into the ShootState has been removed as it is not needed anymore. (#2798, @plkokanov)
• [OPERATOR] Shoot Kube apiserver mounts ssl directories instead of files to avoid creation of empty files. (#2791, @danielfoehrKn)
  • This includes directory /etc/ssl/.
• [OPERATOR] Please update your provider extension to version >= 1.16.0. Previous provider extensions remove the /etc/ssl/ directory mount from the kube-apiserver deployment via webhook . This is a problem when the Shoot Kubernetes version > 1.17 and CSI is enabled, as the Shoot API Server could require the Root CAs in /etc/ssl/ to make requests to OIDC providers or webhook endpoints. (#2791, @danielfoehrKn)
  • The Gardenlet now always adds the /etc/ssl/ mount - a provider extension with a previous version would remove the /etc/ssl/ mount from the deployment if the Shoot Kubernetes version > 1.17 and CSI is enabled.
• [OPERATOR] Fixes a bug that could cause Shoot garbage collection to fail in certain cases. (#2782, @danielfoehrKn)
• [OPERATOR] An issue causing TryUpdate and TryUpdateStatus funcs to do not retry on conflict in a corner case is now resolved. (#2770, @swilen-iwanow)
• [DEVELOPER] provide values gardener.garden.clusterIdentity and gardener.seed.clusterIdentity for controller registration helm charts. (#2851, @MartinWeindel)
  • The old values gardener.garden.identity and gardener.seed.identity are deprecated and will be removed in a future release.
• [DEVELOPER] An integration test for the BackupBucket extension controller has been added and will run as part of make test{-cov}. (#2830, @rfranzke)
• [DEVELOPER] The go race detector was enabled for unit tests. (#2809, @timebertt)

[dependency-watchdog]

Improvements

• [OPERATOR] Fix panic during shoot spec and status check. (gardener/dependency-watchdog#29, @amshuman-kr)
• [OPERATOR] Improve logging while scaling. (gardener/dependency-watchdog#24, @amshuman-kr)
• [OPERATOR] Probe is stopped for clusters that are hibernating, or in hibernation or are still waking up from hibernation (gardener/dependency-watchdog#23, @ggaurav10)

[logging]

Action Required

• [OPERATOR] Flag QueSegmentSize is renamed to QueueSegmentSize (gardener/logging#67, @vlvasilev)

Improvements

• [OPERATOR] New flags are added to extract kubernetes metadata from a tag entry in the log. The metadata consist of pod_name, namespace and container_id. (gardener/logging#67, @vlvasilev)
• [OPERATOR] Logs without kubernetes metadata can be dropped via setting a flag DropLogEntryWithoutK8sMetadata. (gardener/logging#67, @vlvasilev)
• [OPERATOR] When "Out OF Order Timestamp" occurs the log message prints latest timestamp, incoming timestamp, host and label set of the log (gardener/logging#66, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.11.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.11.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.11.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.11.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.11.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.11.0

[gardener]

Action Required

• [DEVELOPER] The flow package's progress reporter option has been changed to return the new ProgressReporter interface. You can call flow.NewImmediateProgressReporter with your reporter function as a replacement. (2df2fec1d2e25bd00aafa1436923422ee56a2b51)

Most notable changes

• [OPERATOR] It is now possible to delay the progress reporting for shoot operations by setting the .controllers.shoot.progressReportPeriod field in the gardenlet component config. This might helpful in large landscape where a lot of shoots exist to limit the number of updates to the Shoots' status sections. (2df2fec1d2e25bd00aafa1436923422ee56a2b51)
• [OPERATOR] Machines without .spec.providerID or .status.node will no longer be persisted in the Worker' .status.state field. This is to prevent unnecessary updates to the ShootState resources. (5b92cdc561d37c61660c4a2e5bdab47c16fcc847)

Improvements

• [OPERATOR] The shoot's .status.conditions[].lastUpdateTime is no longer continuously updated. Note that if the gardenlet does not renew its lease anymore then the gardener-controller-manager will, after a certain threshold, set the Shoot's conditions to Unknown to indicate that the displayed information is most likely outdated. (ffb2c7c66c6790bbdece39fe9f31c89e5f43fb6a)
• [OPERATOR] The BackupBucket and BackupEntry resources in the garden cluster are now only reconciled once per 24h by the gardenlet. (db3c82472ff256c2048d21a3dcb766596478fb76)
• [OPERATOR] Defrag shooted seed etcd-main every day. This is to temporarily mitigate the high load on seed apiserver from recent changes. (b50bbb1f36b8f023f3d43bede2e7e52f4a299fa8)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.10.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.10.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.10.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.10.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.10.2

[gardener]

Improvements

• [OPERATOR] An issue causing the 'VPN Connection' tile in Grafana to show always 'No data' is now fixed. (88efab317290b9585453f8c6dd0f7ab5cd96eba4)
• [OPERATOR] Make VPA scale modes for gardener-apiserver's hvpa configurable. (91b74c2c04445cdfb634595f84962061c3d448ec)
• [OPERATOR] The VPA for the nginx addon is now set to guarantee at least 100m cpu and 128Mi memory for the controller pod. (501fe69153fe0a1e44594427568eaa99a1339dc7)
• [OPERATOR] An issue preventing an admission plugin config (.spec.kubernetes.kubeAPIServer.admissionPlugins[*].config) to be specified for the Kubernetes apiserver is now fixed. (3e898aaf87fa693762af4444a73d8b4fed810daa)
• [OPERATOR] An issue where the custom admission plugin settings might of one shoot cluster can be applied to other shoots reconciled by the same seed cluster has been fixed. (be6af93c008409f48d9ea8d5b217cc07afe95dab)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.10.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.10.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.10.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.10.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.10.1

[gardener]

Most notable changes

• [OPERATOR] Gardener can now support shoot clusters with Kubernetes version 1.19. In order to allow creation/update of 1.19 clusters you will have to update the version of your provider extension(s) to a version that supports 1.19 as well. Please consult the respective releases and notes in the provider extension's repository. (#2799, @rfranzke)
• [OPERATOR] Due to the fact that the Kubernetes community no longer builds hyperkube images on their own we have created gardener/hyperkube. It produces Docker images containing only the kubelet and kubectl binaries which are used to bootstrap the shoot worker nodes. Please note that this means that new Kubernetes versions are now only supported by Gardener if there is a corresponding release on the gardener/hyperkube repository. (#2799, @rfranzke)
• [OPERATOR] Introduces Certificate Rotation for the Gardenlet. (#2542, @danielfoehrKn)

Improvements

• [USER] Fixed an error in the KubeApiServerLatency alert (#2776, @wyb1)
• [USER] Workers that configure a kubelet data volume can now use gardenlinux OS (#2775, @guydaichs)
• [OPERATOR] extensions/pkg/predicate.Or has been deprecated in favor of sigs.k8s.io/controller-runtime/pkg/predicate.Or. (#2797, @timebertt)
• [OPERATOR] gardener.cloud:system:administrators are now allowed to list namespaces, manage RBACs, admission webhooks and apiservices. (#2793, @vpnachev)
• [OPERATOR] Loki is upgraded to version 1.6.0 (#2780, @Kristian-ZH)
• [OPERATOR] Improve alerting for operators. Alerts should fire less frequently and are now grouped by the service label instead of cluster which should also reduce the amount of alerts sent (#2776, @wyb1)
• [OPERATOR] There is new dashboard for fluent-bit in garden's Grafana (#2769, @Kristian-ZH)
• [OPERATOR] ContainerRuntimes are now annotated with gardener.cloud/operation=restore during the restore phase of Control Plane Migration and their state (if any) is copied from the ShootState to the CRs' status.state field. (#2762, @plkokanov)
• [OPERATOR] It is not possible a seed cluster to be deleted, directly or via removal of the shoot.gardener.cloud/use-as-seed annotation on the shoot, if the seed is still used by a BackupBucket or it is hosting the control plane of a shoot cluster. (#2732, @vpnachev)

[logging]

Action Required

• [OPERATOR] Because the dynamic host field contains only the namespace the flags DynamicHostPrefix and DynamicHostSuffix must be set to http://<loki-service> and .svc:3100/loki/api/v1/push (gardener/logging#64, @vlvasilev)

Improvements

• [OPERATOR] Upgrade fluent-bit version to 1.5.4. (gardener/logging#64, @vlvasilev)
• [OPERATOR] Implement buffered Loki client. (gardener/logging#64, @vlvasilev)
• [OPERATOR] A mitigation has been implemented for the out of order error in the fluent bit plugin. It can be enabled by setting the flag ReplaceOutOfOrderTS to true. (gardener/logging#64, @vlvasilev)
• [OPERATOR] The usage of mutex locks in the custom plugin dispatching the logs between the different loki instances have been improved (gardener/logging#64, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.10.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.10.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.10.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.10.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.10.0

[gardener]

Action Required

• [OPERATOR] With this version of Gardener, a validation for supported container runtimes (CR) and container runtime interfaces(CRI) per machine image version has been introduced. To prevent disruptions for creation and update of shoot cluster using non default CR and CRI, the CloudProfiles needs to be enhanced with the list of supported CR and CRI per machine image version. An example can be found here (#2137, @vpnachev)

Most notable changes

• [USER] Gardener now checks if referenced DNS provider secrets (.spec.dns.providers[*].secretName) exist in the project namespace during shoot creation and update requests. Requests will be denied if the referenced secret is not available. (#2761, @timuthy)
• [USER] The experimental Kyma addon has been removed from Gardener, i.e., the setting experimental.addons.shoot.gardener.cloud/kyma annotation has no effect anymore. Existing Kyma installations will remain deployed/untouched. (#2701, @rfranzke)
• [OPERATOR] A new controller for Events related to Shoot objects has been added to the Gardener Controller Manager (disabled by default). It can be used to extend the live-span of events regarding shoot clusters (the live-span of all other events can be configured separately). Please find more information in this document. (#2649, @BeckerMax)

Improvements

• [USER] A documentation for the Shoot status has been added. (#2765, @ialidzhikov)
• [USER] An issue that prevented regular project admins from managing non-human users has been resolved. (#2763, @rfranzke)
• [USER] An issue causing Shoot reconciliation to fail at Maintain shoot annotations with optimistic lock error message is now mitigated. (#2746, @ialidzhikov)
• [USER] The number of concurrent controller syncs of the kube-controller-manager of Shoot clusters has been increased to allow faster processing of events. (#2740, @rfranzke)
• [USER] When the VPA for shoots is disabled then the CustomResourceDefinitions are no longer deleted (they will remain in the system, together with all the VerticalPodAutoscaler objects - if you don't need them anymore you can remove them with kubectl delete crd <crd-names>). (#2715, @rfranzke)
• [USER] A bug showing 403 Forbidden responses when creating new Projects has been fixed. (#2699, @rfranzke)
• [OPERATOR] The gardenlet now reports leader election events to the Seed cluster instead of the Garden cluster. (#2772, @timebertt)
• [OPERATOR] The generic worker actuator now also reports failed machines from the corresponding machine deployment in case the shoot cluster is being hibernated. Earlier scale down issues during hibernation were not reported to users, e.g. if something was wrong with the configured cloud provider account and thus the machine deletion was denied. (#2759, @timuthy)
• [OPERATOR] The shoot task annotation is now updated as soon as the respective task has completed successfully to prevent recurring executions in case the whole shoot reconciliation flow fails. (#2757, @rfranzke)
• [OPERATOR] The kube-scheduler is now auto-restarted in the shoot maintenance time window, similar to other controllers. (#2756, @rfranzke)
• [OPERATOR] A bug has been fixed that caused the REST Mapper to rediscover the available API resources very often. (#2752, @timebertt)
• [OPERATOR] Deploy logging stack earlier in the reconciliation flow. (#2750, @Kristian-ZH)
• [OPERATOR] The explicit terminationGracePeriodSeconds configuration of the Gardener components has been removed. (#2749, @rfranzke)
• [OPERATOR] Konnectivity tunnel proxy agent and server are now on version v0.0.12 (#2748, @zanetworker)
• [OPERATOR] Add monitoring for API Server Watches (#2743, @wyb1)
• [OPERATOR] Remove gardener-seed-admission Mutationgwebhookconfiguration and the mutating pod functionality of the seed-admission-controller (#2735, @vlvasilev)
• [OPERATOR] Kubernetes dependencies are now updated to v0.17.11. (#2728, @ialidzhikov)
• [OPERATOR] gardener-apiserver Deployment does now define a readiness probe. (#2728, @ialidzhikov)
• [OPERATOR] The resource requests for the fluent-bit DaemonSet running in the seed clusters have been increased. (#2723, @vpnachev)
• [OPERATOR] An issues has been fixed which caused the Gardenlet to exit ungracefully during the shoot reconciliation. (#2708, @timuthy)
• [OPERATOR] The error code mapping has been extended to categorize certain common issues upfront and furnish them with error codes. (#2702, @rfranzke)
• [OPERATOR] The gardenlet now spreads Shoot health checks to avoid running into rate limits directly after startup. (#2700, @timebertt)
• [OPERATOR] Control plane health checks have been added for VPA components of shoot clusters. These are executed regularly as soon as the shoot uses the Vertical-Pod-Autoscaling feature (https://github.com/gardener/gardener/blob/master/docs/usage/shoot_autoscaling.md#vertical-pod-auto-scaling). (#2698, @timuthy)
• [OPERATOR] The VPA components in the seed are now vertically auto-scaled as well. (#2696, @rfranzke)
• [OPERATOR] A bug has been fixed that might have caused a 24h absence of the VPA for shooted seeds. (#2695, @rfranzke)
• [OPERATOR] Entries with empty data are no longer added to the ShootState. (#2694, @plkokanov)
• [OPERATOR] A bug in the OpenAPI specification exposed by the Gardener API server has been fixed. (#2682, @rfranzke)
• [OPERATOR] A bug has been fixed which could cause a nil pointer exception in case the v1.8 Gardenlet tries to delete a shoot that wasn't reconciled yet. (#2681, @rfranzke)
• [OPERATOR] Objects into which controllerinstallations and extensions are retrieved during reconciliation by the ControllerInstallation controller and ShootState Sync controller are now created for each reconciliation, instead of the same object being reused multiple times per resource kind. (#2679, @plkokanov)
• [OPERATOR] An issue has been fixed which caused the Gardenlet to exit ungracefully due to the missing shoot cluster identify in the .status. (#2678, @timuthy)
• [OPERATOR] An issues has been fixed which prevented the bootstrapping of new seed clusters. (#2676, @timuthy)
• [OPERATOR] The fluent-bit DaemonSet is now tolerating the taint node-role.kubernetes.io/master with effect NoSchedule. (#2671, @einfachnuralex)
• [OPERATOR] ControllerInstallations are no longer created for ControllerRegistrations that are in deletion. (#2612, @vpnachev)
• [OPERATOR] Now every machine image version in the CloudProfile can specify list of supported container runtime interfaces and container runtimes. (#2137, @vpnachev)
• [DEVELOPER] The kube-apiserver used for the local garden development does now properly forward the client information to the gardener-apiserver. (#2764, @rfranzke)

[gardener-resource-manager]

Most notable changes

• [DEVELOPER] The new resources.gardener.cloud/keep-object annotation can be used on resources managed by ManagedResource objects in order to keep them in the system in case they get removed from the ManagedResource or the ManagedResource itself is being deleted. (gardener/gardener-resource-manager#73, @rfranzke)

[hvpa-controller]

Most notable changes

• [DEVELOPER] Check if OOMKilled pod has latest resource values before overriding stabilisation (gardener/hvpa-controller#78, @ggaurav10)
• [DEVELOPER] Consider hpa scale out limited if hpa is not deployed (gardener/hvpa-controller#77, @ggaurav10)

[logging]

Improvements

• [OPERATOR] Add Timeout, MaxBackoff and MinBackoff wait settings to the fluent-bit-to-loki output plugin. (gardener/logging#60, @vlvasilev)
• [OPERATOR] logs routing depends on the cluster resources in the seed (gardener/logging#59, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.9.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.9.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.9.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.9.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.9.0

[gardener]

Improvements

• [USER] An issue that prevented regular project admins from managing non-human users has been resolved. (a57580ebc5971e0cfdf83763020ee5268dc29894)
• [USER] An issue causing Shoot reconciliation to fail at Maintain shoot annotations with optimistic lock error message is now mitigated. (23fc2d1b225f0826b0799fb6e509fe8270f21d73)
• [OPERATOR] ControllerInstallations are no longer created for ControllerRegistrations that are in deletion. (d20c522d4b1dc52354e1bc4383ab25f1954911cf)
• [OPERATOR] A bug has been fixed that caused the REST Mapper to rediscover the available API resources very often. (0a0b5536e33bc96f7d7508970a8743ee671d7d3c)
• [OPERATOR] The shoot task annotation is now updated as soon as the respective task has completed successfully to prevent recurring executions in case the whole shoot reconciliation flow fails. (69d00f1571e6c2ec2e77bda0cbed44b3d4044835)
• [OPERATOR] The explicit terminationGracePeriodSeconds configuration of the Gardener components has been removed. (c5abfc414349139983d796e88f100f66911f09b7)
• [OPERATOR] gardener-apiserver Deployment does now define a readiness probe. (f3193b7987cbc5e2eb7ee81eb20e07cb96994b02)
• [OPERATOR] Kubernetes dependencies are now updated to v0.17.11. (23cb76ae5aa4d866fa65c040c73027da8e7494a9)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.8.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.8.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.8.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.8.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.8.2

[gardener]

Improvements

• [USER] When the VPA for shoots is disabled then the CustomResourceDefinitions are no longer deleted (they will remain in the system, together with all the VerticalPodAutoscaler objects - if you don't need them anymore you can remove them with kubectl delete crd <crd-names>). (36c59ebc9412cfc92b0cea455c4af68916f3b9ee)
• [USER] A bug showing 403 Forbidden responses when creating new Projects has been fixed. (d95b2a539ada68c3f7d2901cdb1dec77d3a56ce3)
• [OPERATOR] Objects into which controllerinstallations and extensions are retrieved during reconciliation by the ControllerInstallation controller and ShootState Sync controller are now created for each reconciliation, instead of the same object being reused multiple times per resource kind. (#2680, @timuthy)
• [OPERATOR] An issue has been fixed which caused the Gardenlet to exit ungracefully due to the missing shoot cluster identify in the .status. (#2680, @timuthy)
• [OPERATOR] The fluent-bit DaemonSet is now tolerating the taint node-role.kubernetes.io/master with effect NoSchedule. (#2680, @timuthy)
• [OPERATOR] An issues has been fixed which caused the Gardenlet to exit ungracefully during the shoot reconciliation. (9cfbccbef82c0bd69f6fc05da41c235b6e5027fa)
• [OPERATOR] A bug has been fixed that might have caused a 24h absence of the VPA for shooted seeds. (966ba44d2b4d4841cb50d5f9b7fae0f327b989f5)
• [OPERATOR] The error code mapping has been extended to categorize certain common issues upfront and furnish them with error codes. (0af164f5f918a7041a4305f675d1728d6deb53b3)
• [OPERATOR] The gardenlet now spreads Shoot health checks to avoid running into rate limits directly after startup. (8574804d47d68370b82d59a7944d09199f0acc73)
• [OPERATOR] Control plane health checks have been added for VPA components of shoot clusters. These are executed regularly as soon as the shoot uses the Vertical-Pod-Autoscaling feature (https://github.com/gardener/gardener/blob/master/docs/usage/shoot_autoscaling.md#vertical-pod-auto-scaling). (863cefa09d07b6b47f97a928bc98c8a84f609706)
• [OPERATOR] A bug in the OpenAPI specification exposed by the Gardener API server has been fixed. (deade9f4990300226178e5696aa4842c8c426c9f)
• [OPERATOR] A bug has been fixed which could cause a nil pointer exception in case the v1.8 Gardenlet tries to delete a shoot that wasn't reconciled yet. (7f78bbe9684fd5894df3685e9a6dac6b5cee11e1)

[gardener-resource-manager]

Most notable changes

• [DEVELOPER] The new resources.gardener.cloud/keep-object annotation can be used on resources managed by ManagedResource objects in order to keep them in the system in case they get removed from the ManagedResource or the ManagedResource itself is being deleted. (gardener/gardener-resource-manager#73, @rfranzke)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.8.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.8.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.8.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.8.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.8.1

[gardener]

Action Required

• [USER] The Kibana service showing control plane logs per shoot cluster is no longer available. The logs are now available in the Grafana dashboard next to the monitoring. For more details, check this document. (#2515, @vlvasilev)
• [OPERATOR] The shoot reconciliation does now create DNSOwner objects in the seed cluster to manage the ownership of the DNS entries. You should run at least v0.7.16 of the external-dns-management extension. (#2576, @swilen-iwanow)
• [OPERATOR] Gardener API Server has a new mandatory flag for defining desired cluster identity of the Garden cluster that is --cluster-identity. In case gardener is operated by helm - charts for the identity of the clusters are also provided without default values, meaning that they should be filled by the Gardener operators/admin. (#2471, @swilen-iwanow)
• [DEVELOPER] Extension developers that use the webhook framework with the url mode do now have to explicitly specify the port as part of their supplied --webhook-config-url flag (earlier, the webhook server port was implicitly used). (#2665, @rfranzke)
• [DEVELOPER] The local-garden's kube-apiserver cert has been updated to include host.docker.internal as an alternative DNS Name, so clients running in docker containers can successfully validate the TLS cert when talking to the local garden. If you have copied the local-garden kubeconfig to somewhere else, please update your copy with the newly generate one. (#2641, @timebertt)
• [DEVELOPER] The contents of the extensions/hack folder have been merged into the hack folder. (#2623, @rfranzke)

Most notable changes

• [USER] It is now possible to configure the kube-apiserver's --default-watch-cache-size and --watch-cache-sizes flags via the Shoot API (spec.kubernetes.kubeAPIServer.watchCacheSizes). Please see this document for an example and consult the kube-apiserver command-line reference in case you plan on configuring it for your Shoot. (#2668, @timebertt)
• [USER] A new uam role was introduced for Projects (next to owner, admin, and viewer). Members with this role will be bound to the uam custom RBAC verb for the respective Project. Only users bound to this verb are now allowed to add/modify/remove human users or groups from the .spec.members[] list of the Project. Please find more information here. (#2611, @rfranzke)
• [USER] Already deprecated dns.garden.sapcloud.io/provider, dns.garden.sapcloud.io/domain, shoot.garden.sapcloud.io/expirationTimestamp, shoot.garden.sapcloud.io/tasks, garden.sapcloud.io/createdBy, shoot.garden.sapcloud.io/sync-period, shoot.garden.sapcloud.io/ignore annotations and shoot.garden.sapcloud.io/status label are now removed. (#2603, @ialidzhikov)
• [USER] When the NodeLocalDNS feature gate is enabled and a migration from IPTables to IPVS is performed then newer pods will be configured to use the node-local DNS while older pods will still use the non-cached CoreDNS server in the cluster. To enable older pods to use the node-local you have to restart the pods. (#2528, @zanetworker)
• [OPERATOR] The label used to list the internal-domain, default-domain and openvpn Secrets is now gardener.cloud/role. If you manually manage these Secrets (e.g you are not using the controlplane chart), please make sure that they have the required label. (#2603, @ialidzhikov)
• [OPERATOR] The logging stack deployed by Gardener in the seed clusters is now using Loki and Grafana instead of Elasticsearch and Kibana. Once Gardener is updated to this version, the old logging solution will stop working as each shoot will get the new solution with the next reconciliation. Note, old logs will not be preserved. For more details, check this document. (#2515, @vlvasilev)
• [OPERATOR] Clusters in the Gardener topology (Garden, Seed and Shoot), now have ConfigMaps deployed in the kube-system namespace, that hold the cluster identity. Cluster identities for the Shoot and Seed are also visible in their status.clusterIdentity fields. (#2471, @swilen-iwanow)
• [DEVELOPER] A utility function that reconciles the shoot webhook configurations has been added to the generic ControlPlane actuator package. It is recommended for all extensions that have shoot webhooks to call this function before starting the control loops to ensure that the webhook configurations are updated in case the ports change. (#2663, @rfranzke)
• [DEVELOPER] A bug has been fixed causing the BackupEntry deletion to stuck infinitely if the referred secret does not exist. (#2659, @rfranzke)
• [DEVELOPER] You can now use the controller-runtime cache of our ClientSets (kubernetes.Interface.Cache()) to obtain informers for arbitrary API objects (via GetInformer/GetInformerForKind) to construct controllers. (#2581, @timebertt)

Improvements

• [USER] A bug has been fixed which prevented the server certificate of the shoot kube-apiservers that run on a seed with disabled shoot DNS from being generated correctly. (#2643, @rfranzke)
• [USER] A bug has been fixed that could lead to stuck shoot reconciliations in case the .spec.provider.infrastructureConfig was changed multiple times while another shoot reconciliation operation was still in progress. (#2619, @rfranzke)
• [USER] Users can now specify values for systemReserved and kubeReserved in the kubelet configuration as documented here: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ (#2574, @guydaichs)
• [USER] Add grafana dashboard for VPA when the shoot VPA feature is enabled (#2571, @wyb1)
• [OPERATOR] The leader election performed by the gardener-resource-manager deployments in shoot namespaces in the seed is now happening less frequently to prevent overloading the seed's API server unnecessarily. (#2667, @rfranzke)
• [OPERATOR] The default webhook server port for gardener-seed-admission-controller and vpa-admission-controller is now changed to 10250. (#2660, @stoyanr)
• [OPERATOR] gardener-apiserver Deployment does now define a liveness probe. (#2647, @ialidzhikov)
• [OPERATOR] It is now allowed to set an expiration date for the latest machine image version in the CloudProfiles. (#2640, @ialidzhikov)
• [OPERATOR] An issue causing the seed controller to delete VPA RBAC for ShootedSeed is now fixed. (#2639, @ialidzhikov)
• [OPERATOR] The cached controller-runtime clients have been disabled for Shoot clients to decrease the gardenlet's memory footprint in case the CachedRuntimeClients feature gate is enabled. (#2637, @timebertt)
• [OPERATOR] The control plane migration task order is rearranged to ensure the deletion of all objects. (#2636, @kris94)
• [OPERATOR] The Shoot care controller has been optimized to leverage the cached Seed client instead of talking directly to the API server. (#2635, @timebertt)
• [OPERATOR] gardener-apiserver does now support a field selector for ControllerInstallations by spec.registrationRef.name and spec.seedRef.name. (#2634, @ialidzhikov)
• [OPERATOR] Fixes a bug in the maintenance controller that could lead to machine images to not be updated if the Shoot has multiple worker pools. (#2630, @danielfoehrKn)
• [OPERATOR] During deletion of a Shoot, the gardenlet does not redeploy the Worker extension resource anymore, as this sometimes caused leaking resources that blocked the deletion of the Shoot's namespace. (#2626, @timebertt)
• [OPERATOR] added machineclasses CRD for out-of-tree machine controllers (#2625, @MartinWeindel)
• [OPERATOR] A bug has been fixed that could cause the gardener-controller-manager to panic if it tries to maintain a Shoot whose .metadata.annotations=nil. (#2617, @rfranzke)
• [OPERATOR] A bug was fixed that prevented the log collection for control plane components that belong to shoots whose purpose was changed from testing to something else or for those that were woken up. (#2615, @rfranzke)
• [OPERATOR] Fixed a bug where istiod cannot listen on 443 due to insufficient privileges. (#2613, @mvladev)
• [OPERATOR] An issue has been fixed which prevented not required ControllerInstallations from being deleted. As a side effect, it also blocked the deletion of Seed resources. (#2610, @timuthy)
• [OPERATOR] Gardener now validates the extension kinds configured in .spec.resources[].kind in ControllerRegistrations. (#2610, @timuthy)
• [OPERATOR] The gardenlet's /healthz endpoint has been improved to be more stable under certain circumstances like CPU throttling. (#2609, @timebertt)
• [OPERATOR] The VPA for shoots running in the seed is now correctly scaled down when the shoot is being hibernated. (#2606, @rfranzke)
• [OPERATOR] Fix a bug where creation of a shoot with ingress addon might fail because of still not created worker. (#2599, @vpnachev)
• [OPERATOR] A bug has been fixed, which caused the cloud-config secret in the Shoot to not get updated correctly after a change to the worker config in case the CachedRuntimeClients feature gate was activated. (#2591, @timebertt)
• [OPERATOR] A bug has been fixed, which caused Seed clients not to be invalidated properly on Seed deletion. (#2587, @timebertt)
• [OPERATOR] An issue causing fluent-bit Pods to be restarted because of a new ConfigMap checksum when the CachedRuntimeClients feature gate is enabled, is now fixed. (#2583, @vlvasilev)
• [OPERATOR] A bug has been fixed, which caused the discovered Plant region to alternate between region and zone of the Nodes. (#2582, @timebertt)
• [OPERATOR] Wait until MCM deployment is rolled out before proceeding with other reconciliation tasks. (#2579, @prashanth26)
• [OPERATOR] A bug was fixed that caused the machine image version to be overwritten in case a Shoot was updated with a specified image name but with an unspecified image version. (#2570, @timebertt)
• [OPERATOR] A bug has been fixed, that caused that newly created BackupBuckets/BackupEntries where not reconciled by the gardenlet immediately when the CachedRuntimeClients feature gate was enabled. (#2568, @timebertt)
• [OPERATOR] The default QPS setting for Seed Clients in the gardenlet has been increased to adapt to the actual amount concurrent of API calls. (#2567, @timebertt)
• [OPERATOR] Shoot resource now allows configuring following machine-controller parameters: DrainTimeout, HealthTimeout, CreationTimeout, MaxEvictRetries, NodeConditions. (#2563, @hardikdr)
• [OPERATOR] Fixed a bug, that caused the tunnel secrets not to be deleted, in case the used tunnel has changed. (#2562, @timebertt)
• [OPERATOR] Fix secret to backupBucket and backupEntry extension resource mapper. (#2560, @swapnilgm)
• [OPERATOR] Update Istio to 1.6.4. Fix CVE-2020-8663 by setting overload.global_downstream_max_connections on the istio-ingress gateway. (#2556, @mvladev)
• [OPERATOR] Node-local DNS is now supported and can be enabled with the NodeLocalDNS feature gate in the gardenlet's component configuration. More information can be found in the documentation (https://github.com/gardener/gardener/blob/master/docs/usage/node-local-dns.md). (#2528, @zanetworker)
• [OPERATOR] Introduces a RollingUpdate condition in the generic worker actuator (condition.Type RollingUpdate) . Gardener provider extensions write this condition to the Worker CRD. (#2459, @danielfoehrKn)
• [OPERATOR] The generic worker actuator more reliably waits for rolling updates to finish. Waits until all updated machines joined the cluster and until old machines are deleted. Also fixes a stale cache bug that leads to not waiting for the rolling update to complete. (#2459, @danielfoehrKn)
• [OPERATOR] The generic worker actuator detects and restarts 'stuck' machine controller manager pods. (#2459, @danielfoehrKn)
• [OPERATOR] If automatic cross provider scheduling is desired, it is possible know to specify a new seed selector field providers for a cloud profile to enable scheduling on seeds running on different providers. (#2169, @mandelsoft)
  • It is observed if the scheduling method Selector has been chosen, which is the new default now.
• [DEVELOPER] Extension developers who want to use different ports for their provider extension webhook server and the corresponding server port can now specify the service port with the new --webhook-config-service-port command line flag. If it's not present the service port is defaulted to the webhook server port (i.e., old behaviour is preserved). (#2665, @rfranzke)
• [DEVELOPER] The version information in docker images has been updated to correctly display version.major without the v-Prefix and version.gitTreeState as clean. (#2608, @timebertt)
• [DEVELOPER] It is possible now to add a values-test.yaml file to helm charts to specify default values for chart checks, which will be merged into the default values.yaml when running hack/check-charts.sh. This is useful for the case, that charts have a {{ required ... }} statement, but don't specify default values in values.yaml. (#2584, @timebertt)
• [DEVELOPER] The Nodeless Development Environment also works on windows, using WSL2 and docker for windows (#2578, @guydaichs)
• [DEVELOPER] Adds Migrator/Restorer interfaces to the botanist shoot components (#2511, @plkokanov)
• [DEVELOPER] Added GEP-12 for dynamic OIDC webhook authenticator. (#2481, @mvladev)

[autoscaler]

Improvements

• [OPERATOR] Add topology.kubernetes.io labels to be ignored when comparing similar node groups. (gardener/autoscaler#50, @hardikdr)
• [OPERATOR] Prepended v in the version. (gardener/autoscaler#42, @hardikdr)

[gardener-resource-manager]

Most notable changes

• [OPERATOR] It is now possible to specify the leader election settings via the following command line parameters: --leader-election-lease-duration (default: 15s), --leader-election-renew-deadline (default: 10s), --leader-election-retry-period (default: 2s). (gardener/gardener-resource-manager#72, @rfranzke)
• [DEVELOPER] Resources annotated with resources.gardener.cloud/delete-on-invalid-update=true will now be deleted in case the Gardener-Resource-Manager fails to update them and receives an 422 Unprocessable Entity error. This error is usually sent by the Kubernetes API server in case its static validation fails. (gardener/gardener-resource-manager#69, @rfranzke)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.8.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.8.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.8.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.8.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.8.0

[gardener]

Improvements

• [USER] A bug has been fixed which prevented the server certificate of the shoot kube-apiservers that run on a seed with disabled shoot DNS from being generated correctly. (514b3bfdf302dca8f955ff8079479390dff5850f)
• [OPERATOR] An issue preventing gardenlet to wait for network.extensions.gardener.cloud deletion is now fixed. (#2656, @ialidzhikov)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.7.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.7.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.7.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.7.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.7.4

[gardener]

Improvements

• [OPERATOR] Fixed a bug where istiod cannot listen on 443 due to insufficient privileges. (#2616, @mvladev)
• [OPERATOR] An issue causing the seed controller to delete VPA RBAC for ShootedSeed is now fixed. (f5ff01432b412f413fcd3ba25d4e98aa35918ff7)
• [OPERATOR] Fixes a bug in the maintenance controller that could lead to machine images to not be updated if the Shoot has multiple worker pools. (715e5daed388aab8bc2876507d0a8122b694e80c)
• [OPERATOR] During deletion of a Shoot, the gardenlet does not redeploy the Worker extension resource anymore, as this sometimes caused leaking resources that blocked the deletion of the Shoot's namespace. (9d111b2e0aa1117aebdb44777bd11ad94c697ea0)
• [OPERATOR] A bug has been fixed that could cause the gardener-controller-manager to panic if it tries to maintain a Shoot whose .metadata.annotations=nil. (6dd88034db8605ffa1548ccd1f12b3b1ddee7652)
• [OPERATOR] The gardenlet's /healthz endpoint has been improved to be more stable under certain circumstances like CPU throttling. (5e2718a824dfbd4115265658b89c19ad473fc0d6)
• [OPERATOR] The VPA for shoots running in the seed is now correctly scaled down when the shoot is being hibernated. (09e83067ae4476f4e79f49c2c5e2043498a36eb6)
• [OPERATOR] Fix a bug where creation of a shoot with ingress addon might fail because of still not created worker. (672ebba95b9e699f2fb1fd697010df00e9d99c0b)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.7.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.7.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.7.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.7.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.7.3

[gardener]

Improvements

• [OPERATOR] A bug has been fixed, which caused the discovered Plant region to alternate between region and zone of the Nodes. (449d1213094cb4c3ce5856ac94f578e1d207354a)
• [OPERATOR] An issue causing fluent-bit Pods to be restarted because of a new ConfigMap checksum when the CachedRuntimeClients feature gate is enabled, is now fixed. (8da5b60f988d4e9edb8e92c407765987ac2721a3)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.7.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.7.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.7.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.7.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.7.2

[gardener]

Improvements

• [OPERATOR] A bug was fixed that caused the machine image version to be overwritten in case a Shoot was updated with a specified image name but with an unspecified image version. (e8d683da195e177518c05bfe54d71b554172eb7e)
• [OPERATOR] A bug has been fixed, that caused that newly created BackupBuckets/BackupEntries where not reconciled by the gardenlet immediately when the CachedRuntimeClients feature gate was enabled. (202b83aedac0481473d40efe7d11a334c4e2e349)
• [OPERATOR] Update Istio to 1.6.4. Fix CVE-2020-8663 by setting overload.global_downstream_max_connections on the istio-ingress gateway. (9838a4ce4b849d2352d3e95ff92bd102fd56d04c)
• [OPERATOR] The default QPS setting for Seed Clients in the gardenlet has been increased to adapt to the actual amount concurrent of API calls. (3504b00e62e2eedc83155c2718f1bcb6a89b0d59)
• [OPERATOR] Fix secret to backupBucket and backupEntry extension resource mapper. (cb7627fe4c2c1ddb9e2b8d1e9d9ae3352e0a9cc0)
• [OPERATOR] Fixed a bug, that caused the tunnel secrets not to be deleted, in case the used tunnel has changed. (f5cce4593d0a65e5ac572e95ed88f428c261d2b9)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.7.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.7.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.7.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.7.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.7.1