[gardener]

🐛 Bug Fixes

• [USER] metrics-server's version is updated from v0.4.1 to v0.4.2 to adopt upstream fix that was causing metrics-server to be unavailable for a while after rolling update of Nodes. (#3516, @ialidzhikov)
• [OPERATOR] The affinity section is removed from the Loki StatefulSet for the integration tests (#3526, @vlvasilev)

🏃 Others

• [OPERATOR] An issue causing Shoots to be marked as Failed (and no longer retried) on transient not found error is now fixed. (#3508, @ialidzhikov)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.16.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.16.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.16.2
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.16.2
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.16.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.16.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue causing the generic Worker actuator to not wait until the finalizer of the out-of-tree machine controller provider is removed from the credentials secret is now fixed. (#3498, @ialidzhikov)

🏃 Others

• [OPERATOR] Keep VPA namespace env variable in sync in all cases. If they are out of sync between the admission controller and the updater, the updater might become inactive and stop actively scaling targets that have update mode Auto or Recreate. (#3481, @amshuman-kr)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.16.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.16.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.16.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.16.1
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.16.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.16.1

[gardener]

⚠️ Breaking Changes

• [USER] The already deprecated annotation keys confirmation.garden.sapcloud.io/deletion and shoot.garden.sapcloud.io/operation are no longer respected by Gardener components. If you are still using the deprecated annotation keys, please switch the the equivalents from the new API group - respectively confirmation.gardener.cloud/deletion and gardener.cloud/operation. (#3375, @ialidzhikov)
• [OPERATOR] The check for stale extension health checks is now switched from controllers.shootCare.staleExtensionHealthCheckThreshold to controllers.shootCare.staleExtensionHealthChecks.{enabled,threshold} in the GardenletConfiguration. It is now configurable and enabled by default. (#3390, @ialidzhikov)
• [OPERATOR] Deprecated ingress hostnames i.e., AlertManager - au.<shoot-name>.<project-name>.<seed-ingress-domain>, Grafana - gu.<shoot-name>.<project-name>.<seed-ingress-domain>, go.<shoot-name>.<project-name>.<seed-ingress-domain>, Prometheus - p.<shoot-name>.<project-name>.<seed-ingress-domain> were removed and will not be reachable anymore. Please use the hostnames introduced with Gardener v0.34.0 instead. (#3380, @timuthy)
• [OPERATOR] gardenlet's shoot-care-control is now using the gardener.cloud/role label key (until now it was garden.sapcloud.io/role) to perform health checks on controlplane components. Make sure you have first upgraded to at least Gardener v1.14 before you upgrade to this version of Gardener. (#3350, @ialidzhikov)
• [OPERATOR] With the activated cluster-autoscaler during roll-outs, following are the minimum versions required for different provider-extensions: gardener-extension-provider-aws v1.16.0, gardener-extension-provider-openstack v1.12.0, gardener-extension-provider-azure v1.14.0, gardener-extension-provider-gcp v1.12.0, gardener-extension-provider-alicloud v1.18.0, gardener-extension-provider-vsphere v0.1.0. (#3332, @hardikdr)
• [DEVELOPER] A new mutating webhook for the cloudprovider secret has been added in the extensions library. With this change, the EnsurerContext of the genericmutator package has been moved to a separate context package. Please adapt your usage of genericmutator accordingly. (#3348, @kon-angelo)
• [DEPENDENCY] ⚠️ Go dependencies to kubernetes/* and kubernetes-sigs/controller-runtime were updated to v0.19.6 and v0.7.0 respectively. This imposes a lot of consequent breaking changes to go projects vendoring gardener/gardener. If your project/extension vendors gardener/gardener, please read the dedicated section in this issue carefully when upgrading your dependencies. (#3393, @timebertt)
• [DEPENDENCY] pkg/utils/secrets.BasicAuthSecretConfig does no longer allow generating bcrypt password hash. The corresponding functionality is now removed. (#3365, @ialidzhikov)

✨ New Features

• [USER] Logs from VerticalPodAutoscaler are accessible via the Vertical Pod Autoscaler dashboard in Grafana. (#3456, @Kristian-ZH)
• [USER] The shoot reconciliation flow is now waiting until all worker nodes have executed the most recent cloud-config user data. Similarly, the shoot care controller checks if the last successfully applied cloud-config user data on all nodes is outdated or not (and reports this in the EveryNodeReady condition). Please note that both features are only available for new nodes. (#3396, @rfranzke)
• [USER] The cluster-autoscaler is now activated even during rolling-update of the shoot clusters. The change in machine-controller-manager of adding the cluster-autoscaler.kubernetes.io/scale-down-disabled annotation during rolling-update is required, in order for autoscaler to not scale-down worker-pools (coming with machine-controller-manager 0.34.0). (#3332, @hardikdr)
• [OPERATOR] The specification and the configuration files of the cloud-config-downloader.service systemd service are now updated regularly with the original OperatingSystemConfig. (#3449, @vpnachev)
• [OPERATOR] It is now possible to trigger a restart of systemd services on particular shoot worker nodes by annotating the corresponding Node object with worker.gardener.cloud/restart-systemd-services=kubelet, for example. (#3396, @rfranzke)
• [OPERATOR] The .spec.revisionHistoryLimit is now set to 1 for Deployments. (#3374, @rfranzke)
• [OPERATOR] New SeedKubeScheduler feature gate is added to gardenlet. When enabled, it deploys a custom kube-scheduler in gardener-kube-scheduler namespace of Seed clusters with Kubernetes version 1.17 or greater. The scheduler assigns Shoot control plane Pods to Nodes with higher resource utilization, resulting in better bin-packing of control planes. (#3243, @mvladev)
• [DEVELOPER] Nodeless local dev setups now run with etcd v3.4.14 and K8s v1.20.2. (#3426, @timuthy)
• [DEPENDENCY] The ConfigMaps and Secrets used to store the config and state of terraform now have owner reference to the Infrastructure resource. (#3275, @vpnachev)

🐛 Bug Fixes

• [USER] Fixed a bug where service, pod or node CIDRs that are private network (RFC1918) or carrier-grade NAT (RFC6598) IPv4 blocks would produce an invalid allow-to-private-networks networkpolicy. (#3462, @mvladev)
• [USER] A bug has been fixed that prevented shoot clusters from coming up in case .spec.kubernetes.allowPrivilegedContainers=false. (#3409, @rfranzke)
• [OPERATOR] An issue in the API validation has been fixed which prevented the managed ingress feature for seeds being enabled. (#3448, @BeckerMax)
• [OPERATOR] A bug has been fixed which prevented proper auto-scaling of components under control of HVPA. (#3446, @rfranzke)
• [OPERATOR] Fix gardener-seed-admission controller, etcd backup-restore and extension parsers time format (#3429, @vlvasilev)
• [OPERATOR] The generic Worker actuator does now wait until the machine-controller-manager finalizer is removed from the credentials secret that is referenced from the machine classes. (#3425, @ialidzhikov)
• [OPERATOR] Node Problem Detector is now matched by gardener.cloud--allow-to-dns and gardener.cloud--allow-to-apiserver networkpolicies can run with deny-all networkpolicy in kube-system namespace. (#3424, @mvladev)
• [OPERATOR] fix CRD for extension types to allow storing anything in status.state. (#3422, @MartinWeindel)
• [OPERATOR] Fixes a bug causing newly created Seeds to fail during bootstrap (#3400, @BeckerMax)
• [OPERATOR] A side-car container is added to kube-proxy that deletes the incorrect conntrack table entries which sometime occur after restart of kube-proxy and prevent the establishment of a tcp connection to the api-server. (#3395, @DockToFuture)
• [OPERATOR] Fixed a bug of the managed istio feature flag where the istio rolebinding was created in the wrong namespace. (#3382, @danielfoehrKn)
• [OPERATOR] An issue has been fixed which caused unwanted restarts for Grafana instances. (#3379, @timuthy)
• [OPERATOR] A bug has been fixed in gardener-controller-manager's Project controller that can lead to a continuous reconciliation of Project resources if they are stuck in Terminating state. (#3371, @rfranzke)
• [OPERATOR] An issue causing a NetworkPolicy to do not allow egress from prometheus Pod to alertmanager and vpa-exporter Pods is now fixed. (#3370, @ialidzhikov)
• [OPERATOR] An issue causing gardenlet to do not properly compute the .status.clusterIdentity field is now fixed. (#3366, @ialidzhikov)
• [OPERATOR] KonnectivityTunnel's stability is improved and now handles kube-apiserver autoscaling. It properly sets --server-count of konnectivity-server on such event. (#3267, @mvladev)
• [DEVELOPER] The Seed and Shoot logging stack deletion is separated in two functions to avoid accidental deletion of cluster scoped resources. (#3436, @vlvasilev)
• [DEPENDENCY] A bug in the extension library that was preventing the deletion of TF secret and configmaps with empty state is now fixed. (#3423, @vpnachev)
• [DEPENDENCY] Go dependency kubernetes-sigs/controller-runtime was updated to v0.7.1. (#3408, @timuthy)

📖 Documentation

• [OPERATOR] Enhance documentation for Gardenlet's /healthz endpoint. (#3359, @danielfoehrKn)
• [OPERATOR] AWS-specific annotations set on the istio-ingressgateway Service are now deprecated and are going to be removed in the next release. Please use the Seed's spec.settings.loadBalancerServices.annotations field to set or overwrite those annotations. For shoot.gardener.cloud/use-as-seed annotated Shoot clusters, see this PR. (#3185, @mvladev)

🏃 Others

• [USER] Component and Container fields are added in the logging dashboards for more flexible log queries. (#3456, @Kristian-ZH)
• [USER] Webhooks acting on configmaps in the kube-system namespace and webhooks with a TimeoutSeconds>15 for problematic resources are now also blocking Maintenance and Hibernation operations. Please consult this doc for more details. (#3413, @timebertt)
• [OPERATOR] Gardener now considers the seed.spec.ingress.domain field when passing the value via gradener.seed.ingressDomain to ControllerRegistration charts. (#3441, @timuthy)
• [OPERATOR] Add CPU throttling to the "Kubernetes Pods" Grafana dashboard (#3432, @istvanballok)
• [OPERATOR] TestDefinitions have been added that contains disruptive tests (#3411, @schrodit)
• [OPERATOR] NumberOfBatchIDs for the fluent-bit-to-loki plugin is set to 5 numbers. (#3402, @vlvasilev)
• [OPERATOR] Enable fluent-bit privileged escalation for the integration test via "gardener.privileged" PodSecurityPolicy (#3357, @vlvasilev)
• [DEVELOPER] The golang version is updated to 1.15.7. (#3442, @ialidzhikov)
• [DEVELOPER] Certain insensitive terms were removed from the source code and inline documentation to follow inclusive language best practices. (#3368, @timuthy)
• [DEVELOPER] Unit tests are now limited to a timeout of 2 minutes per test suite. (#3363, @timebertt)
• [DEPENDENCY] The extensions library is now registering webhooks for both seeds and shoots with a 10s timeout. (#3440, @rfranzke)

📰 Noteworthy

• [USER] The kubectl get shoot table view was adapted and does no longer show the DOMAIN column. Instead, it shows the provider type as well as the last operation type. (#3460, @rfranzke)
• [USER] If the kube-apiserver deployment of a shoot cluster does not become ready then Gardener does now evaluate and return the most recent complete logs of the newest pod. It will be part of the .status.last{Operation,Errors}. As a consequence, start-up failures due to invalid feature gate or runtime config settings is being treated as configuration problem error. (#3353, @rfranzke)
• [OPERATOR] The ingress domain configuration for Seeds is now immutable. (#3394, @BeckerMax)
• [OPERATOR] The gardenlet does not fail the liveness probe when the seed resource configured in the Gardenlet's config has been deleted. (#2925, @danielfoehrKn)
• [DEPENDENCY] The terraformer library in extensions/pkg/terraformer does now adopt still running pods instead of deleting them and waiting for their deletion. This enables a faster and more reliable status/result propagation and prevents unhelpful error messages. Infrastructure extension developers do not need to wait for a clean Terraform environment in their Delete() function anymore as the library is now handling this case out-of-the-box. (#3349, @rfranzke)
• [DEPENDENCY] During the extension webhook registration, when a namespace is provided via the --webhook-config-namespace flag, the webhook config is enhanced with an owner reference pointing to the provided namespace. This will lead to auto-cleanup of the webhook config when the extension is uninstalled from a seed (earlier, the webhook config was orphaned even after uninstallation from a seed). (#3341, @rfranzke)

[ingress-default-backend]

🏃 Others

• [OPERATOR] The health endpoint is now configurable by setting the env var HEALTH_ENDPOINT. The default (/healthy) has not been changed. (gardener/ingress-default-backend#13, @Diaphteiros)

[logging]

🏃 Others

• [OPERATOR] Batch IDs are configurable via NumberOfBatchIDs. (gardener/logging#83, @vlvasilev)
• [OPERATOR] Add ControllerSyncTimeout to control the informer sync period. Prior it was infinity time. (gardener/logging#83, @vlvasilev)
• [OPERATOR] ReplaceOutOfOrderTS is replaces by SortByTimestamp. The timestamp is no longer replaced. Instead the logs are sorted by their timestamp. (gardener/logging#83, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.16.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.16.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.16.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.16.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.16.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.16.0

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue in the API validation has been fixed which prevented the managed ingress feature for seeds being enabled. (4bfccaef216043a0324a674aaef0efd02dc4b0e7)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.5
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.5
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.5
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.5
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.5

[gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which prevented proper auto-scaling of components under control of HVPA. (3d0859f121100a8bcca8fe533d8b1fefa8b4a8b7)
• [OPERATOR] fix CRD for extension types to allow storing anything in status.state. (f29a08a04a6a5fe0aab55abda33d66de73ee4905)
• [OPERATOR] The generic Worker actuator does now wait until the machine-controller-manager finalizer is removed from the credentials secret that is referenced from the machine classes. (b8cbfee96a2ff7fe816c7ee8ab351682cae8f343)
• [DEVELOPER] The Seed and Shoot logging stack deletion is separated in two functions to avoid accidental deletion of cluster scoped resources. (#3437, @vlvasilev)

🏃 Others

• [OPERATOR] Gardener now considers the seed.spec.ingress.domain field when passing the value via gradener.seed.ingressDomain to ControllerRegistration charts. (#3443, @timuthy)
• [OPERATOR] Fix gardener-seed-admission controller, etcd backup-restore and extension parsers time format. (283ee10c7eab82248f196f0402327d54e7b730d9)
• [DEVELOPER] The golang version is updated to 1.15.7. (6dab5ea88fea476f2c3d824fe8ac25238661c69a)

📰 Noteworthy

• [OPERATOR] The ingress domain configuration for Seeds is now immutable. (ba65cf6bb6184491f828a2b22be9fa96ef4c809a)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.4
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.4
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.4

[gardener]

🐛 Bug Fixes

• [USER] A bug has been fixed that prevented shoot clusters from coming up in case .spec.kubernetes.allowPrivilegedContainers=false. (#3410, @rfranzke)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.3
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.3
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.3

[gardener]

🐛 Bug Fixes

• [OPERATOR] Fixes a bug causing newly created Seeds to fail during bootstrap (#3401, @BeckerMax)
• [OPERATOR] A side-car container is added to kube-proxy that deletes the incorrect conntrack table entries which sometime occur after restart of kube-proxy and prevent the establishment of a tcp connection to the api-server. (243cfebaa3a1d9edbb4e46349639dfdfb776f15f)

🏃 Others

• [OPERATOR] An issue has been fixed which caused unwanted restarts for Grafana instances. (#3404, @ialidzhikov)
• [OPERATOR] NumberOfBatchIDs for the fluent-bit-to-loki plugin is set to 5 numbers. (#3403, @vlvasilev)
• [OPERATOR] Fixed a bug of the managed istio feature flag where the istio rolebinding was created in the wrong namespace. (848a8b9d011daf257b618af1c9fe0b3044ce9f3b)

[logging]

🏃 Others

• [OPERATOR] Batch IDs are configurable via NumberOfBatchIDs. (gardener/logging#83, @vlvasilev)
• [OPERATOR] Add ControllerSyncTimeout to control the informer sync period. Prior it was infinity time. (gardener/logging#83, @vlvasilev)
• [OPERATOR] ReplaceOutOfOrderTS is replaces by SortByTimestamp. The timestamp is no longer replaced. Instead the logs are sorted by their timestamp. (gardener/logging#83, @vlvasilev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.2
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.2
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.2

[gardener]

🐛 Bug Fixes

• [OPERATOR] An issue causing a NetworkPolicy to do not allow egress from prometheus Pod to alertmanager and vpa-exporter Pods is now fixed. (3d27d2e8e836e0517311461aab7320d142bc8338)
• [OPERATOR] An issue causing gardenlet to do not properly compute the .status.clusterIdentity field is now fixed. (b9a42571abd8abfe74c734662fe2fc51177ee320)

🏃 Others

• [OPERATOR] A bug has been fixed in gardener-controller-manager's Project controller that can lead to a continuous reconciliation of Project resources if they are stuck in Terminating state. (56b5c5aefa90e14d1e5044fb5cb166cd15deac10)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.1
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.1

[gardener]

⚠️ Breaking Changes

• [OPERATOR] If the nginx-ingress addon for a shoot used as seed is disabled then you can no longer enable it anymore. Instead, use the new managed ingress controller feature. You can find more information about it here. Existing shoots used as seeds with .spec.addons.nginxIngress.enabled=true will continue to work. (#3131, @BeckerMax)

✨ New Features

• [OPERATOR] It is now possible to specify the spec.settings.loadBalancerServices.annotations field for shooted seeds via the "shoot.gardener.cloud/use-as-seed" annotation. You can do this by specifying the loadBalancerServices.annotations.* option - for example loadBalancerServices.annotations.service.beta.kubernetes.io/aws-load-balancer-type=nlb. (#3344, @ialidzhikov)
• [OPERATOR] The gardener admission controller now exposes metrics (#3293, @wyb1)
• [OPERATOR] Gardener now offers to manage a dedicated ingress controller for seed clusters (earlier, this was a manual operator task when registering seeds). You can find more information about it here. (#3131, @BeckerMax)
• [DEVELOPER] Gardener can now support shoot clusters with Kubernetes version 1.20. In order to allow creation/update of 1.20 clusters you will have to update the version of your provider extension(s) to a version that supports 1.20 as well. Please consult the respective releases and notes in the provider extension's repository. (#3296, @rfranzke)

🐛 Bug Fixes

• [USER] An race issue causing immediate wake up after hibernation to fail is now fixed. The hibernation is now waiting until the kube-apiserver Service is cleaned up. (#3289, @ialidzhikov)
• [OPERATOR] A bug that was renewing the bootstrap token secret on each reconciliation has been fixed. (#3323, @vpnachev)
• [OPERATOR] An issue has been fixed which did not enable VPA for the aggregate Prometheus Pod in new seed clusters. (#3312, @timuthy)
• [OPERATOR] By default, gardener-apiserver now invokes in-tree admission plugins before invoking the webhook plugins. (#3298, @timebertt)
• [OPERATOR] An issue has been fixed that prevented the execution of the Kube-API-Server's configured preStop hooks for >=1.19.x clusters. (#3295, @timuthy)
• [OPERATOR] Gardener health checks now take the effective Shoot specification into consideration if .spec.maintenance.confineSpecRollout is used. Earlier, EveryNodeReady or ControlPlaneHealthy conditions reported an invalid state if the specification was changed but not yet effective due to a rollout during shoot maintenance (confineSpecRollout: true). (#3286, @timuthy)
• [DEPENDENCY] Ensure a stable order of self-registered webhooks in extensions to avoid unnecessary rollouts of control plane components. (#3320, @timebertt)

📖 Documentation

• [USER] API reference documentation for kubernetes types now points to version v1.19. (#3303, @mvladev)
• [OPERATOR] Gardener's scheduler documentation has been enhanced. It concisely explains the algorithm used to determine seed candidates. (#3316, @timuthy)

🏃 Others

• [OPERATOR] The Loki initialDelaySeconds for the readinessProbe is reduces to 80 seconds. (#3333, @vlvasilev)
• [OPERATOR] The vpa-admission-controller and vpa-updater pods are now ensured with some minimal CPU and memory resources. (#3330, @vpnachev)
• [OPERATOR] Gardener will now check seed clusters for VPA functionality as a prerequisite. (#3312, @timuthy)
• [OPERATOR] Upgrade Prometheus to v2.23.0 (#3297, @wyb1)
• [OPERATOR] Change pod anti-affinity to preferredDuringSchedulingIgnoredDuringExecution for gardener-seed-admission-controller deployment in the garden namespaces of seed clusters. (#3294, @hardikdr)
• [OPERATOR] The pre-delivered cluster role gardener.cloud:admin now contains full access permissions for Events and ResourceQuotas. (#3291, @timuthy)
• [OPERATOR] Add panels to the Kubernetes API Server Details Dashboard for dropped requests. (#3284, @wyb1)
• [OPERATOR] Alerts are added for the custom metrics for fluent-bit GardenerLoki plugin (#3283, @Kristian-ZH)
• [OPERATOR] Required connections from Gardenlet to the Garden cluster has been reduced which will have positive effects on scalability and costs. (#3277, @timuthy)
• [DEPENDENCY] Guestbook integration test dependencies are now fetched from bitnami repo instead of deprecated/shutdown helm repo. (#3314, @dguendisch)
• [DEPENDENCY] Making the implementation of the function GetMachineControllerManagerCloudCredentials in the WorkerDelegate optional. Alternatively, extensions can now use the field in the machine class spec.credentialsSecretRef so that all machine classes refer to the same secret from the Worker field spec.secretRef. See here for more details. (#3308, @danielfoehrKn)

📰 Noteworthy

• [USER] The version for the nginx-ingress addon for shoots has been updated to v0.41.2 ONLY for Kubernetes 1.20 shoot clusters. All shoot clusters with Kubernetes < 1.20 will remain with the current v0.22.0 version. Please be reminded that the nginx-ingress addon is not recommended for production scenarios and that you should deploy (+ customize) your own ingress controller instead. Please use it only for development/evaluation purposes. (#3315, @rfranzke)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.15.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.15.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.15.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.15.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.15.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.15.0

[gardener]

🐛 Bug Fixes

• [USER] An race issue causing immediate wake up after hibernation to fail is now fixed. The hibernation is now waiting until the kube-apiserver Service is cleaned up. (3f896cd7a8cd26e34aa8789f786d5c89be08b668)
• [OPERATOR] An issue has been fixed that prevented the execution of the Kube-API-Server's configured preStop hooks for >=1.19.x clusters. (3103cd59df2849506ed256aa05187fd7e0b0006d)
• [OPERATOR] A bug that was renewing the bootstrap token secret on each reconciliation has been fixed. (21fa2785d45e735f0922eabc330e08fc1b754767)
• [DEPENDENCY] Ensure a stable order of self-registered webhooks in extensions to avoid unnecessary rollouts of control plane components. (65bfb70f0705591263bd99c54b3869e1080281ff)

🏃 Others

• [OPERATOR] By default, gardener-apiserver now invokes in-tree admission plugins before invoking the webhook plugins. (#3306, @jia-jerry)
• [OPERATOR] The pre-delivered cluster role gardener.cloud:admin now contains full access permissions for Events and ResourceQuotas. (9367a6968ed9c81c7d3b8d1b94f6a98971b66ee1)
• [DEPENDENCY] Guestbook integration test dependencies are now fetched from bitnami repo instead of deprecated/shutdown helm repo. (92bc456a72028c7f42fe19586e5f40471e1b3658)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.14.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.14.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.14.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.14.1
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.14.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.14.1

[gardener]

⚠️ Breaking Changes

• [DEPENDENCY] The WorkerDelegate must implement method GetMachineControllerManagerCloudCredentials returning map with cloud credential keys and values just like they are used by the machine-controller-manager. (#3224, @vpnachev)
• [DEPENDENCY] The deprecated functions in the terraformer library (SetVariablesEnvironment and GenerateVariablesEnvironment) have been removed. (#3223, @timebertt)
• [DEPENDENCY] The Terraformer functions have been changed to allow passing proper contexts. Please adapt your usage accordingly. (#3223, @timebertt)
• [DEPENDENCY] The terraformer library was switched to logr instead of logrus in order to have more consistent and readable logging in the infrastructure controllers of provider extensions. Please adapt your usage accordingly. (#3223, @timebertt)

✨ New Features

• [USER] The shoot reconciler sets the conditions to Progressing after it finished a successful reconciliation, and the care controller starts to re-evaluate the health status after this happened. This helps end-users to better understand whether their cluster is indeed healthy after a reconciliation. Earlier, it could take up to 30s / 1m (based on the configured care controller sync period) until the actual status is reflected. (#3251, @rfranzke)
• [OPERATOR] The shoot controller inside the gardenlet has been adapted to cater with large Gardener landscapes: (#3242, @rfranzke)
  • 1. When the gardenlet has already reconciled a shoot cluster during its maintenance time window then it doesn't reconcile it again. Instead, it computes a random duration for the next time window and requeues the shoot. Already reconciled shoots are those whose last reconciliation was less then 24h ago.
  • 1. When the gardenlet is (re)started then it does no longer reconcile all shoots immediately whose maintenance time windows are met. Instead, it computes a random time for the current time window and requeues the shoot ("jittering", i.e., spreading the load). This will have the effect that not all shoots are getting reconciled at the same time right after startup.

🐛 Bug Fixes

• [USER] The KUBERNETES_SERVICE_HOST environment variable injected when APIServerSNI is enabled no longer includes a trailing dot (being a Fully Qualified Domain Name) due to several homebrew kubernetes clients not properly handling it and sending wrong server name when initiating a TLS conneciton. (#3235, @mvladev)
• [OPERATOR] apieserver-proxy now uses system-node-critical priority class. Memory limit is also increased to avoid OOM killer. (#3282, @mvladev)
• [OPERATOR] A bug has been fixed that caused the vpa-admission-controller to not being able to update its status (inside Lease object) when its enabled for shoot clusters. (#3265, @rfranzke)
• [OPERATOR] Fix an error during bootstrapping of fresh Seeds (#3262, @BeckerMax)
• [OPERATOR] A worker controller is now ensuring that all machine class secrets have up-to-date cloud credentials. (#3224, @vpnachev)
• [OPERATOR] A bug has been fixed which can lead to Seeds not getting ready when an image vector overwrite for the etcd-druid is configured. (#3212, @rfranzke)

🏃 Others

• [USER] Support scale to/from zero for MCM OOT providers - AWS, Azure. (#3276, @prashanth26)
• [USER] The severity of the user exposed logs is unified and recognizable by the Grafana. (#3270, @vlvasilev)
• [OPERATOR] The target cache of gardener-resource-manager instances running in the Shoot control plane is disabled now. (#3268, @timebertt)
• [OPERATOR] Gardener has improved infrastructure processing procedures in oder to avoid unnecessary reconciliation cycles. (#3255, @timuthy)
• [OPERATOR] Add Loki multitenancy integration test. (#3253, @vlvasilev)
• [OPERATOR] Istio is updated to 1.18.0. (#3250, @mvladev)
• [OPERATOR] Parse the time zone of a log when reading it from the node /var/log/containers directory. (#3219, @vlvasilev)
• [OPERATOR] When fluent-bit containers runs the tail plugin starts to read a file from the head(like it was prior fluent-bit 1.6). (#3219, @vlvasilev)
• [OPERATOR] Make the readiness and liveness probe fail after 30 seconds and the liveness probes starts after 90 seconds. (#3219, @vlvasilev)
• [OPERATOR] get, list and watch for Pods are removed from the fluent-bit RBAC as no longer needed. (#3219, @vlvasilev)
• [OPERATOR] Upgrade Prometheus to v2.22.2. Sometimes Prometheus would have the error mmap: invalid argument. Prometheus v2.22.1+ provides a fix for this issue. (#3213, @wyb1)
• [OPERATOR] metrics-server, node-problem-detector and vpn-shoot now have dnsPolicy: Default set to them to remove dependency to coredns. (#3211, @mvladev)
• [OPERATOR] Sort logs to fix out of order issue (#3188, @Kristian-ZH)
• [OPERATOR] The output plugin exposes custom metrics (#3188, @Kristian-ZH)
• [OPERATOR] Modified fluent-bit dashboard to include the new metrics (#3188, @Kristian-ZH)
• [OPERATOR] Fluent-bit tail plugin DB synchronization is set to FULL to avoid log duplication when fluent-bit pod is restarted. (#3091, @vlvasilev)
• [OPERATOR] Loki chunk_target_size option is set to 1536000 bytes as recommended by Grafana (#3091, @vlvasilev)
• [DEVELOPER] Integration test for the logging is added simulating seed with 100 shoots (#2996, @vlvasilev)

📰 Noteworthy

• [USER] The Shoot garbage collector now also deletes failed Pods with the reason OutOf* in the Seed namespace and the kube-system namespace of the Shoot. (#3248, @timebertt)
• [USER] The system components that were previous specifying label garden.sapcloud.io/role: (optional-addon|monitoring|system-component) are now adapted to specify gardener.cloud/role: (optional-addon|monitoring|system-component). (#3220, @ialidzhikov)
• [OPERATOR] Forbid control plane migration between Seeds with different cloud providers. (#3254, @plkokanov)
• [OPERATOR] The gardenlet enqueues shooted seeds immediately (without configured jitter) when the shooted seed's spec was changed or when the config in the use-as-seed annotation was changed. This enabled a faster rollout of the gardenlet. (#3249, @rfranzke)
• [OPERATOR] gardenlet is now restarted if APIServerSNI is enabled on the Seed cluster. (#3226, @mvladev)
• [OPERATOR] The Shoot namespace in the Seed no longer specifies label garden.sapcloud.io/role: shoot. (#3220, @ialidzhikov)
• [OPERATOR] Upgraded etcd version from v3.3.17 to v3.4.13 and moved from quay.io/coreos/etcd to Gardener-specific custom etcd image eu.gcr.io/gardener-project/gardener/etcd. ⚠️ This will cause an etcd restart. (#3205, @gardener-robot-ci-2)

[autoscaler]

🏃 Others

• [USER] Support scale to/from zero for MCM OOT providers - AWS, Azure. (gardener/autoscaler#65, @prashanth26)
• [OPERATOR] Ignore gardener-specific and csi-specific labels while comparing nodegroups (gardener/autoscaler#62, @hardikdr)

[etcd-backup-restore]

🏃 Others

• [OPERATOR] Validator now double checks latest revision by starting an embedded etcd if DB-based revision check fails. This can potentially avoid unnecessary data restoration when etcd terminates abnormally. (gardener/etcd-backup-restore#275, @ishan16696)
• [OPERATOR] Fix missing alternate full snapshots for some unhibernating shoots. (gardener/etcd-backup-restore#272, @shreyas-s-rao)
• [OPERATOR] Added support for OpenShift Container Storage (OCS) S3 storage type. (gardener/etcd-backup-restore#261, @stoyanr)
• [OPERATOR] Fixed the issue with consecutive restoration if backup-restore sidecar doesn't restart in between. (gardener/etcd-backup-restore#259, @amshuman-kr)
• [OPERATOR] Clarify manual backup restore process (gardener/etcd-backup-restore#224, @jfortin-sap)
• [OPERATOR] Fix snapshot metric initialization. (gardener/etcd-backup-restore#223, @shreyas-s-rao)
• [DEVELOPER] Add TestMachinery integration. (gardener/etcd-backup-restore#249, @shreyas-s-rao)

📰 Noteworthy

• [USER] Add support for Dell EMC ECS object store with S3 protocol (gardener/etcd-backup-restore#254, @lcavajani)
• [OPERATOR] Added CLI-flags (max-call-send-message-size, max-request-bytes and max-txn-ops) to enable restoration for delta snapshots with large amount of data (large number of events or events with large data). (gardener/etcd-backup-restore#282, @abdasgupta)
• [DEVELOPER] Revendor etcd library to v3.4.13 and change import paths to go.etcd.io/etcd for etcd and go.etcd.io/bbolt for bbolt. (gardener/etcd-backup-restore#269, @shreyas-s-rao)
• [DEVELOPER] Integration tests can be executed on any given Kubernetes cluster using the make integration-test-cluster target against the cluster pointed to by the environment variable INTEGRATION_TEST_KUBECONFIG. (gardener/etcd-backup-restore#225, @shreyas-s-rao)

[etcd-druid]

🏃 Others

• [OPERATOR] Added support for OpenShift Container Storage (OCS) S3 storage type. (gardener/etcd-druid#98, @stoyanr)

[gardener-resource-manager]

✨ New Features

• [OPERATOR] gardener-resource-manager now logs its own version on startup or when executed with --version. (gardener/gardener-resource-manager#96, @timebertt)
• [DEVELOPER] Docker images built by make docker-images are now tagged and build with the commit hash appended to the version. (gardener/gardener-resource-manager#96, @timebertt)
• [DEVELOPER] The cache of the kubernetes client for the target cluster can now be disabled via the --target-disable-cache flag. (gardener/gardener-resource-manager#95, @timebertt)

🏃 Others

• [OPERATOR] gardener-resource-manager now uses a DynamicRESTMapper, which will reduce the amount of explicit discovery calls and faster reconciliation loops and some cases. (gardener/gardener-resource-manager#95, @timebertt)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.14.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.14.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.14.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.14.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.14.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.14.0

[gardener]

🏃 Others

• [OPERATOR] The shoot controller inside the gardenlet has been adapted to cater with large Gardener landscapes:
  1. When the gardenlet has already reconciled a shoot cluster during its maintenance time window then it doesn't reconcile it again. Instead, it computes a random duration for the next time window and requeues the shoot. Already reconciled shoots are those whose last reconciliation was less then 24h ago.
  2. When the gardenlet is (re)started then it does no longer reconcile all shoots immediately whose maintenance time windows are met. Instead, it computes a random time for the current time window and requeues the shoot ("jittering", i.e., spreading the load). This will have the effect that not all shoots are getting reconciled at the same time right after startup. (38bf4669b2f642642f11b1e525dee62db29becf3)
• [USER] Containerd is supported in regions where gcr.io container registry can't be accessed. (#3182, @jia-jerry)
• [USER] Fixed NetworkPolicy gardener.cloud--allow-dns to allow traffic from Pods with hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet. (#3163, @mvladev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.9
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.9
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.9
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.9
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.9
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.9

[gardener]

Improvements

• [OPERATOR] The shoot controller inside the gardenlet has been adapted to cater with large Gardener landscapes:
  1. When the gardenlet has already reconciled a shoot cluster during its maintenance time window then it doesn't reconcile it again. Instead, it computes a random duration for the next time window and requeues the shoot. Already reconciled shoots are those whose last reconciliation was less then 24h ago.
  2. When the gardenlet is (re)started then it does no longer reconcile all shoots immediately whose maintenance time windows are met. Instead, it computes a random time for the current time window and requeues the shoot ("jittering", i.e., spreading the load). This will have the effect that not all shoots are getting reconciled at the same time right after startup. (98ddb6783d91243f33ca893000e6ae9f7798a811)
• [USER] The KUBERNETES_SERVICE_HOST environment variable injected when APIServerSNI is enabled no longer includes a trailing dot (being a Fully Qualified Domain Name) due to several Homebrew Kubernetes clients not properly handling it and sending wrong server name when initiating a TLS connection. (#3236, @mvladev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.13.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.13.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.13.2
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.13.2
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.13.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.13.2

[gardener]

Improvements

• [OPERATOR] A bug has been fixed which can lead to Seeds not getting ready when an image vector overwrite for the etcd-druid is configured. (88a620fd039ebc62a4f72279951d480f570d3f1b)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.13.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.13.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.13.1
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.13.1
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.13.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.13.1

[gardener]

Action Required

• [USER] We are preparing a change that will lead to validation errors when the Project resource contains duplicates in the .spec.members[] list. For the time being, duplicates in this list are merged into a single member automatically by the Gardener API Server. In the future, this will no longer happen, instead, a validation error will be returned if a user sends a Project resource with duplicate members. Please adapt your API usage to not send any of such resources. (#3137, @rfranzke)
• [USER] The project controller is now adapted to accept and maintain only the project namespace labels from the new API group - gardener.cloud/role=project and project.gardener.cloud/name=<project-name>. Until now the project controller was accepting and maintaining also the labels from the old API groups - garden.sapcloud.io/role=project and project.garden.sapcloud.io/name=<project-name>. With this change, the project controller removes the namespace labels associated to the old API group. If you are still using these deprecated labels, you need to adapt your machinery. (#3094, @ialidzhikov)
• [USER] The already deprecated annotations shoot.garden.sapcloud.io/use-as-seed and shoot.garden.sapcloud.io/ignore-alerts are no longer respected by the corresponding Gardener components. If you are still using these deprecated annotations, you need to adapt your machinery to use respectively shoot.gardener.cloud/use-as-seed and shoot.gardener.cloud/ignore-alerts. You need to be careful with use-as-seed annotation as if you don't adapt to the new annotation and update your Gardener version to the one that no longer respects the deprecated annotation, this will be handled as deletion of the ShootedSeed and will mark the Seed for deletion. (#3094, @ialidzhikov)
• [USER] A new webhook mutatingwebhookconfigurations.admissionregistration.k8s.io is deployed for all APIServerSNI- enabled clusters. It adds KUBERNETES_SERVICE_HOST environment variable pointing to the upstream Kube API Server. To disable this behavior: (#3082, @mvladev)
  • label your Pods with apiserver-proxy.networking.gardener.cloud/inject: disable
  • or label the entire namespace with apiserver-proxy.networking.gardener.cloud/inject: disable
  • or label your Shoot resource with alpha.featuregates.shoot.gardener.cloud/apiserver-sni-pod-injector: disable to disable it cluster-wide.
• [USER] For APIServerSNI-enabled clusters, Pods talking to the Kube API Server need to be allowed to connect to coredns running in kube-system namespace in order to resolve the hostname of the Kube API server. It also needs to have access to the IP from the default/kubernetes endpoint and the upstream IP of of the kube-apiserver. (#3082, @mvladev)
  • If the Pod:
  • is not matched by any NetworkPolicy - no action is required.
  • is not injected with KUBERNETES_SERVICE_HOST, because the feature is disabled - no action is required.
  • is matched by NetworkPolicies allowing ingress to coredns in kube-system and allows traffic top the default/kubernetes endpoint and the upstream upstream IP of of the kube-apiserver - no action is required.
  • is matched by NetworkPolicies that do not allow access to coredns in kube-system and/or do not allows traffic top the default/kubernetes endpoint and/or the upstream upstream IP of of the kube-apiserver - a NetworkPolicy allowing such egress must be added e.g.:

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-to-apiserver
    spec:
      podSelector: {}
      egress:
      - to:
        - ipBlock:
            cidr: <IP from default/kubernetes endpoint>/32
        - ipBlock:
            cidr: <ip from apiserver FQDN e.g. nslookup api.foo.bar>/32
      - ports:
        - port: 8053
          protocol: UDP
        - port: 8053
          protocol: TCP
        to:
        - podSelector:
            matchExpressions:
            - key: k8s-app
              operator: In
              values:
              - kube-dns
          namespaceSelector:
            matchLabels:
              gardener.cloud/purpose: kube-system
      policyTypes:
      - Egress
      - Ingress
    

• [DEVELOPER] Terraformer.SetVariablesEnvironment has been deprecated in favor of Terraformer.SetEnvVars. Please adapt your usage of the terraformer library accordingly. (#3204, @timebertt)
• [DEVELOPER] The generic worker actuator interface does now include a new function MachineClass() runtime.Object that needs to be implemented. It is similar to MachineClassList() runtime.Object with the difference that it does not return the list object but the machine class object itself. (#3178, @rfranzke)
• [DEVELOPER] The CleanupLeakedClusterRoles function has been removed from the generic worker actuator package. You can find more information about it here and here. (#3178, @rfranzke)
• [DEVELOPER] A temporary workaround during the Cluster resource sync to the Seed by setting a fake Shoot status to prevent ShootNotFailed predicate in the extensions library from reacting false negatively is now cleaned up. Before upgrading to this version of Gardener, make sure that all of the extensions in your environment that use the ShootNotFailed predicate vendor github.com/gardener/[email protected] or above (that contains https://github.com/gardener/gardener/pull/2265). (#3097, @ialidzhikov)

Most notable changes

• [USER] The Shoot now has a new constraint with type MaintenancePreconditionsSatisfied which indicates whether it's safe to maintain a shoot (see this document to get an overview what happens during maintenance). End-users should check this information to properly configure their clusters in order to avoid problems. (#3173, @rfranzke)
• [USER] It is now possible to configure the kube-apiserver's --max-requests-inflight and --max-mutating-requests-inflight flags by setting the .spec.kubernetes.kubeAPIServer.requests.max{Non}MutatingInflight fields (default: {400}/200) in the Shoot specification. (#3141, @rfranzke)
• [USER] It is now possible to configure the kube-controller-manager's --pod-eviction-timeout flag by setting the .spec.kubernetes.kubeControllerManager.podEvictionTimeout field (default: 2m0s) in the Shoot specification. (#3139, @rfranzke)
• [OPERATOR] The controlplane Helm chart for Gardener does now expose a few more configuration options for the gardener-apiserver: (#3207, @rfranzke)
  • .Values.global.apiserver.goAwayChance configures the --goaway-chance flag.
  • .Values.global.apiserver.http2MaxStreamsPerConnection configures the --http2-max-streams-per-connection flag.
  • .Values.global.apiserver.shutdownDelayDuration configures the --shutdown-delay-duration flag.
  • .Values.global.requests.maxNonMutatingInflight configures the --max-requests-inflight flag.
  • .Values.global.requests.maxMutatingInflight configures the --max-mutating-requests-inflight flag.
  • .Values.global.requests.minTimeout configures the --min-request-timeout flag.
  • .Values.global.requests.timeout configures the --request-timeout flag.
  • .Values.global.watchCacheSizes.default configures the --default-watch-cache-size flag.
  • .Values.global.watchCacheSizes.resources[] configures the --watch-cache-size flag.
• [OPERATOR] A bug has been fixed that might led to orphaned machine resources in the shoot namespace in the seed that are stuck with the machine-controller-manager finalizer. (#3178, @rfranzke)
• [OPERATOR] ManagedIstio and APIServerSNI can now be optionally configured via the new sni configuration in GardenletConfiguration, see the example configuration. This allows to use installation of Istio where the the ingressgateway is in another namespace. (#3143, @mvladev)
• [OPERATOR] It is now possible to configure the TTL used for DNSEntry objects in the shoot controller via the gardenlet's component config (.controllers.shoot.dnsEntryTTLSeconds, default: 120). (#3142, @rfranzke)
• [OPERATOR] It is now possible exclude specific Projects from the stale checks by annotating their related Namespaces with project.gardener.cloud/skip-stale-check=true. (#3136, @rfranzke)
• [OPERATOR] The gardenlet rollout caused by shooted seed registrations is now spread by default within [0,5m]. You can overwrite this jitter period in the gardenlet's component configuration (.controllers.shootedSeedRegistration.syncJitterPeriod). (#3135, @rfranzke)
• [OPERATOR] It is now possible to overwrite the feature gates in the gardenlet configuration for shooted seeds without the no-gardenlet option by setting featureGates.<name>={true,false}. (#3134, @rfranzke)
• [OPERATOR] The audit policy config map reference protection controller introduced with v1.12.0 is now disabled by default. You can explicitly enable it in the gardener-controller-manager's component configuration by setting .controllers.shootReference.protectAuditPolicyConfigMaps=true. (#3117, @rfranzke)
• [OPERATOR] A new webhook mutatingwebhookconfigurations.admissionregistration.k8s.io is deployed for all APIServerSNI-enabled clusters. It's running as a sidecar to the KubeAPI Server. (#3082, @mvladev)
• [OPERATOR] The scheduler ensures that only candidate seeds with available capacity for shoots are considered during scheduling. The resource capacity and reservations can be configured in the gardenlet's component configuration. By default, the capacity for shoots in a seed is 200. (#3075, @stoyanr)
• [OPERATOR] Operators can now define a default ResourceQuota resource which is automatically created in project related namespaces. Please consult the documentation (/docs/concepts/controller-manager.md) for more information. (#3072, @timuthy)
• [OPERATOR] apiserver-proxy overload's manager is removed. (#3062, @mvladev)

Improvements

• [USER] An issue has been fixed that can cause Shoots from being stuck in deletion due to invalid .spec.dns configuration. (#3168, @rfranzke)
• [USER] A bug has been fixed that can cause Shoot resources from being stuck in Delete Succeeded state. (#3167, @rfranzke)
• [USER] Containerd is supported in regions where gcr.io container registry can't be accessed. (#3164, @jia-jerry)
• [USER] Fixed NetworkPolicy gardener.cloud--allow-dns to allow traffic from Pods with hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet. (#3162, @mvladev)
• [USER] The gardenlet's shoot controller does now forget about the rate limiting when a shoot's deletion timestamp is set. This is to make starting deletion operations faster. (#3144, @rfranzke)
• [USER] Project admins and viewers are now allowed to read corev1.ResourceQuota objects. (#3132, @timuthy)
• [USER] Gardener now allows to edit a Shoot's metadata (including confirming the deletion), even if some referenced object (e.g. audit policy) has already been deleted. (#3116, @timebertt)
• [USER] Missing audit policy ConfigMaps for Shoots are now ignored when trying to redeploy the kube-apiserver in the shoot deletion flow. (#3115, @rfranzke)
• [USER] A bug that was preventing custom CA certificates to be installed on the shoot nodes is now fixed. (#3113, @vpnachev)
• [USER] An issue causing CoreDNS dashboard to show always 'No Data' is now fixed. (#3089, @wyb1)
• [USER] gardener-controller-manager's Shoot reference controller now also handles audit policy ConfigMap references. (#3071, @ialidzhikov)
• [USER] An issue causing spec.kubernetes.kubelet.kubeReserved.pid field of the Shoot to be set for Kubernetes versions that don't support the corresponding feature is now fixed. (#3059, @ialidzhikov)
• [USER] Node exporter properly reports filesystem size for operating systems that use an xfs filesystem (#3053, @wyb1)
• [OPERATOR] Fix a bug where allow-to-seed-apiserver might not include the IP from the KUBERNETES_SERVICE_HOST environment variable of the gardenlet. (#3203, @mvladev)
• [OPERATOR] Gardener triggers an infrastructure reconciliation during maintenance also for hibernated clusters. This ensures that the infrastructure is always up-to-date, even for long-term hibernated clusters. (#3196, @timuthy)
• [OPERATOR] Gardenlet now logs the HTTP response of failed shoot health checks for checkAPIServerAvailability. (#3195, @timuthy)
• [OPERATOR] Traffic is now allowed to cluster dns and the node local ipvs address to resolve a dns resolution issue with the NodeLocalDNS feature for dns names in control plane pods. (#3184, @DockToFuture)
• [OPERATOR] When existing Namespaces are adopted for Projects then they will now be configured to remain even after the Project is being deleted later again. Earlier, such namespaces were also deleted together with the Project. Please note that this only takes effect for newly adopted project namespaces. (#3179, @rfranzke)
• [OPERATOR] A bug in shoot deletion flow that was stuck waiting the kube-controller-manager to be scaled up when it is already deleted is now fixed. (#3176, @vpnachev)
• [OPERATOR] The golang version is updated to 1.15.5. (#3175, @ialidzhikov)
• [OPERATOR] metrics-server is upgraded to v0.4.1 and readiness and liveness probes now use http instead of tcp. (#3174, @mvladev)
• [OPERATOR] Fluent-bit version upgraded to 1.6.4. (#3165, @vlvasilev)
• [OPERATOR] Add Readiness and Liveness probe to the fluent-bit DaemonSet. (#3165, @vlvasilev)
• [OPERATOR] Change the name of the gardener custom fluent-bit-to-loki plugin from loki to gardenerloki to avoid any plugin collisions with future version of the fluent-bit. (#3165, @vlvasilev)
• [OPERATOR] Add minimum resource limit for the hvpa controller vpa (#3154, @BeckerMax)
• [OPERATOR] The federated seed controller syncing the ShootState is no longer sends events for ShootState resources as they are not evaluated in a meaningful way anyways. (#3149, @rfranzke)
• [OPERATOR] An issue has been fixed which caused the logging stack to skip logs for certain extension pods. (#3146, @Kristian-ZH)
• [OPERATOR] An error has been fixed which caused the seed reconciliation (bootstrap) to fail if ManagedIstio is not enabled. (#3145, @timuthy)
• [OPERATOR] A bug has been fixed that resulted in a stuck Seed deletion due to orphaned ManagedResources in the garden namespace. (#3133, @rfranzke)
• [OPERATOR] When disabling APIServerSNI feature gate, existing LoadBalancer ports from ManagedIstio are not removed until all existing SNI-enabled Shoot clusters are migrated. (#3125, @mvladev)
• [OPERATOR] A bug has been fixed that caused the gardenlet to deploy further instances of itself with its own self-generated server certificate. It prevents undesired redeployments of these further instances. (#3114, @rfranzke)
• [OPERATOR] Increase kube-controller-manager VPA minAllowed (#3107, @ggaurav10)
• [OPERATOR] gardenlet no longer sets the Shoots status to Failed too early when operation cannot be initialized (#3106, @ialidzhikov)
• [OPERATOR] istio-ingressgateway now uses KEEPALIVE to downstream LoadBalancers to prevent idle timeout issues. (#3104, @mvladev)
• [OPERATOR] The vpn deployment rolling strategy is improved so that the new pod is created before the old one is deleted. (#3100, @vpnachev)
• [OPERATOR] A bug has been fixed that prevented the Loki HVPA recommendations from not being reverted. (#3098, @Kristian-ZH)
• [OPERATOR] apiserver-proxy now uses tcp keepalive every 55 seconds to prevent idle timeouts between it and the SNI LoadBalancer. (#3092, @mvladev)
• [OPERATOR] A race condition in Gardener's helm chart (/charts/gardener/controlplane) has been fixed. Earlier, the deployed ValidatingWebhookConfiguration potentially blocked the creation of Gardener ServiceAccounts. The validation is now excluded from namespaces with the label app: gardener. (#3088, @timuthy)
  • ℹ️ Please make sure you either let /charts/gardener/controlplane also deploy the garden namespace, or add the label app=gardener to the namespace yourself.
• [OPERATOR] A cache issue preventing Shoot reference controller of gardener-controller-manager to do not reconcile sometimes references for newly created Shoots when the CachedRuntimeClients feature gate is enabled is now fixed. (#3087, @ialidzhikov)
• [OPERATOR] Fix a bug, where the shoot deletion controller tries to scale-up the already deleted or never created kube-controller-manager deployment. (#3077, @vpnachev)
• [OPERATOR] Gardener now deletes (Cluster)RoleBindings of system components or addons, that were changed to an invalid state by endusers to be able to reconcile them back to the desired state. (#3074, @timebertt)
• [OPERATOR] Remove egress restrictions for vpn-shoot pod as it was incompatible with the cilium network plugin. (#3073, @DockToFuture)
• [OPERATOR] A bug that was preventing the ShootState resource to be updated with newly generated secrets is now fixed. (#3069, @vpnachev)
• [OPERATOR] Remove pod panels from node-details dashboard. These panels only showed pods running in the kube-system namespace so the data is not very relevant and only leads to confusion. (#3068, @wyb1)
• [OPERATOR] The kube-controller-manager VPA now has minAllowed values to prevent VPA from scaling it down too much. (#3057, @timebertt)
• [OPERATOR] It is again allowed the seed taints seed.gardener.cloud/disable-capacity-reservation, seed.gardener.cloud/disable-dns and seed.gardener.cloud/invisible to be used. Note, these taints have been replaced by seed.spec.settings fields and there is no special semantic behind them anymore. (#2970, @vpnachev)
• [OPERATOR] Migrating a Shoot from a Seed which has DNS disabled to a Seed which has DNS enabled will now generate a default domain name for the Shoot, if the Shoot's DNS section is not marked as unmanaged. (#2969, @plkokanov)
• [OPERATOR] Attempting to migrating a Shoot with spec.dns != nil to a Seed which has DNS disabled is now forbidden and will return an error. (#2969, @plkokanov)
• [DEVELOPER] github.com/gardener/gardener/extensions/pkg/predicate.Or (which was deprecated in favor of sigs.k8s.io/controller-runtime/pkg/predicate.Or) is now removed. (#3111, @ialidzhikov)

[gardener-resource-manager]

Action Required

• [DEVELOPER] New api module is now available. Get it with go get github.com/gardener/gardener-resource-manager/api (gardener/gardener-resource-manager#86, @mvladev)

Most notable changes

• [OPERATOR] The ManagedResource's .status.conditions[].lastUpdateTime is no longer continuously updated. This will greatly reduce the number of update calls to the kube-apiserver/etcd. (gardener/gardener-resource-manager#90, @rfranzke)
• [OPERATOR] The gardener-resource-manager does now feature a /healthz endpoint that can be used as part of a liveness probe configuration. It listens on port 8081 by default (configurable via the --health-bind-address flag). (gardener/gardener-resource-manager#81, @rfranzke)
• [DEVELOPER] The following Golang dependencies have been revendored: (gardener/gardener-resource-manager#80, @rfranzke)
  • github.com/gardener/gardener (v1.9.0 -> v1.11.3)
  • github.com/gardener/hvpa-controller (v0.2.5 -> v0.3.1)
  • github.com/onsi/ginkgo (v1.12.1 -> v1.14.0)
  • k8s.io/* (v0.16.8 -> v0.18.10)
  • sigs.k8s.io/controller-runtime (v0.5.5 -> v0.6.3)
• [DEVELOPER] The Golang version has been updated to v1.15.3. (gardener/gardener-resource-manager#80, @rfranzke)
• [DEVELOPER] The base image for the Docker image has been updated to alpine v3.12.1. (gardener/gardener-resource-manager#80, @rfranzke)

Improvements

• [OPERATOR] gardener-resource-manager is now using PATCH (instead of UPDATE) to add/remove a finalizer. (gardener/gardener-resource-manager#87, @ialidzhikov)
• [OPERATOR] An issue has been fixed which caused failing health checks for CustomResourceDefinitions of version v1. (gardener/gardener-resource-manager#85, @timuthy)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.13.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.13.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.13.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.13.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.13.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.13.0

[gardener]

Improvements

• [OPERATOR] The federated seed controller syncing the ShootState is no longer sends events for ShootState resources as they are not evaluated in a meaningful way anyways. (e20f0f38eb16566f0f2320ec892c3fadcb10df46)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.8
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.8
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.8
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.8
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.8
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.8

[gardener]

Improvements

• [OPERATOR] An issue has been fixed which caused the logging stack to skip logs for certain extension pods. (ce92d4cc5c85a7446951e481f5bb36e4bb68b92e)
• [OPERATOR] An error has been fixed which caused the seed reconciliation (bootstrap) to fail if ManagedIstio is not enabled. (7750dc943f1e022139bdaed735afac033cf5d8c5)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.7
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.7
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.7
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.7
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.7
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.7

[gardener]

Improvements

• [USER] A bug that was preventing custom CA certificates to be installed on the shoot nodes is now fixed. (e1ff02207ab7fb1a37dde804ba78452bf4eab514)
• [OPERATOR] When disabling APIServerSNI feature gate, existing LoadBalancer ports from ManagedIstio are not removed until all existing SNI-enabled Shoot clusters are migrated. (#3126, @mvladev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.6
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.6
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.6
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.6
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.6
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.6

[gardener]

Most notable changes

• [OPERATOR] The audit policy config map reference protection controller introduced with v1.12.0 is now disabled by default. You can explicitly enable it in the gardener-controller-manager's component configuration by setting .controllers.shootReference.protectAuditPolicyConfigMaps=true. (3db1c41726dc5f669e015f294b690d330b55bbf1)

Improvements

• [USER] Missing audit policy ConfigMaps for Shoots are now ignored when trying to redeploy the kube-apiserver in the shoot deletion flow. (ed6604017be6a0105af5297e55c0f4b1f5ed4f1d)
• [USER] Gardener now allows to edit a Shoot's metadata (including confirming the deletion), even if some referenced object (e.g. audit policy) has already been deleted. (96fc32d62dd147b02154b3215864304b942d1d4d)
• [OPERATOR] A bug has been fixed that prevented the Loki HVPA recommendations from not being reverted. (eac7f29e42698665c56776a23dc26ed944d484af)
• [OPERATOR] A bug has been fixed that caused the gardenlet to deploy further instances of itself with its own self-generated server certificate. It prevents undesired redeployments of these further instances. (0b51cbf81c3cd515b6eff6d1b166bc49bf4c419f)
• [OPERATOR] Increase kube-controller-manager VPA minAllowed (92882ffb9501878241e4c0e14675d568c21bb234)
• [OPERATOR] The vpn deployment rolling strategy is improved so that the new pod is created before the old one is deleted. (16ffd5ed26282ff41bbe34e84f4c1187c8d260c2)
• [OPERATOR] A race condition in Gardener's helm chart (/charts/gardener/controlplane) has been fixed. Earlier, the deployed ValidatingWebhookConfiguration potentially blocked the creation of Gardener ServiceAccounts. The validation is now excluded from namespaces with the label app: gardener. (fccb4bf02ce66ba4fc5c8892134879fe773795e5)
  • ℹ️ Please make sure you either let /charts/gardener/controlplane also deploy the garden namespace, or add the label app=gardener to the namespace yourself.

[gardener-resource-manager]

Improvements

• [OPERATOR] An issue has been fixed which caused failing health checks for CustomResourceDefinitions of version v1. (gardener/gardener-resource-manager@314c262952d8caf413de4ad5045b5185b9724aa3)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.5
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.5
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.5
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.5
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.5

[gardener]

Improvements

• [USER] An issue causing CoreDNS dashboard to show always 'No Data' is now fixed. (20dc1acb3489b655494f5b03abb1335c0545327f)
• [OPERATOR] istio-ingressgateway now uses KEEPALIVE to downstream LoadBalancers to prevent idle timeout issues. (#3105, @mvladev)
• [OPERATOR] apiserver-proxy now uses tcp keepalive every 55 seconds to prevent idle timeouts between it and the SNI LoadBalancer. (#3093, @mvladev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.12.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.12.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.12.4
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.12.4
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.12.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.12.4