[gardener]

Action Required

• [USER] If APIServerSNI feature gate is enabled by the Gardener administrators, a TLS client with support for server name indication must be used when talking to Shoot API servers. Alternatively, unmanaged DNS provider should be used. (#2406, @mvladev)
• [DEVELOPER] The creation and usage of kubernetes clientsets in controllers across gardener/gardener has been refactored. Please use the newly introduced ClientMaps to retrieve clientsets for all kind of clusters instead of creating new clientsets each time to leverage the clients' caches. (#2449, @tim-ebert)
• [DEVELOPER] Kubernetes clientsets (pkg/client/kubernetes.Interface) now return a cached controller-runtime client from Interface.Client() if kubernetes.UseCachedRuntimeClients has been set to true (defaults to false). The cached clients have to be started before the first usage by a call to Interface.Start(). If you use a ClientMap to retrieve the clientset, this is done automatically. (#2449, @tim-ebert)

Most notable changes

• [USER] Fixed a bug where for Shoot clusters >= 1.17, kube-apiserver did not have any root CA bundles. This resulted in failure to verify x509 certificates when attempting to send traffic to OIDC discovery endpoint or other endpoints. (#2508, @mvladev)
• [USER] The upstream community vertical-pod-autoscaler component is now supported for shoot clusters by setting .spec.kubernetes.verticalPodAutoscaler.enabled=true (disabled by default). More information can be found in this document. (#2478, @rfranzke)
• [USER] It is now correctly advertised in the OpenAPI specification that the name property for dataVolumes in the Shoot spec is a required field. (#2463, @rfranzke)
• [OPERATOR] A bug which blocked APIserver deployments on the same node (due to hostPort usage) is now fixed. (#2477, @zanetworker)
• [OPERATOR] A new FeatureGate CachedRuntimeClients has been added to gardenlet, gardener-controller-manager and gardener-scheduler. If enabled via the respective component config, the components use cached clients for their API calls wherever possible. (#2449, @tim-ebert)
• [OPERATOR] The gardener-controller-manager does now automatically delete stale Projects that are no longer in use. By default, only Projects older than 30d which are unused for at least 14d will be auto-deleted after 90d. However, the concrete values depend on the configuration of the respective Gardener landscape. You can find more information in this document. (#2446, @rfranzke)
• [OPERATOR] Enabling the feature gate APIServerSNI can cause the kube-controller-manager to be scaled down to 0 for 15 minutes. This is a known issue and it's going to be resolved in a future release of dependency-watchdog. (#2406, @mvladev)
• [OPERATOR] The feature gate APIServerSNI implementing GEP-8 is now available at alpha state. This allows for only one LoadBalancer in a Seed cluster to be used for all Shoot clusters in it. It's recommended to use in conjunction with ManagedIstio feature gate as the feature requires Istio to be installed in the Seed cluster. (#2406, @mvladev)
• [DEVELOPER] The Terraformer can now deal with output types other than String in the Terraform state. (#2460, @timuthy)

Improvements

• [USER] It is now possible to restrict core components from running on a worker pool by specifying systemComponents.allow: false in the pool definition. (#2480, @jannickfahlbusch)
  • System components deployed by extensions (Such as typha from calico) need a separate adaptation and are not covered by this change.
• [USER] The .spec.maintenance settings are now correctly defaulted when a Shoot is being created without any such configuration. (#2464, @rfranzke)
• [OPERATOR] Added vpa for hvpa-controller (#2553, @ggaurav10)
• [OPERATOR] Fixed possible race condition when updating the ShootState. (#2543, @plkokanov)
• [OPERATOR] An issue has been fixed, which caused the deletion of hibernated Shoots to be blocked if the KonnectivityTunnel has been enabled while the Shoot was hibernated. (#2540, @tim-ebert)
• [OPERATOR] Failed shoot conditions are set to Progressing status for the configured conditionThresholds time after a successful shoot reconciliation. This is to prevent false negative status reports shortly after reconciliations. (#2535, @rfranzke)
• [OPERATOR] Fixes a bug that could lead to defaulting a machine image of a Shoot to a preview version. (#2534, @danielfoehrKn)
• [OPERATOR] Conditions are now not only pardoned for Create/Delete operations but also for processing Reconcile operations in case there aren't any last errors. (#2533, @rfranzke)
• [OPERATOR] It is now possible to configure the enabled FeatureGates for the gardener-scheduler via the respective values in the gardener/controlplane chart. (#2531, @tim-ebert)
• [OPERATOR] The istiod validating webhook on Seeds is now exposed at port 443, allowing it to function properly in GKE clusters. (#2529, @tim-ebert)
• [OPERATOR] Adapts values for hvpa's LimitsRequestsGapScaleParams to latest hvpa-controller version (#2521, @ggaurav10)
• [OPERATOR] Backupentry is now properly ignored when trying to annotate it with gardener.cloud/operation=migrate if it has already been deleted from the cluster. (#2520, @plkokanov)
• [OPERATOR] An issue has been fixed which caused Gardener to delete on-demand extensions prematurely. (#2517, @timuthy)
• [OPERATOR] Removed the generic tolerations for all taints from control-plane component deployments. This is required for dedicated worker pool nodes to host only ETCD pods if gardener/kupid extension is deployed in the seed clusters. (#2507, @amshuman-kr)
• [OPERATOR] Fixed a bug, that caused gardener-controller-manager and gardenlet to panic if the Kubeconfig referenced by a Plant or Seed is empty. (#2504, @tim-ebert)
• [OPERATOR] Fixed a bug that leads to Shoots not receiving a force minor version update when the Kubernetes AutoUpdate is enabled. (#2490, @danielfoehrKn)
• [OPERATOR] Added the metrics shoot:container_network_transmit_bytes_total_apiserver:sum and shoot:container_network_receive_bytes_total_apiserver:sum which will be useful in observing the network traffic for all shoots. (#2488, @wyb1)
• [OPERATOR] ManagedIstio is updated to 1.6.3 (#2487, @mvladev)
• [OPERATOR] A bug has been fixed that prevented the HPA for istio to work as expected when the ManagedIstio feature gate was enabled. (#2486, @rfranzke)
• [OPERATOR] Konnectivity tunnel is now updated to v0.0.10. (#2484, @zanetworker)
• [OPERATOR] Runtime metrics for Pods and Nodes are now also available in environments which don't support a domain name resolution for worker nodes. (#2468, @timuthy)
• [OPERATOR] An issue has been fixed which prevented the retry operation for shoots from working reliably in case of a reconciliation. (#2467, @timuthy)
• [OPERATOR] CoreDNS pods are now protected by a PDB during machine upgrades and should reside on different nodes for HA. (#2466, @zanetworker)
• [OPERATOR] ControllerInstallations are not removed from the Seed if there is at least one Shoot referring it in the spec.seedName or status.seedName. (#2456, @swilen-iwanow)
• [OPERATOR] It is now possible to keep a Namespace in the system even when the related Project is deleted by annotating the Namespace with namespace.gardener.cloud/keep-after-project-deletion=true. (#2436, @rfranzke)
• [OPERATOR] Secrets deployed by gardener in the Shoot's Control Plane are now saved/loaded to/from the ShootState. (#2359, @plkokanov)
• [DEVELOPER] Docker images built by make docker-images are now tagged and build with the commit hash appended to the version. (#2500, @tim-ebert)
• [DEVELOPER] ChartApplier's Delete and ManifestReader's DeleteManifest now support passing TolerateErrorFunc option which can be used to tolerate certain errors - e.g. using TolerateNoMatchError can be useful in situations where a deleting a custom resource, but its CRD is already removed. (#2496, @mvladev)
• [DEVELOPER] General information about Gardener Enhancement Proposals (GEPs) have been added. Please consult this documentation for more information. (#2479, @timuthy)
• [DEVELOPER] Minor fixes for hook-me.sh script (#2473, @prashanth26)
• [DEVELOPER] The EnsureCleanedUp and WaitForCleanEnvironment funcs are now exported via the Terraformer interface. (#2461, @tim-ebert)

[dependency-watchdog]

Improvements

• [OPERATOR] Minimize throttling for happy path of probes (when targets do not need to be updated). (gardener/dependency-watchdog#20, @amshuman-kr)
  • For example, load the current replicas via the local cache, add jitter to the probe intervals to spread out host apiserver call, make client-go QPS and Burst configurable via CLI flags and export load-related metrics.

[gardener-resource-manager]

Most notable changes

• [OPERATOR] Please ensure, that gardener-resource-manager has the required permissions to also update secrets now. (gardener/gardener-resource-manager#54, @tim-ebert)

Improvements

• [USER] All resources managed by resource-manager now have resource-manager.gardener.cloud/description annotation with DO NOT EDIT warning. (gardener/gardener-resource-manager#46, @mvladev)
• [OPERATOR] Missing RBAC rules for updating and patching secrets were added. (gardener/gardener-resource-manager#71, @tim-ebert)
• [OPERATOR] A bug has been fixed, that caused new Services to get assigned a different ClusterIP than specified. (gardener/gardener-resource-manager#68, @tim-ebert)
• [OPERATOR] Fixed a bug, that caused the deletion of Services to be blocked. (gardener/gardener-resource-manager#65, @tim-ebert)
• [OPERATOR] Fixed a bug, that caused the deletion of RBAC resources to be blocked. (gardener/gardener-resource-manager#63, @tim-ebert)
• [OPERATOR] A forceful reconciliation for managed resources can now be triggered by the annotation gardener.cloud/operation: reconcile. (gardener/gardener-resource-manager#57, @timuthy)
• [OPERATOR] The Gardener-Resource-Manager does now try to recover unhealthy managed resources (condition ResourcesHealthy) by reconciling the resource. (gardener/gardener-resource-manager#57, @timuthy)
• [OPERATOR] gardener-resource-manager now properly removes its finalizer from secrets, that are not referenced by a ManagedResource anymore. (gardener/gardener-resource-manager#54, @tim-ebert)
• [OPERATOR] A bug has been fixed, which could lead to a situation, where a ManagedResource is falsely indicating a "Ready" state for a short period of time. (gardener/gardener-resource-manager#51, @tim-ebert)
• [OPERATOR] The ManagedResource CRD features a new field .spec.deletePersistentVolumeClaims. If set to true, gardener-resource-manager will delete PVCs belonging to managed StatefulSets, when they are deleted. (gardener/gardener-resource-manager#50, @tim-ebert)
• [OPERATOR] gardener-resource-manager now also injects labels specified in .spec.injectLabels into the .spec.volumeClaimTemplates of new StatefulSets. (gardener/gardener-resource-manager#49, @tim-ebert)
• [OPERATOR] gardener-resource-manager now deletes resources with DeletePropagationForeground to cascade the deletion to their dependents (e.g. to clean up Jobs created by a CronJob). (gardener/gardener-resource-manager#48, @tim-ebert)
• [OPERATOR] The logs of gardener-resource-manager have been reworked to contain less unnecessary error entries. (gardener/gardener-resource-manager#45, @tim-ebert)
• [OPERATOR] gardener-resource-manager now keeps the status of managed objects to prevent overwriting the status of CRs that don't have a status subresource. (gardener/gardener-resource-manager#44, @tim-ebert)
• [OPERATOR] gardener-resource-manager now keeps the replicas and/or resource requirements of Deployments and StatefulSets if they are scaled horizontally and/or vertically by an HPA or HVPA respectively. (gardener/gardener-resource-manager#44, @tim-ebert)
• [OPERATOR] The ResourcesApplied condition of ManagedResources now includes all errors, that occurred while applying/deleting managed objects if there were any. (gardener/gardener-resource-manager#43, @tim-ebert)
• [OPERATOR] A bug has been fixed, which made gardener-resource-manager fail to apply all new objects, if there were conflicting changes on those objects, instead of retrying the update request. (gardener/gardener-resource-manager#42, @tim-ebert)
• [OPERATOR] gardener-resource-manager now adds finalizers to Secrets referenced in ManagedResources to prevent Secrets from being deleted accidentally. (gardener/gardener-resource-manager#41, @tim-ebert)
• [OPERATOR] gardener-resource-manager now makes use of a caching client for talking to the targeted API server, which reduces its network traffic. (gardener/gardener-resource-manager#40, @tim-ebert)
• [OPERATOR] gardener-resource-manager handling for Jobs is now improved. (gardener/gardener-resource-manager#37, @ialidzhikov)
• [OPERATOR] Service merge now handles headless ClusterIP services and ExternalName services. .spec.healthCheckNodePort is only set if the service is of type LoadBalancer with .spec.externalTrafficPolicy: Local (gardener/gardener-resource-manager#35, @mvladev)
• [OPERATOR] A bug has been fixed, that caused new Services to get assigned a different ClusterIP than specified. (gardener/gardener-resource-manager@276bf6d3fbc265e3dc4a32286a6cc0fdf3dd8280)
• [OPERATOR] gardener-resource-manager handling for Jobs is now improved. (gardener/gardener-resource-manager@6e40fe5d7253ece0a562086254939f139f1719f1)
• [DEVELOPER] The new --always-update command line parameter (default: false) allows to configure whether to always send a PUT request for managed resources regardless of whether their desired state differs from their actual state. (gardener/gardener-resource-manager#70, @rfranzke)

[hvpa-controller]

Improvements

• [OPERATOR] Minor bug fix: Use vpa scale policies correctly (gardener/hvpa-controller#73, @ggaurav10)
• [OPERATOR] Change handling of LimitsRequestsGapScaleParams: Use max of value and percentage gaps, instead of min (gardener/hvpa-controller#72, @ggaurav10)
• [OPERATOR] Now HVPA doesn't take VPA recommendations into account if VPA condition has ConfigUnsupported, ConfigDeprecated or LowConfidence set to true (gardener/hvpa-controller#68, @ggaurav10)
• [OPERATOR] Removing "Temporary/fast fix to enable scale down even if vpaWeight == 0" as we have better ways to optimise cost now (gardener/hvpa-controller#64, @ggaurav10)
• [OPERATOR] Ignore minChange configuration while overriding scale up stabilisation. This ensures that full VPA recommendations are applied in case the target pods are OOMKilled or restarted due to livenessProbe failure, no matter what. (gardener/hvpa-controller#61, @amshuman-kr)
• [OPERATOR] Consider HPA to be limited if we have seen oomkill or liveness probe fails already. This change makes HVPA controller scale the app vertically more actively, ignoring the HPA's status condition type ScalingLimited. (gardener/hvpa-controller#57, @ggaurav10)
• [OPERATOR] Consider HPA scale out to be limited in case, overrideScaleUpStabilization is set in the status and hpa weight is 0 so that full VPA recommendation is immediately applied. (gardener/hvpa-controller#56, @ggaurav10)
• [OPERATOR] Add ci master build status and go report card badges (gardener/hvpa-controller#54, @ggaurav10)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.7.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.7.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.7.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.7.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.7.0

[gardener]

Improvements

• [OPERATOR] An issue has been fixed which caused Gardener to delete on-demand extensions prematurely. (b04a5ec4eae5d0f50d744c13747fd1db85ba7f34)

[dependency-watchdog]

Improvements

• [OPERATOR] Minimize throttling for happy path of probes (when targets do not need to be updated). (gardener/dependency-watchdog#20, @amshuman-kr)
  • For example, load the current replicas via the local cache, add jitter to the probe intervals to spread out host apiserver call, make client-go QPS and Burst configurable via CLI flags and export load-related metrics.

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.6
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.6
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.6
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.6
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.6

[gardener]

Improvements

• [OPERATOR] Fixed a bug that leads to Shoots not receiving a force minor version update when the Kubernetes AutoUpdate is enabled. (#2514, @vpnachev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.5
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.5
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.5
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.5

[gardener]

Improvements

• [OPERATOR] Fixed a bug that leads to Shoots not receiving a force minor version update when the Kubernetes AutoUpdate is enabled. (#2512, @vpnachev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.5
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.5
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.5
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.5

[gardener]

Most notable changes

• [OPERATOR] A bug which blocked APIserver deployments on the same node (due to hostPort usage) is now fixed. (f8a5056de1a3a8fe7cdee939172ba79b2386a395)

[gardener-resource-manager]

Improvements

• [OPERATOR] gardener-resource-manager handling for Jobs is now improved. (gardener/gardener-resource-manager@6e40fe5d7253ece0a562086254939f139f1719f1)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.4

[gardener]

Improvements

• [OPERATOR] An issue has been fixed which prevented the retry operation for shoots from working reliably in case of a reconciliation. (6d3b429ea50926bc737bdd5cd99d7a8a33b3c270)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.3

[gardener]

Improvements

• [OPERATOR] An issue has been fixed which prevented the retry operation for shoots from working reliably in case of a reconciliation. (a4322e3e49032bdd757ac1336ad462b92595b782)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.4
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.4
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.4
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.4

[gardener]

Improvements

• [USER] The .spec.maintenance settings are now correctly defaulted when a Shoot is being created without any such configuration. (e70b7294a9bd574bf4cf7926c9a51b9eaaf7054c)
• [DEVELOPER] The EnsureCleanedUp and WaitForCleanEnvironment funcs are now exported via the Terraformer interface. (7bb6e1c03c1f7a5f359bc8fcbf11028ede816aad)
• [DEVELOPER] The Terraformer can now deal with output types other than String in the Terraform state. (ae5ddf003f96dcf3e494f5dae2242257d3aa262c)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.2

[gardener]

Most notable changes

• [DEVELOPER] The Terraformer can now deal with output types other than String in the Terraform state. (8b47bb55612308a412ce2a9a6b7c864fed73c1af)

Improvements

• [OPERATOR] An issue has been fixed which caused failed shoot clusters to not be deleted successfully after a retry had been triggered. (#2445, @timuthy)
• [OPERATOR] Fixes a bug in the Gardener to include CRI information in the OperatingSystemConfig CRD. Os-extensions depend on that information to generate CRI specific files and systemd.services. In an edge case that could also lead to the containerd.service to not be enabled. (#2421, @danielfoehrKn)
• [DEVELOPER] The EnsureCleanedUp and WaitForCleanEnvironment funcs are now exported via the Terraformer interface. (c960b497333131630d0a7ee6e5408e6e0de0a37b)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.3

[gardener]

Improvements

• [OPERATOR] The health check for the tunnel pod is now falling back to the vpn-shoot pod if no pod labeled with type=tunnel is found. (#2448, @vpnachev)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.1

[gardener]

Action Required

• [OPERATOR] The discovery section has been removed completely from the GardenletConfiguration, SchedulerConfiguration, ControllerManagerConfiguration and the respective charts in charts/gardener as it is no longer needed. Please adapt your chart usages and/or component configurations accordingly. (#2415, @tim-ebert)
• [OPERATOR] The minimum required version for the Garden Cluster was increased to 1.16. (#2370, @tim-ebert)
• [DEVELOPER] The pkg/client/kubernetes.ApplierInterface interface has been renamed to Applier. Please adapt your usages accordingly when revendoring. (#2417, @tim-ebert)
• [DEVELOPER] You can now obtain a ChartRenderer and a ChartApplier by using the respective functions of the ClientSet interface (pkg/client/kubernetes.Interface). Please use the provided functions instead of constructing new ChartRenderers and ChartAppliers wherever needed. (#2417, @tim-ebert)

Most notable changes

• [USER] Shoot resources do now support specifying tolerations for taints on seeds in the .spec.tolerations field. Only tolerations that were whitelisted by the corresponding Project's .spec.tolerations.whitelist field, or by the global configuration (controlled by Gardener administrators) are allowed to be used. Please read more about it in this document. (#2384, @rfranzke)
• [OPERATOR] The Gardenlet supports a new KonnectivityTunnel feature gate (disabled by default). If enabled then the network connection between the shoot control plane in the seed and the shoot worker nodes will be established from shoot->seed for >= 1.18 shoot clusters (instead of seed->shoot like earlier). Furthermore, in this case the additional "vpn-shoot" load balancer in the shoot will no longer be needed. Please note that the feature is in alpha state and might be promoted in future Gardener releases. (#2251, @zanetworker)
• [DEVELOPER] The oscommon library for OperatingSystemConfig extension controllers was enhanced to allow providing additional values for the to-be-rendered template. These additional values can be computed out of the OperatingSystemConfig resource, for example, out of the providerConfig. (#2420, @rfranzke)

Improvements

• [USER] A bug has been fixed that prevented the correct transportation of the machine image specific configuration in .spec.provider.workers[].machine.image.providerConfig to the respective extension controller. (#2438, @rfranzke)
• [USER] It is now possible to configure the KUBE_MAX_PD_VOLS variable for the kube-scheduler using the .spec.kubernetes.kubeScheduler.kubeMaxPDVols flag. Please find more information here. Note that using this field is considered alpha-/experimental-level and is on your own risk. You should be aware of all the side-effects and consequences when changing it. (#2413, @rfranzke)
• [USER] The Shoot resource now supports changing the failSwapOn flag under the kubelet section, default is true. (#2411, @saggir)
• [USER] A bug has been fixed, which caused the kube-proxy to fail starting up for clusters with .spec.kubernetes.allowPrivilegedContainers=false. (#2395, @tim-ebert)
• [OPERATOR] Adds the kernel version to the metric shoot:node_operating_system:sum (#2443, @wyb1)
• [OPERATOR] An issue has been fixed which caused failed shoot clusters to not be deleted successfully after a retry had been triggered. (#2435, @timuthy)
• [OPERATOR] Upgrade grafana to 7.0.3 (#2428, @wyb1)
• [OPERATOR] Fixes a bug in the Gardener to include CRI information in the OperatingSystemConfig CRD. Os-extensions depend on that information to generate CRI specific files and systemd.services. In an edge case that could also lead to the containerd.service to not be enabled. (#2421, @danielfoehrKn)
• [OPERATOR] An issue preventing extensionsv1alpha1.BackupEntry to be deleted is now fixed. (#2419, @ialidzhikov)
• [OPERATOR] Istio is updated to 1.6.0 and status-port of ingress gateway is changed to 15021. (#2418, @mvladev)
• [OPERATOR] An issue has been fixed which prevented clusters from hibernating successfully. (#2410, @timuthy)
• [OPERATOR] An issues has been resolved which caused missing annotations for control plane pods of test shoots (.spec.purpose: testing). (#2402, @timuthy)
• [OPERATOR] The lastOperation of the infrastructure resource after successful restoration is no longer set to Migrate=Success. (#2400, @plkokanov)
• [OPERATOR] Shoots with 'incomplete DNS configuration' errors are now flagged with the ERR_CONFIGURATION_PROBLEM error code. (#2398, @rfranzke)
• [OPERATOR] The handling when evaluating errors from third-party controllers/components is more conservative now. It will wait until a certain grace period is exceed before actually revealing and reporting the error to the Shoot resource. (#2397, @rfranzke)
• [OPERATOR] Fixed a bug that lead to omitting the details message why a health check failed when writing the Extension CRD conditions. (#2394, @danielfoehrKn)
• [OPERATOR] Upgrade VPA to 0.8.0 (#2392, @wyb1)
• [OPERATOR] A new document providing an overview of the existing in-tree admission plugins in Gardener has been added. (#2384, @rfranzke)
• [OPERATOR] It is now possible to configure annotations for Services of type LoadBalancer created in the seed clusters. These annotations will be injected into each of these Service resources. They can be configured in the .spec.settings.loadBalancerServiceAnnotations field in Seed resources. Please consult this document for more information. (#2380, @rfranzke)
• [OPERATOR] References to resources (usually secrets) can now be added to the Shoot and referred to by name in various Shoot components, e.g. extension provider configurations. (#2360, @stoyanr)
• [OPERATOR] The NetworkPolicy 'allow-to-seed-apiserver' has been improved to only allow the seed apiserver endpoint IPs. A new controller in the Gardenlet creates and updates the aforementioned NetworkPolicy. (#2339, @danielfoehrKn)
• [OPERATOR] It is now possible to use a Kubernetes Cluster with private api server endpoints as a Seed cluster. (#2339, @danielfoehrKn)
• [DEVELOPER] The golang Kubernetes dependencies have been upgraded to v1.17.6. (#2370, @tim-ebert)
• [DEVELOPER] If you are using the nodeless dev-setup, you can now register gardener-controller-manager's validating webhooks by running hack/local-development/dev-setup-register-gardener --with-webhooks. (#2363, @tim-ebert)

[dependency-watchdog]

Improvements

• [OPERATOR] Probes are now restarted only if any of the target kubeconfigs actually changed. Also, the kubeconfig secrets are loaded from the informer rather than hitting the host apiserver (seed) with GET calls. Also, logged OS signal for debugging. (gardener/dependency-watchdog#16, @amshuman-kr)
• [OPERATOR] Log flags and configuration. (gardener/dependency-watchdog#13, @amshuman-kr)
• [OPERATOR] The release tags from now are prefixed with v. (gardener/dependency-watchdog#11, @ialidzhikov)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.6.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.6.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.6.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.6.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.6.0

[gardener]

Improvements

• [OPERATOR] An issue has been fixed which prevented clusters from hibernating successfully. (fb5be8b85416cdb82f165d1a8cc37f7d69ee523b)
• [OPERATOR] An issues has been resolved which caused missing annotations for control plane pods of test shoots (.spec.purpose: testing). (f7c2d264b0508ad9f74a7a8bc556fdaf278a1615)
• [OPERATOR] An issue preventing extensionsv1alpha1.BackupEntry to be deleted is now fixed. (b2979647a6a1104feae07e0cd7aba17b1441344f)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.2

[gardener]

Improvements

• [USER] A bug has been fixed, which caused the kube-proxy to fail starting up for clusters with .spec.kubernetes.allowPrivilegedContainers=false. (1f87b4414e934bad5004a44b21c3ef912b63260e)
• [OPERATOR] Shoots with 'incomplete DNS configuration' errors are now flagged with the ERR_CONFIGURATION_PROBLEM error code. (e0fc7d18da8873de6a2e1448d6acbdc44dc25f0e)
• [OPERATOR] The handling when evaluating errors from third-party controllers/components is more conservative now. It will wait until a certain grace period is exceed before actually revealing and reporting the error to the Shoot resource. (66c5c2824cda31a2b73603efdaaad6f8613e263f)
• [OPERATOR] Fixed a bug that lead to omitting the details message why a health check failed when writing the Extension CRD conditions. (858cb7ad5c9550125a39472a664f7f8f266a9845)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.1

[gardener]

Action Required

• [OPERATOR] The minimum required Kubernetes version for the garden cluster is now 1.12. Please make sure this your garden cluster is of at least this version before upgrading Gardener. (#2151, @swilen-iwanow)
• [DEVELOPER] Extension controllers using extensions/pkg/controller/error.RequeueAfterError need to make sure that the cause error is properly logged on their side. (#2351, @ialidzhikov)
• [DEVELOPER] The old testframework has been completely removed. All extension repositories that previously used the networkolicies generator have to be regenerated when this change is vendored. (#2344, @schrodit)
• [DEVELOPER] Extension controllers for Network and ContainerRuntime CRDs have to implement the Restore and Migrate operations. (#2276, @swilen-iwanow)

Most notable changes

• [OPERATOR] The Gardenlet does now run a HTTPS server that serves a /healthz and /metrics endpoint. You should generate a server certificate for the gardenlet, gardenlet.garden, gardenlet.garden.svc hosts. The bind address, port, and TLS certificate paths are configurable in its component config. Also, the gardenlet Helm chart was enhanced with a liveness probe that targets the /healthz endpoint. (#2309, @rfranzke)
• [USER] The SecurityContextDeny admission plugin is no longer allowed to be used for shoots as it conflicts with the PodSecurityPolicy admission plugin which is enabled by default. (#2346, @rfranzke)
• [USER] Shoot clusters which are flagged with ERR_INFRA_UNAUTHORIZED or ERR_CONFIGURATION_PROBLEM error codes are now automatically set to Failed status. This means that they won't be retried automatically unless you annotate the Shoot with gardener.cloud/operation=retry. All other error codes will lead to automatic retries for at most 12h before the shoot is set to Failed. (#2333, @rfranzke)
• [USER] Hibernation blocking due to Mutating/Validating webhooks is now improved and enforced for hooks with failurePolicy: Fail and operation CREATE, UPDATE or * for the following resources: (#2270, @mvladev)
  • apiservices
  • apiservices/status
  • certificatesigningrequests
  • certificatesigningrequests/approval
  • certificatesigningrequests/status
  • clusterrolebindings
  • clusterroles
  • configmaps (only for kube-system namespace)
  • controllerrevisions (only for kube-system namespace)
  • customresourcedefinitions
  • customresourcedefinitions/status
  • daemonsets (only for kube-system namespace)
  • daemonsets/status (only for kube-system namespace)
  • deployments (only for kube-system namespace)
  • deployments/scale (only for kube-system namespace)
  • endpoints
  • leases
  • namespaces
  • namespaces/finalize
  • namespaces/status
  • networkpolicies (only for kube-system namespace)
  • nodes
  • nodes/status
  • pods (only for kube-system namespace and shoot.gardener.cloud/no-cleanup=true,orgin=gardener labels)
  • pods/status (only for kube-system namespace and shoot.gardener.cloud/no-cleanup=true,orgin=gardener labels)
  • podsecuritypolicies
  • priorityclasses
  • replicasets (only for kube-system namespace)
  • replicasets/scale (only for kube-system namespace)
  • replicasets/status (only for kube-system namespace)
  • rolebindings (only for kube-system namespace)
  • roles (only for kube-system namespace)
  • secrets (only for kube-system namespace)
  • serviceaccounts (only for kube-system namespace)
  • services
  • services/status
• [OPERATOR] The default value for retryDuration has been changed from 24h to 12h. Hence, Gardenlet tries to reconcile shoots with erroneous operations for 12 hours (by default). After this period of time only the retry operation, a .spec change, or a rollout of a new Gardenlet version re-triggers a reconciliation. (#2324, @timuthy)
• [OPERATOR] The following taints have been deprecated and will be removed in a future version: (#2315, @rfranzke)
  • seed.gardener.cloud/disable-capacity-reservation in favour of the new .spec.settings.excessCapacityReservation.enabled field.
  • seed.gardener.cloud/invisible in favour of the new .spec.settings.scheduling.visible field.
  • seed.gardener.cloud/disable-dns in favour of the new .spec.settings.shootDNS.enabled field.
  • The .controllers.seed.reserveExcessCapacity setting in the component config of the Gardenlet has been removed in favour of the new settings field mentioned earlier.
  • Please check this document for further information about the seed settings.
• [OPERATOR] The ControllerRegistration object does now allow better control for deployment/deletion of extension controllers to seed clusters (policy, seed selector, ...). You might want to look into this document. (#2278, @rfranzke)
• [OPERATOR] Added new feature gate ManagedIstio to gardenlet. When enabled it deploys a customized installation of istio on Seed clusters. Disabling it once enabled does not remove any installed resources. (#2273, @mvladev)
  • The installation deploys istiod in the istio-system namespace and istio ingress gateway in istio-ingress namespace. mTLS is enforced and Services, VirtualServices and DestinationRules are not exported and advertised by default and therefore it must be explicitly enabled either via the networking.istio.io/exportTo: "*" annotation or .exportTo: ["*"].

Improvements

• [USER] An issue causing an additional reconciliation after successful flow execution for newly created Shoots is now fixed. (#2364, @ialidzhikov)
• [USER] A bug has been fixed that caused the gardener-controller-manager to incorrectly delete extension ClusterRoles of other projects when reconciling a Project that doesn't have a member with the same extension role. (#2352, @rfranzke)
• [USER] The Shoot specification does now have a new .spec.seedSelector field which allows to provide label selector. Only seeds whose labels match will be considered for scheduling decisions. (#2340, @rfranzke)
• [USER] The bootstrapping of new shoot worker nodes has been made more reliable. (#2313, @rfranzke)
• [USER] It is now possible to opt-out of/disable globally enabled extensions for Shoots by setting .spec.extensions[] = {type: <extension-type>, disabled: true}. (#2278, @rfranzke)
• [OPERATOR] Some grafana dashboards have been changed to use a rate of 2m instead of 1m. (#2386, @wyb1)
• [OPERATOR] A bug has been fixed that made the gardener-scheduler considering protected seeds for shoots outside the garden namespace, potentially making it trying to schedule a shoot to such a seed (which will fail forever). (#2382, @rfranzke)
• [OPERATOR] Now when the excess capacity reservation is disabled, the reservation pods are deleted. (#2365, @vpnachev)
• [OPERATOR] Fixed a bug when deploying ContainerRuntime custom resources for multiple worker pools. (#2357, @danielfoehrKn)
• [OPERATOR] Extension Reconcilers do no longer log the same error which the controller-runtime is supposed to log too. This should prevent given error to appear multiple times in the logs. (#2351, @ialidzhikov)
• [OPERATOR] Resources (usually secrets) required by extension controllers can now be added to the extension status, persisted in the ShootState, and migrated from the source seed to the destination seed. (#2350, @stoyanr)
• [OPERATOR] A bug that was preventing the Gardenlet from bootstrapping seed clusters if the HVPA feature gate is disabled was fixed. (#2345, @rfranzke)
• [OPERATOR] It is now possible to add arbitrary labels to the .spec.regions[].labels field in the CloudProfile, e.g. to provide more information about reliability, access restrictions, etc. (#2340, @rfranzke)
• [OPERATOR] The check for stale extension healthchecks is now also disabled by default in gardenlet chart values.yaml. (#2337, @ialidzhikov)
• [OPERATOR] The error reporting for shoot clusters has been improved. Configuration problems or similar issues which occur during shoot reconciliation are now instantly visible in the shoot resource. (#2336, @timuthy)
• [OPERATOR] Adjust default concurrent-sync settings for the Gardenlet controllers ShootState and ControllerInstallationRequired causing too many goroutines to be created. Also fixed a bug during worker creation to further reduce the amount of workers being created. (#2331, @danielfoehrKn)
• [OPERATOR] Fix the Seed enqueue in the federated seed controller for initially not-bootstrapped Seeds. Now Seeds are properly enqueued when their status changes from not-bootstrapped to successfully bootstrapped. (#2330, @danielfoehrKn)
• [OPERATOR] Update of the temporary, experimental Kyma addon to latest Kyma version 1.12.0. It can be installed onto shoot clusters out-of-the-box by annotating the Shoot with experimental.addons.shoot.gardener.cloud/kyma=enabled. Be aware that we won't provide upgrades or customization, and that this addon is temporary and will be removed in a future version of Gardener again. Its purpose is to ease the Kyma installation and to show-case which features it provides. It is by no means a production-ready setup. Also, please note that, once enabled, the Kyma addon can never be disabled again. The only way to get rid of it is to delete the shoot cluster. You can check the status of the installation by using kubectl get installation/kyma-installation -o jsonpath="{'Status: '}{.status.state}{', description: '}{.status.description}". (#2326, @a-thaler)
• [OPERATOR] An issue has been fixed with prevented failed shoots from being excluded from reconciliation after the retryDuration is exceeded. (#2324, @timuthy)
• [OPERATOR] Exclude logs from testing purpose or hibernated shoots (#2320, @vlvasilev)
• [OPERATOR] Removed safe-to-evict cluster-autoscaler annotation from etcd-events pods. (#2317, @georgekuruvillak)
• [OPERATOR] The kube-apiserver deployment of shoots is now scheduled with anti-affinity and a pod disruption budget of 1. (#2310, @rfranzke)
• [OPERATOR] An issue preventing gardenlet to reflect unhealthy Shoot conditions in the shoot.gardener.cloud/status label for newly created Shoots is now fixed. (#2308, @ialidzhikov)
• [OPERATOR] Fixed a bug in the health check library that leads to too many health checks being executed when the Extension.Status changes. (#2307, @danielfoehrKn)
• [OPERATOR] Fixed the health check condition.lastTransitionTime in the health check library. (#2307, @danielfoehrKn)
• [OPERATOR] The Gardenlet sets an additional annotation to Extension CRDs during reconciliation to guarantee an update event for the watching clients. (#2290, @danielfoehrKn)
• [OPERATOR] Fixes a bug in the extension libraries that could lead to duplicate reconciliation of extension resources. When respecting the operation annotation set by the Gardenlet during reconciliation, extension controllers now only watch the Extension CRD. (#2290, @danielfoehrKn)
• [OPERATOR] Fixes a bug in the extension library of all extension resources that lead to not stopping the reconciliation of extension resources when the Shoot is in 'failed' state (Shoot.Status.lastOperation.state = Failed). (#2279, @danielfoehrKn)
• [OPERATOR] An issue has been fixed that may have resulted in the deletion of extension controllers from a seed cluster although the seed still had extension objects that the extension controller was responsible for. (#2261, @rfranzke)
• [OPERATOR] Restrictions on kube-proxy are lifted to allow switching of kube-proxy mode (IPTables, IPVS) for k8s cluster > 1.16. (#2238, @DockToFuture)
• [OPERATOR] Seed lifecycle controller uses Lease object to report Seeds' readiness (#2151, @swilen-iwanow)
• [OPERATOR] Updating maintenance integration tests for minor Kubernetes updates. (#2109, @danielfoehrKn)
• [DEVELOPER] A utility function for cleaning up orphaned ClusterRoles for the machine-controller-manager was added to the extensions/pkg/controller/worker/genericactuator package. (#2378, @rfranzke)
• [DEVELOPER] The generic Worker actuator does now exit its reconciliation flows early if it detects an error during the machine reconciliation. This allows to faster propagate problems to the end-user. (#2348, @rfranzke)
• [DEVELOPER] The extension health check library does now allow individual health checks to return the Progressing status. This allows to provide more accurate status information and less false negative health reports. (#2289, @rfranzke)
• [DEVELOPER] It is now possible to add a global cleanup function for integration tests. (#2283, @schrodit)
• [DEVELOPER] It is now possible to add a dedicated AfterTest function to test cases to run a specific function when the test has finished. (#2283, @schrodit)
• [DEVELOPER] Extend the Extensions actuator interface with Migrate and Restore (#2277, @vlvasilev)

[etcd-druid]

Most notable changes

• [USER] Bump default etcd-backup-restore image version to v0.9.1. This fixes the false alerts for FullBackupFailed on etcd pod restart. ⚠️ Etcd pod will be restarted in next reconcile. (gardener/etcd-druid#56, @georgekuruvillak)
• [DEVELOPER] ⚠️ Etcd-druid NO MORE adds the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to etcd pods. Please make use of .spec.annotations to configure such annotation. (gardener/etcd-druid#55, @swapnilgm)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.5.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.5.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.5.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.5.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.5.0

[gardener]

Improvements

• [USER] A bug has been fixed that caused the gardener-controller-manager to incorrectly delete extension ClusterRoles of other projects when reconciling a Project that doesn't have a member with the same extension role. (d9799ba074767f58e1d26642387e3cc4aaef9750)
• [OPERATOR] A bug that was preventing the Gardenlet from bootstrapping seed clusters if the HVPA feature gate is disabled was fixed. (6790d072c08b1db22d223f8854c3619c263c572b)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.3.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.3.3
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.3.3
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.3.3
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.3.3

[gardener]

Improvements

• [USER] A bug has been fixed that caused the gardener-controller-manager to incorrectly delete extension ClusterRoles of other projects when reconciling a Project that doesn't have a member with the same extension role. (83acc93fedc5f35b6cd95ea879fa2d3b9a2fb6b2)
• [OPERATOR] Fixed a bug when deploying ContainerRuntime custom resources for multiple worker pools. (b9a920ab6256f68561837627c2cdfb823bd0160e)
• [OPERATOR] A bug that was preventing the Gardenlet from bootstrapping seed clusters if the HVPA feature gate is disabled was fixed. (654ee8e45e8c214cd3c5ea8e6ac8e5ba879de997)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.4.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.4.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.4.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.4.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.4.2

[gardener]

Improvements

• [USER] The bootstrapping of new shoot worker nodes has been made more reliable. (b729b72fb21d663596b17975da2b619a3e116069)
• [OPERATOR] Fixed a bug in the health check library that leads to too many health checks being executed when the Extension.Status changes. (5507532513a29aefa2249e2e96b83045cfd8cb5d)
• [OPERATOR] Fixed the health check condition.lastTransitionTime in the health check library. (5507532513a29aefa2249e2e96b83045cfd8cb5d)
• [OPERATOR] An issue preventing gardenlet to reflect unhealthy Shoot conditions in the shoot.gardener.cloud/status label for newly created Shoots is now fixed. (2bbb388c80080c85e551cfd13517ef24dccf3cff)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.4.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.4.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.4.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.4.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.4.1

[gardener]

Improvements

• [OPERATOR] Fixes scale down issue with MachineDeployments for certain K8s versions (#2303, @prashanth26)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.3.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.3.2
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.3.2
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.3.2
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.3.2

[gardener]

Most notable changes

• [USER] There are two new error codes that help categorizing a problem occurred during a shoot reconciliation or health check: ERR_INFRA_RESOURCES_DEPLETED indicates that the underlying infrastructure does not have enough resources anymore, and ERR_CONFIGURATION_PROBLEM indicates that the user has misconfigured something and should double-check the specification. (#2237, @rfranzke)
• [USER] It is now possible to confine rollouts of changes/updates to the Shoot specification to the individual maintenance time window. You can set .spec.maintenance.confineSpecUpdateRollout=true to achieve the desired behaviour. Please consult this document to get all information about it. (#2233, @rfranzke)
• [USER] Shoot clusters can now receive forceful minor upgrades when using an expired Kubernetes version. Versions with preview classification are excluded from auto-update functionality for Kubernetes and machine image versions. (#2108, @danielfoehrKn)
• [OPERATOR] Gardener's image vector does no longer contain etcd and etcd-backup-restore. If you want to overwrite these versions you must do it via the etcd-druid. Please consult this document for more information. (#2262, @rfranzke)
• [OPERATOR] The Gardener API server now blocks the removal of Kubernetes and machine image versions from the CloudProfile which are still in use by shoot clusters. (#2106, @danielfoehrKn)
• [DEVELOPER] Gardener will now respect the error codes presented in the extension CRDs' .status.lastError.codes[] field instead of recomputing them. (#2248, @rfranzke)

Improvements

• [USER] A dashboard for CoreDNS is now available in grafana. (#2291, @wyb1)
• [USER] The infrastructure reconciliation for hibernated shoots is now skipped. (#2258, @rfranzke)
• [USER] The shoot health check controller has been improved to produce error codes (if applicable) to the .status.conditions[].codes that help categorizing observed problems. (#2242, @rfranzke)
• [USER] The version of the Kubernetes Dashboard addon has been bumped to v2.0.0. (#2221, @rfranzke)
• [USER] An issue that caused the Shoot's .status.lastOperation.state to be set to Error although no actual reconciliation operation is executed for the Shoot has been fixed. (#2217, @rfranzke)
• [USER] Shoot conditions may now also specify a list of error codes, similar to .status.lastError.codes. (#2212, @rfranzke)
• [USER] Gardener does not block hibernation of a Shoot Cluster anymore, in case it contains Endpoints objects that are reconciled by a custom operator (e.g. knative). (#2205, @tim-ebert)
• [USER] The CPU and memory limits for the metrics-server have been slightly increased to support large clusters. (#2198, @rfranzke)
• [OPERATOR] The validating webhook for the Gardener Seed Admission controller is now exposed at port 443, allowing it to function properly in GKE clusters. (#2300, @rfranzke)
• [OPERATOR] An issue preventing gardenlet to properly reflect the Shoot condition as false for newly created Shoots is now fixed. (#2299, @ialidzhikov)
• [OPERATOR] The CPU limits for the coredns and blackbox-exporter deployments in the shoot have been slightly increased to prevent false negative API server availability reports. (#2286, @rfranzke)
• [OPERATOR] Gardener now validates that the Pod(/Service) network of a shoot cluster does not intersect with the Service(/Pod) network of the assigned seed. (#2282, @timuthy)
• [OPERATOR] blackbox exporter sidecar is removed from the kube apiserver (#2267, @wyb1)
• [OPERATOR] For backwards-compatibility reasons, the gardenlet does not check for stale extension reports per default. To enable, the field controllers.shootCare.staleExtensionHealthCheckThreshold in the Gardenlet configuration file (https://github.com/gardener/gardener/blob/master/example/20-componentconfig-gardenlet.yaml)can be set. (#2266, @danielfoehrKn)
• [OPERATOR] The ShootState synchronization controller does now properly respect ContainerRuntime resources. (#2259, @rfranzke)
• [OPERATOR] Fixed a bug in the healthcheck library that prevents checks after a Shoot has been woken up from hibernation. Gardener extensions require a minor change during the healthcheck registration. (#2249, @danielfoehrKn)
• [OPERATOR] Fix bug that prevented the Shoot reconciliation to wait for the deletion of Extension CRDs. (#2240, @danielfoehrKn)
• [OPERATOR] Issues for Shoot with metadata.generateName are fixed. Shoot name length limit restriction is applied on generateName with random suffix length fixed to 5 as in the kubernetes. Default DNS name is generated using same name generator for shoot. ⚠️ But it will differ from generated shoot name. (#2236, @swapnilgm)
• [OPERATOR] Fixes a bug that prevented proper labeling of worker pool-nodes that have CRI enabled. (#2231, @danielfoehrKn)
• [OPERATOR] A race condition that led to incomplete maintenance operations for shoot clusters has been fixed. (#2229, @rfranzke)
• [OPERATOR] The gardenlet detects outdated health check reports on extension CRDs with a default threshold of 5 minutes in case Gardener extensions stop performing health checks. The threshold can be configured in the Gardenlet configuration. (#2215, @danielfoehrKn)
• [OPERATOR] Remove unused RetrySyncPeriod field (controllers.shoot.retrySyncPeriod) from the Gardenlet configuration. (#2215, @danielfoehrKn)
• [OPERATOR] The kubelet-monitor script running on every worker node is now fixed and properly monitors the kubelet again. (#2214, @rfranzke)
• [OPERATOR] The podAntiAffinity of the fluentd statefulset deployed in the seed clusters is now a soft requirement. (#2213, @Kristian-ZH)
• [OPERATOR] Fix a bug that limits the workers count of a single shoot. (#2210, @aylei)
• [OPERATOR] Edited VPA specific RBAC roles to include get,list and watch for etcd resources by VPA actors (#2204, @georgekuruvillak)
• [OPERATOR] Grafana dashboards for the seed are updated. Removed the cluster overview dashboard since metrics used in this dashboard were removed. Other dashboards are changed to no longer show data on a "Pod level" since pod level metrics have a high cardinality and have been mostly removed from the aggregate-prometheus. (#2202, @wyb1)
• [OPERATOR] The terminationGracePeriodSeconds setting for the Prometheus instance in shoot control planes has been lowered from 300 to 60. (#2199, @wyb1)
• [OPERATOR] Added a test to validate if systemctl on the node's operating system runs without errors. (#2192, @schrodit)
• [OPERATOR] ETCD encryption data is persisted in the ShootState (#2084, @plkokanov)
• [DEVELOPER] The ShootNotFailed predicate in the extensions library does now work as expected. (#2265, @rfranzke)
• [DEVELOPER] ControlPlane, BackupEntry and OperatingSystemConfig controllers support operations for migrate and restore. (#2247, @swilen-iwanow)
• [DEVELOPER] The extensionsv1alpha1.Last{Operation,Error} interfaces were removed - the respective GetLast{Operation,Error}() functions do now return the objects directly instead of the old interfaces. (#2244, @rfranzke)
• [DEVELOPER] The Golang version has been upgraded to 1.14.2. (#2234, @rfranzke)
• [DEVELOPER] A bug in the CSI migration controller has been fixed that may cause the CSIMigration<Provider>Complete feature gate to be set to early. (#2223, @rfranzke)
• [DEVELOPER] The extension library can now be used to create simple validating or mutating webhooks for different K8s types with different handlers. (#2219, @timuthy)
• [DEVELOPER] Extension resource conditions may now also specify a list of error codes, similar to .status.lastError.codes. Gardener will pick them up and merge them into the shoot conditions. (#2212, @rfranzke)
• [DEVELOPER] The containerd test is now skipped for worker pools that are not using the ubuntu operating system. (#2201, @ialidzhikov)
• [DEVELOPER] Extend the infrastructure actuator interface with Migrate and Restore (#2167, @vlvasilev)

[autoscaler]

Improvements

• [USER] Add configurable delay for pod age before considering for scale-up (gardener/autoscaler#33, @hardikdr)
• [OPERATOR] Avoid checking controller references while draining the node. (gardener/autoscaler#44, @hardikdr)
• [OPERATOR] Update azure instance types in cluster-autoscaler. (gardener/autoscaler#41, @hardikdr)
• [OPERATOR] Updated the AWS-instance types used cluster autoscaler. (gardener/autoscaler#40, @hardikdr)
• [OPERATOR] Cluster Autoscaler ignores the worker pools with maxSize set to 0. (gardener/autoscaler#39, @hardikdr)
• [OPERATOR] Autoscaler uses zone-information while scaling-up the node-group from zero (gardener/autoscaler#38, @hardikdr)

[etcd-backup-restore]

Most notable changes

• [USER] HTTP API for triggering out-of-schedule full and delta snapshots now returns snapshot metadata in response body in JSON format. (gardener/etcd-backup-restore#214, @shreyas-s-rao)

Improvements

• [USER] Added new HTTP API for fetching details of latest full and delta snapshots, in JSON format. (gardener/etcd-backup-restore#214, @shreyas-s-rao)
• [USER] Add metrics etcdbr_snapstore_latest_deltas_total and etcdbr_snapstore_latest_deltas_revisions_total to provide information about the delta snapshots since the latest full snapshot in the snapstore. (gardener/etcd-backup-restore#211, @shreyas-s-rao)
• [OPERATOR] Skip the first full snapshot on start, if initial delta snapshot is taken and last full snapshot is recent enough than 24hr. (gardener/etcd-backup-restore#222, @swapnilgm)
• [OPERATOR] Add documentation to force restore etcd data. (gardener/etcd-backup-restore#217, @shreyas-s-rao)
• [OPERATOR] The release tags from now are prefixed with v. (gardener/etcd-backup-restore#210, @ialidzhikov)
• [OPERATOR] Configuring backup-restore server using config file is now supported. (gardener/etcd-backup-restore#208, @swapnilgm)
• [DEVELOPER] Fix integration test setup script. (gardener/etcd-backup-restore#215, @shreyas-s-rao)

[etcd-druid]

Improvements

• [OPERATOR] Removed owner reference from etcd StatefulSet so that HVPA can recommend resource recommendations. VPA does not support StatefulSet having ownerReferences set to another top-level controller . (gardener/etcd-druid#48, @georgekuruvillak)
• [OPERATOR] Updated the etcd resource manifest with priorityClassName to specify the priority of etcd pods. (gardener/etcd-druid#36, @georgekuruvillak)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.4.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.4.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.4.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.4.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.4.0

[gardener]

Improvements

• [USER] Gardener does not block hibernation of a Shoot Cluster anymore, in case it contains Endpoints objects that are reconciled by a custom operator (e.g. knative). (399066787819d4b72052aeb55e3296591c134a74)
• [USER] The CPU and memory limits for the metrics-server have been slightly increased to support large clusters. (f5a718e25aabc78d5602414d20e290fdc664cf86)
• [OPERATOR] A race condition that led to incomplete maintenance operations for shoot clusters has been fixed. (915643d191fa273d55bf5563d668240c5f326f66)
• [OPERATOR] The kubelet-monitor script running on every worker node is now fixed and properly monitors the kubelet again. (652da1151a2c8204a1000d215d4d6e59c47eabe6)
• [OPERATOR] Edited VPA specific RBAC roles to include get,list and watch for etcd resources by VPA actors (6e5cce60e87b81d615e6cad293cc419faea8eeaf)
• [OPERATOR] The terminationGracePeriodSeconds setting for the Prometheus instance in shoot control planes has been lowered from 300 to 60. (8782a64e664c40aed98549509a18a2ecf35debd3)
• [DEVELOPER] A bug in the CSI migration controller has been fixed that may cause the CSIMigration<Provider>Complete feature gate to be set to early. (6c9e30471ac3c8d21f3e53a8ae25b4ad545d33a2)
• [DEVELOPER] The containerd test is now skipped for worker pools that are not using the ubuntu operating system. (4eacfa14c6f821fb4ab71ef3d563358bd5afe8df)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.3.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.3.1
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.3.1
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.3.1
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.3.1