Kubernetes-native system managing the full lifecycle of conformant Kubernetes clusters as a service on Alicloud, AWS, Azure, GCP, OpenStack, vSphere, KubeVirt, Hetzner, EquinixMetal, MetalStack, and OnMetal with minimal TCO.
OTHER License
max-parallel-chunk-uploads
. Default is set to 5. (gardener/etcd-backup-restore#68, @swapnilgm)debug/pprof/*
endpoint to dynamically profile cpu, heap consumption. To enable profiling one has to explicitly set enable-profiling
on server
sub-command. (gardener/etcd-backup-restore#60, @swapnilgm)github.com/Azure/azure-storage-blob-go
as azure storage SDK, (gardener/etcd-backup-restore#76, @swapnilgm).part
i.e.<path-to-etcd-data-dir>.part
. On successful restoration we will replace actual etcd data directory with this. This brings standard and more cleaner approach to restoration. (gardener/etcd-backup-restore#58, @georgekuruvillak)defragmentation-period-in-hours
flag on etcdbrctl server
command. (gardener/etcd-backup-restore#55, @swapnilgm)CloudProfile
resources. (281ed3cce7efebfbca8bf708f60abd5ec3a40548)k8s.gcr.io/kube-addon-manager
: v8.7
-> v8.8
quay.io/prometheus/alertmanager
: v0.15.2
-> v0.15.3
quay.io/prometheus/node-exporter
: v0.16.0
-> v0.17.0
grafana/grafana
: 5.3.0
-> 5.4.2
quay.io/prometheus/blackbox-exporter
: v0.12.0
-> v0.13.0
quay.io/calico/node
: v3.2.3
-> v3.4.0
quay.io/calico/cni
: v3.2.3
-> v3.4.0
quay.io/calico/typha
: v3.2.3
-> v3.4.0
coredns/coredns
: 1.2.2
-> 1.3.0
quay.io/kubernetes-ingress-controller/nginx-ingress-controller
: 0.20.0
-> 0.21.0
1.10.1
. (b4eb1295468dbb47342f4484fdef4f098a1aa05b)v1.11.4
. (#608, @ialidzhikov)initialDelaySeconds
of fluentd's livenessProbe to 4m
.garden
namespace).6.2.4
to 6.5.4
.custom.shoot.sapcloud.io/
. The namespace is also annotated with shoot.garden.sapcloud.io/uid=<value>
. This can be used by controllers running in the seed to extract more information about the shoot. (#600, @vpnachev)Ingresses
and Secrets
. (#599, @timuthy)BackupInfrastructures
resources if the current status is Reconcile (Succeeded)
and if the last reconciliation happened more than the configured syncPeriod
ago. (#594, @timuthy)
BackupInfrastructure
resource with backupinfrastructure.garden.sapcloud.io/operation=reconcile
./version
endpoint. (d74c63a5ff4480a1e565853877037ec9bd55a7cb)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.15.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.15.0
1.10.1
. (f7a284c1530aae78a09e0f997bac3fde0c1c59f3)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.14.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.14.3
/version
endpoint. (9c95d7d9c7381acfb4f50c375223311ab868441c)BackupInfrastructures
resources if the current status is Reconcile (Succeeded)
and if the last reconciliation happened more than the configured syncPeriod
ago. (bb6ccb141fc8b05dc09688500ac1e1af89cee7b3)
BackupInfrastructure
resource with backupinfrastructure.garden.sapcloud.io/operation=reconcile
.gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.14.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.14.2
Ingresses
and Secrets
. (#601, @timuthy)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.14.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.14.1
.subsets
field of the Endpoint
object is empty has been fixed. (gardener/aws-lb-readvertiser#11, @zanetworker)runtime
and application
). This change required that all chart values that previously were on top-level have now moved under the .global
key. However, the structure/types of all existing keys has not changed. Please check the values.yaml
. (#589, @rfranzke)Seed
and Shoot
CIDR verification rules are now enabled. (#567, @mvladev)
Shoot
or Seed
resource created previously with invalid configurations MUST be deleted, before updating to this version. Failing to do so, will cause ANY modification to said resources (including attempted deletion) to be rejected by the apiserver.Logging
feature gate previously then please delete the persistent volumes related to the central ElasticSearch cluster before before upgrading Gardener: kubectl -n garden delete sts elasticsearch-logging
(#530, @vlvasilev)cluster-admin
privileges, i.e., users need now to login with the basic authentication credentials before the dashboard can be used. These credentials can be found in the provided shoot kubeconfig. By default (i.e., without being logged in), the dashboard only has minimal privileges to run properly. (#555, @rfranzke)Progressing
(beside True
, False
, Unknown
). If a health check fails then it first goes to Progressing
state. Only if this state remains for a configured threshold time it switches to False
. This is to compensate expected and normal oscillations in all the involved dynamic systems. (#552, @afritzler).spec.cloud.secretBindingRef
) is now immutable. Rotating the cloud provider credentials is still possible by modifying the referenced secret's data. (#539, @adracus)shoot.garden.sapcloud.io/operation=maintain
. (#528, @timuthy)Ingress
es specifying having a default domains and those labelled with garden.sapcloud.io/purpose=managed-cert
. (#509, @timuthy).spec.hibernation.schedules
field. Multiple schedules can be defined, allowing the shoot to be hibernated and woken up multiple times across a day. The schedules need to be described as cron schedule expressions (example). (#507, @adracus)type=APIServerAvailable
). We plan to add another new condition showing pending/firing alerts will be added as well in the future. (#497, @dkistner)garden.sapcloud.io/[email protected]
. (#527, @mvladev)shoot.garden.sapcloud.io/operation=reconcile
. (#524, @rfranzke)ClusterRole
garden.sapcloud.io:system:administrators
containing all privileges required by a garden administrator. Operators are asked to bind this ClusterRole
themselves to users that should have these permissions. (#518, @rfranzke)Ingress
resources requesting a certificate for Gardener's default domains. This can be enabled by enabling the CertificateManagement
feature gate in the ControllerManagementConfiguration
. Please make sure that the required configuration is placed in the garden
namespace of the garden cluster (please find here an example configuration). (#509, @timuthy)shoot.garden.sapcloud.io/status
label to shoots to indicate the health status of the shoot and to allow filtering. There are three possible values: healthy
, progressing
, unhealthy
. See also: https://github.com/gardener/gardener/pull/552 (#564, @adracus)
shoot.garden.sapcloud.io/unhealthy
is deprecated and will be removed in the future.denial
(either NXDOMAIN or NODATA (name exists, type does not)) or error
(SERVFAIL, NOTIMP, REFUSED, etc. - anything that indicates the remote server is not willing to resolve the request)` messages. (#561, @vlvasilev)ClusterRoleBinding
that allows every node to read every secret, pod, etc. Instead, due to the "node authorizer" plugin, every node is restricted to only read those secrets/pods/... that are assigned or related to its node. (#549, @rfranzke)80m
CPU on every shoot worker node for itself. (#548, @rfranzke)start
or end
field in a shoot's hibernation schedule can now be omitted (although at least one of both needs to be specified). This allows to have schedules that only hibernate or only wake up a shoot. (#543, @adracus)DELETECOLLECTION
requests for shoots and projects. This fixes issues with deleting namespaces in clusters which had registered the Garden APIService
. (#537, @rfranzke)Node
object could not be found. (cc568fb49f0cc0b8ea60ee95f9977356e051fee1)Logging
feature gate has been disabled. (#556, @rfranzke).spec.dns.domain
but set .spec.dns.provider!=unmanaged
. The domain will have the following scheme: <shootName>.<projectName>.<managed-domain>
. This does only work if default domains were registered in the system beforehand. (#547, @timuthy)CloudProfile
's .spec.openstack.requestTimeout
field. (#526, @afritzler)DaemonSet
s and StatefulSet
s has been added. (#523, @dkistner)3.7
to 3.8
. (gardener/ingress-default-backend@0ffdef9c14999028a488b5a708204f1a613a2825)alicloud
Terraform provider plugin has been upgraded from 1.10.0
to 1.22.0
. (gardener/terraformer#15, @jia-jerry)3.7
to 3.8
. (gardener/terraformer@c6fd9284e342fde12ed8a9793d36b05f17f97b7b)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.14.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.14.0
ClusterRoleBinding
that allows every node to read every secret, pod, etc. Instead, due to the "node authorizer" plugin, every node is restricted to only read those secrets/pods/... that are assigned or related to its node. (9a51d5940949836cad37248cd864bd347e27beb8)Logging
feature gate has been disabled. (595a05c51e5801c6fd565b4e6a44ec798ce9694f)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.5
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.5
shoot.garden.sapcloud.io/operation=maintain
. (56019d79f7a3af92bdafb4777bf32fbc12a1409c)shoot.garden.sapcloud.io/operation=reconcile
. (a9dc08cfbb80f9e014e39d577d21049dca62b6cc)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.4
DELETECOLLECTION
requests for shoots and projects. This fixes issues with deleting namespaces in clusters which had registered the Garden APIService
. (9a9b3448bd50aac69d47359a380ed5996346ef2f)Node
object could not be found. (8b7917485300b5ba0bb3d3166ea983f005f8babb)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.3
.subsets
field of the Endpoint
object is empty has been fixed. (gardener/aws-lb-readvertiser#11, @zanetworker)ClusterRole
garden.sapcloud.io:system:administrators
containing all privileges required by a garden administrator. Operators are asked to bind this ClusterRole
themselves to users that should have these permissions. (34be67c917d53da14a83991ed85f3950444cb4e2)3.7
to 3.8
. (gardener/ingress-default-backend@0ffdef9c14999028a488b5a708204f1a613a2825)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.2
gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.1
default/kubernetes
endpoint and keeps it up-to-date. This requires users to deploy their kube-apiserver with --endpoint-reconciler-type=none
. Moreover, the AWS LB readvertiser needs a kubeconfig having privileges to talk to the kube-apiserver. Please consider the latest example manifest for instructions how to deploy the AWS LB readvertiser. (gardener/aws-lb-readvertiser#10, @zanetworker)autoScalerMin=autoScalerMax=0
for all worker pools). The only way to hibernate a shoot cluster is setting .spec.hibernation.enabled=true
as of now. Please adapt your scripts and change all existing shoot cluster specification accordingly before upgrading Gardener. (#471, @rfranzke)Project
resource out of properly labelled Namespace
s has been removed. Upgrading from a Gardener version prior 0.11.0 is not supported. Also, this Gardener version is not compatible with Gardener Dashboard versions prior 1.23.0. (#480, @rfranzke)downscaleDelay
to 24h
and tolerance
to 0.2
for seeds. These flags must now be specified in the Shoot
specifications of the individual seeds. (#399, @amshuman-kr)kubernetes-dashboard
and nginx-ingress
are now deprecated. The cluster-autoscaler addon is automatically enabled if at least one worker pool specifies max>min
. Please find more individual recommendations/explanations in the example shoot resource. (#471, @rfranzke)Deployment
s, DaemonSet
s, StatefulSet
s) in both the seed and the shoot. (#457, @Adracus)quay.io/coreos/etcd
: v3.3.9
-> v3.3.10
quay.io/prometheus/alertmanager
: v0.15.1
-> v0.15.2
quay.io/prometheus/prometheus
: v2.3.2
-> v2.4.3
quay.io/coreos/kube-state-metrics
: v1.3.1
-> v1.4.0
grafana/grafana
: 5.2.2
-> 5.3.0
quay.io/calico/node
: v3.2.1
-> v3.2.3
quay.io/calico/cni
: v3.2.1
-> v3.2.3
quay.io/calico/typha
: v3.2.1
-> v3.2.3
k8s.gcr.io/kubernetes-dashboard-amd64
: v1.8.3
-> v1.10.0
jtblin/kube2iam
: 0.10.1
-> 0.10.4
quay.io/kubernetes-ingress-controller/nginx-ingress-controller
: 0.17.1
-> 0.20.0
10s
to 5m
to reduce load on the backing blob storages. (#443, @georgekuruvillak)hyperkube
image will be preloaded in advance. Rolling out this change will result in a rolling update of all worker nodes. (#429, @Adracus)TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
. (#428, @rfranzke)spec.kubernetes.allowPrivilegedContainers
to false
in the Shoot
resource: kubectl patch shoot my-shoot -p '{"spec":{"kubernetes":{"allowPrivilegedContainers":false}}}'
(#226, @mvladev)Project
resource has been changed to support management of project members (by setting .spec.members
, please take a look at the example Project
resource). (#480, @rfranzke)Logging
feature gate called (disabled by default). If enabled, Gardener will deploy one logging stack for every seed cluster and one per shoot control plane. The seed cluster's garden
namespace will get a separate ElasticSearch + Kibana instance serving the logs of all the seed's pods running not running in shoot namespaces. The dedicated shoot logging stack covers the logs of the shoot's control plane. Application logs are of no concern, the logging stack is meant to be used by Gardener operators (similar to the dedicated shoot monitoring stacks). (#176, @vlpanov)Quota
objects can now be fetched by kubectl get squota
to avoid conflicts with Kubernetes' core/v1/ResourceQuota
resources. (#468, @rfranzke)kube-controller-manager
and cloud-controller-manager
exist. (#415, @rfranzke)Shoot
specification. (#399, @amshuman-kr)shoot.garden.sapcloud.io/ignore
annotation needs to be removed first. (e56134333d0f7fb09849b798eabe5b5798a43cfb)usable
field has been added to machineType
s and volumeType
s in CloudProfile
s (defaults to true
). With this flag it is possible to prevent that any shoot uses a non-usable machine or volume type. Existing shoots may continue to use a non-usable type, but no other shoot can be updated or created with it. (#488, @mliepold)12h
(this will help to reduce the amount of alerts being send). (#485, @mvladev)container_memory_working_set_bytes
to for memory calculations/display. (#482, @mvladev)
gardener-external-admission-controller
in the seed. (#464, @timuthy)1Gi
to 1.5Gi
. (#458, @swapnilgm).spec.maintenance.timeWindow
object`) instead of regularly checking whether it can start maintaining or not. (#456, @rfranzke)cluster-autoscaler
, machine-controller-manager
and cloud-controller-manager
have been added. (#432, @mvladev)cluster-autoscaler
, machine-controller-manager
and cloud-controller-manager
have been added. (#430, @mvladev)APIService
s are only available of version v1beta1
. As a consequence, Gardener now uses APIService
s v1beta1
instead of v1
. (#404, @timuthy)clientConnection
configuration of the controller-manager's componentconfig file into account, effectively allowing faster updates to resources in the Garden cluster. (70ff4c73ea2a181a5847bea2b13f3f257ed8fad1)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.13.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.13.0
gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.12.4
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.12.4
1Gi
to 1.5Gi
. (2c5471f945ed9a19eb747df1130b396d324b0ec0)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.12.3
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.12.3
gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.12.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.12.2
gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.12.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.12.1
type
field has been removed from the StorageClass
deployed in OpenStack Shoots. Moreover, the StorageClass
es deployed by Gardener are now reconciled and not only created once anymore. (#395, @afritzler)PersistentVolume
s are now added by Gardener instead of the Kubernetes initializer admission plugin (to temporarily mitigate a runtime panic that would otherwise happen in the Kubernetes cloud-controller-manager (see https://github.com/kubernetes/kubernetes/issues/68996 for more details)). (#392, @rfranzke)CustomResourceDefinition
s and APIService
s if they have not been cleaned up regularly by their responsible controllers within 5 minutes. (#391, @Adracus)kube-dns
with CoreDNS
in all Shoot clusters. kube-dns
is becoming deprecated, and CoreDNS
is the new default DNS plugin for Kubernetes (as it is more pluggable and provides more features). (#364, @zanetworker)24h
. (#397, @amshuman-kr)Shoot
resources is fixed. (8950d68efb5725f6127af875b8050cacad9170f4)5m
-> 15m
and Upscale: 3m
-> 1m
. (#389, @rfranzke)v3.2.1
. (#380, @databus23)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.12.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.12.0
Shoot
resources is fixed. (4cd8b818488bd2377ad1fbb4a3e5541c07aead1e)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.11.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.11.1
Deployment
resource you can influence how the rolling update of your worker pools behaves. Please find more information about these values here. (#351, @zanetworker).spec.hibernation.enabled
field in the Shoot specification for triggering hibernation/wake-up. The current behavior (setting min=max=0
for all worker groups) is still supported but deprecated and will be removed in a future version. (#350, @timuthy)kube-system
namespace of every Shoot cluster. This enables the horizontal pod autoscaling feature of Kubernetes out of the box. Users that have previously deployed the metrics-server on their own do not longer need to care about it (as it's now managed by Gardener). Gardener will overtake the responsibility of maintaining it. (#347, @rfranzke)Project
resource (see this for an example). A project results in a Kubernetes namespace into which Shoot clusters can be deployed. The Gardener controller manager features a webhook endpoint for protecting direct deletion of namespace belonging to a project. For that, it now exposes a HTTPS server which requires a TLS certificate having gardener-controller-manager
, gardener-controller-manager.garden
, gardener-controller-manager.garden.svc
as valid SANs. Please check the Gardener componentconfig example and the Gardener Helm chart values for how to specify the new configuration flags. Please do also make sure that the Garden cluster has enabled the ValidatingAdmissionWebhook
admission plugin. Similar to Shoot
s, deleting a Project
resource requires to annotate it with confirmation.garden.sapcloud.io/deletion=true
in a previous step. (#330, @rfranzke)shoot.garden.sapcloud.io/operation=maintain
(even outside the maintenance time window). (1e5cc01e1e6e2433dea68adab556a82d82bead06)ec2:Describe*
to ec2:DescribeInstances
. See #178 for more details. (f1bfc09b7eff5a1a0b54a977d6fe0290b4d294d6)minReplicas=maxReplicas=1
for the kube-apiservers of shooted Seed clusters has been fixed. (317b4e2)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.11.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.11.0
⚠️ Gardener was depending on a yaml2json
binary which was downloaded on every worker machine in order to construct the kubelet's kubeconfig and to apply the machine's cloud-config. We are now no longer depending on yaml2json
(instead, we are now working inside Docker containers to not rely on external binaries but rather on stable Docker images). Gardener MUST be updated as the yaml2json
binary is no longer available at its download URL which results in the fact that no new machines can join the cluster (as the cloud-config cannot be read/extracted). This will result in a rolling update of all machines for all clusters.
gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.10.2
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.10.2
minReplicas=maxReplicas=1
for the kube-apiservers of shooted Seed clusters has been fixed. (aa3a93a9)gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:0.10.1
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:0.10.1