typhoon

• Kubernetes v1.18.3
• Use Kubelet TLS bootstrap with bootstrap token authentication (#713)
  • Enable Node Authorization and NodeRestriction to reduce authorization scope
  • Renew Kubelet certificates every 72 hours
• Update etcd from v3.4.7 to v3.4.9
• Update Calico from v3.13.1 to v3.14.0
• Add CoreDNS node affinity preference for controller nodes (#188)
• Deprecate CoreOS Container Linux support (no OS updates after May 2020)
  • Use a fedora-coreos module for Fedora CoreOS
  • Use a container-linux module for Flatcar Linux

AWS

• Fix Terraform plan error when controller_count exceeds AWS zones (e.g. 5 controllers) (#714)
  • Regressed in v1.17.1 (#605)

Azure

• Update Azure subnets to set address_prefixes list (#730)
  • Fix warning that address_prefix is deprecated
  • Require terraform-provider-azurerm v2.8.0+ (action required)

DigitalOcean

• Promote DigitalOcean to beta on both Fedora CoreOS and Flatcar Linux

Fedora CoreOS

• Fix Calico install-cni crashloop on Pod restarts (#724)
  • SELinux enforcement requires consistent file context MCS level
  • Restarting a node resolved the issue as a previous workaround

AWS

• Support Fedora CoreOS image streams (#727)
  • Add os_stream variable to set the stream to stable (default), testing, or next
  • Remove unused os_image variable

Google

• Support Fedora CoreOS image streams (#723)
  • Add os_stream variable to set the stream to stable (default), testing, or next
  • Deprecate os_image variable. Manual image uploads are no longer needed

Flatcar Linux

Azure

• Use the Flatcar Linux Azure Marketplace image
  • Restore #664 (reverted in #707) but use Flatcar Linux new free offer (not byol)
• Change os_image to use a flatcar-stable default

Google

• Promote Flatcar Linux to beta

Addons

• Update nginx-ingress from v0.30.0 to v0.32.0
  • Add support for IngressClass
• Update Prometheus from v2.17.1 to v2.18.1
  • Update kube-state-metrics from v1.9.5 to v1.9.6
  • Update node-exporter from v1.0.0-rc.0 to v1.0.0-rc.1
• Update Grafana from v6.7.2 to v7.0.0

• Kubernetes v1.18.2
• Choose Fedora CoreOS or Flatcar Linux (action required)
  • Use a fedora-coreos module for Fedora CoreOS
  • Use a container-linux module for Flatcar Linux
• Change Container Linux modules' defaults from CoreOS Container Linux to Flatcar Container Linux (#702)
  • CoreOS Container Linux won't receive updates after May 2020

Fedora CoreOS

• Fix bootstrap race condition from SELinux unshared content label (#708)

Azure

• Add support for Fedora CoreOS (#704)

DigitalOcean

• Fix race condition creating firewall allow rules (#709)

Flatcar Linux

AWS

• Change os_image default from coreos-stable to flatcar-stable (#702)

Azure

• Change os_image to be required. Recommend uploading a Flatcar Linux image (action required) (#702)
• Disable Flatcar Linux Azure Marketplace image support (breaking, #707)
  • Revert to manual uploading until marketplace issue is closed (#703)

Bare-Metal

• Recommend changing os_channel from coreos-stable to flatcar-stable

Google

• Change os_image to be required. Recommend uploading a Flatcar Linux image (action required) (#702)

DigitalOcean

• Change os_image to be required. Recommend uploading a Flatcar Linux image (action required) (#702)
• Fix race condition creating firewall allow rules (#709)

• Kubernetes v1.18.1
• Choose Fedora CoreOS or Flatcar Linux (action recommended)
  • Use a fedora-coreos module for Fedora CoreOS
  • Use a container-linux module with OS set to Flatcar Linux
• Update etcd from v3.4.5 to v3.4.7
• Change kube-proxy and calico or flannel to tolerate specific taints (#682)
  • Tolerate master and not-ready taints, rather than tolerating all taints
• Update flannel from v0.11.0 to v0.12.0 (#690)
• Fix bootstrap when networking mode flannel (non-default) is chosen (#689)
  • Regressed in v1.18.0 changes for Calico (#675)
• Rename Container Linux controller_clc_snippets to controller_snippets for consistency (#688)
• Rename Container Linux worker_clc_snippets to worker_snippets for consistency
• Rename Container Linux clc_snippets (bare-metal) to snippets for consistency
• Drop support for gitRepo volumes

Azure

• Fix Azure worker UDP outbound connections (#691)
  • Fix Azure worker clock sync timeouts

DigitalOcean

• Add support for Fedora CoreOS (#699)

Addons

• Refresh Prometheus rules/alerts and Grafana dashboards (#692)
• Update Grafana from v6.7.1 to v6.7.2

• Kubernetes v1.18.0
• Update etcd from v3.4.4 to v3.4.5
• Switch from upstream hyperkube image to individual images (#669)
  • Use upstream k8s.gcr.io kube-apiserver, kube-controller-manager, kube-scheduler, and kube-proxy container images
  • Use poseidon/kubelet to package the upstream Kubelet binary and dependencies as a container image (checksummed, automated build)
  • Add quay.io/poseidon/kubelet as a Typhoon distributed artifact in the security policy
  • Update base images from debian 9 to debian 10
  • Background: Kubernetes will stop releasing the hyperkube container image and provide the Kubelet as a binary for packaging
• Choose Fedora CoreOS or Flatcar Linux (action recommended)
  • Use a fedora-coreos module for Fedora CoreOS
  • Use a container-linux module with OS set for Flatcar Linux (varies, see docs)
  • CoreOS Container Linux won't receive updates after May 2020
• Add support for Fedora CoreOS snippets (terraform-provider-ct v0.5+) (#686)
• Recommend updating terraform-provider-ct plugin from v0.4.0 to v0.5.0
• Set Fedora CoreOS log driver back to the default journald (#681)
• Deprecate asset_dir variable and remove docs (#678)
• Deprecate support for gitRepo volumes. A future release will drop support.

AWS

• Fix Fedora CoreOS AMI to filter for stable images (#685)
  • Latest Fedora CoreOS testing or bodhi-update images could be chosen depending on the region

Bare-Metal

• Update default os_stream from testing to stable

Google Cloud

• Known: Use of stale Fedora CoreOS image may require terraform re-apply during bootstrap (#687)

DigitalOcean

• Rename image variable to os_image for consistency (#677) (action required)

Addons

• Update Prometheus from v2.16.0 to v2.17.1
• Update Grafana from v6.6.2 to v6.7.1

• Kubernetes v1.17.4
• Update etcd from v3.4.3 to v3.4.4
  • On Container Linux, fetch using the docker transport format (#659)
• Update CoreDNS from v1.6.6 to v1.6.7 (#648)
• Update Calico from v3.12.0 to v3.13.1

AWS

• Promote Fedora CoreOS to stable (#668)
• Allow VPC route table extension via reference (#654)
• Fix worker_node_labels on Fedora CoreOS (#651)
• Fix automatic worker node delete on shutdown on Fedora CoreOS (#657)

Azure

• Upgrade to terraform-provider-azurerm v2.0+ (action required)
  • Change worker_priority from Low to Spot if used (action required)
  • Switch to Azure's new Linux VM and Linux VM Scale Set resources
  • Set controller's Azure disk caching to None
  • Associate subnets (in addition to NICs) with security groups (aesthetic)
• Add support for Flatcar Container Linux (#664)
  • Requires accepting Flatcar Linux Azure Marketplace terms

Bare-Metal

• Add worker_node_labels map variable for per-worker node labels (#663)
• Add worker_node_taints map variable for per-worker node taints (#663)

DigitalOcean

• Add support for Flatcar Container Linux (#644)

Google Cloud

• Promote Fedora CoreOS to beta (#668)
• Fix worker_node_labels on Fedora CoreOS (#651)
• Fix automatic worker node delete on shutdown on Fedora CoreOS (#657)

Addons

• Update nginx-ingress from v0.28.0 to v0.30.0
• Update Prometheus from v2.15.2 to v2.16.0
  • Refresh Prometheus rules and alerts
  • Add a BlackboxProbeFailure alert
  • Update kube-state-metrics from v1.9.4 to v1.9.5
  • Update node-exporter from v0.18.1 to v1.0.0-rc.0
• Update Grafana from v6.6.1 to v6.6.2
  • Refresh Grafana dashboards
• Remove Container Linux Update Operator (CLUO) addon example (#667)
  • CLUO hasn't been in active use in our clusters and won't be relevant
    beyond Container Linux. Requires patches for use on Kubernetes v1.16+

• Kubernetes v1.17.3
• Update Calico from v3.11.2 to v3.12.0
• Allow Fedora CoreOS clusters to pass CNCF conformance suite
  • Set Docker log driver to json-file as a workaround
• Try a Fedora CoreOS or Flatcar Linux cluster, alongside your stable CoreOS Container Linux clusters (recommended)

AWS

• Promote Fedora CoreOS to beta

Bare-Metal

• Promote Fedora CoreOS to beta
• Add Fedora CoreOS kernel arguments initrd and console (#640)

Google Cloud

• Add initial Terraform module for Fedora CoreOS (#632)
• Add initial support for Flatcar Container Linux (#639)

Addons

• Update nginx-ingress from v0.27.1 to v0.28.0
• Update kube-state-metrics from v1.9.3 to v1.9.4
• Update Grafana from v6.5.3 to v6.6.1

• Kubernetes v1.17.2

AWS

• Promote Fedora CoreOS from preview to alpha

Bare-Metal

• Promote Fedora CoreOS from preview to alpha
• Update Fedora CoreOS images location
  • Use Fedora CoreOS production download streams
  • Use live PXE kernel and initramfs images

Addons

• Update nginx-ingress from v0.26.1 to v0.27.1 (#625)
  • Change runAsUser from 33 to 101 for alpine-based image
• Update kube-state-metrics from v1.9.2 to v1.9.3

• Kubernetes v1.17.1
• Update CoreDNS from v1.6.5 to v1.6.6 (#602)
• Update Calico from v3.10.2 to v3.11.2 (#604)
• Inline Kubelet service on Container Linux nodes (#606)
• Disable unused Kubelet 127.0.0.1:10248 healthz listener (#607)
• Enable kube-proxy metrics and allow Prometheus scrapes
  • Allow TCP/10249 traffic with worker node sources

AWS

• Update Fedora CoreOS AMI filter for fedora-coreos-31 (#620)

Google

• Allow terraform-provider-google v3.0+ (#617)
  • Only enforce v2.19+ to ease migration, as no v3.x features are used

Addons

• Update Prometheus from v2.14.0 to v2.15.2
  • Add discovery for kube-proxy service endpoints
• Update kube-state-metrics from v1.8.0 to v1.9.2
• Reduce node-exporter DaemonSet tolerations (#614)
• Update Grafana from v6.5.1 to v6.5.3

• Kubernetes v1.17.0
• Manage clusters without using a local asset_dir (#595)
  • Change asset_dir to be optional. Remove the variable to skip writing assets locally (action recommended)
  • Allow keeping cluster assets only in Terraform state (pluggable, encryption) and allow terraform apply from stateless automation systems
  • Improve asset unpacking on controllers
  • Obtain kubeconfig from Terraform module outputs
• Replace usage of template_dir with templatefile function (#587)
  • Require Terraform version v0.12.6+ (action required)
• Update CoreDNS from v1.6.2 to v1.6.5 (#588)
  • Add health lameduck option to wait before shutdown
• Update Calico from v3.10.1 to v3.10.2 (#599)
• Reduce pod eviction timeout for deleting pods on unready nodes from 5m to 1m (#597)
  • Present since v1.13.3, but mistakenly removed in v1.16.0
• Add CPU requests for control plane static pods (#589)
  • May provide slight edge case benefits and aligns with upstream

Google

• Use new google_compute_region_instance_group_manager version block format
  • Fixes warning that instance_template is deprecated
  • Require terraform-provider-google v2.19.0+ (action required)

Addons

• Update Grafana from v6.4.4 to v6.5.1
• Add pod networking details in dashboards (#593)
• Add node alerts and Grafana dashboard from node-exporter (#591)
• Reduce Prometheus high cardinality time series (#596)

• Kubernetes v1.16.3
• Update etcd from v3.4.2 to v3.4.3 (#582)
• Upgrade Calico from v3.9.2 to v3.10.1
  • Allow advertising service ClusterIPs to peer routers via a BGPConfiguration
• Switch kube-proxy from iptables to ipvs mode (#574)

Addons

• Update Prometheus from v2.13.0 to v2.14.0
  • Refresh rules, alerts, and dashboards from upstreams
• Remove addon-resizer from kube-state-metrics (#575)
• Update Grafana from v6.4.2 to v6.4.4

• Kubernetes v1.16.2
• Update etcd from v3.4.1 to v3.4.2 (#570)
• Update Calico from v3.9.1 to v3.9.2
  • Default to using Calico and supporting NetworkPolicy on all platforms

Azure

• Change default networking provider from "flannel" to "calico" (#573)

Bare-Metal

• Add controllers and workers as typed lists of machine detail objects (#566)
  • Define clusters' machines cleanly and with Terraform v0.12 type constraints (action required, see PR example)
  • Remove controller_names, controller_macs, and controller_domains variables
  • Remove worker_names, worker_macs, and worker_domains variables

DigitalOcean

• Change default networking provider from "flannel" to "calico" (#573)

Addons

• Update Grafana from v6.4.1 to v6.4.2
• Change CLUO label from "app" to "name"

• Kubernetes v1.16.1
• Update etcd from v3.4.0 to v3.4.1
• Update Calico from v3.8.2 to v3.9.1
• Add Terraform v0.12 variables types (#553, #557, #560, #556, #562)
  • Deprecate cluster_domain_suffix variable

AWS

• Add worker_node_labels variable to set initial worker node labels (#550)
• Add node_labels variable to internal workers pool module (#550)
• For Fedora CoreOS, detect most recent AMI in the region

Azure

• Promote networking provider Calico VXLAN out of experimental (set networking = "calico")
• Add worker_node_labels variable to set initial worker node labels (#550)
• Add node_labels variable to internal workers pool module (#550)
• Change workers module default vm_type to Standard_DS1_v2 (followup to #539)

Bare-Metal

• For Fedora CoreOS, use new kernel, initrd, and raw paths (#563)
• Fix Terraform missing comma error (#549)
• Remove deprecated container_linux_oem variable (#562)

DigitalOcean

• Promote networking provider Calico VXLAN out of experimental (set networking = "calico")
• Fix Terraform missing comma error (#549)

Google Cloud

• Add worker_node_labels variable to set initial worker node labels (#550)
• Add node_labels variable to internal workers module (#550)

Addons

• Update Prometheus from v2.12.0 to v2.13.0
  • Fix Prometheus etcd target discovery and scraping (#561, regressed with Kubernetes v1.16.0)
• Update kube-state-metrics from v1.7.2 to v1.8.0
• Update nginx-ingress from v0.25.1 to v0.26.1 (#555)
  • Add lifecycle hook to allow draining for up to 5 minutes
• Update Grafana from v6.3.5 to v6.4.1

• Kubernetes v1.16.0 (#543)
  • Read about several Kubernetes API deprecations!
  • Remove legacy node role labels (no longer shown in kubectl get nodes)
  • Rename node labels to node.kubernetes.io/master and node.kubernetes.io/node (migratory)
• Migrate control plane from self-hosted to static pods (#536)
  • Run kube-apiserver, kube-scheduler, and kube-controller-manager as static pods on each controller
  • kubectl edits to kube-apiserver, kube-scheduler, and kube-controller-manager are no longer possible (change)
  • Remove bootkube, self-hosted pivot, and pod-checkpointer
• Update CoreDNS from v1.5.0 to v1.6.2 (#535)
• Update etcd from v3.3.15 to v3.4.0
• Recommend updating terraform-provider-ct plugin from v0.3.2 to v0.4.0

Azure

• Change default controller_type to Standard_B2s (#539)
  • B2s is cheaper by $17/month and provides 2 vCPU, 4GB RAM
• Change default worker_type to Standard_DS1_v2 (#539)
  • F1 is previous generation. DS1_v2 is newer, similar cost, and supports Low Priority mode

Addons

• Update Grafana from v6.3.3 to v6.3.5

• Kubernetes v1.15.3
• Update etcd from v3.3.13 to v3.3.15
• Update Calico from v3.8.1 to v3.8.2

AWS

• Enable root block device encryption by default (#527)
  • Require terraform-provider-aws v2.23+ (action required)

Addons

• Update Prometheus from v2.11.0 to v2.12.0
  • Update kube-state-metrics from v1.7.1 to v1.7.2
• Update Grafana from v6.2.5 to v6.3.3
  • Use stable IDs for etcd, CoreDNS, and Nginx Ingress dashboards (#530)
• Update nginx-ingress from v0.25.0 to v0.25.1
  • Fix Nginx security advisories

• Kubernetes v1.15.2
• Update Calico from v3.8.0 to v3.8.1
• Add new load balancing, TCP/UDP, and firewall docs (#523)

Addons

• Add new Grafana dashboards for CoreDNS and Nginx Ingress Controller (#525)

• Kubernetes v1.15.1
• Upgrade Calico from v3.7.3 to v3.8.0
  • Enable CNI bandwidth plugin for traffic shaping
• Run kube-apiserver with lower privilege user (nobody) (#506)
• Relax terraform-provider-ct version constraint (v0.3.2+)
  • Allow provider versions below v1.0.0 (e.g. upgrading to v0.4)

Azure

• Fix to add all controller nodes to the apiserver load balancer backend address pool (#518)
  • kube-apiserver availability relied on the 0th controller

Google Cloud

• Allow controller nodes to span more than 3 zones if available in a region (#504)
• Eliminate extraneous controller instance groups in single-controller clusters (#504)
• Raise network deletion timeout from 4m to 6m (#505)

Addons

• Update Prometheus from v2.10.0 to v2.11.0
  • Refresh rules, alerts, and dashboards from upstreams
  • Update kube-state-metrics from v1.6.0 to v1.7.1
• Update Grafana from v6.2.4 to v6.2.5
• Update nginx-ingress from v0.24.1 to v0.25.0
  • Support networking.k8s.io/v1beta1 apiVersion

Bonus

On AWS and bare-metal, a Fedora CoreOS preview is available to try (announcement).

• Kubernetes v1.15.0
• Migrate from Terraform v0.11 to v0.12.x (action required!)
  • Migration instructions for Terraform v0.12
• Require terraform-provider-ct v0.3.2+ to support Terraform v0.12 (action required)
• Update Calico from v3.7.2 to v3.7.3
• Remove Fedora Atomic modules (deprecated in March) (#501)

AWS

• Require terraform-provider-aws v2.7+ to support Terraform v0.12 (action required)
• Allow using Flatcar Linux Edge by setting os_image to "flatcar-edge"

Azure

• Require terraform-provider-azurerm v1.27+ to support Terraform v0.12 (action required)
• Avoid unneeded rotations of Regular priority virtual machine scale sets
  • Azure only allows eviction_policy to be set for Low priority VMs. Supporting Low priority VMs meant when Regular VMs were used, each terraform apply rolled workers, to set eviction_policy to null.
  • Terraform v0.12 nullable variables fix the issue so plan does not produce a diff.

Bare-Metal

• Require terraform-provider-matchbox v0.3.0+ to support Terraform v0.12 (action required)
• Allow using Flatcar Linux Edge by setting os_channel to "flatcar-edge"

DigitalOcean

• Require terraform-provider-digitalocean v1.3+ to support Terraform v0.12 (action required)
• Change the default worker_type from s-1vcpu1-1gb to s-1vcpu-2gb

Google Cloud

• Require terraform-provider-google v2.5+ to support Terraform v0.12 (action required)

Addons

• Update Grafana from v6.2.1 to v6.2.4
• Update node-exporter from v0.18.0 to v0.18.1

v1.14.3

• Kubernetes v1.14.3
• Update CoreDNS from v1.3.1 to v1.5.0
  • Add ready plugin to improve readinessProbe
• Fix trailing slash in terraform-render-bootkube version (#479)
• Recommend updating terraform-provider-ct plugin from v0.3.1 to v0.3.2 (#487)

AWS

• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Azure

• Replace azurerm_autoscale_setting with azurerm_monitor_autoscale_setting (#482)
  • Require terraform-provider-azurerm v1.22+ (action required)
• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Bare-Metal

• Recommend updating terraform-provider-matchbox plugin from v0.2.3 to v0.3.0 (#487)

Google Cloud

• Rename worker pool module count variable to worker_count (#485) (action maybe)
  • count will become a reserved variable name in Terraform v0.12

Addons

• Update Prometheus from v2.9.2 to v2.10.0
• Update Grafana from v6.1.6 to v6.2.1

• Kubernetes v1.14.2
• Update etcd from v3.3.12 to v3.3.13
• Upgrade Calico from v3.6.1 to v3.7.2
• Change VXLAN port from 8472 (kernel default) to 4789 (IANA)

AWS

• Only set internal VXLAN rules when networking is "flannel" (default: calico)

Azure

• Allow choosing Calico as the network provider (experimental) (#472)
  • Add a networking variable accepting "flannel" (default) or "calico"
  • Use VXLAN encapsulation since Azure doesn't support IPIP

DigitalOcean

• Allow choosing Calico as the network provider (experimental) (#472)
  • Add a networking variable accepting "flannel" (default) or "calico"
  • Use VXLAN encapsulation since DigitalOcean doesn't support IPIP
• Add explicit ordering between firewall rule creation and secure copying Kubelet credentials (#469)
  • Fix race scenario if copies to nodes were before rule creation, blocking cluster creation

Addons

• Update Prometheus from v2.8.1 to v2.9.2
  • Update kube-state-metrics from v1.5.0 to v1.6.0
• Update node-exporter from v0.17.0 to v0.18.0
• Update Grafana from v6.1.3 to v6.1.6
• Reduce nginx-ingress Role RBAC permissions (#458)

• Kubernetes v1.14.1

Addons

• Update Grafana from v6.1.1 to v6.1.3
• Update nginx-ingress from v0.23.0 to v0.24.1