AKS

This release is rolling out to all regions

Service Updates

• Azure CLI 2.0.74 released with key AKS changes
  • https://github.com/Azure/azure-cli/releases/tag/azure-cli-2.0.74
  • Added --load-balancer-sku parameter to aks create command, which allows for
    creating AKS cluster with SLB
  • Added --load-balancer-managed-outbound-ip-count,
    --load-balancer-outbound-ips and --load-balancer-outbound-ip-prefixes
    parameters to aks [create|update] commands, which allow for updating load
    balancer profile of an AKS cluster with SLB
  • Added --vm-set-type parameter to aks create command, which allows to
    specify vm types of an AKS Cluster (vmas or vmss)

Release Notes

• Bug Fixes
  • Fixed an issue where the node pool count rendered in the portal would be incorrect when not using the multiple node pools feature.
  • Fixed an issue to ensure a cluster upgrade will upgrade both the control plane and agent pools for clusters using VMSS, but not multiple agent pools.
  • Resolved an issue with cluster upgrades that could remove existing
    diagnostics settings and data erroneously.
  • Fixed an issue where AKS was not validating user defined taint formats per agent pool resulting in
    failures at cluster creation time.
• Behavioral Changes
  • Increased the reserved CPU cores for kubelets to scale proportionally to cores available on the kubelet's host node. Read more about AKS resource reservation here.
• Preview Features
  • Fixed an issue where AKS was not enforcing the minimum Kubernetes version
    required at additional agent pool creation time when using the multiple node pools feature.
  • Fixed an issue where creating new agent pools will overwrite the route
    table and customers would lose their route table rules. Fixes issue #1212.

This release is rolling out to all regions

Service Updates

• The announced updates to default new clusters to VMSS/SLB configurations is
  under way, if you are using the aks-preview Azure CLI extension,
  all clusters created are now defaulted to VMSS & SLB.
• AKS Kubernetes 1.10 support will be end-of-lifed on Oct 25, 2019
• AKS Kubernetes 1.11 & 1.12 support will be end-of-lifed on Dec 9, 2019
• New Documentation additions:
  • Authenticate with Azure Container Registry from AKS
  • Security hardening in AKS virtual machine hosts
• The AKS team is pleased to announce the new aks-periscope tool.
  • AKS Periscope will allow AKS customers to run initial diagnostics and
    collect logs into an Azure Blob storage account to help them analyze and
    identify potential problems.
  • For more information please see: https://aka.ms/AKSPeriscope

Release Notes

• New Features
  • AKS now GA in the Azure US Gov Virginia region.
    • https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-is-now-available-in-azure-government/
  • Control of egress traffic for cluster nodes in AKS is now GA
    • This feature allows you to restrict outbound network communication for
      you cluster as required for compliance or other secure use-cases.
    • https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic
• Known Issues:
  • Clusters that do not have PSPs enabled upgrading to Kubernetes 1.15 will fail
    • https://github.com/Azure/AKS/issues/1220
• Bug Fixes
  • An issue where excessively logs (eg node/status patch events) were being
    emitted to the audit logs stream and stored. Customer should now see
    greatly reduced audit log volume
• Preview Features
  • The --control-plane-only flag has been added to the aks-preview extension - this command
    will force the upgrade of the customers control plane without simultaneously
    upgrading the other nodepools. This functionality is only supported for
    multi-pool clusters.
    • See: https://github.com/Azure/azure-cli-extensions/blob/master/src/aks-preview/HISTORY.md

Service Updates

• AKS Kubernetes 1.10 support will end-of-lifed on Oct 25, 2019

  • https://azure.microsoft.com/en-us/updates/kubernetes-1-10-x-end-of-life-upgrade-by-oct-25-2019/

• AKS Kubernetes 1.11 & 1.12 support will be end-of-lifed on Dec 9th, 2019

  • Kubernetes 1.15 support on AKS will go GA in Oct 2019
  • Post with additional information will be linked.

• Known issue: Service principal not found: https://github.com/Azure/AKS/issues/1206

• Resolved: AKS workers continually reboot: https://github.com/Azure/AKS/issues/1204

• Kubernetes 1.16 breaking API changes: https://github.com/Azure/AKS/issues/1205

• New Features

  • VMSS backed AKS clusters are now GA
    • VMSS is the underlying compute resource type which enables features such
      as cluster autoscaler, but is a separate feature.
    • See https://docs.microsoft.com/azure/virtual-machine-scale-sets/overview
      and https://docs.microsoft.com/azure/aks/cluster-autoscaler
      for more information.
    • NOTE: Official support in the Azure CLI for AKS+VMSS will be released
      on 2019-09-24 (version 2.0.74 https://github.com/Azure/azure-cli/milestone/73)
  • Standard Load Balancer support (SLB) is now GA
    • See https://docs.microsoft.com/azure/aks/load-balancer-standard
      for documentation.
    • NOTE: Official support in the Azure CLI for AKS+SLB will be released
      on 2019-09-24 (version 2.0.74 https://github.com/Azure/azure-cli/milestone/73)
  • Support for the following VM SKUs is now released: Standard_D48_v3,
    Standard_D48s_v3, Standard_E48_v3, Standard_E48s_v3, Standard_F48s_v2,
    Standard_L48s_v2, Standard_M208ms_v2, Standard_M208s_v2

• Bug Fixes

  • Fixed an issue where cluster upgrade could be blocked when the "managedBy"
    property is missing from the node resource group.
  • Fixed an issue where ingress controller network policy would block all
    egress traffic when assigned to pods (using label selectors).

• Behavioral Changes

  • Review the planned changes for new cluster creation defaults referenced in
    Release 2019-08-26

• Preview Features

  • Fixed an issue where multiple nodepool clusters would use the incorrect
    version(s) and block upgrades.
  • Fixed an issue where AKS would incorrectly allow customers to specify
    different versions for multiple nodepools.
  • Fixed an issue where the incorrect node count would be returned or fail to
    update when using multiple node pools

• Preview Features

  • Kubernetes 1.15 is now in Preview (1.15.3)

• Bug Fixes

  • A bug where kube-svc-redirect would crash due to an invalid bash line has been fixed.
  • A recent Kubernetes dashboard change to enable self-signed certs has been reverted due to browser issues.
  • A bug where the OMSAgent pod would fail scheduling on a user tainted node has been fixed with proper toleration on the OMSAgent pod.
  • A preview bug allowing more than 8 node pools to be created has been fixed to enforce a max of 8 node pools per cluster.
  • A preview bug that would change the primary node pool when adding a new node pool has been fixed.

• Behavioral Changes

  • Review the planned changes for new cluster creation defaults referenced in Release 2019-08-26

• Component Updates

  • aks-engine has been updated to v0.40.0
    • https://github.com/Azure/aks-engine/releases/tag/v0.40.0

This release is rolling out to all regions

• Features
  • Added prometheus annotation to coredns to facilitate metric port discovery
    • For more info please check: https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config#overview-of-configurable-prometheus-scraping-settings
• Bug Fixes
  • Fixed bug with older 1.8 clusters that was preventing clusters from upgrading.
    • Important: this was a best effort fix since these cluster versions are out of support. Please upgrade to a currently supported version
    • For information on how AKS handles Kubernetes version support see:
      Supported Kubernetes versions in Azure
  • Removed the default restricted Pod Security Policy to solve race condition with containers not seeing the user in their config. This policy can be applied by customers.
  • Fixed a bug with kube-proxy, ip-masq-agent and kube-svc-redirect where in certain scenarios they could try to access iptables at the same time.
• Preview Features
  • CLI extension updated for new Standard Load Balancer (SLB) and VM Scale Set (VMSS) Parameters:
    • --vm-set-type Agent pool vm set type. VirtualMachineScaleSets or AvailabilitySet.
    • --load-balancer-sku - Azure Load Balancer SKU selection for your cluster. Basic or Standard.
    • --load-balancer-outbound-ip-prefixes - Comma-separated public IP prefix resource IDs for load balancer outbound connection. Valid for Standard SKU load balancer cluster only.
    • --load-balancer-outbound-ips - Comma-separated public IP resource IDs for load balancer outbound connection. Valid for Standard SKU load balancer cluster only.
    • --load-balancer-managed-outbound-ip-count - Desired number of automatically created and managed outbound IPs for load balancer outbound connection. Valid for Standard SKU load balancer cluster only.
• Behavioral Changes
  • Starting from 2019-09-18, the preview CLI extension will default new cluster creates to VM Scale-Sets and Standard Load Balancers (VMSS/SLB) instead of VM Availability Sets and Basic Load Balancers (VMAS/BLB).
  • Starting from 2019-11-04 the official CLI and Azure Portal will default new cluster creates to VMSS/SLB instead of VMAS/BLB.
  • These client defaults changes are important to be aware of due to:
    • SLB will automatically assign a public IP to enable egress. This is a requirement placed by Azure Standard Load Balancers, to learn more about Standard vs. Basic, read here.
    • SLB enables bringing your own IP address to be used, you will be able to define these with new parameters.
    • The capability to use an SLB without any public IP assigned is on the roadmap plan.
    • You may still provision a basic load balancer by specifying "basic" for the "loadbalancersku" property at cluster create time.
    • Read more at https://aka.ms/aks/slb
• Component Updates
  • aks-engine has been updated to v0.39.2
    • https://github.com/Azure/aks-engine/releases/tag/v0.39.2
  • Azure Monitor for Containers Agent updated to 2019-08-22 release: https://github.com/microsoft/Docker-Provider/releases

This release is rolling out to all regions

Please Note: This release includes new Kubernetes versions 1.13.10 &
1.14.6 these include the fixes for CVEs CVE-2019-9512 and
CVE-2019-9514. Please see our customer guidance

• Bug Fixes
  • New kubernetes versions released to fix CVE-2019-9512 and CVE-2019-9514
    • Kubernetes 1.14.6
    • Kubernetes 1.13.10
  • Fixed Azure Network Policy bug with multiple labels under a matchLabels selector.
  • Fix for CNI lock timeout issue caused due to race condition in starting telemetry process.
  • Fixed issue creating AKS clusters using supported Promo SKUs
• Component Updates
  • aks-engine has been updated to v0.38.8
    • https://github.com/Azure/aks-engine/releases/tag/v0.38.8
  • Azure CNI has been updated to v1.0.25

This release is rolling out to all regions

• Bug Fixes

  • Several bug fixes for AKS NodePool creation and other CRUD operations.
  • Fixed audit log bug on older < 1.9.0 clusters.
    • Important: this was a best effort fix since these cluster versions are out of support. Please upgrade to a currently supported version
    • For information on how AKS handles Kubernetes version support see:
      Supported Kubernetes versions in Azure
  • Improved error messaging for VM size errors, including actions to take.
  • Fixed for PUT request bug that caused an unexpected API restart.

• Behavioral Changes

  • AKS has released an API update, documentation available here: https://docs.microsoft.com/en-us/rest/api/aks/managedclusters
    • Important: With this API update there are changes to the API whitelisting API. This is now under ManagedClusterAPIServerAccessProfile, where previously it was a top level property.

This release is rolling out to all regions

Please Note: This release includes new Kubernetes versions 1.13.9 &
1.14.5 (GA today) these include the fixes for CVEs CVE-2019-11247 and
CVE-2019-11249. Please see our customer guidance

• New Features
  • Kubernetes 1.14 is now GA (1.14.5)
    • As of Monday August 12th (2019-08-12) customers running Kubernetes 1.10.x
      have 60 days (2019-10-14) to upgrade to a supported release. Please see
      AKS supported versions document for more information.
  • Kubernetes Audit log
    support is now GA.
• Bug Fixes
  • Fixed an issue where creating a cluster with a custom subnet would return an
    HTTP error 500 vs 400 when the subnet could not be found.
• Behavioral Changes
• Preview Features
  • Fixed an issue where customers could not create a new node pool with AZs
    even if they were already using SLBs.
  • Fixed an issue where VMSS cluster commands could return the incorrect node
    count.
• Component Updates
  • aks-engine has been updated to v0.38.7
    • https://github.com/Azure/aks-engine/releases/tag/v0.38.7

This release is rolling out to all regions

• New Features

  • Customers may now create multiple AKS clusters using ARM templates
    regardless of what region the clusters are located in.

• Bug Fixes

  • AKS has resolved the issue(s) with missing metrics in the default
    metrics blade.
  • An issue where the --pod-max-pids was set to 100 (maximum) for clusters
    and re-applied during upgrade causing pthread_create() failed (11: Resource temporarily unavailable) pod start failures was fixed.
    • See https://github.com/Azure/aks-engine/pull/1623 for more information

• Preview Features

  • AKS is now in Public Preview in the Azure Government (Fairfax, VA)
    region. Please note the following:
    • Azure Portal support for AKS is in progress, for now customers must use the
      Azure CLI for all cluster operations currently.
    • AKS preview features are not supported in Azure Government currently and will
      be supported when those features are GA.
  • Fixed an issue where a delete request for a locked VMSS node would get an
    incorrect and unclear InternalError failure - the error message and error
    code have both been fixed.
  • Fixed an issue with egress filtering where managed AKS pods
    would incorrectly use the IP address to connect instead of the FQDN.
  • Fixed an issue with the SLB preview where AKS allowed the customer to
    provide an IP address already in use by another SLB.
  • An issue that prevented customers from using normal cluster operations
    on multiple node pool clusters with a single VMSS pool has been fixed.

• Component Updates

  • AKS-Engine has been updated to v0.38.4
    • https://github.com/Azure/aks-engine/releases/tag/v0.38.4

This release is currently being rolled out to all regions

• Preview Features
  • An issue where New Windows node pools in existing cluster would not get
    updated Windows versions has been fixed.
  • TCP reset has been set for all new clusters using the SLB preview.
  • An issue where AKS would trigger a scale operation requested on a previously
    deleted VMSS cluster has been fixed.
• Component Updates
  • AKS-Engine has been updated to v0.38.3

This release is currently being rolled out to all regions

Important behavioral change: All AKS clusters are being updated to pull all needed container images for cluster operations from Azure Container Registry, this means if you have custom allow/deny lists, port filtering, etc you will need to update your network configuration to allow ACR.

Please see the documentation for more
information including all required AKS cluster ports and URLs

• New Features

  • Support for the M, NC_promo and DS_v3 Azure Compute VM SKUs has been added.

• Bug Fixes

  • Fixed an issue with clusters created in Canada and Australia regions between
    2019-07-09 and 2019-07-10 as well as US region clusters created on 2019-07-10
    where customers would receive error: Changing property 'platformFaultDomainCount' is not allowed errors.

• Behavioral Changes

  • The error message returned to users when attempting to create clusters with
    an unsupported Kubernetes version in that region has been fixed.
  • Noted above, AKS has moved all container images required by AKS clusters for
    cluster CRUD operations have been moved to Azure Container Registry. This
    means that customers must update allow/deny rules and ports. See:
    Required ports and addresses for AKS clusters

• Preview Features

  • Fixed a VMSS cluster upgrade failure that would return: Changing property 'type' is not allowed.
  • An issue where az aks nodepool list would return the incorrect node count
    has been resolved.

• Component Updates

  • The Azure Monitor for Container agent has been updated to the 2019-07-09 release
    • Please see the release notes.

• New Features
  • Kubernetes versions 1.11.10 and 1.13.7 have been added. Customers
    are encouraged to upgrade.
    • For information on how AKS handles Kubernetes version support see:
      Supported Kubernetes versions in Azure
  • The az aks update-credentials command now supports Azure tenant migration of your
    AKS cluster. Follow the instructions in Choose to update or create a service principal and then execute the Update AKS cluster with new credentials command passing in the --tenant-id argument.
• Behavioral Changes
  • All new clusters now have --protect-kernel-defaults enabled.
    • See: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
• Preview Features
  • Kubernetes 1.14.3 is now available for preview users.
  • Azure availability zone support is now in public preview.
    • This feature enables customers to distribute their AKS clusters across
      availability zones providing a higher level of availability.
    • Please see AKS previews for additional information.
  • For all previews, please see the previews document for opt-in
    instructions and documentation links.
• Component Updates
  • aks-engine has been updated to version 0.37.5
    • https://github.com/Azure/aks-engine/releases/tag/v0.37.5
  • Azure CNI has been updated to version 1.0.22
  • Moby has been updated to 3.0.5 from 3.0.4
    • Note that this version number is Azure specific, the Moby project does not
      have official releases / release numbers.

Q: Why is it called "canning" if it's storing things in glass containers and not cans?
A: Because renaming it at this point would be jarring.

• Bug Fixes
  • Fixed an issue with az aks update-credentials where the command would
    not take special characters and nodes would get incorrect values.
    Note that double quote " , backslash \, ampersand &, and angle quotations <>
    are still NOT allowed to be used as password characters.
  • Fixed an issue with update-credentials where the command would not work for VMSS clusters
    with more than 10 instances.
  • AKS now has validation to check for Resource Locks when performing Scale and Upgrade operations.
  • Fixed an issue where GPU nodes could fail to install the GPU driver due to ongoing
    background apt operations.
  • Adjusted the timeout value for Service Principal update based on the number of nodes in the
    cluster, to accommodate larger clusters.
• New Features
  • AKS now supports OS disk sizes of up to 2048GiB.
  • Persistent Tags
    • Custom tags can now be passed to AKS and will persisted onto the MC infrastructure Resource Group.
      Note: They will NOT be applied to all child resources in that RG, aka VMs, VNets, disks, etc.
• Preview Features
  • Windows Node Pools
    • AKS updated Windows default image to latest windows patch release.
  • API server authorized IP ranges
    • The max number of API server authorized IP ranges has now increased to 100.
• Component Updates
  • AKS-Engine has been updated to v0.35.6
    • This change includes a new AKS VHD with the Linux Kernel CVE fixes. See more:
      https://github.com/Azure/AKS/issues/
    • This new VHD also fixes broken IPv6 support for the host.

• Bug Fixes
  • Fixed an issue that could result in a failed service principal update and
    AKS cluster creation.
  • Fixed an issue where deploying AKS clusters using ARM templates without a
    defined Service Principal would incorrectly pass validation.
• Preview Features
  • Azure Standard load balancer support is now in public preview.
    • This has been a long awaited feature which enables selection of the SKU
      type offered by Azure Load Balancer to be used with your AKS cluster. Please see
      AKS previews for additional information.
  • For all previews, please see the previews document for opt-in
    instructions and documentation links.
• Component Updates
  • The Azure Monitor for Container agent has been updated to the 2019-06-14 release
    • Please see the release notes.

• Behavioral Changes

  • Important: Change in UDR and subnet behavior
    • When using Kubenet with a custom subnet, AKS now checks if there is an
      existing associated route table.
    • If that is the case AKS will NOT attach the kubenet RT/Routes automatically
      and they should be added manually to the existing RT.
    • If no Route Table exists AKS will automatically attach the kubenet RT/Routes.

• Preview Features

  • A bug where users could not scale VMSS based clusters after disabling the
    cluster autoscaler has been fixed.
  • A missing CRD for calico-enabled clusters (#1042) has been fixed.

• Bug Fixes

  • Kubernetes taints and tolerations are now supported in all AKS regions.
    • Taints & Tolerations are preserved for current cluster nodes and
      through upgrades, however they are not preserved through scale (up,
      down) operations.

• Preview Features

  • A bug that prevented cluster agent pool deletions due to VMSS creation
    failures has been fixed.
  • A bug preventing the cluster autoscaler from working with nodepool enabled
    clusters (one or more nodepools) has been fixed.
  • A bug where the NSG would not be reset as needed during a nodepool create
    request has been fixed.

• Behavioral changes

  • AKS removed all weak CBC suite ciphers for API server. More info: https://blog.qualys.com/technology/2019/04/22/zombie-poodle-and-goldendoodle-vulnerabilities

• Component Updates

  • AKS-Engine has been updated to v0.35.4

• New Features

  • AKS is now available in both China East 2 / China North 2 Azure Regions.
  • AKS is now available in South Africa North
  • The L and M series Virtual Machines are now supported

• Component Updates

  • AKS-Engine has been updated to version 0.35.3
  • CoreDNS has been upgraded from 1.2.2 to version 1.2.6

• Preview Features

  • A bug where users could not delete an agent pool containing VMSS nodes if
    the VMSS node creation fails has been fixed.

Q: How do you organize an outer space party?
A: You planet.

• Behavioral Changes

  • The 192.0.2.0/24 IP block is now reserved for AKS use. Clusters created in
    a VNet that overlaps with this block will fail pre-flight validation.

• Bug Fixes

  • An issue where users running old AKS clusters attempting to upgrade would
    get a failed upgrade with an Internal Server Error has been fixed.
  • An issue where Kubernetes 1.14.0 would not show in the Azure Portal or AKS
    Preview CLI with the 'Preview' or 'isPreview' tag has been resolved.
  • An issue where customers would get excessive log entries due to missing
    Heapster rbac permissions has been fixed.
    • https://github.com/Azure/AKS/issues/520
  • An issue where AKS clusters could end up with missing DNS entries resulting
    in DNS resolution errors or crashes within CoreDNS has been resolved.

• Preview Features

  • A bug where the AKS node count could be out of sync with the VMSS node count
    has been resolved.
  • There is a known issue with the cluster autoscaler preview and multiple
    agent pools. The current autoscaler in preview is not compatible with
    multiple agent pools, and could not be disabled. We have fixed the issue
    that blocked disabling the autoscaler. A fix for mutliple agent pools and
    the cluster autoscaler is in development.

Q. How much money does a pirate pay for corn?
A. A buccaneer.

• Window node support for AKS is now in Public Preview

  • Blog post: https://aka.ms/aks/windows
  • Support and documentation:
    • Documentation: https://aka.ms/aks/windowsdocs
    • Issues may be filed on this Github repository (https://github.com/Azure/AKS)
      or raised as a Sev C support request. Support requests and issues for
      preview features do not have an SLA / SLO and are best-effort only.
  • Do not enable preview featured on production subscriptions or clusters.
  • For all previews, please see the previews document for opt-in
    instructions and documentation links.

• Bug fixes

  • An issue impacting Java workloads where pods running Java workloads would
    consume all available node resources instead of the defined pod resource
    limits defined by the user has been resolved.
    • https://bugs.openjdk.java.net/browse/JDK-8217766
    • AKS-Engine PR for fix: https://github.com/Azure/aks-engine/pull/1095

• Component Updates

  • AKS-Engine has been updates to v0.35.1

Q: Why did the Clydesdale give the pony a glass of water?
A: Because he was a little horse!

• New Features
  • Shared Subnets are now supported with Azure CNI.
    • Users may bring / provide their own subnets to AKS clusters
    • Subnets are no longer restricted to a single subnet per AKS cluster, users
      may now have multiple AKS clusters on a subnet.
    • If the subnet provided to AKS has NSGs, those NSGs will be preserved and
      used.
      • Warning: NSGs must respect: https://aka.ms/aksegress or the
        cluster might not come up or work properly.
    • Note: Shared subnet support is not supported with VMSS (in preview)
• Bug Fixes
  • A bug that blocked Azure CNI users from setting maxPods above 110 (maximum
    of 250) and that blocked existing clusters from scaling up when the value
    was over 110 for CNI has been fixed.
  • A validation bug blocking long DNS names used by customers has been fixed.
    For restrictions on DNS/Cluster names, please see
    https://aka.ms/aks-naming-rules

Q: Did you hear that I’m reading a book about anti-gravity?
A: It’s impossible to put down.