v1.3.0-alpha.1

Known Issues

• Clusters provisioned with Kubernetes 1.22 or upgraded from 1.21 to 1.22 using KubeOne 1.3.0-alpha.1 use a metrics-server version incompatible with Kubernetes 1.22. This might cause issues with deleting Namespaces that manifests by the Namespace being stuck in the Terminating state. This can be fixed by upgrading to KubeOne 1.3.0-rc.0 and running kubeone apply.

Added

• Add support for Kubernetes 1.22 (#1447, #1456)
• Add support for the kubeadm v1beta3 API. The kubeadm v1beta3 API is used for all Kubernetes 1.22+ clusters. (#1457)

Changed

Fixed

• Fix adding second container to the machine-controller-webhook Deployment (#1433)
• Extend restart API server script to handle failing crictl logs due to missing symlink. This fixes the issue with kubeone apply failing to restart the API server containers when provisioning or upgrading the cluster (#1448)

Updated

• Update Go to 1.16.7 (#1441)
• Update machine-controller to v1.35.1 (#1440)
• Update Hetzner CCM to v1.9.1 (#1428)
  • Add HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP=true to the environment if the network is configured
• Update DigitalOcean CCM to v0.1.33 (#1429)

Terraform Configs

• Inherit the firmware settings from the template VM in the Terraform configs for vSphere (#1445)

v1.3.0-alpha.0

Attention Needed

• [BREAKING/ACTION REQUIRED] The kubeone reset command requires an explicit confirmation like the apply command starting with this release
  • Running the reset command requires typing yes to confirm the intention to unprovision/reset the cluster
  • The command can be automatically approved by using the --auto-approve flag
• [BREAKING/ACTION REQUIRED] Upgrade Terraform to 1.0.0. The minimum Terraform version as of this KubeOne release is v1.0.0. (#1368)
• [BREAKING/ACTION REQUIRED] Use AdmissionRegistration v1 API for machine-controller-webhook. The minimum supported Kubernetes version is now 1.16. (#1290)
  • Since AdmissionRegistartion v1 got introduced in Kubernetes 1.16, the minimum Kubernetes version that can be managed by KubeOne is now 1.16. If you're running the Kubernetes clusters running 1.15 or older, please use the older release of KubeOne to upgrade those clusters
• KubeOne Addons can now be organized into subdirectories. It currently remains possible to put addons in the root of the addons directory, however, this is option is considered as deprecated as of this release. We highly recommend all users to reorganize their addons into subdirectories, where each subdirectory is for YAML manifests related to one addon.

Added

API

• Add new kube-proxy configuration API (#1420)
  • This API allows users to switch kube-proxy to IPVS mode, and configure IPVS properties such as strict ARP and scheduler
  • The default kube-proxy mode remains iptables
• Add support for Encryption Providers (#1241, #1320)
• Add support for specifying a custom Root CA bundle (#1316)

Features

• Docker to containerd automated migration (#1362)
• Automatically renew Kubernetes certificates when running kubeone apply if they're supposed to expire in less than 90 days (#1300)
• Ignore preexisting static manifests kubeadm preflight error (#1335)
• Add a new kubeone config images list subcommand to list images used by KubeOne and kubeadm. This command replaces the image loader script (#1334)
• Add containerd support for Flatcar clusters (#1340)
• Add support for running Kubernetes clusters on Amazon Linux 2 (#1339)

Addons

• Implement a mechanism for embedding YAML addons into KubeOne binary (#1387)
• Support organizing addons into subdirectories (#1364)
• Add a new KubeOne addon for handling unattended upgrades of the operating system (#1291)
• Add a new KubeOne addon for deploying the Hetzner CSI plugin (#1418)

Changed

CLI

• [BREAKING/ACTION REQUIRED] The kubeone reset command requires an explicit confirmation like the apply command starting with this release
  • Running the reset command requires typing yes to confirm the intention to unprovision/reset the cluster
  • The command can be automatically approved by using the --auto-approve flag

Bug Fixes

• Fix missing ClusterRole rule for cluster autoscaler (#1331)
• Fix missing confirmation for reset (#1251)
• Remove CNI patching (#1386)
• Fix subsequent apply failures if CABundle is enabled (#1404)
• Fix kubeone reset error when trying to list Machines (#1416)

Updated

• [BREAKING/ACTION REQUIRED] Upgrade Terraform to 1.0.0. The minimum Terraform version as of this KubeOne release is v1.0.0. (#1368, #1376)
• Use latest available (wildcard) docker and containerd version (#1358)
• Upgrade machinecontroller to v1.33.0 (#1391)
• Upgrade machine-controller addon apiextensions to v1 API (#1423)
• Upgrade calico-vxlan CNI plugin addon to v3.19.1 (#1403)
• Update Go to 1.16.1 (#1267)

Addons

• Replace the Canal CNI Go template with an embedded addon (#1405)
• Replace the WeaveNet Go template with an embedded addon (#1407)
• Replace the NodeLocalDNS template with an addon (#1392)
• Replace the metrics-server CCM Go template with an embedded addon (#1411)
• Replace the machine-controller Go template with an embedded addon (#1412)
• Replace the DigitalOcean CCM Go template with an embedded addon (#1396)
• Replace the Hetzner CCM Go template with an embedded addon (#1397)
• Replace the Packet CCM Go template with an embedded addon (#1401)
• Replace the OpenStack CCM Go template with an embedded addon (#1402)
• Replace the vSphere CCM Go template with an embedded addon (#1410)

v1.2.3

Changed

Bug Fixes

• Pass the -node-external-cloud-provider flag to the machine-controller-webhook. This fixes the issue with the worker nodes not using the external CCM on the clusters with the external CCM enabled (#1380)
• Disable repo_gpgcheck for the Kubernetes yum repository. This fixes the cluster provisioning and upgrading failures for CentOS/RHEL caused by yum failing to install Kubernetes packages (#1304)

Checksums

SHA256 checksums can be found in the kubeone_1.2.3_checksums.txt file.

v1.2.2

Changed

Bug Fixes

• Fix AWS config for terraform 0.15 (#1372)
  • AWS terraform config now works under terraform 0.15+ (including 1.0)
• Update machinecontroller to v1.30.0 (#1370)
  • machinecontroller to v1.30.0 relaxes docker / containerd version constraints
• Relax docker/containerd version constraints (#1371)

Checksums

SHA256 checksums can be found in the kubeone_1.2.2_checksums.txt file.

v1.2.1

Check out the changelog for the v1.2.0 release for more information about what changes were introduced in the 1.2 release.

Changed

Bug Fixes

• Install cri-tools (crictl) on Amazon Linux 2. This fixes the issue with provisioning Kubernetes and Amazon EKS-D clusters on Amazon Linux 2 (#1282)

Checksums

SHA256 checksums can be found in the kubeone_1.2.0-rc.1_checksums.txt file.

v1.2.0

KubeOne v1.2.0

The latest KubeOne 1.2.0 release is focused on community driven improvements and paving the way for future releases that incorporate even more features. We have been adding quite some alpha level features that will be improved and graduated in the future releases.

Please work through the Attention Needed section carefully and for more details read the v1.2.0 changelog.

Major Highlights

Add support for Kubernetes 1.20

Ongoing support for the latest Kubernetes version will give users access to the latest features and improvements. The Kubernetes 1.20 brings many new features, improvements, and fixes. We recommend checking out the official announcement for more details about the latest release.

containerd support

As of Kubernetes 1.20, Dockershim — a component that connects Kubelet and Docker, is deprecated. Starting with Kubernetes 1.23, it’ll not be possible to use Docker on Kubernetes nodes. Instead, a Container Runtime Interface (CRI) compatible container runtime must be used.

Containerd is one of CRI-compatible runtimes, and starting with this KubeOne release, it’s possible to provision clusters using containerd. We’ll also provide a migration path for existing clusters created by KubeOne in one of the upcoming releases.

Please check the Attention Needed section for more details about the upcoming changes related to the container runtimes.

Attention Needed

• [BREAKING/ACTION REQUIRED] Starting with the KubeOne 1.3 release, the kubeone reset command will require an explicit confirmation like the apply command
  • Running the reset command will require typing yes to confirm the intention to unprovision/reset the cluster
  • The command can be automatically approved by using the --auto-approve flag
  • The --auto-approve flag has been already implemented as a no-op flag in this release
  • Starting with this release, running kubeone reset will show a warning about this change each time the reset command is used
• [BREAKING/ACTION REQUIRED] Disallow and deprecate the PodPresets feature
  • If you're upgrading a cluster that uses the PodPresets feature from Kubernetes 1.19 to 1.20, you have to disable the PodPresets feature in the KubeOne configuration manifest
  • The PodPresets feature has been removed from Kubernetes 1.20 with no built-in replacement
  • It's not possible to use the PodPresets feature starting with Kubernetes 1.20, however, it currently remains possible to use it for older Kubernetes versions
  • The PodPresets feature will be removed from the KubeOneCluster API once Kubernetes 1.19 reaches End-of-Life (EOL)
  • As an alternative to the PodPresets feature, Kubernetes recommends using the MutatingAdmissionWebhooks.
• [BREAKING/ACTION REQUIRED] Support for CoreOS has been removed from KubeOne and machine-controller
  • CoreOS has reached End-of-Life on May 26, 2020
  • As an alternative to CoreOS, KubeOne supports Flatcar Linux
  • We recommend migrating your CoreOS clusters to the Flatcar Linux or other supported operating system
• [BREAKING/ACTION REQUIRED] Default values for OpenIDConnect has been corrected to match what's advised by the example configuration
  • Previously, there were no default values for the OpenIDConnect fields
  • This might only affect users using the OpenIDConnect feature
• Kubernetes has announced deprecation of the Docker (dockershim) support in the Kubernetes 1.20 release. It's expected that Docker support will be removed in Kubernetes 1.22 or 1.23
  • All newly created clusters running Kubernetes 1.21+ will be provisioned with containerd instead of Docker
  • Automated migration from Docker to containerd is currently not available, but is planned for one of the upcoming KubeOne releases
  • We highly recommend using containerd instead of Docker for all newly created clusters. You can opt-in to use containerd instead of Docker by adding containerRuntime configuration to your KubeOne configuration manifest:

    containerRuntime:
      containerd: {}

    For the configuration file reference, run kubeone config print --full.

Known Issues

• Provisioning a Kubernetes or Amazon EKS-D cluster on Amazon Linux 2 will fail due to missing crictl binary. This bug has been fixed in the v1.2.1 release.
• Upgrading an Amazon EKS-D cluster will fail due to kubeadm preflight checks failing. We're investigating the issue and you can follow the progress by checking the issue #1284.

Changelog since v1.2.0-rc.1

There have been no changes since the KubeOne v1.2.0-rc.1 release.

Changelog since v1.1.0

For the complete changelog since the KubeOne v1.1.0 release, please check the CHANGELOG document.

Checksums

SHA256 checksums can be found in the kubeone_1.2.0_checksums.txt file.

v1.2.0-rc.1

Changed

General

• Build KubeOne using Go 1.16.1 (#1268, #1267)

Checksums

SHA256 checksums can be found in the kubeone_1.2.0-rc.1_checksums.txt file.

v1.2.0-rc.0

Attention Needed

• [BREAKING/ACTION REQUIRED] Starting with the KubeOne 1.3 release, the kubeone reset command will require an explicit confirmation like the apply command
  • Running the reset command will require typing yes to confirm the intention to unprovision/reset the cluster
  • The command can be automatically approved by using the --auto-approve flag
  • The --auto-approve flag has been already implemented as a no-op flag in this release
  • Starting with this release, running kubeone reset will show a warning about this change each time the reset command is used

Changed

General

• Warn about kubeone reset requiring explicit confirmation starting with KubeOne 1.3 (#1252)

Checksums

SHA256 checksums can be found in the kubeone_1.2.0-rc.0_checksums.txt file.

v1.2.0-beta.1

Attention Needed

• [Breaking] Support for CoreOS has been removed from KubeOne and machine-controller
  • CoreOS has reached End-of-Life on May 26, 2020
  • As an alternative to CoreOS, KubeOne supports Flatcar Linux
  • We recommend migrating your CoreOS clusters to the Flatcar Linux or other supported operating system
• [Breaking] Default values for OpenIDConnect has been corrected to match what's advised by the example configuration
  • Previously, there were no default values for the OpenIDConnect fields
  • This might only affect users using the OpenIDConnect feature
• [Breaking] Disallow and deprecate the PodPresets feature
  • [Action Required] If you're upgrading a cluster that uses the PodPresets feature from Kubernetes 1.19 to 1.20, you have to disable the PodPresets feature in the KubeOne configuration manifest
  • The PodPresets feature has been removed from Kubernetes 1.20 with no built-in replacement
  • It's not possible to use the PodPresets feature starting with Kubernetes 1.20, however, it currently remains possible to use it for older Kubernetes versions
  • The PodPresets feature will be removed from the KubeOneCluster API once Kubernetes 1.19 reaches End-of-Life (EOL)
  • As an alternative to the PodPresets feature, Kubernetes recommends using the MutatingAdmissionWebhooks.

Added

• Add support for Kubernetes 1.20
  • Previously, we've shared that there is an issue affecting newly created clusters where the first control plane node is unhealthy/broken for the first 5-10 minutes. We've investigated the issue and found out that the issue can be successfully mitigated by restarting the first API server. We've implemented a task that automatically restarts the API server if it's affected by the issue (#1243, #1245)
• Add support for Debian on control plane and static worker nodes (#1233)
  • Debian is currently not supported by machine-controller, so it's not possible to use it on worker nodes managed by machine-controller

Changed

API Changes

• [Breaking] Default values for OpenIDConnect has been corrected to match what's advised by the example configuration (#1235)
  • Previously, there were no default values for the OpenIDConnect fields
  • This might only affect users using the OpenIDConnect feature
• [Breaking] Disallow and deprecate the PodPresets feature (#1236)
  • [Action Required] If you're upgrading a cluster that uses the PodPresets feature from Kubernetes 1.19 to 1.20, you have to disable the PodPresets feature in the KubeOne configuration manifest
  • The PodPresets feature has been removed from Kubernetes 1.20 with no built-in replacement
  • It's not possible to use the PodPresets feature starting with Kubernetes 1.20, however, it currently remains possible to use it for older Kubernetes versions
  • The PodPresets feature will be removed from the KubeOneCluster API once Kubernetes 1.19 reaches End-of-Life (EOL)
  • As an alternative to the PodPresets feature, Kubernetes recommends using the MutatingAdmissionWebhooks.

General

• Add rsync on CentOS and Amazon Linux (#1240)

Bug Fixes

• Drop mounting Flexvolume plugins into the OpenStack CCM. This fixes the issue with deploying the OpenStack CCM on the clusters running Flatcar Linux (#1234)
• Ensure all credentials are available to be used in addons. This fixes the issue with the Backups addon not working on non-AWS providers (#1248)

Updated

• Update machine-controller to v1.25.0 (#1238)

Removed

• [Breaking] Support for CoreOS has been removed from KubeOne and machine-controller (#1232)
  • CoreOS has reached End-of-Life on May 26, 2020
  • As an alternative to CoreOS, KubeOne supports Flatcar Linux
  • We recommend migrating your CoreOS clusters to the Flatcar Linux or other supported operating system

Checksums

SHA256 checksums can be found in the kubeone_1.2.0-beta.1_checksums.txt file.

v1.2.0-beta.0

Attention Needed

• Kubernetes has announced deprecation of the Docker (dockershim) support in the Kubernetes 1.20 release. It's expected that Docker support will be removed in Kubernetes 1.22
  • All newly created clusters running Kubernetes 1.21+ will be provisioned with containerd instead of Docker
  • Automated migration from Docker to containerd is currently not available, but is planned for one of the upcoming KubeOne releases
  • We highly recommend using containerd instead of Docker for all newly created clusters. You can opt-in to use containerd instead of Docker by adding containerRuntime configuration to your KubeOne configuration manifest:

    containerRuntime:
      containerd: {}

    For the configuration file reference, run kubeone config print --full.

Known Issues

• Provisioning Kubernetes 1.20 clusters results with one of the control plane nodes being unhealthy/broken for the first 5-10 minutes after provisioning the cluster. This causes KubeOne to fail to create MachineDeployment objects because the machine-controller-webhook service can't be found. Also, one of the NodeLocalDNS pods might get stuck in the crash loop.
  • KubeOne currently still doesn't support Kubernetes 1.20. We do not recommend provisioning 1.20 clusters or upgrading existing clusters to Kubernetes 1.20
  • We're currently investigating the issue. You can follow the progress in the issue #1222

Added

• Add support for containerd container runtime (#1180, #1188, #1190, #1205, #1227, #1229)
  • Kubernetes has announced deprecation of the Docker (dockershim) support in the Kubernetes 1.20 release. It's expected that Docker support will be removed in Kubernetes 1.22
  • All newly created clusters running Kubernetes 1.21+ will default to containerd instead of Docker
  • Automated migration from Docker to containerd is currently not available, but is planned for one of the upcoming KubeOne releases

Changed

Bug Fixes

• Fix wrong legacy Docker version on RPM systems (#1191)

Terraform Configs

• Replace GoBetween load-balancer in vSphere Terraform example by keepalived (#1217)

Addons

• Fix DNS resolution issues for the Backups addon (#1179)

Checksums

SHA256 checksums can be found in the kubeone_1.2.0-beta.0_checksums.txt file.