Improvements

Rook v1.6.3 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Ensure correct devices are started for OSDs after node restart (#7951, @BlaineEXE)
• Write reconcile results to events on the CephCluster CR (#7222, @iamniting)
• Updated dashboard ingress example for networking v1 (#7933, @travisn)
• Remove obsolete gateway type setting in object store CRD (#7919, @satoru-takeuchi)
• Support specifying only public network or only cluster network or both (#7546, @rohan47)
• Generate same operator deployment for OKD as OCP (#7898, @RyuunoAelia)
• Ensure correct hostpath lock for OSD integrity (#7886, @satoru-takeuchi)
• Improve resilience of mon failover if operator is restarted during failover (#7884, @travisn)
• Disallow overriding the liveness probe handler function (#7889, @leseb)
• Actively update the service endpoint for external mgr (#7875, @leseb)
• Remove obsolete CSI statefulset template path vars from K8s 1.13 (#7877, @Rakshith-R)
• Create crash collector pods after mon secret created (#7867, @subhamkrai)
• OSD controller only updates PDBs during node drains instead of any OSD down event (#7726, @sp98)
• Allow heap dump generation when logCollector sidecar is not running (#7847, @leseb)
• Add nullable to object gateway settings (#7857, @travisn)

Improvements

Rook v1.6.2 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Set base Ceph operator image and example deployments to v16.2.2 (#7829, @BlaineEXE)
• Update snapshot APIs from v1beta1 to v1 (#7711, @Rakshith-R)
• Documentation for creating static PVs (#7782, @Rakshith-R)
• Allow setting primary-affinity for the OSD (#7807, @synarete)
• Remove unneeded debug log statements (#7526, @parth-gr)
• Preserve volume claim template annotations during upgrade (#7835, @travisn)
• Allow re-creating erasure coded pool with different settings (#7820, @subhamkrai)
• Double mon failover timeout during a node drain (#7801, @sp98)
• Remove unused volumesource schema from CephCluster CRD (#7813, @travisn)
• Set the device class on raw mode osds (#7815, @travisn)
• External cluster schema fix to allow not setting mons (#7789, @leseb)
• Add phase to the CephFilesystem CRD (#7752, @jshen28)
• Generate full schema for volumeClaimTemplates in the CephCluster CRD (#7631, @BlaineEXE)
• Automate upgrades for the MDS daemon to properly scale down and scale up (#7445, @jshen28)
• Add Vault KMS support for object stores (#7385, @thotz)
• Ensure object store endpoint is initialized when creating an object user (#7633, @alimaredia)
• Support for OBC operations when RGW is configured with TLS (#7764, @thotz)
• Preserve the OSD topology affinity during upgrade for clusters on PVCs (#7759, @travisn)
• Unify timeouts for various Ceph commands (#7719, @satoru-takeuchi)
• Allow setting annotations on RGW service (#7598, @thotz)
• Expand PVC size of mon daemons if requested (#7715, @iamniting)

Improvements

Rook v1.5.11 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Set operator base image and default Ceph version to v15.2.11 (#7733, @travisn)
• Deploy Ceph-CSI v3.2.2 with latest base image including CVE fixes (#7810, @Madhu-1)
• Improve node watcher for deploying new OSDs (#7462, @parth-gr)
• Fix bucket health check where SSL is enabled for RGW (#7331, @thotz)
• Detect the topology affinity for portable OSDs during upgrade (#7759, @travisn)
• Ensure object store endpoint is initialized for user (#7633, @alimaredia)

Improvements

Rook v1.6.1 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Disable host networking by default in the CSI plugin with option to enable (#7356, @Rakshith-R)
• Fix the schema for erasure-coded pools so replication size is not required (#7662, @travisn)
• Improve node watcher for adding new OSDs (#7568, @parth-gr)
• Operator base image updated to v16.2.1 (#7713, @leseb)
• Deployment examples updated to Ceph v15.2.11 (#7733, @travisn)
• Update Ceph-CSI to v3.3.1 (#7724, @Madhu-1)
• Allow any device class for the OSDs in a pool instead of restricting the schema (#7718, @travisn)
• Fix metadata OSDs for Ceph Pacific (#7703, @BlaineEXE)
• Allow setting the initial CRUSH weight for an OSD (#7472, @synarete)
• Fix object store health check in case SSL is enabled (#7331, @thotz)
• Upgrades now ensure latest config flags are set for MDS and RGW (#7681, @leseb)
• Suppress noisy RGW log entry for radosgw-admin commands (#7663, @BlaineEXE)

Major Themes

v1.6.0 is a minor release with features primarily for the Ceph operator.

K8s Version Support

Kubernetes supported versions: 1.11 and newer

Upgrade Guides

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph

Breaking Changes

Removed Storage Providers

Each storage provider is unique and requires time and attention to properly develop and support. After much discussion with the community, we have decided to remove three storage providers from Rook in order to focus our efforts on storage providers that have active community support. See the project status for more information. These storage providers have been removed:

• CockroachDB
• EdgeFS
• YugabyteDB

Ceph

• Support for creating OSDs via Drive Groups was removed. Please refer to the Ceph upgrade guide for migration instructions.

Features

Ceph

• Ceph Pacific (v16) support, including features such as:
  • Multiple Ceph Filesystems
  • Networking dual stack
• CephFilesystemMirror CRD to support mirroring of CephFS volumes with Pacific
• Ceph CSI Driver
  • CSI v3.3.0 driver enabled by default
  • Volume Replication Controller for improved RBD replication support
  • Multus support
  • GRPC metrics disabled by default
• Ceph RGW
  • Extended the support of vault KMS configuration
  • Scale with multiple daemons with a single deployment instead of a separate deployment for each rgw daemon
• OSDs:
  • LVM is no longer used to provision OSDs as of Nautilus 14.2.14 Octopus 15.2.9, and Pacific 16.2.0, simplifying the OSDs on raw devices, except for encrypted OSDs and multiple OSDs per device.
  • More efficient updates for multiple OSDs at the same time (in the same failure domain) to speed up upgrades for larger Ceph clusters
• Multiple Ceph mgr daemons are supported for stretch clusters and other clusters where HA of the mgr is critical (set count: 2 under mgr in the CephCluster CR)
• Pod Disruption Budgets (PDBs) are enabled by default for Mon, RGW, MDS, and OSD daemons. See the disruption management settings.
• Monitor failover can be disabled, for scenarios where maintenance is planned and automatic mon failover is not desired
• CephClient CRD has been converted to use the controller-runtime library

Improvements

Rook v1.5.10 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update Ceph-CSI to v3.2.1 (#7506, @shaas)
• Use latest Ceph API for setting dashboard and rgw credentials (#7641, @subhamkrai)
• Redact secret info from reconcile diffs in debug logs (#7630, @BlaineEXE)
• Continue to get available devices if failed to get a device info (#7608, @satoru-takeuchi)
• Include RGW pods in list for rescheduling from failed node (#7537, @rohan47)
• Enforce pg_auto_scaler on rgw pools (#7513, @leseb)
• Prevent voluntary mon drain while another mon is failing over (#7442, @sp98)
• Avoid restarting all encrypted OSDs on cluster growth (#7489, @leseb)
• Set secret type on external cluster script (#7473, @leseb)
• Fix init container "expand-encrypted-bluefs" for encrypted OSDs (#7466, @leseb)
• Fail pool creation if the sub failure domain is the same as the failure domain (#7284, @leseb)
• Set default backend for vault and remove temp key for encrypted OSDs (#7454, @leseb)

Improvements

Rook v1.5.9 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Properly add CephCluster schema to support bucket health checks (#7330, @thotz)
• Avoid overlapping OSD placement for PVC and non-PVCs (#7415, @subhamkrai)
• Correct RBAC for multus in the helm chart (#7340, @rohan47)
• During uninstall skip cleanup if cluster is not configured correctly (#7414, @sp98)
• Enable the PG auto repair module (#7391, @leseb)
• Set pool quota with K8s quantity format (#7397, @fritchie)
• Proper JSON parsing for object store configuration (#7354, @satoru-takeuchi)
• Disable CSI GRPC metrics by default (#7379, @Madhu-1)
• Add OSD flapping alert (#7358, @anmolsachan)
• Add OSD slow ops alert (#7417, @anmolsachan)
• Prometheus query to avoid many-to-many match error (#7273, @anmolsachan)
• Only raise CephCluster conditions that represent current state (#7315, @travisn)
• Handle SSL cases for RGW's liveness probe (#7334, @thotz)
• Improved vault warnings for encrypted OSDs (#7337, @leseb)
• Stop managing labels of monitoring resources (#7323, @umangachapagain)
• Detect standby mgr for external clusters (#7217, @aruniiird)

Improvements

Rook v1.5.8 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update operator and example manifests to use ceph/ceph:v15.2.9 (#7302, @BlaineEXE)
• Consistently force delete Rook pods on unresponsive nodes (#7266, @rohan47)
• OSD encryption improvements with Vault (#7292, @leseb)
• Ability to set pool quotas in bytes or objects (#7264, @fritchie)
• Fix SIGSEGV when failing to get object store user (#7280, @satoru-takeuchi)
• Enforce portable OSDs in same topology as osd prepare job (#7256, @travisn)
• Do not merge nodeAffinity for OSDs between device sets and non-device sets (#7192, @subhamkrai)
• During OSD removal archive crash dumps (#7001, @crombus)
• Add tolerations for all daemons to the cleanup job (#7234, @travisn)
• Helm fix for casing on enableCephFSSnapshotter (#7249, @subhamkrai)
• Do not override existing Vault keys for encrypted OSDs (#7245, @leseb)
• Create new OSDs before updating existing OSDs (#6926, @BlaineEXE)
• Helm fix to allow multiple filesystems (#7224, @TomHellier)
• Add the secure endpoint to the object store user CR status (#7191, @thotz)
• Set default values to allow partial override of liveness probes (#7215, @subhamkrai)

Improvements

Rook v1.5.7 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• CSI Troubleshooting Guide (#7157, @subhamkrai)
• Print device information in OSD prepare logs (#7194, @leseb)
• Expose vault curl error in the OSD init container for KCS configurations (#7193, @leseb)
• Prevent re-using a device to configure an OSD on PVC from a previous cluster (#7170, @leseb)
• Remove crash collector if all Ceph pods moved off a node (#7160, @leseb)
• Add helm annotation to keep CRDs in the helm chart during uninstall (#7162, @onedr0p)
• Bind mgr modules to all interfaces instead of pod ip (#7151, @travisn)
• Check for orchestration cancellation while waiting for all OSDs to start (#7112, @sp98)
• Skip pdb reconcile on create and delete events (#7155, @sp98)
• Silence harmless errors in log when the operator is still initializing (#7056, @leseb)
• Add --extra-create-metadata flag to the CSI driver (#7147, @Madhu-1)
• Add deviceClass to the object store schema (#7132, @travisn)
• Simplify the log-collector container name (#7133, @travisn)
• Skip csi detection if CSI is disabled (#6866, @jshen28)
• Remove Rook pods stuck in terminating state on a failed node (#6999, @rohan47)
• Timeout for rgw configuration to prevent stuck object store when no healthy OSDs (#7075, @sp98)
• Update lib bucket provisioner for OBCs (#7086, @satoru-takeuchi)

Improvements

Rook v1.5.6 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Add option to set FSGroupPolicy for CSI PVCs (#6875, @Madhu-1)
• Add --public-addr argument to the MDS daemon (#6701, @sp98)
• Only commit the maxMonID after starting a mon daemon for mon disaster recovery (#7084, @travisn)
• Allow specifying timeout for waiting for an OSD to upgrade (#6833, @sp98)
• Prevent operator crash with host networking and when the node is not found (#7083, @travisn)
• Avoid unnecessary OSD restarts by consistently ordering the pod spec args (#7071, @travisn)
• Allow removal of unhealthy OSDs on PVCs (#6982, @travisn)
• Fix ProgressingCompleted CephCluster status condition type (#7059, @ron1)
• Enable the log collector for RGW and MDS if desired (#7044, @leseb)
• Update the RBD mirror daemon(s) when the cluster is upgraded (#7041, @leseb)
• Ability to configure multiple filesystems in the helm chart (#7008, @TomHellier)
• Allow disabling the snapshotter sidecar container (#7031, @Madhu-1)
• Avoid potential OSD corruption with a file lock on the host (#6793, @satoru-takeuchi)
• Delete discovery daemon if it is disabled (#6974, @satoru-takeuchi)
• Expose in the CephCluster conditions which OSD is being processed (#6972, @jshen28)
• During object store deletion return success if not found (#6966, @travisn)

Improvements

Rook v1.5.5 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Only restart ceph dashboard module if settings changed (#6950, @travisn)
• Enhance delete cephObjectStoreUser logging (#6691, @iamniting)
• Fix malformations in common-external.yaml (#6951, @BlaineEXE)
• Allow disabling the liveness check for the MDS daemonset (#6883, @jshen28)
• Add devices to the CRD schema at overall storage level (#6938, @travisn)
• Apply fallback resources and placement for OSDs in device sets (#6918, @subhamkrai)
• Change external rgw detection, not relying on cluster (#6888, @Zempashi)
• Suppress reconcile error after operator restart (#6925, @travisn)
• Update internal K8s client to version 1.20.0 (#6853, @subhamkrai)
• Requeue clusterDisruption controller more proactively (#6749, @sp98)
• Allow using lvm batch for ceph 14.2.15 (#6831, @jshen28)

Improvements

Rook v1.4.9 is a patch release limited in scope and focusing on small feature additions and bug fixes.

This patch release updates the Rook CRDs to v1 as part of the migration to Helm 3. While we have tested this helm upgrade scenario, you may want to consider upgrading to 1.5 where there has been more comprehensive testing with this conversion rather than deploy v1.4.9 with helm.

Ceph

• Update to Helm 3 and convert deprecated v1beta1 resources to v1 (#6910, @travisn)
• Add devices to schema at overall storage level (#6938, @travisn)
• Update operator base image and example manifests to Ceph v15.2.8 (#6847, @travisn)
• Tune fast device class for OSD on PVC in the Azure (#6303, @subhamkrai)
• RGW service selector should not change during upgrade (#6742, @travisn)

Improvements

Rook v1.5.4 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

If you are installing OSDs on partitions, please see #6849 for details on the latest Ceph patch releases that do not support partitions.

• Endless reconcile in clusters where all nodes do not configure OSDs (#6796, @travisn)
• Update Ceph-CSI install default to v3.2.0 (#6769, @Madhu-1)
• OBC is not properly created, resource name may not be empty (#6788, @BlaineEXE)
• Update examples and operator to Ceph v15.2.8 (#6847, @travisn)
• Remove duplicate env values in the Ceph mgr pod (#6801, @binoue)
• Apply deviceClass properly to non-PVC OSDs (#6815, @travisn)
• Allow OSDs on both PVCs and raw devices in the same cluster (#6822, @travisn)
• Namespace meta-comments in manifests to more easily enable multiple Rook clusters (#6809, @BlaineEXE)
• Updated thresholds of OSD capacity alerts (#6785, @anmolsachan)
• Stretch cluster waits for OSDs to stabilize before enabling arbiter (#6806, @travisn)
• Add mgr caps for the csi-rbd-node client (#6765, @Madhu-1)
• Allow stretch cluster arbiter to have separate placement from other mons (#6761, @travisn)
• Merge placement node affinity for stretch clusters (#6759, @travisn)

Improvements

Rook v1.5.3 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Buckets from OBCs were being deleted when upgrading from v1.5.1 to v1.5.2. The related change was reverted, with bucket handling delayed until the next release. (#6767, @BlaineEXE)
• Adding a new node to the cluster was not automatically triggering adding of new OSDs (#6768, @shaas)
• Ability to set the prometheus port for external clusters (#6714, @leseb)
• Fix cleanupPolicy schema error in the cluster.yaml example (#6763, @leseb)

v1.5.2 is taken down due to a potential data loss with OBCs in #6767

Improvements

Rook v1.5.2 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Enable collecting Ceph logs from the host in case the pods restarted multiple times (#6679, @leseb)
• Support OSD metadata device passed by-id (#6696, @leseb)
• Ability to abort and restart orchestration when the cluster CR is modified (#6693, @leseb)
• Object bucket provisioning error handling (#6699, @BlaineEXE)
• Object workloads failed briefly during Rook upgrades until RGW pods updated (#6742, @travisn)
• Update Ceph manifests and operator base image to v15.2.7 (#6726, @travisn)
• Avoid duplicate labels when merging custom labels (#6720, @galexrt)
• Properly apply custom labels on CSI components (#6702, @galexrt)
• Uninstall cleanup ignores pending Ceph daemon pods (#6719, @sp98)
• Update the Controller Runtime version to 0.6 (#6568, @aruniiird)

Improvements

Rook v1.4.8 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update base operator image and example manifests to Ceph v15.2.7 (#6690, @travisn)
• Merge custom labels properly with other labels in the spec (#6720, @galexrt)
• Uninstall cleanup ignores ceph daemon pods that are in pending state (#6719, @sp98)
• Orchestration is aborted and restarted if the cluster CR is updated (#6693, @leseb)
• Restore mon clusterIP if the service is missing in disaster recovery scenarios (#6658, @travisn)
• Set the RGW deployment version label (#6610, @BlaineEXE)
• Add privileged securityContext to CephFS provisioner (#6561, @Madhu-1)

Improvements

Rook v1.5.1 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Ceph-CSI updated to the latest v3.1.2 by default (#6676, @Madhu-1)
• PDBs for OSDs redesigned to allow multiple nodes in a zone to drain (#6497, @sp98)
• Snapshot scheduling for mirrored pools (#6553, @leseb)
• Restore mon clusterIP if the service is missing (#6558, @travisn)
• Multus support for CSI in a non-default namespace (#6396, @rohan47)
• Add external cluster script to the toolbox (#6648, @leseb)

Major Themes

v1.5.0 is a minor release with features primarily for the Ceph operator.

K8s Version Support

• Kubernetes supported versions: 1.11 and newer

Upgrade Guides

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph

Breaking Changes

Ceph

• The CRDs have been separated from common.yaml into crds.yaml to give more flexibility for upgrades. New non-helm deployments require installation of the crds.yaml.
• The discovery daemon is disabled by default in new clusters since the discovery is not necessary in most clusters.
  • Enable the discovery daemon if devices are being added to nodes and you want to automatically configure OSDs on new devices without restarting the operator.
• Rook will now disallow an even number of Ceph mons. Mons require an odd number for a healthy quorum.
• CRDs have been updated to use apiextensions.k8s.io/v1 Kubernetes API from the deprecated apiextensions.k8s.io/v1beta1.

Features

Core

• Discover Agent: custom labels can be added to the agent pods

Ceph

• Ceph Cluster
  • The storage capacity of the cluster is updated periodically in the CephCluster CR status
  • Encryption support is available with a Key Management Service (Vault)
  • (experimental): Support for Stretch clusters that allow mons and OSDs to work reliably across two data centers
• Ceph Block Pool
  • Improved mirroring support for block pools
  • New replicasPerFailureDomain setting to allow for a number of replicas in a secondary failure domain, primarily for stretch clusters
• Ceph Filesystem
  • preserveFilesystemOnDelete option replaces preservePoolsOnDelete to avoid deleting the filesystem during uninstall
    • Rook will treat the deprecated preservePoolsOnDelete: true as if the user intends preserveFilesystemOnDelete: true
• Helm chart
  • Updated to Helm v3.4 for build and testing
  • Added a crds.enabled setting to allow the CRDs to be managed separately from the Helm chart. By default the CRDs are still installed
• Ceph CSI
  • Allow custom labels to be added to DaemonSet and Deployment Pods

Improvements

Rook v1.4.7 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Log warning about v14.2.13 being an unsupported Ceph version due to errors creating new OSDs (#6545, @leseb)
• Disaster recovery guide for PVCs (#6452, @tareqhs)
• Set the deviceClass for OSDs in non-PVC clusters (#6545, @leseb)
• External cluster script to fail if prometheus port is not default (#6504, @leseb)
• Remove the osd pvc from the osd purge job (#6533, @travisn)
• External cluster script added additional checks for monitoring endpoint (#6473, @aruniiird)
• Ignore Ceph health error MDS_ALL_DOWN during reconciliation (#6494, @LalitMaganti)
• Add optional labels to mon pods (#6515, @yuseinishiyama)
• Assert type for logging errors before using it (#6503, @leseb)
• Check for orphaned mon resources with every reconcile (#6493, @travisn)
• Update the mon PDBs if the maxUnavailable changed (#6469, @sp98)

NFS

• Update documentation and examples (#6455, @prksu)

Improvements

Rook v1.4.6 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Support IPv6 single-stack (#6283, @sp98)
• Only start a single CSI provisioner in single-node test clusters (#6437, @satoru-takeuchi)
• Raw mode OSD on LV-backed PVC (#6184, @satoru-takeuchi)
• Capture ceph-volume detailed log in non-pvc scenario on failure (#6426, @leseb)
• Add --upgrade option to external cluster script (#6392, @aruniiird)
• Capture stderr when executing ceph commands and write to log (#6395, @leseb)
• Reduce the retry count for the bucket health check for more accurate status (#6408, @leseb)
• Prevent closing of monitoring channel more than once (#6369, @raghavendra-talur)
• Check underlying block status for encrypted OSDs (#6367, @leseb)