Improvements

Rook v1.7.9 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

This release includes the new migration tool from flex to CSI. The conversion is required before clusters can be upgraded to v1.8 where flex volumes will not be supported.

• docs: Guide for migrating from flex to csi volumes (#9222, @subhamkrai)
• build: Add pv-migrator tool to rook image in 1.7.x branch (#9274, @subhamkrai)
• helm: Addition of mountOptions into storage class configuration (#9287, @TomHellier)
• osd: Update existing OSDs with deviceClass (#9259, @parth-gr)
• cephfs-mirror: Various fixes for random bootstrap peer import errors (#9264, @leseb)
• helm: Allow configurability of ingress version (#9237, @TomHellier)
• osd: Check if osd is safe-to-destroy before removal from the osd purge job (#9230, @leseb)
• object: Configure object store with correct zone name for multisite (#9249, @olivierbouffet)
• monitoring: Update label on prometheus resources (#9243, @LittleFox94)
• core: Add expected capabilities to OpenShift security context (#9236, @leseb)
• nfs: Only set the pool size when it exists and always run default pool creation for NFS (#9224, @leseb)
• monitoring: Allow overriding monitoring labels (#9240, @LittleFox94)
• helm: Use a more explicit value for ingress for K8s 1.18 (#9280, @TomHellier)
• object: Create system user with correct realm for external cluster (#9143, @olivierbouffet)
• docs: Add details about HPA via KEDA (#9202, @thotz)

See the pending release notes to for an overview of features coming in v1.8.

Improvements

Rook v1.7.8 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• docs: add OMAP quick fix warning to the upgrade guide (#9187, @BlaineEXE)
• mon: Set cluster name to mon cluster (#9203, @parth-gr)
• osd: Add privileged support for OpenShift to blkdevmapper securityContext (#9191, @Omar007)
• osd: Set reduced blkdevmapper capabilities (#9158, @Omar007)
• helm: Set correct ingress endpoint protocol for https and http (#9181, @travisn)
• rgw: Allow downgrade of rgw daemon (#9137, @travisn)
• helm: Enable cephfs volume expansion by default (#9128, @travisn)
• core: Fail if config dir creation fails (#9122, @leseb)
• osd: Increase wait timeout for osd prepare cleanup (#9116, @travisn)
• pool: Allow more data chunks in an EC pool (#9107, @travisn)

Improvements

Rook v1.6.11 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• rgw: Allow reconcile to complete even during a downgrade (#9137, @travisn)
• docs: Add OMAP quick fix warning to the upgrade guide (#9187, @BlaineEXE)
• multus: Do not build all the args to remote exec cmd (#8860, @leseb)
• multus: do not fail on keys deletion (#8868, @leseb)

Improvements

Rook v1.7.7 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• docs: Support ephemeral volumes with Ceph CSI RBD and CephFS driver (#9055, @humblec)
• core: Allow downgrade of all daemons consistently (#9098, @travisn)
• core: Reconcile once instead of multiple times after the cluster CR is edited (#9091, @leseb)
• nfs: Add pool setting CR option (#9040, @leseb)
• ceph: Trigger 'CephMonQuorumLost' alert when mon quorum is down (#9068, @aruniiird)
• rgw: Updated livenessProbe and readinessProbe (#9080, @satoru-takeuchi)
• mgr: Do not set the balancer mode on pacific (#9063, @leseb)
• helm: Add appVersion property to the charts (#9051, @travisn)
• rgw: Read tls secret hint for insecure tls (#9020, @leseb)
• ceph: Ability to set labels on the crash collector (#9044, @leseb)
• core: Treat cluster as not existing if the cleanup policy is set (#9041, @travisn)
• docs: Document failover and failback scenarios for applications (#8411, @Yuggupta27)
• ceph: Update endpoint with IP for external RGW server (#9010, @thotz)

Full Changelog: https://github.com/rook/rook/compare/v1.7.6...v1.7.7

Improvements

Rook v1.7.6 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• core: only merge stderr on error (#8995, @leseb)
• nfs: remove RADOS options from CephNFS and use .nfs pool (#8501, @josephsawaya)
• csi: fix comment for the provisioner and clusterID (#8990, @Madhu-1)
• mon: Enable mon failover for the arbiter in stretch mode (#8984, @travisn)
• monitoring: fixing the queries for alerts 'CephMgrIsAbsent' and 'CephMgrIsMissingReplicas' (#8985, @aruniiird)
• osd: fix kms auto-detection when full TLS (#8867, @leseb)
• csi: add affinity to csi version check job (#8965, @Rakshith-R)
• pool: remove default value for pool compression (#8966, @leseb)
• monitoring: handle empty ceph_version in ceph_mon_metadata to avoid raising misleading alert (#8947, @GowthamShanmugam)
• nfs: remove RADOS options from CephNFS and use .nfs pool (#8501, @josephsawaya)
• osd: print the c-v output when inventory command fails (#8971, @leseb)
• helm: remove chart content not in common.yaml (#8884, @BlaineEXE)
• rgw: replace period update --commit with function (#8911, @BlaineEXE)
• rgw: fixing ClientID of log-collector for RGW instance (#8889, @parth-gr)
• mon: run ceph commands to mon with timeout (#8939, @leseb)
• osd: do not hide errors (#8933, @leseb)
• rgw: use trace logs for RGW admin HTTP info (#8937, @BlaineEXE)

Full Changelog: https://github.com/rook/rook/compare/v1.7.5...v1.7.6

Improvements

Rook v1.7.5 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• Update csi sidecar references to the latest versions (#8820, @humblec)
• No longer install the VolumeReplication CRDs from Rook (#8845, @travisn)
• Initialize rbd block pool after creation (#8923, @Rakshith-R)
• Close stdoutPipe for the discovery daemon (#8917, @subhamkrai)
• Add documentation to recover a pod from a lost node (#8742, @subhamkrai)
• Increasing the auto-resolvable alerts delay to 15m (#8896, @aruniiird)
• Change CephAbsentMgr to use 'up' query (#8882, @aruniiird)
• Adding 'namespace' field to the needed ceph queries (#8901, @aruniiird)
• Update period if period does not exist (#8828, @BlaineEXE)
• Do not fail on KMS keys deletion (#8868, @leseb)
• Do not build all the multus args to remote exec cmd (#8860, @leseb)
• Fix external script when passing monitoring list (#8807, @leseb)
• Use insecure TLS for bucket health check (#8712, @leseb)
• Add PVC privileges to the rook-ceph-purge-osd service account (#8833, @ashangit)
• Fix the example of local PVC-based cluster (#8846, @satoru-takeuchi)
• Add signal handling for log collector (#8806, @leseb)
• Prometheus rules format changes (#8774, @aruniiird)
• Add namespace to ceph node down query (#8793, @aruniiird)

Improvements

Rook v1.7.4 is a patch release limited in scope and focusing on small feature additions and bug fixes to the Ceph operator.

• Add missing error type check to exec (#8751, @BlaineEXE)
• Raise minimum supported version of Ceph-CSI to v3.3.0 (#8803, @humblec)
• Set the Ceph v16.2.6 release as the default version (#8743, @leseb)
• Pass region to newS3agent() (#8766, @thotz)
• Remove unnecessary CephFS provisioner permission (#8739, @Madhu-1)
• Configurable csi provisioner replica count (#8801, @Madhu-1)
• Allow setting the default storageclass for a filesystem in the helm chart (#8771, @kubealex)
• Retry object health check if creation fails (#8708, @BlaineEXE)
• Use the admin socket for the mgr liveness probe (#8721, @jmolmo)
• Correct the CephFS mirroring documentation (#8732, @leseb)
• Reconcile OSD PDBs if allowed disruption is 0 (#8698, @sp98)
• Add peer spec migration to upgrade doc (#8435, @BlaineEXE)
• Fix lvm osd db device check (#8267, @lyind)
• Refactor documentation to simplify for the Ceph provider (#8693, @travisn)
• Emphasize unit tests in the development guide (#8685, @BlaineEXE)

Improvements

Rook v1.6.10 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Reconcile OSD PDB if allowed disruptions are 0 (#8698, @sp98)
• Merge tolerations for the OSDs if specified in both all and osd placement (#8630, @subhamkrai)
• External cluster script compatibility with python2 (#8623, @aruniiird)
• Do not check ok-to-stop when OSDs are in CLBO (#8583, @leseb)
• Fix panic when recreating the csidriver object (#8582, @Madhu-1)

Improvements

Rook Ceph v1.7.3 is a patch release limited in scope and focusing on small feature additions and bug fixes.

• Cassandra and NFS have moved to their own repos. All improvements in this repo starting from this release will only be for the Ceph storage provider. (#8619, @BlaineEXE)
• Image list for offline installation can be found in images.txt (#8596, @subhamkrai)
• Add networking.k8s.io/v1 Ingress chart compatibility (#8666, @hall)
• Modify the log info when ok to continue fails (#8675, @subhamkrai)
• Print the output on errors from ceph-volume (#8670, @leseb)
• Add quota and capabilities configuration for CephObjectStore users (#8211, @thotz)
• Fix pool deletion when uninstalling a multus cluster configuration (#8659, @leseb)
• Use node externalIP if no internalIP defined (#8653, @JrCs)
• Fix CephOSDCriticallyFull and CephOSDNearFull monitoring alert queries (#8668, @Muyan0828)
• Fix CephMonQuorumAtRisk monitoring alert query (#8652, @anmolsachan)
• Allow an even number of mons (#8636, @travisn)
• Create a pod disruption budget for the Ceph mgr deployment when two mgrs are requested (#8593, @parth-gr)
• Fix error message in UpdateNodeStatus (#8629, @hiroyaonoe)
• Avoid multiple reconciles of ceph cluster due to the ipv4 default setting (#8638, @leseb)
• Avoid duplicate ownerReferences (#8615, @YZ775)
• Auto grow OSDs size on PVCs based on prometheus metrics (#8078, @parth-gr)
• External cluster configuration script fixed for backward compatibility with python2 (#8623, @aruniiird)
• Fix vault kv secret engine auto-detection (#8618, @leseb)
• Add ClusterID and PoolID mappings between local and peer cluster (#8626, @sp98)
• Set the filesystem status when mirroring is not enabled (#8609, @travisn)

Improvements

Rook v1.7.2 s a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Merge toleration for osd/prepareOSD pod if specified both places (#8566, @subhamkrai)
• Fix panic when recreating the csidriver object (#8582, @Madhu-1)
• Build with latest golang v1.16.7 (#8540, @BlaineEXE)
• Do not check ok-to-stop when OSDs are in CLBO (#8583, @leseb)
• Convert util.NewSet() to sets.NewString() (#8584, @parth-gr)
• Add support for update() from lib-bucket-provisioner (#8514, @thotz)
• Signal handling with context (#8441, @leseb)
• Make storage device config nullable (#8552, @BlaineEXE)
• Allow K8s version check on prerelease versions (#8561, @subhamkrai)
• Add permissions to rook-ceph-mgr role for osd removal in rook orchestator (#8568, @josephsawaya)
• Use serviceAccountName as the key in ceph csi templates (#8546, @humblec)
• Consolidate the calls to set mon config (#8590, @travisn)

NFS

• Upgrade nfs-ganesha to 3.5 version (#8534, @kam1kaze)

Improvements

Rook v1.6.9 s a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Make storage device config nullable (#8552, @BlaineEXE)
• Build with latest golang v1.16.7 (#8540, @BlaineEXE)
• Refuse to failover the arbiter mon on stretch clusters (#8520, @travisn)
• Add an option to enable/disable merge all placement (#8381, @subhamkrai)
• Update ancillary monitoring resources (#8406, @BlaineEXE)
• Updated mon health check goroutine for reconfiguring patch values (#8370, @parth-gr)
• Releases for v1.6 are now based on Github actions instead of Jenkins (#8525 #8564, @subhamkrai)

Improvements

Rook v1.7.1 s a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update Ceph CSI version to v3.4.0 (#8425, @Madhu-1)
• Add ability to specify the CA bundle for RGW (#8492, @degorenko)
• Remove unused mon timeout cli flags (#8489, @leseb)
• Add an option to enable/disable merge all placement (#8381, @subhamkrai)
• Refuse to failover the arbiter mon on stretch clusters (#8520, @travisn)
• Improve topology example of cluster on local pvc (#8491, @satoru-takeuchi)

Major Themes

v1.7.0 is a minor release with features primarily for the Ceph operator.

K8s Version Support

Kubernetes supported versions: 1.11 and newer.

Upgrade Guides

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph

Breaking Changes

Ceph

• Clusters with multiple filesystems will need to update their Ceph version to Pacific. The Operator configuration option ROOK_ALLOW_MULTIPLE_FILESYSTEMS has been removed in favor of simply verifying the Ceph version is at least Pacific where multiple filesystems are fully supported.

Features

Ceph

• Official Ceph images are now being published to quay.io. To pick up the latest version of Ceph, update your
  CephCLuster spec field image must be updated to point to quay. See the example cluster.
• Add support for creating Hybrid Storage Pools.
  • A hybrid storage pool creates a CRUSH rule for choosing the primary OSD for high performance
    devices (ssd, nvme, etc) and the remaining OSD for low performance devices (hdd).
  • See the design and Ceph docs for more details.
• Add support CephFS mirroring peer configuration. See the configuration for more details.
• Add support for Kubernetes TLS secrets for referring TLS certs needed for the Ceph RGW server.
• Stretch clusters are considered stable
  • Ceph v16.2.5 or greater is required for stretch clusters
• The use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. See the mirroring section in the CephBlockPool spec for more details.
• Add user data protection when deleting Rook-Ceph Custom Resources. See the design for detailed information.
  • A CephCluster will not be deleted if there are any other Rook-Ceph Custom resources referencing
    it with the assumption that they are using the underlying Ceph cluster.
  • A CephObjectStore will not be deleted if there is a bucket present. In addition to protection
    from deletion when users have data in the store, this implicitly protects these resources from
    being deleted when there is a referencing ObjectBucketClaim present.

Cassandra

• CRDs converted from v1beta1 to v1
  • Schema is generated from the internal types for more complete validation
  • Minimum K8s version for the v1 CRDs is K8s 1.16

NFS

• CRDs converted from v1beta1 to v1
  • Schema is generated from the internal types for more complete validation
  • Minimum K8s version for the v1 CRDs is K8s 1.16

Future Version Changes

To give you plenty of time to plan ahead, the following changes will be made in v1.8 (planned for November 2021):

• The minimum K8s version will be increased from v1.11 to v1.16
• Support for Ceph Nautilus will be removed
• The Rook Flex driver will be removed. Stay tuned for a tool to convert volumes from flex to CSI.

Improvements

Rook v1.6.8 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Re-enable lvm mode for OSDs on disks. See details to know if your OSDs are affected by unexpected partitions (#8319, @BlaineEXE)
• Update test to watch for v1 cronjob instead of v1beta1 (#8356, @travisn)
• Update PodDisruptionBudget from v1beta1 to v1 (#7977, @parth-gr)
• Add support for tls certs via k8s tls secrets for rgw (#8243, @thotz)
• Create correct ClusterRoleBinding for helm chart in namespace other than rook-ceph (#8344, @ashangit)
• If two mgrs, ensure services are reconciled with the cluster (#8330, @travisn)
• Proxy rbd commands when multus is enabled (#8339, @leseb)
• Proxy ceph command when multus is configured (#8272, @leseb)
• Ensure OSD keyring exists at OSD pod start (#8155, @subhamkrai)
• Add an example of a pvc-based ceph cluster on bare metal (#7969, @satoru-takeuchi)
• Mount /dev for the OSD daemon on lv-backed pvc (#8304, @satoru-takeuchi)
• Add ceph cluster context for lib bucket provisioning reconcile (#8310, @leseb)
• Create PDBs for all rgw and cephfs (#8301, @satoru-takeuchi)
• Always rehydrate the access and secret keys (#8286, @leseb)
• Fix PDB of RGW instances (#8274, @satoru-takeuchi)
• Ability to disable pool mirroring (#8215, @sp98)
• Fetch rgw port from the CephObjectStore the OBC (#8244, @thotz)
• Enable debug logging for adminops client log level is debug (#8208, @thotz)
• Update blockPoolChannel before starting the mirror monitoring (#8222, @sp98)
• Scaling down nfs deployment was failing (#8250, @travisn)

Improvements

Rook v1.6.7 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Ignore atari partitions for OSDs when scanning disks. This is a partial fix for multiple OSDs being created unexpectedly per disk, causing OSD corruption. See details to know if your OSDs are affected (#8195, @BlaineEXE)
• Update CSIDriver object from betav1 to v1 (#8029, @Yuggupta27)
• Retry cluster reconcile immediately after cancellation (#8237, @travisn)
• Avoid operator resource over-usage when configuring RGW pools and memory limits are applied (#8238, @subhamkrai)
• Remove k8s.io/kubernetes as a code dependency (#7913, @abursavich)
• Silence harmless errors if the operator is still initializing (#8227, @leseb)
• If MDS resource limits are not set, assign mds_cache_memory_limit = resource requests * 0.8 (#8180, @subhamkrai)
• Do not require rgw instances spec for external clusters (#8219, @leseb)
• Add tls support to external rgw endpoint (#8092, @leseb)
• Stop overwriting shared livenessProbe when overridden (#8206, @llamerada-jp)
• Update cluster-on-pvc example for proper OSD scheduling (#8199, @travisn)

Improvements

Rook v1.6.6 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update csi sidecar images to latest release (#8125, @Madhu-1)
• Update csi node-driver-registrar to latest release (#8190, @Rakshith-R)
• Evict a mon if colocated with another mon (#8181, @travisn)
• Enable logging in legacy LVM OSD daemons (#8175, @leseb)
• Do not leak key encryption key to the log (#8173, @leseb)
• Read and validate CSI params in a goroutine (#8140, @Rakshith-R)
• Only require rgw-admin-ops user when an RGW endpoint is provided (#8164, @leseb)
• Avoid unnecessary OSD restarts when multus is configured (#8142, @BlaineEXE)
• Use cacert if no client cert/key are present for OSD encryption with Vault (#8157, @leseb)
• Mons in stretch cluster should be assigned to a node when using dataDirHostPath (#8147, @travisn)
• Support cronjob v1 for newer versions of K8s to avoid deprecated v1beta1 (#8114, @subhamkrai)
• Initialise httpclient for bucketchecker and objectstoreuse (#8139, @thotz)
• Activate osd container should use correct host path for config (#8137, @travisn)
• Set device class for already present osd deployments (#8134, @degorenko)
• No need for --force when creating filesystem (#8130, @subhamkrai)
• Expose enableCSIHostNetwork correctly in the helm chart (#8074, @Madhu-1)
• Add RBAC for mgr to create service monitor (#8118, @travisn)
• Update operator internal controller runtime and k8s reference version (#8087, @subhamkrai)

Improvements

Rook v1.6.5 is a patch release limited in scope and focusing on small feature additions and bug fixes.

We are happy to announce the availability of a Helm chart to configure the CephCluster CR.
Please try it out and share feedback! We would like to declare it stable in v1.7.

Ceph

• Experimental Helm chart for CephClusters (#7778, @henryzhangsta)
• Disable insecure global id if no insecure clients are detected. If insecure clients are still required, see these instructions. (#7746, @travisn)
• Enable host networking by default in the CSI driver due to issues with client IO hangs when the driver restarts (#8102, @Rakshith-R)
• Add a disaster recovery guide for an accidentally deleted CephCluster CR (#8040, @travisn)
• Do not fail prepareOSD job if devices are not passed (#8098, @degorenko)
• Ensure MDS and RGW are upgraded anytime the ceph image changes (#8060, @sp98)
• External cluster config enables v1 address type when enabling v2 (#8083, @aruniiird)
• Create object pools in parallel for faster object store reconcile (#8082, @leseb)
• Fix detection of delete event reconciliation (#8086, @BlaineEXE)
• Use RGW admin API for s3 user management (#7998, @leseb)

Improvements

Rook v1.6.4 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Support for separate tolerations and affinities for rbd and cephfs CSI drivers (#8006, @Rakshith-R)
• Update ceph version to 15.2.13 (#8004, @leseb)
• External cluster upgrades fix for CRD schema (#8042, @leseb)
• Build with golang 1.16 instead of 1.15 (#7945, @subhamkrai)
• Retry starting CSI drivers on initial failure (#8020, @Rakshith-R)
• During uninstall stop monitoring rbd mirroring before cleanup (#8031, @leseb)
• Update the backend path for RGW transit engine (#8008, @thotz)
• If reducing mon count only remove one extra mon per health check (#8011, @travisn)
• Parse radosgw-admin json properly for internal commands (#8000, @satoru-takeuchi)
• Expand OSD PVCs only if the underlying storage class allow expansion (#8001, @iamniting)
• Allow the operator log level to be changed dynamically (#7976, @subhamkrai)
• Pin experimental volume replication to release-v0.1 branch (#7985, @Madhu-1)
• Remove '--site-name' arg when creating bootstrap peer token (#7986, @leseb)
• Do not configure external metric endpoint if not present (#7974, @leseb)
• Helm chart to allow multiple filesystems (#7930, @SerialVelocity)
• Rehydrate the bootstrap peer token secret on monitor changes (#7935, @leseb)

Improvements

Rook v1.5.12 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Fix OSD hostpath to prevent risk of data corruption on restart (#7886, @satoru-takeuchi)
• Double the mon failover timeout (to 20 minutes) during node drain (#7801, @sp98)
• Improve reliability of mon failover when the operator is restarted during failover (#7884, @travisn)
• Allow heap dump generation when logCollector sidecar is not running (#7847, @leseb)