Improvements

Rook v1.4.5 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Update the CSI driver to v3.1.1 (#6340, @Madhu-1)
• Fix drive group deployment failure (#6267, @BlaineEXE)
• Fix OBC upgrade from 1.3 to 1.4 external cluster (#6353, @leseb)
• Remove user unlink while deleting OBC (#6338, @thotz)
• Enable RBAC in the helm chart for enabling monitoring (#6352, @Strangerxxx)
• Disable encryption keyring parameter not necessary after opening block (#6350, @leseb)
• Improve reconcile performance in clusters with many OSDs on PVCs (#6330, @travisn)
• Only one external cluster secret supported in import script (#6343, @leseb)
• Allow OSD PVC template name to be set to any value (#6307, @travisn)
• OSD prepare job was failing due to low aio-max-nr setting (#6284, @leseb)
• During upgrade assume a pod spec changed if diff checking fails (#6272, @travisn)
• Merge config from rook-config-override configmap to the default global config file (#6252, @ashangit)

Improvements

Rook v1.4.4 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Upgrade to v1.4.3 for cluster-on-pvc hung due to changing label selectors on the mons (#6256, @travisn)
• Remove osd status configmap for nodes with long names (#6235, @fanjunwei)
• Allow running rgw daemons from an external cluster (#6226, @leseb)

Improvements

Rook v1.3.11 is a patch release limited in scope to a single bug fix.

Ceph

• The Ceph-CSI driver was being unexpectedly removed by the garbage collector in some clusters. For more details to apply a fix during the upgrade to this patch release, see these steps. (#6162, @Madhu-1)

Improvements

Rook v1.4.3 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• The Ceph-CSI driver was being unexpectedly removed by the garbage collector in some clusters. For more details to apply a fix during the upgrade to this patch release, see these steps. (#6162, @Madhu-1)
• Add storageClassDeviceSet label to osd pods (#6225, @subhamkrai)
• DNS suffix issue for OBCs in custom DNS suffix clusters (#6234, @subhamkrai))
• Cleanup mon canary pvc if the failover failed (#6224, @travisn)
• Only enable mgr init container if the dashboard is enabled (#6198, @leseb)
• cephobjectstore monitoring goroutine must be stopped during uninstall (#6208, @subhamkrai)
• Remove NParts and Cache_Size from MDCACHE block in the NFS configuration (#6207, @varshar16)
• Purge a down osd with a job created by the admin (#6127, @xenolinux)
• Do not use label selector on external mgr service (#6142, @leseb)
• Allow uninstall even if volumes still exist with a new CephCluster setting (#6145, @raghavendra-talur)

Improvements

Rook v1.4.2 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Improve check for LVM on the host to allow installing of OSDs (#6175, @leseb)
• Set the OSD prepare resource limits (#6118, @travisn)
• Allow memory limits below recommended settings (#6116, @travisn)
• Use full DNS suffix for object endpoint with OBCs (#6170, @leseb)
• Remove the CSI driver lifecycle preStop hook (#6141, @Madhu-1)
• External cluster optional settings for provisioners (#6048, @aruniiird)
• Operator watches nodes that match OSD placement rules (#6156, @galexrt)
• Allow user to add labels to the cluster daemon pods (#6084 #6082, @galexrt)
• Fix vulnerability in package golang.org/x/text (#6136, @subhamkrai)
• Add expansion support for encrypted osd on pvc (#6126, @leseb)
• Do not use realPath for OSDs on PVCs (#6120, @leseb)
• Example object store manifests updated for consistency (#6123, @subhamkrai)
• Separate topology spread constrinats for osd prepare jobs and osd daemons (#6103, @travisn)
• Pass CSI resources as strings in the helm chart (#6104, @Madhu-1)
• Improve callCephVolume() for list and prepare (#6059, @leseb)
• Improved multus support for the CSI driver configuration (#5740, @rohan47)
• Object store healthcheck yaml examples (#6090, @leseb)
• Add support for wal encrypted device on pvc (#6062, @leseb)
• Updated helm usage in documentation (#6086, @galexrt)
• More details for RBD Mirroring documentation (#6083, @galexrt)

Improvements

Rook v1.3.10 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Use full DNS suffix for object endpoint with OBCs (#6170, @leseb)
• Remove the CSI driver lifecycle preStop hook (#6141, @Madhu-1)
• Check for removed node in the crash collector controller (#6063, @leseb)
• External cluster optional settings for provisioners (#6048, @aruniiird)
• External cluster validation skips RGW bucket index (#6039, @leseb)
• External cluster fixed health check (#6018, @leseb)
• Require pod anti affinity for the CSI driver provisioners (#5462, @Madhu-1)

Improvements

Rook v1.4.1 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Integrated Ceph-CSI 3.1 with CephFS snapshotting and cloning. To enable this feature, if you had already updated to v1.4.0, re-apply upgrade-from-v1.3-apply.yaml in the upgrade guide. (#6017, @humblec)
• Enabled encryption of metadata OSDs on PVCs (#5977, @leseb)
• Fix psp clusterrole name for helm chart (#6028, @travisn)
• Mon failover should first scale down the failed mon before starting the new mon (#6029, @travisn)
• Use version of lvm command compatible with older versions of lvm (#6060, @leseb)
• Enable the object store in the Ceph dashboard (#4676, @thotz)
• Remove crash collector daemon on removed nodes (#6063, @leseb)
• Add support for wal device for OSDs on PVCs (#5580, @silverhandy)
• Check channel is open before closing during shutdown (#6033, @Madhu-1)
• Skip object validation for external clusters (#6039, @leseb)
• Add crushRoot to the pool CRD schema (#6014, @mridulv)
• Removed the storage provisioners usage from prometheus rules (#6019, @anmolsachan)
• Fix the health check for external clusters (#6018, @leseb)

Cassandra

• Fix a crash when applying annotations (#6056, @travisn)

Major Themes

v1.4 is a minor release with features primarily for the Ceph operator and critical updates to the NFS and Cassandra operators.

K8s Version Support

• Kubernetes supported versions: 1.11 and newer

Upgrade Guides

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph

Breaking Changes

Ceph

• Alpha snapshots from the Ceph-CSI driver are no longer supported. Only Beta snapshots are now supported.
• The operator will require the presence of the lvm2 package on the host where OSDs will run in some scenarios. If not available, the deployment will fail. The package is required for the OSDs to start after a node restart. This does not apply to OSDs created with the storageClassDeviceSet since v1.3.
• Several operator settings have moved from the operator deployment (operator.yaml) into the CephCluster CR:
  • Previous settings in the operator deployment: ROOK_CEPH_STATUS_CHECK_INTERVAL, ROOK_MON_HEALTHCHECK_INTERVAL, ROOK_MON_OUT_TIMEOUT. These settings will still be applied for backward compatibility, but it is recommended to update the CephCluster CR with the settings if you have changed the defaults.
  • For the new settings in the CephCluster CR, refer to the health settings
• RBD Mirroring: The settings for the RBD mirroring daemons have moved from the CephCluster CR into a new CephRBDMirror CR. If you have deployed RBD mirroring, you will need to create the new CR to manage the RBD mirroring daemons.
  To transition, you can inject the new rbd mirror CR with the desired count of daemons and delete the previously managed rbd mirror deployments manually.

Features

Ceph

• Ceph-CSI 3.0 is deployed by default
  • Multi Architecture docker images are published (amd64 and arm64)
  • Create/Delete beta snapshot for RBD, while support for Alpha snapshots is removed.
  • Create PVCs from RBD snapshots and PVCs
  • Support ROX volumes for RBD and CephFS
• The dashboard for the ceph object store will be enabled if the dashboard module is enabled.
• An admission controller enhances CRD validations (Experimental)
  • The admission controller is not enabled by default.
  • Support for Ceph CRDs is provided. Some validations for CephClusters are included and a framework for additional validations is in place for other CRDs.
• RGW Multisite is available through new CRDs for zones, zone groups, and realms. (Experimental)
• CephObjectStore CRD changes:
  • Health displayed in the Status field
  • Run health checks on the object store endpoint by creating a bucket and writing to it periodically.
  • The endpoint is stored for reference in the Status field
• OSD changes:
  • OSDs on PVC now support multipath and crypt device types.
  • OSDs on PVC can now be encrypted by setting encrypted: true on the storageClassDeviceSet.
  • OSDs can now be provisioned using Ceph's Drive Groups definitions for Ceph Octopus v15.2.5+.
  • OSDs can be provisioned on the device path such as /dev/disk/by-path/pci-HHHH:HH:HH.H with colons (:)
• A new CephRBDMirror CR will configure the RBD mirroring daemons. The RBD mirror settings were previously included in the CephCluster CR.
• Multus support is improved, though still in experimental mode
  • Added support for the Whereabouts IPAM
• CephCluster CRD changes:
  • Converted to use the controller-runtime framework
  • Added settings to configure health checks as well as pod liveness probes.
• CephBlockPool CRD has a new field called parameters which allows to set any Ceph pool property on a given pool
• OBC changes:
  • Updated the lib bucket provisioner version to support multithreading
  • Added support for quota, have options for object count and total size.
• Prometheus monitoring for external clusters is now possible, refer to the external cluster section
• The operator will check for the presence of the lvm2 package on the host where OSDs will run. If not available, the prepare job will fail. This will prevent issues of OSDs not restarting on node reboot.
• Added a new label ceph_daemon_type to Ceph daemon pods.
• Added a toolbox job example for running a script with Ceph commands, similar to running commands in the Rook toolbox.

Cassandra

There were two updates to the Cassandra operator with CVE fixes:

• Updated Jolokia javaagent from 1.6.0 to 1.6.2
• Updated Base image from Alpine 3.8 to 3.12

NFS

Several updates have been made in this release for the NFS operator:

• Conversion of the internal controllers to the K8s controller runtime.
• Creation of an admission controller to validate all input to the NFS CR.

Improvements

Rook v1.3.9 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Avoid removing a healthy mon unexpectedly when the operator restarts (#5868, @travisn)
• External cluster export script enhancements (#5952 #5931 #5833 #5899, @leseb @aruniiird)
• Toleration for CSI version check job (#5881, @Madhu-1)
• Remove obsolete CSI metadatastore and mountcache flags (#5870, @Madhu-1)
• Report cluster status error on heatlh check timeout (#5856, @leseb)
• Pool CR settings must be strings (#5849, @travisn)

Improvements

Rook v1.3.8 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Helm chart templating update (#5660, @nyarly)
• Update to the latest Ceph Nautilus release v14.2.10 (#5767, @travisn)
• Capture logging from the OSD daemons (#5770, @travisn)
• Ignore pod anti-affinity for the detect-version job placement (#5820, @tapih)
• Resync crds when updated from 1.2 (#5780, @leseb)
• Toolbox connection allowed for independent mode (#5819, @vasukulkarni)

Improvements

Rook v1.3.7 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Force delete stuck mons in every reconcile instead of only in the health check (#5720, @travisn)
• During mon failover remove the backing pvc (#5698, @travisn)
• Fix for the devicePathFilter during OSD creation (#5504, @satoru-takeuchi)
• External support for object store users (#5664, @leseb)
• Added warning and critical alerts for all the Ceph PVCs (#5682, @anmolsachan)
• Added CephClusterReadOnly alert and updated utilization alerts (#5673, @anmolsachan)

Cassandra

• Update base image to alpine:3.12 (#5749, @galexrt)

Improvements

Rook v1.3.6 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Increase OSD PVC size when the template size increases (#5309, @travisn)
• Update CSI driver to v2.1.2 (#5603, @Madhu-1)
• Autocorrect the mon failover id if incorrect (#5668, @travisn)
• Add the ability to set any pool property (#5623, @leseb)
• OSD metadata pvcs implemented with a map instead of an index (#5639, @travisn)
• Clean up OBC resources in case of failure (#5509, @thotz)
• Do not activate the crash mgr module (#5622, @leseb)
• Always enable the balancer mgr module for Octopus (#5622, @leseb)
• Configurable Service Port Names (#5595, @afflom)
• Ensure waiting for clean PGs during OSD upgrade in small clusters (#5606, @leseb)
• Increase the liveness probe start delay on OSDs (#5492, @leseb)

NFS

• Set the log level to info instead of debug (#5637, @aberfeldy)

Improvements

Rook v1.3.5 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Use internal node IP for mon endpoint when host networking enabled (#5594, @leseb)
• Set preferred anti-affinity for RGW when host networking not enabled (#5564, @ashangit)
• Do not delete OBC user for retain policy if the owner (#5511, @thotz)
• OBC additionalConfig field initialization (#5577, @leseb)
• Start the CSI driver in parallel of cluster configuration (#5572, @travisn)
• Handle node failure more gracefully (#5556, @travisn)
• Document how to find whether there is a filesystem on a device (#5557, @satoru-takeuchi)
• Cleanup disks during uninstall if the confirmation flag is set (#5257, @leseb)
• Add missing security context to crash collector for OpenShift (#5548, @leseb)
• Do not enforce osd-memory-target (#5529, @leseb)
• Default the device set template name to data (#5524, @travisn)

Improvements

Rook v1.3.4 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Finalizer for OBC cleanup (#5436, @sp98)
• Remove invalid MDS deactivate command during upgrade (#5278, @rajatsingh25aug)
• Enable verbose logging for LVM commands (#5515, @leseb)
• Set external creds if admin key is available (#5507, @leseb)
• Fail more gracefully for an unsupported Ceph version (#5503, @leseb)
• Set pg_num_min on new rgw metadata pools (#5489, @travisn)
• Object store deployment failed to start on openshift (#5468, @leseb)
• Relax OBC error handling and user deletion (#5465, @leseb)
• Create missing secret on external cluster (#5450, @leseb)
• Python script to generate needed external cluster resources (#5388, @aruniiird)
• Docs: clarify required version of helm for upgrades (#5445, @binoue)
• CSI priority class example update (#5443, @Madhu-1)
• Set test default pool size to one (#5428, @travisn)
• Remove invalid verbose params from lv activate (#5438, @travisn)

Improvements

Rook v1.3.3 is a patch release limited in scope and focusing on small feature additions and bug fixes.

Ceph

• Multipath support for OSDs on PVCs (#5386, @travisn)
• OSD on PVC with portable=false remained in pending state after node upgrade (#5420, @travisn)
• RGW with SSL certificate failing after upgrade (#5265, @thotz)
• Set ceph/ceph:v14.2.9 as the base image for the operator (#5413, @travisn)
• Allow the CSI driver version to be undetected if unsupported flag is set (#5412, @travisn)
• Allow setting PriorityClassName for CSI pods (#4848, @Madhu-1)
• Log the operator settings and their source (#5405, @travisn)
• Add support for OBCs on external cluster (#5372, @leseb)
• OSDs on PVs failing during deactivate if underlying device is unavailable (#5398, @leseb)
• Update CSI version to v2.1.1 (#5395, @Madhu-1)
• Force delete portable OSD pods if stuck (#5348 #5414, @travisn)
• Add resource request and limit for CSI pods (#5287, @Madhu-1)
• Enable pg_autoscaler module in default examples (#5343, @travisn)
• Deploying fails with runAsNonRoot when PSP enabled (#5293, @BlaineEXE)
• Remove specific rgw configuration on mon configuration database when scaling down (#5185, @ashangit)

EdgeFS

• Provide a way to configure ADS (#5360, @dyusupov)
• Check for empty raw disks during discovery (#5244, @dyusupov)

Improvements

Rook v1.3.2 is a patch release limited in scope and focusing on bug fixes.

Ceph

• MetadataDevice was failing the OSD creation (#5300, @travisn)
• Helm chart updates to be in sync with yaml manifests (#5297, @binoue)
• Update Ceph CSI driver to v2.1.0 (#5289, @Madhu-1)
• Restore the cluster-test.yaml example (#5305, @travisn)
• Properly reconcile CephCluster if named differently from namespace (#5302, @travisn)
• Enable running a toolbox job with the toolbox script (#5284, @travisn)
• Remove need for admin key for external cluster configuration (#4917, @leseb)
• Configure mgr modules in parallel of OSD reconcile (#5280, @travisn)
• Set default manifests to configure Ceph v14.2.9 (#5281, @travisn)
• Add requireSafeReplicaSize in CephObjectStore CRD (#5260, @zoetrope)
• Add seccomp annotations to the PSP (#5263, @surajssd)
• Revert mgr keyring to minimal privilege (#2529, @umangachapagain)
• Codegen needs to copy to pkg subfolder (#5218, @travisn)
• CephFS provisioner fails to provision a Persistent Volume after upgrading (#4740, @rajatsingh25aug)
• Add default value for the nodeSelector in the helm chart (#5236, @zerkms)
• Skip reconcile events where the spec is not updated (#5219, @travisn)
• Add support for updating filesystem pool replicated size (#3144, @rajatsingh25aug)
• Docs: Troubleshooting an OSD that can't be created on a previously configured device (#5209, @travisn)
• Docs: Fix secret env keys on ObjectBucketClaim (#5286, @terorie)

Improvements

Rook v1.3.1 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Stop the pool controller from staying in a reconcile loop (#5173, @leseb)
• Update the rgw service port during upgrade (#5228, @travisn)

Major Themes

• Ceph, EdgeFs, YugabyteDB, and Cassandra features.
• Minio operator removed from Rook.

Action Required

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph
• EdgeFS
• Cassandra

Breaking Changes

Ceph

Several breaking changes are introduced for the Ceph operator in this release. See the upgrade guide for migration instructions if these issues apply to your cluster.

• Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3.
• Support for directory-based (FileStore) OSDs is removed. Raw partitions are supported as an alternative.
• Support for OSDs created in Rook v0.9 or earlier is removed (created with Rook partitions instead of ceph-volume).

Minio

• The minio operator is removed from Rook

Notable Features

• Kubernetes supported versions are 1.11 and newer
  • Integration tests are run on 1.11, 1.13-1.18
• Rook is now built with Go modules in golang 1.13

Ceph

• Ceph Octopus is now supported, with continued support for Nautilus.
  • Ceph Nautilus 14.2.5 is the minimum supported version
  • Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3.
• OSD changes:
  • Added support for creating OSDs on raw partitions for Ceph v14.2.8 and newer.
  • Drop support for (a) unsafe directory-based OSDs, (b) Filestore OSDs (in sustaining mode in Ceph), and (c) Rook OSDs created in v0.9 or earlier. For more details refer to the corresponding issue.
  • OSD on PVC now supports a metadata device, refer to the cluster on PVC section or the corresponding issue.
  • OSD on PVC now supports PVC expansion, if the size of the underlying block increases the Bluestore main block and the overall storage capacity will grow up.
  • OSD on PVC doesn't use LVM anymore to configure OSD, but solely relies on the entire block device, done here.
  • Specific devices for OSDs can now be specified using the full udev path (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX) instead of the device name.
  • OSD on PVC CRUSH device storage class can now be changed by setting an annotation "crushDeviceClass" on the "data" volume template. See "cluster-on-pvc.yaml" for example.
  • Support for directory-based (FileStore) OSDs is removed. Raw partitions are supported as an alternative.
• Ceph-CSI changes:
  • CSI v2.0.0 is the minimum supported version
  • CSI v2.0.1 is the default version installed with Rook v1.3.0
  • CSI drivers can now be configured using "rook-ceph-operator-config" ConfigMap.
    ConfigMap can be used in mix with the already existing Env Vars defined in operator deployment manifest. Precedence will be given to ConfigMap in case of conflicting configurations.
• Rook will now refuse to create pools with replica size 1 unless requireSafeReplicaSize is set to false.
• Rook Ceph cleanupPolicy will clean up the dataDirHostPath only after user confirmation. For more info about CleanUpPolicy read the design as well as the documentation cleanupPolicy
• Rook monitor, mds and osd now have liveness probe checks on their respective sockets
• Object stores can be created without specifying pools in the CephObjectStore CR to allow for pool management outside of Rook such as in the dashboard.
• Operator logs are more concise with the Ceph commands only printing at debug level.
• Various improvements to the integration tests including tests for an external cluster and for a cluster running on PVCs.
• Pools can now be configured to inline compress the data using the compressionMode parameter. Support added here
• Ceph OSDs in Octopus do not use the host PID namespace, but the PID namespace of the pod (more security). The OSD does not see running host processes anymore.
• Placement of all the ceph daemons now supports topologySpreadConstraints.
• Rook is now capable of working with Multus to expose dedicated interfaces to pods. For more information please refer to the network configuration doc.

EdgeFS

• Enhanced support for Time zones: Geo-distributed name space can be properly identified in case of Kubernetes stretched clusters.
• Microsoft Windows CIFS/SMB CRD: Orchestrate Windows CIFS/SMB Scale-Out service managed by Rook SMB CRD.

YugabyteDB

• Master and TServer pods for YugabyteDB will have resources requests and limits specified as per YugabyteDB recommendations. This will help avoid the soft/hard memory limit issue.

Cassandra

• Added JMX Prometheus exporter support.

Improvements

Rook v1.2.7 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Apply the expected lower PG count for rgw metadata pools (#5091, @travisn)
• Reject devices smaller than 5GiB for OSDs (#5089, @leseb)
• Add extra check for filesystem to skip boot volumes for OSD configuration (#5022, @leseb)
• Avoid duplication of mon pod anti-affinity (#4998, @travisn)
• Update service monitor definition during upgrade (#5078, @umangachapagain)
• Resizer container fix due to misinterpretation of the cephcsi version (#5073, @Madhu-1)
• Set ResourceVersion for Prometheus rules (#4528, @galexrt)
• Upgrade doc clarification for RBAC related to the helm chart (#5054, @PCatinean)

Improvements

Rook v1.2.6 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Update default Ceph version to v14.2.8 (#4960, @leseb)
• Fix for OSDs on PVCs that were crashing on Ceph v14.2.8 (#4960, @leseb)
• Mount /udev so the osds can discover device info (#5001, @leseb)
• Query for the topology.kubernetes.io labels in K8s 1.17 or newer for the CRUSH hierarchy (#4989, @travisn)
• Log a warning when useAllNodes is true, but nodes are defined in the cluster CR (commit, @travisn)