Improvements

Rook v1.12.6 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• osd: Prevent to fallback to LVM mode in OSD on LVM creation (#12165, @satoru-takeuchi)
• build: Reduce Go module dependencies significantly by removing unnecessary package dependencies (#13035, @BlaineEXE)
• osd: Use bash script to run OSDs if the flappingRestartIntervalHours is set (#13031, @sp98)
• mon: Write the assimilate-conf config to a valid operator working directory (#13033, @travisn)
• mds: improve the MDS liveness probe calls (#12860, @parth-gr)
• csi: Add mountinfo path for the CephFS daemonset (#13011, @Madhu-1)
• osd: Print warning message if no matching node found in nodes list for an OSD (#13046, @wb14123)

Improvements

Rook v1.12.5 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• external: Fix return value for client.healthchecker user keyring (#13001, @parth-gr)
• object: Create cosi user for each object store (#12633, @thotz)
• mgr: Allow more than two mgrs (#12895, @travisn)
• core: Use crash profile in crash daemon keyring (#12970, @travisn)
• multus: Allow node profiles in validation test (#12968, @BlaineEXE)
• file: Disable active standby when set to false (#12981, @travisn)
• mgr: Adding support for prometheus endpoint configuration (#12926, @rkachach)
• external: Update healthchecker caps for rbd command (#12941, @subhamkrai)
• docs: Review the teardown guide for updated guidelines (#12921, @travisn)
• osd: Support encrypted OSDs on partitons (#12924, @satoru-takeuchi)
• helm: Clusterrole to add objectbucketclaims to user facing roles (#12329, @jouve)
• build: update s5cmd to version v2.2.1 (#12898, @obnoxxx)
• core: Import packages only once (#12909, @testwill)
• docs: Improving minikube developer setup documentation (#12984, @rkachach)

Improvements

Rook v1.12.4 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• core: Restart ceph daemons when network updated (#12791, @subhamkrai)
• helm: Update docs/comments to reflect reef support (#12872, @mjnagel)
• exporter: Bind to all interfaces if IPv6 is enabled (#12886, @matthewpi)
• object: Unique username for OBC even when preceding OBC was retained (#12884, @haslersn)
• multus: Allow using NADs without inspectable CIDRs (#12778, @BlaineEXE)
• discover: Fix discover daemonset default value for discovery interval (#12844, @lbailleul)
• operator: Move discover pod settings to the operator configmap (#12840 #12691, @subhamkrai)
• build: Correct release version for toolbox based on operator in release-1.12 branch (#12853, @travisn)
• osd: Replace existing OSDs to be upgraded when new store is available (#12770, @sp98)
• ci: Simplify docs-check workflow by extracting functionality (#12816, @obnoxxx)
• csi: Enable csi-addons-side when dependent CRDs are deployed (#12487, @subhamkrai)
• monitoring: Set port for exporter servicemonitor (#12825, @weirdwiz)

Improvements

Rook v1.11.11 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• object: Unique username for OBC even when preceding OBC was retained (#12884, @haslersn)
• object: Avoid creating same bucket for two different OBC (#12804, @thotz)
• csi: Add csi pods to the list to force delete pod on an unavailable node (#12681, @Madhu-1)
• operator: Fix formatting of some logger methods (#12666, @polyedre)

Improvements

Rook v1.12.3 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• object: Avoid creating same bucket for two different OBCs (#12804, @thotz)
• object: Allow creating an object user in different namespace (#12730, @travisn)
• external: rename --cluster_name parameter to --k8s_cluster_name (#12811, @parth-gr)
• core: Return valid CIDR IP for blocklisting (#12797, @Madhu-1)
• ci: Add K8s v1.28 to test suite (#12748, @subhamkrai)
• csi: Set ceph cluster as ControllerRef for holder daemonset (#12724, @Madhu-1)
• build: Add support for Go version 1.21 in addition to 1.20 (#12788, @subhamkrai)
• build: Update go modules (#12663, @BlaineEXE)
• build: Remove go modules under apis directory (#12764, @travisn)
• helm: Documentation uses values.yaml instead of values-override.yaml (#12734, @parth-gr)
• osd: Implement replacing OSDs for when the backend store may change in future (#12507, @sp98)
• mon: Support mon backing storage in different storage classes across zones (#12384, @ideepika)
• monitoring: Add "for" to CephOSDDownHigh alert (#12731, @cjyar)
• operator: Log operator config configmap settings only when changed (#12679, @subhamkrai)
• build: Update vault and other indirect imports (#12672, @subhamkrai)
• helm: Support helm chart install in OpenShift clusters (#12675, @parth-gr)

Improvements

Rook v1.12.2 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• file: Handle filesystem add pool error for Reef (#12725, @Madhu-1)
• nfs: Run dbus-daemon sidecar as dbus user instead of root (#12698, @nixpanic)
• multus: Add sample job manifest for multus config validation (#12495, @Nikhil-Ladha)
• csi: Add CSI pods to the list to force delete when stuck terminating (#12681, @Madhu-1)
• csi: Configurable maxUnavailable for CephFS CSI Daemonset for faster updates (#12678, @karthik-us)
• mon: Cleanup ParseMonEndpoints from mon package (#12561, @henrydavies1)
• helm: Configure additional ServiceMonitor labels (#12605, @jouve)
• operator: Fix formatting of some logger methods (#12666, @polyedre)
• object: Fix s5cmd documentation for s3 endpoint verification (#12624, @parth-gr)

Improvements

Rook v1.12.1 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• docs: Add note in upgrade guide about cephfs driver's breaking change (#12615, @Rakshith-R)
• osd: Use cp -a command for copying init-containers binary (#12501, @Sheetalpamecha)
• external: added an optional flag for enabling v2 mon port (#12534, @Nikhil-Ladha)
• external: set RequireMsgr2 if v2 port is configured (#12537, @parth-gr)
• multus: Revise kube config/client loading (#12611, @BlaineEXE)
• nfs: Fix null pointer deref in generateSssdSidecarResources() (#12617, @spuiuk)
• external: Create cluster namespace for the external cluster (#12508, @parth-gr)
• object: Update ceph COSI driver image to v0.1.1 (#12562, @thotz)
• object: Do not add objectstore endpoint to zone when sync is disabled (#12560, @polyedre)
• nfs: mount krb5.conf and krb5.keytab in the sssd container (#12517, @spuiuk)
• core: Force delete rook-ceph-exporter pod on a down node (#12575, @sp98)
• rbd: Skip PV during node fencing when PV is not backed by Ceph-CSI (#12563, @subhamkrai)
• build: COSI driver is missing from OLM list (#12592, @subhamkrai)
• ci: add detached mode in tmate (#12579, @subhamkrai)

Upgrade Guide

To upgrade from previous versions of Rook, see the Rook upgrade guide.

Breaking Changes

• The minimum supported Kubernetes version is v1.22
• The minimum supported Ceph-CSI driver is 3.8
• Version of golang supported during development is v1.20
• For CephObjectStores, if rgw_run_sync_thread was set manually via ceph config set ...
  the value will be overridden by Rook according to disableMultisiteSyncTraffic

Features

• Add support for Ceph Reef (v18)
• Ceph COSI Driver for provisioning buckets is available in experimental mode
• Ceph CSI version by default is v3.9
• Automate recovery of RBD (RWO) volumes following node loss
  • Requires CSI-add-ons
  • Requires the K8s v1.26 feature for non-graceful node shutdown
• Multus network validation tool
• Security improvements by dropping container capabilities
• Allow disabling ObjectBucketClaim and ObjectBucketNotification controllers
• Various improvements for the external Ceph cluster configuration script
• NFS
  • Add experimental support for RGW as a backend for CephNFSes
  • Add support for the monitoring endpoint present in NFS-Ganesha v5.1 (Ceph Reef v18 and up)
  • Fix various bugs in kerberos support

Improvements

Rook v1.11.10 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• object: Align spelling of user admin capabilities with ceph (#12460, @pgoron)
• csi: Update csi holder daemonset template for updating at next restart (#12462, @Madhu-1)
• core: Remove vault server module dependencies (#12455, @travisn)
• external: Support custom ceph keyring file (#12502, @neoaggelos)
• external: Pin the default csi subvolume (#12477, @parth-gr)
• external: Add support for IPV6 ceph daemons (#12143, @parth-gr)
• external: Make the import script idempotent (#12457, @mysiki)
• ci: Fix ci test encryption-pvc-kms-vault-token-auth (#12467, @subhamkrai)

Improvements

Rook v1.11.9 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• multus: Fix "deletecollection" permission not present (#12437, @sudharsanomprakash)
• dashboard: Remove deprecated kubernetes.io/ingress.class annotation (#12418, @Jeansen)
• external: Make import script idempotent (#12417, @parth-gr)
• exporter: Ignore failed deletion of service monitor (#12430, @travisn)
• multus: Add config file for validation tool (#12396, @BlaineEXE)
• object: Clarify success message when reconciling CephObjectStoreUser (#12406, @polyedre)
• docs: Update storage architecture diagram (#12252, @galexrt)
• operator: Add ceph image version label to PVC (#12372, @YZ775)
• object : Add SSL ref in cephobjectstore user secret (#12341, @thotz)

Improvements

Rook v1.11.8 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• helm: add EC Block Pool config in helm chart (#12324, @Javlopez)
• pool: Add .mgr pool to the stretch cluster examples (#12360, @travisn)
• nfs: Add Spec.Security.Kerberos.DomainName to the CRD to configure /etc/idmapd.conf (#12220, @spuiuk)
• mgr: Removing unnecessary rook-ceph-mgr rbac entries (#12337, @rkachach)
• core: typo in logs to print fullname of CephCluster (#12217, @takirala)
• core: empty ceph-daemons-sock-dir for osd onPVC (#12299, @avanthakkar)
• docs: prevent to delete other clusters data on cluster deletion (#12334, @satoru-takeuchi)
• docs: improve external doc format (#12383, @parth-gr)
• docs: Suggest qemu driver for minikube on apple silicon (#11722, @BlaineEXE)

Improvements

Rook v1.11.7 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• core: Delete exporter resources if ceph version is not supported (#12271, @avanthakkar)
• external: FQDN should be persisted instead of using the ip endpoint (#12264, @parth-gr)
• object: Implement more capabilities for object store users (#12256, @thotz)
• test: Add CI e2e test for multus validation test (#12282, @BlaineEXE)
• core: Use default-* logging flags for ceph daemons so they can be overridden (#12302, @Javlopez)
• helm: Add exporter resource entry to ceph cluster documentation (#12251, @galexrt)
• mgr: Allow other namespaces in the ServiceMonitor resource (#12293, @kerryeon)
• object: Add missing cephcluster spec addition in object controller (#12273, @thotz)
• monitoring: Service monitor should not use mgr_role label (#12268, @travisn)
• test: Allow specifying custom nginx image for multus validation (#12231, @iPraveenParihar)
• operator: Pull multus validation test images before test (#12211, @BlaineEXE)
• rbdmirror: Ensure rbd mirror daemon is upgraded (#12247, @travisn)

Improvements

Rook v1.11.6 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• osd: Support expanding lvm osd on pvc (#12164, @satoru-takeuchi)
• monitoring: Skip creating the service monitor for the exporter if monitoring is not enabled (#12216, @travisn)
• docs: Generate documentation for CRDs (#12110 #12179, @Javlopez)
• core: Add termination grace period for exporter pods (#12215, @avanthakkar)
• csi: servicemonitor for rook-ceph csi drivers (#12170, @jouve)
• monitoring: Configurable option to disable prometheus metrics (#12193, @travisn)
• mgr: Default to active mgr label if only one mgr is running (#12137, @travisn)
• osd: Allow scanning devices with filter (#11976, @Javlopez)
• core: Disable controller runtime metrics server (#12194, @Madhu-1)
• mgr: Use mgr_role dynamic label to tag the active ceph manager (#11845, @rkachach)
• operator: use KUBECONFIG context for cli if present (#12192, @BlaineEXE)
• external: fix rgw multisite config check (#12182 #12238, @parth-gr)
• operator: validate multus validation networks in cli (#12187, @BlaineEXE)
• operator: Fix package logger name for rookcli (#12186, @BlaineEXE)
• ceph: Unset the encryption configuration before updating the setting (#12181, @Madhu-1)

Improvements

Rook v1.11.5 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• mgr: Retry creating ceph dashboard credentials (#12149, @parth-gr)
• nfs: Reduce size CephNFS CRD from unnecessary file volume sources (#12155, @BlaineEXE)
• core: Update k8s API references to more recent version (#12161, @subhamkrai)
• test: Add multus validation test routine to rook binary (#12069, @BlaineEXE)
• external: check that the pool and cluster name is provided (#12132, @parth-gr)
• core: Skip OBC controllers if not needed based (#12075, @sp98)
• Add an ingress for Ceph object stores (#12109, @jouve)
• core: Disable the exporter service (#12118, @avanthakkar)
• nfs: Fixes for mounting CephNFS using Kerberos auth (#12086, @spuiuk)

Improvements

Rook v1.11.4 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• core: Update default image to Ceph v17.2.6 (#12068, @travisn)
• core: Disable the Ceph exporter daemon (#12077, @avanthakkar)
• helm: Add option to scale down rook operator (#12048, @TomHellier)
• helm: Drop snapshot.storage.k8s.io/v1beta1 (#12051, @sathieu)
• external: Add support for RGW multisite in external cluster script (#12037, @parth-gr)
• external: Do not require the monitoring endpoint (#12061, @neoaggelos)
• external: Allow creating pools with special characters in name (#12056, @parth-gr)
• external: Do not enforce rbd, cephfs and rgw flags for the external cluster (#12028, @parth-gr)
• core: Use cluster ID for ns lookup on exported multi-cluster service (#12064, @sp98)
• docs: Add scenario for deleted namespace to the disaster recovery guide (#11895, @gaord)
• mgr: Failed to update the port of dashboard (#11932, @zhucan)

Improvements

Rook v1.11.3 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• csi: Make AttachRequired as configurable for RWX volumes (#11899, @Madhu-1)
• nfs: Add support for nfs-ganesha metrics monitoring (#12007, @synarete)
• mgr: Add option to disable the prometheus mgr module (#11980, @thenamehasbeentake)
• object: Check OBC provisioner for bucket notification (#11975, @thotz)
• external: Make rgw call separate from cephfs and rbd in export script (#11947, @parth-gr)
• core: Update vault pkg to 1.13.1 (#12013, @subhamkrai)
• core: Fix config format for msgr2 ipv6 monitors (#11993, @heliochronix)
• osd: Handle global or node-local device class configuration correctly (#11966, @satoru-takeuchi)
• csi: IPv6 compatibility for requiring msgr2 (#11992, @travisn)
• mon: Remove condition to use 6790 mon port (#11963, @sp98)

Improvements

Rook v1.10.13 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• osd: Handle global or node-local device class configuration correctly (#11966, @satoru-takeuchi)
• manifest: Add missing quote (#11880, @DjVinnii)
• object: Make OBC genUserID unique across clusters (#11665, @BlaineEXE)
• file: Check if the filesystem exists before checking dependencies (#11221, @zhucan)
• core: On crash pod ensure rook version label is not set (#11760, @gaord)

Improvements

Rook v1.11.2 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• osd: Implemented encryption key rotation (#11749, @Rakshith-R)
• core: Remove unnecessary ceph-conf-dir volume mount from exporter (#11950, @avanthakkar)
• core: Set key rotation default in code instead of in CRDs (#11951, @travisn)
• external: Use f-strings for formatting (#11944, @Sheetalpamecha)
• core: Use msgr2 if compression is enabled (#11928, @uhthomas)
• ci: Skip building csv on arm64 (#11906, @subhamkrai)
• osd: Validate and remove duplicate topology labels (#11823, @parth-gr)
• rgw: RGW dashboard can be disabled in the object CR (#11908, @thenamehasbeentake)
• external: Pool and metadata EC pools were reversed in scripts (#11919, @dragon2611)
• rgw: Skip objectstore name length validation when cluster is external (#11911, @parth-gr)
• nfs: Network mode can be set separately for cephcluster and nfs (#11777, @taxilian)
• csi: Update port to 3300 if msgr2 is required (#11859, @travisn)
• core: Add FSID to the additionalPrinterColumns on cephcluster CRD (#11864, @thenamehasbeentake)
• core: Add missing labels in exporter deployment (#11866, @avanthakkar)

Improvements

Rook v1.11.1 is a patch release limited in scope and focusing on feature additions and bug fixes to the Ceph operator.

• ceph: Fix host networking by only adding OSD ports when required for multi-cluster config (#11797, @sp98)
• core: Ceph exporter requires ceph config where OSDs are not running (#11848, @avanthakkar)
• monitoring: Remove prometheus alerts that don't apply to rook (#11842, @travisn)
• mgr: Revert readiness probe and go back to the original sidecar HA implementation (#11829, @rkachach)
• manifest: Align whitespace in example cluster.yaml (#11804, @gauravsitlani)
• external: Add realm support for external cluster (#11584, @parth-gr)
• object: Make OBC genUserID unique across clusters (#11665, @BlaineEXE)
• file: Check if a filesystem exists before checking dependencies during deletion (#11221, @zhucan)
• core: On crash pod ensure rook version label is not set (#11760, @gaord)

Upgrade Guide

To upgrade from previous versions of Rook, see the Rook upgrade guide.

Breaking Changes

• The minimum version of K8s version supported is v1.21.
• The minimum version of the Ceph-CSI driver is v3.7.
• Removed support for MachineDisruptionBudgets, including settings removed from the CephCluster CR:
  • manageMachineDisruptionBudgets
  • machineDisruptionBudgetNamespace
• Versions of golang supported during development are v1.19 and v1.20.

Features

• Ceph-CSI v3.8 is now the version deployed by default with Rook. The driver has a number of important updates to add more storage features available to clients.
• Added setting requireMsgr2 on the CephCluster CR to allow clusters with a kernel of 5.11 or newer to fully communicate with msgr2 and disable the msgr1 port. This allows for more flexibility to enable msgr2 features such as encryption and compression on the wire.
• Change pspEnable default value to false in helm charts, and remove documentation for enabling PSP.
  If still using a version of K8s where PSPs are required, see the v1.10 documentation.
• Object store bucket notifications and topics are now marked as stable features.
• The Ceph exporter daemon is configured as the source of metrics based on performance counters from Ceph daemons. The exporter daemon
  provides more scalability of metrics collection to reduce load on the Ceph mgr.
• Read affinity for RBD volumes is now available, leveraging the krbd map options to allow serving reads from an OSD in proximity to the client, according to OSD locations defined in the CRUSH map and topology labels on nodes.
• Mirroring data across clusters with overlapping networks is now supported. Mon and OSD services will be configured with global IPs across multiple clusters with overlapping CIDRs. The clusters must be configured using an MCS API-compatible applications such as submariner globalnet. This feature is supported for Ceph version v17.2.6 or later.
• The Ceph Mgr standby now is managed with a readiness probe instead of a sidecar. Note that the standby mgr is expected to fail the readiness probe, while the active mgr passes the readiness probe.