OpenEBS v4.6.1 is a patch release on top of v4.6.0, delivering targeted fixes and minor enhancements across the storage engines.
This release includes new patch releases of Replicated PV Mayastor (v2.12.1), Local PV LVM (v1.10.1), Local PV ZFS (v2.11.1) and Local PV Rawfile (v0.15.1). Local PV Hostpath remains at v4.6.0.
Add a --default-format-options node agent argument (lvmNode.defaultFormatOptions in the Helm chart) to set node-level default mkfs options per filesystem, used when a StorageClass does not set formatOptions (openebs/lvm-localpv#506)
Note for users on older kernels: mkfs.xfs 6.5+ enables the nrext64 feature by default, which only kernel 5.19+ can mount. On clusters with older kernels, set lvmNode.defaultFormatOptions.xfs: "-i nrext64=0" (or the per-StorageClass formatOptions) to keep XFS volumes mountable.
ZFS
Add a formatOptions StorageClass parameter for extra mkfs options on first format, and a --default-format-options node agent flag (zfsNode.defaultFormatOptions in the Helm chart) for per-filesystem defaults; ignored when fstype is zfs (openebs/zfs-localpv#760)
Go module path moved to github.com/openebs/zfs-localpv/v2 for semantic import versioning; affects only Go library consumers (openebs/zfs-localpv#756)
The same older-kernel note as LVM above applies; set zfsNode.defaultFormatOptions.xfs: "-i nrext64=0" if any node runs a kernel older than 5.19.
Optional CSI Snapshot Controller - Replicated PV Mayastor, Local PV LVM, Local PV ZFS, and Local PV Rawfile now allow you to disable the bundled CSI snapshot controller through the Helm chart. This avoids conflicts in clusters where a snapshot controller is already managed at the cluster level.
Enhancements
Analytics Configuration Overrides - The OpenEBS Helm charts now accept global overrides for the analytics identifier and key values, making analytics configuration consistent across the Replicated PV Mayastor, Local PV Hostpath, Local PV LVM, Local PV ZFS, and Local PV Rawfile charts.
Replicated Storage (Mayastor)
New Features
Eventing Aggregator
Replicated PV Mayastor now includes the Eventing Aggregator, a new component that collects cluster events and makes them available for querying through the get events command in the Mayastor kubectl plugin. Events can be retrieved from Loki, directly from NATS JetStream, or from a previously collected support bundle for offline analysis, and can be filtered by category, action, node, target, component, pool, volume, replica, and state. Cluster events are also included in the support bundle produced by kubectl openebs dump system. The Eventing Aggregator is enabled by default and works with or without Loki deployed.
Best-Effort Snapshot Restore Policy
A new snapshotRestorePolicy StorageClass parameter controls how a snapshot restore behaves when not every replica pool can host a clone. With the default strict policy, every requested replica must be cloned from the snapshot. With bestEffort, the restore proceeds as long as at least one clone succeeds; the volume comes up under-replicated and the remaining replicas are filled in through a normal rebuild. This allows a restore to succeed when a source pool is full or otherwise unable to host a clone.
DiskPool Handle Rescanning
Replicated PV Mayastor now periodically rescans DiskPool backing device file handles to detect hot-removal on devices that require I/O to surface a removal event, such as those using the AIO and io_uring backends. Previously, removal of an idle device could go undetected. The rescan also refreshes the reported disk size, and both the behaviour and its interval are configurable through the Helm chart.
DiskPool Error Clearing
Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin. This allows a pool to be returned to normal reporting after the underlying condition has been resolved.
Volume and Nexus Label V2
A new versioned on-disk label layout, V2, is introduced alongside the existing V1 layout. V2 reduces the metadata partition reserved at the front of every replica from 4 MiB to 3 MiB and aligns volume capacity to 1 MiB boundaries. It also resolves block-mode backup compatibility with Kasten K10.
For a V2 volume, the requested size is rounded up to the next 1 MiB boundary. A size that is already a multiple of 1 MiB is unchanged, so a request for 10Gi stays 10Gi, while a decimal quantity such as 10G is not MiB-aligned and is rounded up. Each replica additionally reserves 8 MiB for the metadata partition, and the replica total is then aligned up to the cluster size of the pool, which is 4 MiB by default. For example, a 10 MiB volume on a pool with the default cluster size is exposed as a 10 MiB device while each replica consumes 20 MiB of pool space.
A volume created with a size that is not 1 MiB aligned reports a slightly larger size than requested, which is expected behaviour. Because V2 also adds the 8 MiB metadata reservation for every replica, a V2 volume consumes more pool space than the raw requested size, so size your pools accordingly.
The cluster-wide label version is negotiated automatically as the lowest version supported across all registered io-engines, and only ever moves up, so the control plane never creates a label that a node cannot understand. In a cluster where some nodes do not yet support V2, the negotiated version stays at V1 until every node supports it. Existing volumes remain on V1 and require no migration, and resize and snapshot operations preserve the label version of the volume.
Enhancements
TLS Hardening and Certificate Auto-Reload
Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Certificates are hot-reloaded on rotation without a restart, TLS discovery is enabled by default, plain HTTP on the REST service is restricted to health probes only, and the Helm chart exposes the TLS configuration for the public API.
RDMA Capability Detection
The CSI node now checks for the nvme_rdma kernel module before reporting RDMA capability, and node transport capabilities are propagated through registration. Previously a node could be treated as RDMA-capable without the required kernel support.
Asynchronous Bdev Destruction
Block device destruction in the io-engine is now asynchronous, improving the responsiveness of pool and replica teardown.
Additional Helm Chart Options
Additional environment variables are exposed for component customisation, the api-rest health probes use an initialDelaySeconds of 1 for faster readiness, and the maximum Loki ingestion limits have been increased.
Clearer Impact Reporting for Purge Operations
Node and DiskPool purge operations that involve data loss now list the affected volumes and snapshots directly in the reported error, instead of requiring a separate --show-impact run. Snapshot impact is now included alongside volume impact.
Pool Identification on Replica Metrics
Replica metrics now carry pool_name and pool_uuid labels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation.
Bug Fixes and Improvements
Volume Expansion with Undersized Replicas
Resolved an issue where a volume expansion could fail to complete when one or more replicas had not yet been resized. Undersized replicas are now resized before the nexus resize is retried.
Pool Availability During Device Removal and Replica Deletion
Resolved several issues affecting DiskPool availability during device and replica lifecycle events, including a race between pool deregistration and reload, and a race when listing pools while replicas were being destroyed.
Frozen I/O During Nexus Shutdown
Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, child device closure is awaited, and devices are no longer detached on transient I/O submission errors.
Nexus Size Miscalculation
Resolved an issue where the nexus block device size could be off by one block. Volume sizing now accounts for label metadata, so the usable capacity always meets the requested size.
Pool Capacity Accounting for Snapshot Clones
Resolved an issue where pool capacity tallying did not account for the creation of snapshot clones, which could lead to over-commitment of pool space.
Readiness Probe Startup Race
Resolved an issue where a readiness failure could be cached before the first successful probe, causing api-rest to be incorrectly reported as not ready.
Node Rebuild Count Accuracy
Resolved an issue where the node rebuild count was not refreshed on single nexus updates, improving the accuracy of rebuild throttling.
SPDK Fixes
Updated SPDK with fixes for a null pointer dereference and IPv6 transport handling, and resolved an interrupt-mode reactor teardown issue.
Limitations
The IO engine fully utilizes all allocated CPU cores regardless of the actual I/O load, as it runs a poller at full speed.
Each DiskPool is limited to a single block device and cannot span across multiple devices.
The data-at-rest encryption feature does not support rotation of Data Encryption Keys (DEKs).
Known Issues
If a node hosting a pod reboots and the pod lacks a controller (like a Deployment), the volume unpublish operation may not trigger. This causes the control plane to assume the volume is still in use, which leads to fsfreeze operation failure during snapshots.
Workaround: Recreate or rebind the pod to ensure proper volume mounting.
Large pools (for example, 10–20 TiB) may experience extended recovery times after a dirty shutdown of the node hosting the io-engine.
When using Replicated PV Mayastor on Oracle Linux 9 (kernel 5.14.x), servers may unexpectedly reboot during volume detach operations due to a kernel bug (CVE-2024-53170) in the block layer.
This issue is not caused by Mayastor but is triggered more frequently because of its NVMe-TCP connection lifecycle.
Workaround: Upgrade to kernel 6.11.11, 6.12.2, or later, which includes the fix.
Local Storage
ZFS
New Features and Enhancements
Dataset Tuning Parameters for Local PV ZFS - Local PV ZFS StorageClasses now support the atime and logbias parameters, giving you direct control over access-time updates and write-workload optimisation on the underlying ZFS datasets and volumes.
Bug Fixes and Improvements
Image URL Rendering for Local PV ZFS - Resolved an issue where image URLs in rendered manifests were not quoted, which could break rendering for registries whose URLs contain characters that YAML treats specially.
LVM
New Features and Enhancements
Updated CSI Snapshot Components for Local PV LVM - The bundled csi-snapshotter and snapshot-controller components have been updated to v8.2.0.
Bug Fixes and Improvements
Idempotent Volume Expansion for Local PV LVM - Resolved an issue where repeating a volume expansion could behave inconsistently. Node expansion now remains required during resize, so repeated expand operations are idempotent.
ServiceMonitor Manifest Rendering for Local PV LVM - Resolved an issue where the namespace field in the generated ServiceMonitor manifest was incorrectly indented.
Hostpath
New Features and Enhancements
Topology-Constrained StorageClasses for Local PV Hostpath - The Local PV Hostpath Helm chart now allows you to set allowedTopologies on the provisioned StorageClass, so volume placement can be restricted to a defined set of nodes or zones directly from chart values.
HTTP Health Probe for Local PV Hostpath - The Local PV Hostpath provisioner now serves a dedicated HTTP health endpoint, replacing the previous process-based liveness check. This provides a more accurate signal of provisioner health to Kubernetes.
Configurable Kubernetes API Client Rate Limits for Local PV Hostpath - The Kubernetes API client QPS and burst values used by the Local PV Hostpath provisioner are now configurable, allowing provisioning throughput to be tuned in large clusters.
Configurable Helper Pod Image Pull Policy for Local PV Hostpath - The image pull policy for the Local PV Hostpath helper pod can now be set through the Helm chart, providing more control in air-gapped and locally mirrored registry environments.
Bug Fixes and Improvements
XFS Project Quota Cleanup for Local PV Hostpath - Resolved an issue where volume cleanup on XFS could attempt to reset the quota project on named pipes, causing cleanup to fail.
File Permissions in Node Deployment Mode for Local PV Hostpath - Resolved an issue where the configured file permissions mode was not applied to provisioned volumes when the provisioner ran in node deployment mode. Additional volume manager fixes for node deployment mode are also included.
Rawfile
New Features and Enhancements
API Server for Local PV Rawfile - Local PV Rawfile now includes an API server with an OpenAPI specification and a Swagger UI, providing a documented interface for inspecting and interacting with the provisioner.
Bug Fixes and Improvements
Volume Expansion for Local PV Rawfile - Resolved an issue where volume expansion could fail because of unreliable mount output parsing. Mount information is now resolved using findmnt.
Testing
OpenEBS is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS project. Therefore, the use of OpenEBS with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 22.04 LTS (kernel: ubuntu-6.8.0-90-generic)
Tested k8s versions
1.32.10
1.35.6
Known Issues
For Local PV LVM and Local PV ZFS, you may face issues on single-node setups post-upgrade where the controller pod does not enter the Running state due to changes in the manifest and missing affinity rules.
Workaround: Delete the old controller pod to allow scheduling of the new one. This does not occur when upgrading from the previous release.
For Local PV LVM, thin pool capacity is not unmapped or reclaimed and is also not tracked in the lvmnode custom resource. This may result in unexpected behavior.
Upgrade and Backward Incompatibilities
PVC-Level BasePath Override Disabled for Local PV Hostpath - As a security hardening, a BasePath supplied through the cas.openebs.io/config annotation on a PersistentVolumeClaim is now ignored. This prevents a user who can create PersistentVolumeClaims from choosing the directory on the node where the volume is created. Set BasePath on the StorageClass instead. Deployments that depend on the earlier behaviour can restore it with the localpv-provisioner.localpv.allowInsecurePvcBasePathOverride Helm value, which is disabled by default.
Deprecated Local PV Rawfile Helm Values Removed - The top-level dataDirPath and reservedCapacity Helm chart values have been removed. Use the equivalent storage pool specific values instead.
Filesystem-Level Snapshots Removed from Local PV Rawfile - Filesystem-level (Btrfs) snapshot support has been removed. Existing snapshots are not deleted, but they are no longer accessible after the upgrade. Remove any filesystem-level snapshots before upgrading.
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| [openebs](https://www.openebs.io/) ([source](https://github.com/openebs/openebs)) | minor | `4.5.1` → `4.6.1` |
---
### Release Notes
<details>
<summary>openebs/openebs (openebs)</summary>
### [`v4.6.1`](https://github.com/openebs/openebs/releases/tag/v4.6.1)
[Compare Source](https://github.com/openebs/openebs/compare/v4.6.0...v4.6.1)
### OpenEBS 4.6.1 Release Notes
#### Release Summary
OpenEBS v4.6.1 is a patch release on top of v4.6.0, delivering targeted fixes and minor enhancements across the storage engines.
This release includes new patch releases of Replicated PV Mayastor (v2.12.1), Local PV LVM (v1.10.1), Local PV ZFS (v2.11.1) and Local PV Rawfile (v0.15.1). Local PV Hostpath remains at v4.6.0.
#### Engine Versions
| Engine | Version | Release Notes |
| ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| Replicated PV Mayastor | v2.12.1 | [Mayastor v2.12.1 release notes](https://github.com/openebs/mayastor/releases/tag/v2.12.1) |
| Local PV ZFS | v2.11.1 | [Local PV ZFS v2.11.1 release notes](https://github.com/openebs/zfs-localpv/releases/tag/v2.11.1) |
| Local PV LVM | v1.10.1 | [Local PV LVM v1.10.1 release notes](https://github.com/openebs/lvm-localpv/releases/tag/v1.10.1) |
| Local PV Hostpath | v4.6.0 | [Local PV Hostpath v4.6.0 release notes](https://github.com/openebs/dynamic-localpv-provisioner/releases/tag/v4.6.0) |
| Local PV Rawfile | v0.15.1 | [Local PV Rawfile v0.15.1 release notes](https://github.com/openebs/rawfile-localpv/releases/tag/v0.15.1) |
#### Change Summary
#### Replicated Storage (Mayastor)
- Allow labels containing slashes (e.g. Kubernetes-style `domain/key` labels) in the delete, cordon and drain REST routes ([openebs/mayastor-control-plane#1183](https://github.com/openebs/mayastor-control-plane/pull/1183))
- Fix Loki auto-discovery timeout during support bundle collection and raise the default `--timeout` ([openebs/mayastor-extensions#1010](https://github.com/openebs/mayastor-extensions/pull/1010))
- `kubectl mayastor` plugin output now includes an ID column and spurious error strings have been silenced ([openebs/mayastor-extensions#1036](https://github.com/openebs/mayastor-extensions/pull/1036))
Refer to the [Mayastor v2.12.1 release notes](https://github.com/openebs/mayastor/releases/tag/v2.12.1) for detailed changes.
#### Local Storage
#### LVM
- Add a `--default-format-options` node agent argument (`lvmNode.defaultFormatOptions` in the Helm chart) to set node-level default mkfs options per filesystem, used when a StorageClass does not set `formatOptions` ([openebs/lvm-localpv#506](https://github.com/openebs/lvm-localpv/pull/506))
> **Note for users on older kernels:** mkfs.xfs 6.5+ enables the `nrext64` feature by default, which only kernel 5.19+ can mount. On clusters with older kernels, set `lvmNode.defaultFormatOptions.xfs: "-i nrext64=0"` (or the per-StorageClass `formatOptions`) to keep XFS volumes mountable.
#### ZFS
- Add a `formatOptions` StorageClass parameter for extra mkfs options on first format, and a `--default-format-options` node agent flag (`zfsNode.defaultFormatOptions` in the Helm chart) for per-filesystem defaults; ignored when `fstype` is `zfs` ([openebs/zfs-localpv#760](https://github.com/openebs/zfs-localpv/pull/760))
- Go module path moved to `github.com/openebs/zfs-localpv/v2` for semantic import versioning; affects only Go library consumers ([openebs/zfs-localpv#756](https://github.com/openebs/zfs-localpv/pull/756))
> The same older-kernel note as LVM above applies; set `zfsNode.defaultFormatOptions.xfs: "-i nrext64=0"` if any node runs a kernel older than 5.19.
#### Rawfile
- Add support for setting compute resources on the API Server ([openebs/rawfile-localpv#399](https://github.com/openebs/rawfile-localpv/pull/399))
- Fix StorageClass `copyOnWrite` with value `""` to correctly default to auto-detected instead of `false` ([openebs/rawfile-localpv#404](https://github.com/openebs/rawfile-localpv/pull/404))
- Fix controller deployment not being disableable due to an incorrect Helm variable ([openebs/rawfile-localpv#409](https://github.com/openebs/rawfile-localpv/pull/409))
- StorageClass and VolumeSnapshotClass are now updatable via `helm upgrade`
- Add documentation for async volume replication with [VolSync](https://volsync.readthedocs.io/)
#### Hostpath
- No changes; Local PV Hostpath remains at v4.6.0.
### New Contributors
- [@​rohan2794](https://github.com/rohan2794) made their first contribution in [openebs/zfs-localpv#754](https://github.com/openebs/zfs-localpv/pull/754)
### [`v4.6.0`](https://github.com/openebs/openebs/releases/tag/v4.6.0)
[Compare Source](https://github.com/openebs/openebs/compare/v4.5.1...v4.6.0)
### OpenEBS 4.6.0 Release Notes
#### Engine Versions
| Engine | Version | Release Notes |
| ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| Replicated PV Mayastor | v2.12.0 | [Mayastor v2.12.0 release notes](https://github.com/openebs/mayastor/releases/tag/v2.12.0) |
| Local PV ZFS | v2.11.0 | [Local PV ZFS v2.11.0 release notes](https://github.com/openebs/zfs-localpv/releases/tag/v2.11.0) |
| Local PV LVM | v1.10.0 | [Local PV LVM v1.10.0 release notes](https://github.com/openebs/lvm-localpv/releases/tag/v1.10.0) |
| Local PV Hostpath | v4.6.0 | [Local PV Hostpath v4.6.0 release notes](https://github.com/openebs/dynamic-localpv-provisioner/releases/tag/v4.6.0) |
| Local PV Rawfile | v0.15.0 | [Local PV Rawfile v0.15.0 release notes](https://github.com/openebs/rawfile-localpv/releases/tag/v0.15.0) |
#### Change Summary
#### General
##### New Features
- **Optional CSI Snapshot Controller** - Replicated PV Mayastor, Local PV LVM, Local PV ZFS, and Local PV Rawfile now allow you to disable the bundled CSI snapshot controller through the Helm chart. This avoids conflicts in clusters where a snapshot controller is already managed at the cluster level.
##### Enhancements
- **Analytics Configuration Overrides** - The OpenEBS Helm charts now accept global overrides for the analytics identifier and key values, making analytics configuration consistent across the Replicated PV Mayastor, Local PV Hostpath, Local PV LVM, Local PV ZFS, and Local PV Rawfile charts.
#### Replicated Storage (Mayastor)
#### New Features
##### Eventing Aggregator
Replicated PV Mayastor now includes the Eventing Aggregator, a new component that collects cluster events and makes them available for querying through the `get events` command in the Mayastor kubectl plugin. Events can be retrieved from Loki, directly from NATS JetStream, or from a previously collected support bundle for offline analysis, and can be filtered by category, action, node, target, component, pool, volume, replica, and state. Cluster events are also included in the support bundle produced by `kubectl openebs dump system`. The Eventing Aggregator is enabled by default and works with or without Loki deployed.
##### Best-Effort Snapshot Restore Policy
A new `snapshotRestorePolicy` StorageClass parameter controls how a snapshot restore behaves when not every replica pool can host a clone. With the default `strict` policy, every requested replica must be cloned from the snapshot. With `bestEffort`, the restore proceeds as long as at least one clone succeeds; the volume comes up under-replicated and the remaining replicas are filled in through a normal rebuild. This allows a restore to succeed when a source pool is full or otherwise unable to host a clone.
##### DiskPool Handle Rescanning
Replicated PV Mayastor now periodically rescans DiskPool backing device file handles to detect hot-removal on devices that require I/O to surface a removal event, such as those using the AIO and io\_uring backends. Previously, removal of an idle device could go undetected. The rescan also refreshes the reported disk size, and both the behaviour and its interval are configurable through the Helm chart.
##### DiskPool Error Clearing
Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin. This allows a pool to be returned to normal reporting after the underlying condition has been resolved.
##### Volume and Nexus Label V2
A new versioned on-disk label layout, V2, is introduced alongside the existing V1 layout. V2 reduces the metadata partition reserved at the front of every replica from 4 MiB to 3 MiB and aligns volume capacity to 1 MiB boundaries. It also resolves block-mode backup compatibility with Kasten K10.
For a V2 volume, the requested size is rounded up to the next 1 MiB boundary. A size that is already a multiple of 1 MiB is unchanged, so a request for `10Gi` stays `10Gi`, while a decimal quantity such as `10G` is not MiB-aligned and is rounded up. Each replica additionally reserves 8 MiB for the metadata partition, and the replica total is then aligned up to the cluster size of the pool, which is 4 MiB by default. For example, a 10 MiB volume on a pool with the default cluster size is exposed as a 10 MiB device while each replica consumes 20 MiB of pool space.
A volume created with a size that is not 1 MiB aligned reports a slightly larger size than requested, which is expected behaviour. Because V2 also adds the 8 MiB metadata reservation for every replica, a V2 volume consumes more pool space than the raw requested size, so size your pools accordingly.
The cluster-wide label version is negotiated automatically as the lowest version supported across all registered io-engines, and only ever moves up, so the control plane never creates a label that a node cannot understand. In a cluster where some nodes do not yet support V2, the negotiated version stays at V1 until every node supports it. Existing volumes remain on V1 and require no migration, and resize and snapshot operations preserve the label version of the volume.
#### Enhancements
##### TLS Hardening and Certificate Auto-Reload
Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Certificates are hot-reloaded on rotation without a restart, TLS discovery is enabled by default, plain HTTP on the REST service is restricted to health probes only, and the Helm chart exposes the TLS configuration for the public API.
##### RDMA Capability Detection
The CSI node now checks for the `nvme_rdma` kernel module before reporting RDMA capability, and node transport capabilities are propagated through registration. Previously a node could be treated as RDMA-capable without the required kernel support.
##### Asynchronous Bdev Destruction
Block device destruction in the io-engine is now asynchronous, improving the responsiveness of pool and replica teardown.
##### Additional Helm Chart Options
Additional environment variables are exposed for component customisation, the api-rest health probes use an `initialDelaySeconds` of 1 for faster readiness, and the maximum Loki ingestion limits have been increased.
##### Clearer Impact Reporting for Purge Operations
Node and DiskPool purge operations that involve data loss now list the affected volumes and snapshots directly in the reported error, instead of requiring a separate `--show-impact` run. Snapshot impact is now included alongside volume impact.
##### Pool Identification on Replica Metrics
Replica metrics now carry `pool_name` and `pool_uuid` labels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation.
#### Bug Fixes and Improvements
##### Volume Expansion with Undersized Replicas
Resolved an issue where a volume expansion could fail to complete when one or more replicas had not yet been resized. Undersized replicas are now resized before the nexus resize is retried.
##### Pool Availability During Device Removal and Replica Deletion
Resolved several issues affecting DiskPool availability during device and replica lifecycle events, including a race between pool deregistration and reload, and a race when listing pools while replicas were being destroyed.
##### Frozen I/O During Nexus Shutdown
Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, child device closure is awaited, and devices are no longer detached on transient I/O submission errors.
##### Nexus Size Miscalculation
Resolved an issue where the nexus block device size could be off by one block. Volume sizing now accounts for label metadata, so the usable capacity always meets the requested size.
##### Pool Capacity Accounting for Snapshot Clones
Resolved an issue where pool capacity tallying did not account for the creation of snapshot clones, which could lead to over-commitment of pool space.
##### Readiness Probe Startup Race
Resolved an issue where a readiness failure could be cached before the first successful probe, causing api-rest to be incorrectly reported as not ready.
##### Node Rebuild Count Accuracy
Resolved an issue where the node rebuild count was not refreshed on single nexus updates, improving the accuracy of rebuild throttling.
##### SPDK Fixes
Updated SPDK with fixes for a null pointer dereference and IPv6 transport handling, and resolved an interrupt-mode reactor teardown issue.
#### Limitations
- The IO engine fully utilizes all allocated CPU cores regardless of the actual I/O load, as it runs a poller at full speed.
- Each DiskPool is limited to a single block device and cannot span across multiple devices.
- The data-at-rest encryption feature does not support rotation of Data Encryption Keys (DEKs).
#### Known Issues
- If a node hosting a pod reboots and the pod lacks a controller (like a Deployment), the volume unpublish operation may not trigger. This causes the control plane to assume the volume is still in use, which leads to `fsfreeze` operation failure during snapshots.
**Workaround:** Recreate or rebind the pod to ensure proper volume mounting.
- Large pools (for example, 10–20 TiB) may experience extended recovery times after a dirty shutdown of the node hosting the io-engine.
- When using Replicated PV Mayastor on Oracle Linux 9 (kernel 5.14.x), servers may unexpectedly reboot during volume detach operations due to a kernel bug (CVE-2024-53170) in the block layer.
This issue is not caused by Mayastor but is triggered more frequently because of its NVMe-TCP connection lifecycle.
**Workaround:** Upgrade to kernel 6.11.11, 6.12.2, or later, which includes the fix.
#### Local Storage
#### ZFS
##### New Features and Enhancements
- **Dataset Tuning Parameters for Local PV ZFS** - Local PV ZFS StorageClasses now support the `atime` and `logbias` parameters, giving you direct control over access-time updates and write-workload optimisation on the underlying ZFS datasets and volumes.
##### Bug Fixes and Improvements
- **Image URL Rendering for Local PV ZFS** - Resolved an issue where image URLs in rendered manifests were not quoted, which could break rendering for registries whose URLs contain characters that YAML treats specially.
#### LVM
##### New Features and Enhancements
- **Updated CSI Snapshot Components for Local PV LVM** - The bundled `csi-snapshotter` and `snapshot-controller` components have been updated to v8.2.0.
##### Bug Fixes and Improvements
- **Idempotent Volume Expansion for Local PV LVM** - Resolved an issue where repeating a volume expansion could behave inconsistently. Node expansion now remains required during resize, so repeated expand operations are idempotent.
- **ServiceMonitor Manifest Rendering for Local PV LVM** - Resolved an issue where the namespace field in the generated ServiceMonitor manifest was incorrectly indented.
#### Hostpath
##### New Features and Enhancements
- **Topology-Constrained StorageClasses for Local PV Hostpath** - The Local PV Hostpath Helm chart now allows you to set `allowedTopologies` on the provisioned StorageClass, so volume placement can be restricted to a defined set of nodes or zones directly from chart values.
- **HTTP Health Probe for Local PV Hostpath** - The Local PV Hostpath provisioner now serves a dedicated HTTP health endpoint, replacing the previous process-based liveness check. This provides a more accurate signal of provisioner health to Kubernetes.
- **Configurable Kubernetes API Client Rate Limits for Local PV Hostpath** - The Kubernetes API client QPS and burst values used by the Local PV Hostpath provisioner are now configurable, allowing provisioning throughput to be tuned in large clusters.
- **Configurable Helper Pod Image Pull Policy for Local PV Hostpath** - The image pull policy for the Local PV Hostpath helper pod can now be set through the Helm chart, providing more control in air-gapped and locally mirrored registry environments.
##### Bug Fixes and Improvements
- **XFS Project Quota Cleanup for Local PV Hostpath** - Resolved an issue where volume cleanup on XFS could attempt to reset the quota project on named pipes, causing cleanup to fail.
- **File Permissions in Node Deployment Mode for Local PV Hostpath** - Resolved an issue where the configured file permissions mode was not applied to provisioned volumes when the provisioner ran in node deployment mode. Additional volume manager fixes for node deployment mode are also included.
#### Rawfile
##### New Features and Enhancements
- **API Server for Local PV Rawfile** - Local PV Rawfile now includes an API server with an OpenAPI specification and a Swagger UI, providing a documented interface for inspecting and interacting with the provisioner.
##### Bug Fixes and Improvements
- **Volume Expansion for Local PV Rawfile** - Resolved an issue where volume expansion could fail because of unreliable mount output parsing. Mount information is now resolved using `findmnt`.
#### Testing
OpenEBS is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS project. Therefore, the use of OpenEBS with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 22.04 LTS (kernel: ubuntu-6.8.0-90-generic)
- Tested k8s versions
- 1.32.10
- 1.35.6
#### Known Issues
- For Local PV LVM and Local PV ZFS, you may face issues on single-node setups post-upgrade where the controller pod does not enter the `Running` state due to changes in the manifest and missing affinity rules.
**Workaround:** Delete the old controller pod to allow scheduling of the new one. This does not occur when upgrading from the previous release.
- For Local PV LVM, thin pool capacity is not unmapped or reclaimed and is also not tracked in the `lvmnode` custom resource. This may result in unexpected behavior.
#### Upgrade and Backward Incompatibilities
- **PVC-Level BasePath Override Disabled for Local PV Hostpath** - As a security hardening, a `BasePath` supplied through the `cas.openebs.io/config` annotation on a PersistentVolumeClaim is now ignored. This prevents a user who can create PersistentVolumeClaims from choosing the directory on the node where the volume is created. Set `BasePath` on the StorageClass instead. Deployments that depend on the earlier behaviour can restore it with the `localpv-provisioner.localpv.allowInsecurePvcBasePathOverride` Helm value, which is disabled by default.
- **Deprecated Local PV Rawfile Helm Values Removed** - The top-level `dataDirPath` and `reservedCapacity` Helm chart values have been removed. Use the equivalent storage pool specific values instead.
- **Filesystem-Level Snapshots Removed from Local PV Rawfile** - Filesystem-level (Btrfs) snapshot support has been removed. Existing snapshots are not deleted, but they are no longer accessible after the upgrade. Remove any filesystem-level snapshots before upgrading.
#### New Contributors
- [@​jochenseeber](https://github.com/jochenseeber) made their first contribution in [#​4077](https://github.com/openebs/openebs/pull/4077)
- [@​PingouinFerreux](https://github.com/PingouinFerreux) made their first contribution in [#​4292](https://github.com/openebs/openebs/pull/4292)
- [@​susobhandey](https://github.com/susobhandey) made their first contribution in [#​4274](https://github.com/openebs/openebs/pull/4274)
</details>
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC40Ni40IiwidXBkYXRlZEluVmVyIjoiNDQuNzkuMiIsInRhcmdldEJyYW5jaCI6ImZyZXNoLXN0YXJ0IiwibGFiZWxzIjpbXX0=-->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
This PR contains the following updates:
4.5.1→4.6.1Release Notes
openebs/openebs (openebs)
v4.6.1Compare Source
OpenEBS 4.6.1 Release Notes
Release Summary
OpenEBS v4.6.1 is a patch release on top of v4.6.0, delivering targeted fixes and minor enhancements across the storage engines.
This release includes new patch releases of Replicated PV Mayastor (v2.12.1), Local PV LVM (v1.10.1), Local PV ZFS (v2.11.1) and Local PV Rawfile (v0.15.1). Local PV Hostpath remains at v4.6.0.
Engine Versions
Change Summary
Replicated Storage (Mayastor)
domain/keylabels) in the delete, cordon and drain REST routes (openebs/mayastor-control-plane#1183)--timeout(openebs/mayastor-extensions#1010)kubectl mayastorplugin output now includes an ID column and spurious error strings have been silenced (openebs/mayastor-extensions#1036)Refer to the Mayastor v2.12.1 release notes for detailed changes.
Local Storage
LVM
--default-format-optionsnode agent argument (lvmNode.defaultFormatOptionsin the Helm chart) to set node-level default mkfs options per filesystem, used when a StorageClass does not setformatOptions(openebs/lvm-localpv#506)ZFS
formatOptionsStorageClass parameter for extra mkfs options on first format, and a--default-format-optionsnode agent flag (zfsNode.defaultFormatOptionsin the Helm chart) for per-filesystem defaults; ignored whenfstypeiszfs(openebs/zfs-localpv#760)github.com/openebs/zfs-localpv/v2for semantic import versioning; affects only Go library consumers (openebs/zfs-localpv#756)Rawfile
copyOnWritewith value""to correctly default to auto-detected instead offalse(openebs/rawfile-localpv#404)helm upgradeHostpath
New Contributors
v4.6.0Compare Source
OpenEBS 4.6.0 Release Notes
Engine Versions
Change Summary
General
New Features
Enhancements
Replicated Storage (Mayastor)
New Features
Eventing Aggregator
Replicated PV Mayastor now includes the Eventing Aggregator, a new component that collects cluster events and makes them available for querying through the
get eventscommand in the Mayastor kubectl plugin. Events can be retrieved from Loki, directly from NATS JetStream, or from a previously collected support bundle for offline analysis, and can be filtered by category, action, node, target, component, pool, volume, replica, and state. Cluster events are also included in the support bundle produced bykubectl openebs dump system. The Eventing Aggregator is enabled by default and works with or without Loki deployed.Best-Effort Snapshot Restore Policy
A new
snapshotRestorePolicyStorageClass parameter controls how a snapshot restore behaves when not every replica pool can host a clone. With the defaultstrictpolicy, every requested replica must be cloned from the snapshot. WithbestEffort, the restore proceeds as long as at least one clone succeeds; the volume comes up under-replicated and the remaining replicas are filled in through a normal rebuild. This allows a restore to succeed when a source pool is full or otherwise unable to host a clone.DiskPool Handle Rescanning
Replicated PV Mayastor now periodically rescans DiskPool backing device file handles to detect hot-removal on devices that require I/O to surface a removal event, such as those using the AIO and io_uring backends. Previously, removal of an idle device could go undetected. The rescan also refreshes the reported disk size, and both the behaviour and its interval are configurable through the Helm chart.
DiskPool Error Clearing
Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin. This allows a pool to be returned to normal reporting after the underlying condition has been resolved.
Volume and Nexus Label V2
A new versioned on-disk label layout, V2, is introduced alongside the existing V1 layout. V2 reduces the metadata partition reserved at the front of every replica from 4 MiB to 3 MiB and aligns volume capacity to 1 MiB boundaries. It also resolves block-mode backup compatibility with Kasten K10.
For a V2 volume, the requested size is rounded up to the next 1 MiB boundary. A size that is already a multiple of 1 MiB is unchanged, so a request for
10Gistays10Gi, while a decimal quantity such as10Gis not MiB-aligned and is rounded up. Each replica additionally reserves 8 MiB for the metadata partition, and the replica total is then aligned up to the cluster size of the pool, which is 4 MiB by default. For example, a 10 MiB volume on a pool with the default cluster size is exposed as a 10 MiB device while each replica consumes 20 MiB of pool space.A volume created with a size that is not 1 MiB aligned reports a slightly larger size than requested, which is expected behaviour. Because V2 also adds the 8 MiB metadata reservation for every replica, a V2 volume consumes more pool space than the raw requested size, so size your pools accordingly.
The cluster-wide label version is negotiated automatically as the lowest version supported across all registered io-engines, and only ever moves up, so the control plane never creates a label that a node cannot understand. In a cluster where some nodes do not yet support V2, the negotiated version stays at V1 until every node supports it. Existing volumes remain on V1 and require no migration, and resize and snapshot operations preserve the label version of the volume.
Enhancements
TLS Hardening and Certificate Auto-Reload
Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Certificates are hot-reloaded on rotation without a restart, TLS discovery is enabled by default, plain HTTP on the REST service is restricted to health probes only, and the Helm chart exposes the TLS configuration for the public API.
RDMA Capability Detection
The CSI node now checks for the
nvme_rdmakernel module before reporting RDMA capability, and node transport capabilities are propagated through registration. Previously a node could be treated as RDMA-capable without the required kernel support.Asynchronous Bdev Destruction
Block device destruction in the io-engine is now asynchronous, improving the responsiveness of pool and replica teardown.
Additional Helm Chart Options
Additional environment variables are exposed for component customisation, the api-rest health probes use an
initialDelaySecondsof 1 for faster readiness, and the maximum Loki ingestion limits have been increased.Clearer Impact Reporting for Purge Operations
Node and DiskPool purge operations that involve data loss now list the affected volumes and snapshots directly in the reported error, instead of requiring a separate
--show-impactrun. Snapshot impact is now included alongside volume impact.Pool Identification on Replica Metrics
Replica metrics now carry
pool_nameandpool_uuidlabels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation.Bug Fixes and Improvements
Volume Expansion with Undersized Replicas
Resolved an issue where a volume expansion could fail to complete when one or more replicas had not yet been resized. Undersized replicas are now resized before the nexus resize is retried.
Pool Availability During Device Removal and Replica Deletion
Resolved several issues affecting DiskPool availability during device and replica lifecycle events, including a race between pool deregistration and reload, and a race when listing pools while replicas were being destroyed.
Frozen I/O During Nexus Shutdown
Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, child device closure is awaited, and devices are no longer detached on transient I/O submission errors.
Nexus Size Miscalculation
Resolved an issue where the nexus block device size could be off by one block. Volume sizing now accounts for label metadata, so the usable capacity always meets the requested size.
Pool Capacity Accounting for Snapshot Clones
Resolved an issue where pool capacity tallying did not account for the creation of snapshot clones, which could lead to over-commitment of pool space.
Readiness Probe Startup Race
Resolved an issue where a readiness failure could be cached before the first successful probe, causing api-rest to be incorrectly reported as not ready.
Node Rebuild Count Accuracy
Resolved an issue where the node rebuild count was not refreshed on single nexus updates, improving the accuracy of rebuild throttling.
SPDK Fixes
Updated SPDK with fixes for a null pointer dereference and IPv6 transport handling, and resolved an interrupt-mode reactor teardown issue.
Limitations
Known Issues
If a node hosting a pod reboots and the pod lacks a controller (like a Deployment), the volume unpublish operation may not trigger. This causes the control plane to assume the volume is still in use, which leads to
fsfreezeoperation failure during snapshots.Workaround: Recreate or rebind the pod to ensure proper volume mounting.
Large pools (for example, 10–20 TiB) may experience extended recovery times after a dirty shutdown of the node hosting the io-engine.
When using Replicated PV Mayastor on Oracle Linux 9 (kernel 5.14.x), servers may unexpectedly reboot during volume detach operations due to a kernel bug (CVE-2024-53170) in the block layer.
This issue is not caused by Mayastor but is triggered more frequently because of its NVMe-TCP connection lifecycle.
Workaround: Upgrade to kernel 6.11.11, 6.12.2, or later, which includes the fix.
Local Storage
ZFS
New Features and Enhancements
atimeandlogbiasparameters, giving you direct control over access-time updates and write-workload optimisation on the underlying ZFS datasets and volumes.Bug Fixes and Improvements
LVM
New Features and Enhancements
csi-snapshotterandsnapshot-controllercomponents have been updated to v8.2.0.Bug Fixes and Improvements
Hostpath
New Features and Enhancements
allowedTopologieson the provisioned StorageClass, so volume placement can be restricted to a defined set of nodes or zones directly from chart values.Bug Fixes and Improvements
Rawfile
New Features and Enhancements
Bug Fixes and Improvements
findmnt.Testing
OpenEBS is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS project. Therefore, the use of OpenEBS with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 22.04 LTS (kernel: ubuntu-6.8.0-90-generic)
Known Issues
For Local PV LVM and Local PV ZFS, you may face issues on single-node setups post-upgrade where the controller pod does not enter the
Runningstate due to changes in the manifest and missing affinity rules.Workaround: Delete the old controller pod to allow scheduling of the new one. This does not occur when upgrading from the previous release.
For Local PV LVM, thin pool capacity is not unmapped or reclaimed and is also not tracked in the
lvmnodecustom resource. This may result in unexpected behavior.Upgrade and Backward Incompatibilities
BasePathsupplied through thecas.openebs.io/configannotation on a PersistentVolumeClaim is now ignored. This prevents a user who can create PersistentVolumeClaims from choosing the directory on the node where the volume is created. SetBasePathon the StorageClass instead. Deployments that depend on the earlier behaviour can restore it with thelocalpv-provisioner.localpv.allowInsecurePvcBasePathOverrideHelm value, which is disabled by default.dataDirPathandreservedCapacityHelm chart values have been removed. Use the equivalent storage pool specific values instead.New Contributors
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Mend Renovate CLI.
cd29a6eabatoa99043c1bbUpdate Helm release openebs to v4.6.0to Update Helm release openebs to v4.6.1