ADR 003 — ZFS with Boot Environments

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/decisions/003-zfs-with-boot-environments.md. Edit at the source, not here.

ADR 003 — ZFS with Boot Environments

Status: Accepted (staged: secondary pool in Hito 0.3, boot environments in Hito 3) Date: 2026-04-19 Deciders: SYNNET Engineering

Context

SynVirt nodes need a reliable storage solution for:

VM volumes: VM disk images that must support fast snapshots, efficient cloning (for rapid VM provisioning from templates), and replication to a second node.
OS root filesystem: The host OS on a production hypervisor must support safe, rollback-capable upgrades. Applying a kernel update or daemon upgrade that renders the node unbootable is a critical failure scenario.

The storage architecture is introduced in stages. Hito 0.1/0.2 ran entirely on a single ext4 root. Hito 0.3 introduces ZFS as a secondary pool for VM data while keeping ext4 on root. ZFS-on-root with boot environments is deferred to Hito 3 because doing it properly requires a custom installer step that the standard Debian installer does not provide (see Issue 001 for the tracking ticket).

Decision

SynVirt uses a two-pool layout:

rpool (ext4, fixed 40 GB): Root filesystem. Contains /, /boot, the ESP and biosgrub helper partitions. Standard Debian install. Subject to routine apt upgrades; no snapshot / rollback story in Hito 0.x.
localdata (ZFS, rest of the target disk): Mounted at /var/lib/synvirt/local-data. Holds five datasets:
- localdata/vms — VM disk images and configs
- localdata/iso-cache — ISO / image cache
- localdata/backups — local backups
- localdata/ai-models — models for the AI module
- localdata/logs-archive — archived logs

ZFS zvols under localdata/vms are the backing store for DRBD replication in Twin and Cluster modes (DRBD integration lands in Hito 3 with module-storage; Twin mode is wired up later in October 2026).

The ZFS kernel module is installed via DKMS (see ADR 010 for sourcing) and rebuilt automatically on kernel upgrades.

Boot environments on root are deferred to Hito 3. The target end state is ZFS-on-root with automatic boot environment creation before every system-level change, selectable from GRUB on failure. Until Hito 3 lands, rollback-capable OS upgrades are NOT available on SynVirt nodes; operators must plan kernel/daemon upgrades against a snapshot or replica at a level above the OS.

Rationale

Why ZFS for the data pool, starting in Hito 0.3

Copy-on-Write snapshots for VM volumes: ZFS snapshots are instantaneous and space-efficient. Cloning a VM from a template becomes a metadata operation rather than a full disk copy. Snapshots enable crash-consistent VM backups without pausing the guest.
ZFS RAID-Z for data redundancy: In Standalone mode with multiple disks, RAID-Z provides hardware-independent RAID with checksums that detect and correct silent data corruption that traditional RAID (md, hardware) misses.
Integration with DRBD (Hito 3 / Twin mode): ZFS zvols expose a block device interface that DRBD can replicate over the network. This gives us synchronous replication of VM data between nodes with the ZFS checksum layer providing end-to-end data integrity. LINSTOR + DRBD integration itself is module-storage scope in Hito 3; Twin mode (the 2-node HA deployment model) is wired up later.
Mature technology: OpenZFS has been running in production on Linux since 2013. It is deployed at scale in storage appliances, commercial hypervisors, and large-scale data platforms.

Why a separate rpool / localdata split

Reinstall preserves customer data. Because localdata is a distinct pool on its own partition, reinstalling the OS (repaving rpool) leaves VM images, backups, and logs intact.
Upgrade blast radius is bounded. A broken kernel or apt upgrade touches only rpool; localdata is not affected.
Clear operational ownership. Operators can reason about “the OS disk” vs “the data disk” without understanding ZFS boot environments first.

Why defer boot environments to Hito 3

Standard installer does not support ZFS-on-root. Getting ZFS-on-root into a Debian preseed flow requires a custom installer with a ZFS bootloader integration, not a simple preseed recipe. Shipping this in Hito 0.3 would block the milestone on installer work that does not serve the 0.3 scope (see Issue 001).
OpenZFS 2.4.x feature maturity. Block cloning and raidz expansion in 2.4.x are desirable for the Hito 3 architecture and are easier to adopt if we baseline on 2.4.x from day one on root.
Hito 3 couples BE with the staged upgrade pipeline. The boot environment machinery is most valuable when wired into an atomic upgrade workflow (pre-upgrade BE snapshot → upgrade → on-failure reboot into prior BE). Hito 3 is where that workflow lands.

Consequences

DKMS dependency from Hito 0.3: The ZFS kernel module must be rebuilt every time the kernel is upgraded. The DKMS framework handles this automatically, but a failed DKMS build will leave the node without ZFS after a reboot. Pre-upgrade testing of DKMS builds is required in the release process.
No rollback-capable OS upgrades until Hito 3. Operators must treat kernel / daemon upgrades on Hito 0.3 nodes the way they treat them on any ext4-rooted Debian system: back up, test in staging, upgrade the production node with a maintenance window.
Hito 3 migration is a reinstall, not an in-place conversion. Converting an existing Hito 0.3 / Hito 1 / Hito 2 node from ext4-on-root to ZFS-on-root will require a reinstall when Hito 3 lands; localdata is preserved across reinstalls by design, but rpool is repaved. Migration tooling is planned but not committed.
Memory usage: ZFS uses RAM for the ARC (Adaptive Replacement Cache). A minimum of 8 GB RAM is recommended for production nodes running ZFS. This will be reflected in Hito 0.3 hardware requirements.

Alternatives Considered

Btrfs

Pros: Included in the mainline Linux kernel, no DKMS needed, supports snapshots and subvolumes.
Cons: Btrfs RAID-5/6 (the equivalent of RAID-Z) has known data loss bugs that have not been fully resolved as of 2026. The project documentation explicitly warns against using RAID-5/6 in production. Without RAID, Btrfs does not offer the data integrity guarantees we require. Production maturity lags significantly behind ZFS.
Decision: Rejected. RAID-5/6 instability makes it unsuitable for production VM storage.

LVM + ext4 for the data pool

Pros: Extremely well-understood, zero kernel module dependencies, native Debian installer support, fast performance.
Cons: No copy-on-write. Snapshots via LVM are COW at the block level but are slow and consume space unpredictably. No checksums — silent data corruption is undetectable. No boot environments. Thin provisioning is available but adds complexity without the integrity benefits of ZFS.
Decision: Rejected. Lack of CoW snapshots and data integrity checksums are disqualifying for a production hypervisor data pool.

bcachefs

Pros: Native mainline kernel module (no DKMS), CoW, checksums, built-in replication.
Cons: Marked stable in kernel 6.7 (2024). As of 2026 it is less than 2 years into production use at scale. Operational knowledge, tooling, and community resources are immature relative to ZFS. A storage bug in a new filesystem could be catastrophic for customer data.
Decision: Rejected. Too new for a commercial product where customer VM data integrity is a core trust requirement. May be revisited for Hito 4+.

Single-pool ZFS (ZFS-on-root from Hito 0.3)

Pros: Unified storage model, boot environments available from day one, no two-pool cognitive overhead.
Cons: The standard Debian installer does not support ZFS-on-root in preseed. Shipping this in Hito 0.3 would block on a custom installer track that does not serve the 0.3 milestone.
Decision: Rejected for Hito 0.3; adopted as the Hito 3 target end state. See Issue 001 for tracking.