ADR 009: Dual-pool ZFS architecture (ext4 rpool + ZFS localdata)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/decisions/009-zfs-dual-pool-architecture.md. Edit at the source, not here.

ADR 009: Dual-pool ZFS architecture (ext4 rpool + ZFS localdata)

Status

Accepted

Context

Hito 0.3 introduces ZFS on SynVirt nodes for the first time. Two shapes were on the table:

1. Single-pool ZFS-on-root (rpool contains both OS and data, boot environments from day one).
2. Dual-pool split (rpool = small ext4 partition for the OS; localdata = ZFS pool for VM images, backups, caches).

Option 1 has the strongest long-term story (it is the Hito 3 end state, see ADR 003 and Issue 001). But it requires a custom installer step because the stock Debian installer does not configure ZFS-on-root from preseed. Shipping that installer work inside the Hito 0.3 window would block the milestone on an installer track that does not serve its immediate scope.

Option 2 can be implemented entirely inside a stock Debian preseed recipe plus a late_command that runs after the base install: the data partition is initialized as an ext4 placeholder during partman, then wiped and converted to a ZFS pool in the late_command chroot.

Decision

SynVirt 0.3 ships a dual-pool layout on the target disk (GPT, dual-firmware):

Partition	Size	Type	Purpose
1	1 MB	biosgrub	BIOS legacy boot on GPT
2	538 MB	ESP	EFI System Partition (FAT32), /boot/efi
3	40 GB	ext4	Root / (rpool)
4	rest	ZFS	localdata pool

The localdata pool is created in the preseed late_command with:

• ashift=12, autotrim=on
• compression=zstd, atime=off, xattr=sa, acltype=posixacl
• normalization=formD
• Mountpoint /var/lib/synvirt/local-data

Five datasets are created under it: vms, iso-cache, backups, ai-models, logs-archive.

If the ZFS kernel module cannot be loaded inside the debian-installer kernel (DKMS may not have finished rebuilding against the installer kernel at late_command time), pool creation is deferred to a oneshot systemd unit (synvirt-zfs-init.service) that runs on first boot, ordered Before=synvirt-installer.service. A completion sentinel (/var/lib/synvirt/zfs-init.completed) makes the unit idempotent.

Rationale

• Ships in Hito 0.3 with no installer fork. The dual-pool layout lives inside a stock preseed + late_command; no patched installer is required.
• Reinstalls preserve customer data. localdata is on a separate partition from rpool. Reinstalling SynVirt over a node repaves rpool but leaves localdata intact. Combined with the deferred zfs-init unit (which detects an existing pool and skips creation), this gives operators a safe “repave the OS, keep the data” workflow.
• Clear operational model. Operators see two disks-in-one: the OS disk (small, ext4, upgrade-churn) and the data disk (large, ZFS, long-lived). This is a mental model that maps cleanly to standard hypervisor operations.
• Forward-compatible with Hito 3. The localdata pool shape (ashift, feature flags, dataset layout) is already the one we want long-term. Hito 3 adds ZFS-on-root and boot environments on top; it does not rewrite localdata.
• Forward-compatible with Twin/Cluster. ZFS zvols under localdata/vms are exactly the block devices DRBD will replicate in Twin (October 2026) and Cluster (Nov-Dec 2026) modes. LINSTOR + DRBD integration itself lands earlier with module-storage in Hito 3.

Consequences

• Two partitions to care about. Sizing decisions for rpool (40 GB fixed) and localdata (rest of disk) are baked into the partman recipe. Nodes with unusually small disks (< ~45 GB) cannot install. This is enforced upstream by the whiptail selector’s 20 GiB floor plus the 41.5 GB system-partition cost.
• No rollback on the OS partition. ext4 on rpool has no snapshot / boot environment story. See ADR 003 for the Hito 3 plan that adds BE on root.
• ZFS-on-root migration is a reinstall. Moving an existing 0.3 node to the Hito 3 single-pool layout will require a reinstall (localdata preserved). Migration tooling is not committed.
• Deferred init adds a cold-boot latency. When pool creation defers to the synvirt-zfs-init unit, the first boot takes ~10-30 extra seconds to create the pool and datasets before the TUI installer opens. This is logged to /var/log/synvirt-zfs-init.log.
• Disk selection at install time. The target-disk resolution flow (auto-detect vs whiptail radiolist in the advanced entry) is described in the 0.3 build docs. The selected disk is persisted to /var/lib/synvirt/target-disk.conf so the deferred init can find partition 4.

Alternatives Considered

Single-pool ZFS-on-root in Hito 0.3

• Pros: Unified pool model, boot environments available from day one, no two-pool cognitive overhead.
• Cons: Requires a custom installer track (stock Debian preseed cannot configure ZFS-on-root). Blocks 0.3 on installer work that belongs in Hito 3.
• Decision: Rejected for Hito 0.3; adopted as the Hito 3 target.

LVM-backed data pool (no ZFS in 0.3)

• Pros: Zero new kernel modules, minimal installer surface.
• Cons: No CoW snapshots, no checksums, no zvols for Hito 2’s DRBD path. Would require rebuilding the data layer in Hito 0.3+ anyway.
• Decision: Rejected. ZFS is required for the storage roadmap; introducing it later is strictly more work than introducing it now.

Single large ext4 partition that becomes ZFS via in-place

migration

• Pros: Avoids the two-partition split.
• Cons: In-place ext4→ZFS migration is not a supported operation; it requires a full data copy to a second location. Not meaningfully different from a reinstall, and more error-prone.
• Decision: Rejected.