Issue 001: ZFS Boot Environments scope (deferred to Hito 3)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/issues/001-zfs-boot-environments-scope.md. Edit at the source, not here.

Issue 001: ZFS Boot Environments scope (deferred to Hito 3)

Status: Open (tracked for Hito 3, June 2026) Filed: 2026-04-19 Owners: SYNNET Engineering Related: ADR 003, ADR 009

Summary

ZFS-on-root with boot environments was previously part of the ADR 003 decision. It has been removed from the Hito 0.3 scope and explicitly re-scoped to Hito 3 (targeted June 2026). This issue tracks what that entails and the decisions that remain open.

What “Boot Environments” means here

A boot environment (BE) is a bootable ZFS dataset that represents the OS root at a specific point in time. The target operational model for SynVirt is:

Before any system-level change (kernel upgrade, daemon upgrade, apt batch), the upgrade pipeline snapshots the current BE and clones it to a new BE named with the upgrade timestamp / purpose.
The upgrade runs on the new BE. The GRUB menu is rewritten to list the new BE first, with the prior BE one entry down.
On successful boot + health check, the new BE is kept and the prior BE is retained for rollback for N days (N configurable).
On failed boot / failed health check, the operator (or an automatic watchdog) reboots into the prior BE from GRUB. The system is recovered without a rebuild.

Why it is deferred to Hito 3

Shipping ZFS-on-root with boot environments correctly requires all of the following:

A custom installer step that lays down ZFS-on-root under a stock Debian preseed flow. The stock installer does not do this; we would need either a d-i udeb, a first-boot conversion script, or a custom stage-2 installer.
A GRUB integration that enumerates ZFS BEs as menu entries and handles the zfsbootmenu-style rewriting on snapshot / rollback.
An upgrade wrapper around apt / kernel upgrade that creates the pre-upgrade BE, performs the upgrade in the target BE, and writes the GRUB entries before reboot.
A health-check service that runs post-boot, marks the BE “known good” on success, and triggers an automatic rollback on failure.
Operator-facing UI in the web console and CLI for listing, promoting, and pruning BEs.

That is not a Hito 0.3 surface — it is its own milestone. The Hito 0.3 milestone is scoped to introducing ZFS as the localdata secondary pool under a stock preseed. Boot environments are re-docked to Hito 3 so the storage foundation can be tested and run in production under Hito 0.3/1/2 while the BE machinery is built in parallel.

Open decisions for Hito 3

The following are deliberately not answered yet; they are reopened at Hito 3 kickoff:

Bootloader integration. zfsbootmenu vs custom GRUB modifications. Tradeoff: zfsbootmenu is production-proven but introduces a new boot stage; custom GRUB keeps the boot pipeline simpler but requires us to own more of it.
ESP layout. Whether BEs share a single ESP or each BE gets its own ESP copy. Sharing is simpler; per-BE gives stronger rollback guarantees if an upgrade touches the ESP.
Migration path for existing 0.3 nodes. Current plan is reinstall-in-place with localdata preserved. Open question: whether we offer a supported “convert existing ext4 rpool to ZFS rpool without reinstall” tool, and at what cost.
BE retention policy defaults. Number of BEs kept, age-based vs count-based pruning, interaction with localdata snapshot retention.
Health-check contract. What counts as a successful boot. First candidate: synvirt-daemon reaching a ready state within N seconds of boot.

Out of scope for this issue

VM-level snapshots and backups on localdata/vms. That is Hito 1 scope and uses ZFS snapshots directly; it does NOT depend on boot environments.
DRBD replication of localdata/vms. That is Hito 2 scope.
Any BE work on the node before Hito 3. The rpool is ext4 on all 0.x / 1.x / 2.x nodes.

Acceptance criteria (when Hito 3 closes this issue)

This issue is closed when all of the following are true on a Hito 3 node:

A fresh install lays down ZFS-on-root with an initial BE named synvirt-install-<timestamp>.
synvirt upgrade (or the equivalent command) creates a pre-upgrade BE, applies the upgrade, and rewrites the GRUB menu to list the new BE first.
A forced failed-upgrade simulation (e.g. a kernel that panics early) results in a rollback to the prior BE either via the health-check watchdog or via the operator selecting it from GRUB.
The localdata pool is untouched across BE operations.
Operator-facing documentation covers BE lifecycle, rollback, and pruning.