Backup P4 — Peer data plane (design for signature)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/specs/backup-p4-peer-data-plane.md. Edit at the source, not here.

Backup P4 — Peer data plane (design for signature)

Status: SIGNED — §D1–§D11 signed 2026-07-09 (Tony), conditions recorded in §7; P4 code GO issued 2026-07-09. §D12 landed separately (commit 12639057); §D13 remains a future phase. Scope: the “peer” repository kind — protecting one host’s VMs onto another cluster host’s ZFS pool over the mesh. Everything here is design; nothing is implemented. Decisions are numbered §D<n> so they can be signed or amended individually, the way the P3 engine spec’s §2.x decisions were.

This doc is grounded in the code as it stands at commit aaaa73f3 (Phase 6). Every “today” claim carries a file:line so a reviewer can check it.

---

1. What P4 is, and is not

Is: a real data path for RepositoryKind::PeerHost — an origin host pushes its restore-point streams to a peer host, which stores them on one of its ZFS pools; and the inverse, restoring from a peer. It flips the create-job hard block that exists today (§D9).

Is not: a new trust model, a new transport, or a second key holder. It rides the cluster mesh clusterd already owns. It is not resumable in v1 (§D6). It does not change the on-repository backup format (manifest-last file layout, per-stream sha256) — a peer repo stores the same bytes an NFS/LocalZfs repo stores, just on a remote pool.

Explicitly deferred (placement decisions in §D12–§D13): vTPM restore fidelity and SureBackup-style boot verification. Both are real, both are out of P4.

---

2. Ground truth (verified today)

• backupd cannot reach the mesh. synvirt-backupd/Cargo.toml has no synvirt-cluster/tonic/rustls dependency. clusterd is the sole holder of the mesh ed25519 identity + pinned channel; the daemon reaches it only over the clusterd control UDS (daemon/src/api/backup_proxy.rs:56-60: “Peer trust + reachability are owned by clusterd (backupd cannot reach the mesh)”).
• The cross-host byte mover already exists for VM clone: PATH_REMOTE_DATASET_SEND (synvirt-clusterd/src/control_proto.rs:75-78) is a streaming POST over the clusterd control UDS; clusterd bridges it to the peer over the pinned mTLS mesh (synvirt-cluster/src/mesh/federation.rs:428 send_dataset, header-chunk-then-bytes, per-RPC deadline raised to 3600s; receiver receive_dataset at :154). SPKI ed25519 pin over tokio-rustls, server_name carries no trust (mesh/client.rs:38-67); federation is Tier-2-gated. Origin push, no receiver-pull.
• The peer probe already lands in the daemon (not backupd): backup_proxy.rs:61-92 peer_probe_response → clusterd PATH_REMOTE_BACKUP_PROBE → runtime.rs:1239-1261 (dials the peer’s federation service, reachability + identity-pin proof only; free_bytes: None in v1, “dataset stat is a later phase”).
• The create-job block is synvirt-backupd/src/http.rs:224-241 ensure_job_repository: PeerHost => Err(PeerDataPlaneUnavailable(...)) → E_BACKUP_PEER_DATA_PLANE_UNAVAILABLE, 409.
• Guardrail reserved roots (synvirt-backupd/src/zfs.rs): backupd may zfs create only <pool>/synvirt-restore and <pool>/Backups (root only — a third path component is refused, zfs.rs:159-178); receive is clamped to <pool>/synvirt-restore/<run-id> (:142-152). BACKUPS_COMPONENT (zfs.rs:43-47) is documented as holding “both peer-received restore points and local-ZFS repository data”.
• LocalZfs layout (the mirror target): <pool>/Backups/local/<repo-id>/… as plain directories under the single Backups dataset (probe/local.rs:19-27; run dir synvirt/v1/<job-id>/<run-id>/streams/NN.zfs
  • manifest.json, run.rs:866-898).
• PeerConnection.target_dataset (synvirt-backup-core/src/types.rs:391) is required + persisted + wire-stable but only traced today, never dereferenced for I/O (runtime.rs:1233-1248). Reserved for exactly this phase.
• vTPM/nvram are captured implicitly: the parent-filesystem stream is a non-recursive zfs send -Lc of the per-VM filesystem, which carries <slug>-vtpm/, <slug>_VARS.fd, and the domain.xml mirror (manifest.rs:40-42, run.rs:881-898). But restore does not re-establish the swtpm link — see §D12.

---

3. Numbered decisions

§D1 — Transport: daemon-brokered through clusterd. NOT direct backupd↔backupd.

A peer stream never leaves backupd directly. The path is:

origin backupd  ── zfs send bytes (UDS) ──▶  origin daemon
origin daemon   ── PATH_REMOTE_BACKUP_SEND (UDS, streaming) ──▶  origin clusterd
origin clusterd ── tonic mTLS over pinned mesh channel ──▶  peer clusterd
peer clusterd   ── hands byte stream to ──▶  peer daemon (FederationProvider)
peer daemon     ── backup-receive callback (UDS) ──▶  peer backupd
peer backupd    ── writes file under <pool>/Backups/<origin-id>/… ──▶  peer pool

Rationale: direct backupd↔backupd mTLS was rejected because it would make backupd a second holder of mesh trust material — there is no API that hands out a channel or cert, and duplicating the ed25519 identity into backupd violates the single-key-holder invariant clusterd exists to enforce. Brokering mirrors the proven PATH_REMOTE_DATASET_SEND clone path exactly, one layer up.

New clusterd control route: PATH_REMOTE_BACKUP_SEND (origin→peer push of a backup stream) — the backup analogue of PATH_REMOTE_DATASET_SEND, but the receiver writes a file under the Backups namespace, it does not zfs recv into a dataset (the backup format is files, not a received dataset — see §D3). A matching PATH_REMOTE_BACKUP_FETCH for restore (§D8).

backupd stays the engine. It still decides the chain, sizes streams, writes the manifest last, tracks the run, holds the anchor. Only the byte transport for a peer destination is delegated up to the daemon→clusterd broker. The origin backupd opens zfs send and streams stdout to the daemon over a new backupd→daemon relay endpoint on the callback plane (the daemon already holds state.cluster_control); the daemon forwards to clusterd. This preserves the doctrine “streaming zfs send/recv executes in the owning daemon” — backupd still executes the send; the daemon is a byte pipe to the mesh, not the sender.

§D2 — Auth/pinning: unchanged, inherited from clusterd.

SPKI ed25519 pin over tokio-rustls, Tier-2-gated, per the existing federation service. No new certificates, no new trust store, no PKI in backupd. A peer’s identity is its mesh fingerprint (PeerConnection.peer_fingerprint), already what the probe pins against.

§D3 — Receiving-side guardrail: writes ONLY under <pool>/Backups/**, as plain dirs.

The peer backupd writes received bytes exactly like a LocalZfs repo writes local bytes: ensure_backups_namespace(<pool>) creates the single <pool>/Backups dataset (already allowed by the guard), then everything below is plain directories via tokio::fs. No new zfs create permission is needed and the blast-radius guard is unchanged — <pool>/Backups/<origin-id>/… are directories, not datasets, so guard_argv’s two-component create allow-list (zfs.rs:159-178) still covers the whole peer path without widening.

A peer backupd MUST refuse any received path that does not resolve under <pool>/Backups/ (path-canonicalise + prefix-check before the first byte), typed E_BACKUP_SNAPSHOT_FORBIDDEN-style. The origin cannot name an arbitrary destination: it names a pool label (§D5); the origin-id and run subpath are derived by the peer, never sent by the origin.

§D4 — Final sublayout: <pool>/Backups/<origin-id>/synvirt/v1/<job-id>/<run-id>/…

<origin-id> = the origin host’s mesh SPKI fingerprint (stable, collision-free, already the peer-trust primitive; not the hostname, which can change). Under it, the exact same synvirt/v1/<job-id>/<run-id>/streams/NN.zfs

• manifest.json layout a LocalZfs/NFS repo uses — so restore, prune, and the manifest reader are backend-agnostic and need no peer-specific parsing.

Rationale for keying on origin-id: one peer pool may receive from several origin hosts; segregating by origin fingerprint keeps their trees isolated, makes “whose data is this” answerable on the peer, and lets a peer prune or evict one origin without touching another.

§D5 — target_dataset reinterpreted (not removed): the peer POOL LABEL.

The field is wire-stable and reserved; P4 gives it meaning. It becomes the destination pool label on the peer (the operator-facing pool name, resolved to a zpool on the peer at probe + write time), NOT a dataset path. Validation at registration flips from “non-empty string” to:

• non-empty, and
• rejects any value containing / (a pool label is not a path — this is what stops an origin from steering writes outside Backups), and
• at probe time, must resolve to a real, importable pool on the peer (the probe gains a pool-resolution check, §D7).

The wire field name stays target_dataset for append-only stability; its rustdoc + the wizard label change to “destination pool”. (A future cleanup may add a clearer alias; not P4.)

§D6 — Push protocol: streaming, NOT resumable in v1.

One zfs send per stream, piped through the broker, written under a .part name on the peer and atomically renamed on completion — identical commit semantics to the file movers. Manifest-last still holds: the manifest is the last object written, so a push that dies mid-stream never leaves a valid restore point. A failed push discards the peer-side run directory (peer backupd cleans its own <origin-id>/…/<run-id> on abort) and the origin retries as a fresh run on the next occurrence — no partial-resume bookkeeping. Resumable transfer is a named future item, not P4.

§D7 — Receiver capacity / used_bytes.

The probe (peer_probe_response) today returns free_bytes: None. P4 fills it: clusterd’s remote_backup_probe is extended to stat the destination pool’s free space on the peer and return it, so the wizard shows real headroom and §D10 can pre-empt a doomed push. Repository used_bytes = the size of <pool>/Backups/<origin-id>/ on the peer, computed on the peer on demand and returned through the existing recalculate-usage route (which is proxied, so no new public surface). Both are peer-side stats brokered by clusterd; backupd never reaches across.

§D8 — Restore FROM a peer: PULL, brokered symmetrically.

Backup is origin-push; restore is a pull by the restoring host. The restoring host’s daemon asks clusterd (PATH_REMOTE_BACKUP_FETCH) to stream the named run’s files back from the peer into local <pool>/synvirt-restore/<run-id> staging, after which the unchanged D5 path runs: chain validation, staged receive, daemon-owned rename + define, XML sanitisation, new VM, never auto-started, never in-place. The peer is a byte source only; it does not define or start anything. Any host that can see the peer over the mesh can restore (not just the origin) — subject to the same Tier-2 gate.

§D9 — Exact moment the create-job hard block flips.

ensure_job_repository (http.rs:224-241) stops returning PeerDataPlaneUnavailable for a peer repo only when both hold, evaluated per-repo at job-create time (not a global feature flag):

1. Capability present on both ends: the origin’s clusterd advertises PATH_REMOTE_BACKUP_SEND and a fresh probe proves the peer’s clusterd answers it (not Unimplemented). A peer on an older build → block stays, with a message naming the peer and telling the operator to update it.
2. Repository status is Reachable, not DataPlanePending. The DataPlanePending status (types.rs:313-318) is retired for peer repos the moment the data plane ships; until a given repo probes Reachable with the capability, the block holds.

Capability-gating (not a flag) means a half-upgraded fleet can never push to a peer that would drop the bytes.

§D10 — Failure modes.

• Origin dies mid-push: peer discards the incomplete run dir; no manifest = no restore point (manifest-last). Origin’s run is Interrupted by the existing boot reconcile; next occurrence is a fresh run.
• Receiver full (ENOSPC): peer returns a typed error; origin marks the run Failed with E_BACKUP_PEER_FULL and a human message. The §D7 capacity probe SHOULD pre-empt this at schedule time (skip with a visible reason, reusing the Phase-6 skip-note surface) rather than fail mid-stream.
• Cert / SPKI rotation mid-run: the mesh channel drops → the stream errors → run Failed; the next run dials with the rotated pin. backupd holds no cert, so there is nothing in backupd to rotate — the blast radius of a rotation is one failed run, never corruption (manifest-last).
• Peer removed from cluster while it holds restore points: covered by the Phase-6 reconcile discipline for repositories — a peer repo whose peer is gone probes Unreachable; its restore points remain listed (honest) but restore fails typed until the peer returns. delete_repository still refuses while restore points reference it (the Phase-6 hold-lifecycle fix), so an operator is never silently stripped of recoverability.

§D11 — Fleet rollout order: receivers before origins.

Deploy the peer-receive capability to every host before any origin enables a peer job. §D9’s capability probe makes this safe even if violated (a push to an un-upgraded peer is blocked, not dropped), but the intended order is: receivers first, then origins. Per the standing fleet rule, .13 is never touched; rollout is .11/.12 only, and the actual deploy is a separate gated event — this doc does not authorise it.

§D12 — PLACEMENT: vTPM/swtpm restore fidelity is a Phase-5 RESTORE fix, NOT P4.

Confirmed gap (affects all restores, local included — not peer-specific): capture is fine (the parent stream carries <slug>-vtpm/ + <slug>_VARS.fd), but backup_restore.rs only zfs_renames the tree + sanitize_restored_xml rewrites the UUID and /<src>/→/<dst>/ paths (:123, :181-195). Nothing renames the captured <source-slug>-vtpm directory to the restored VM’s <new-slug>-vtpm, and nothing re-establishes the /var/lib/libvirt/swtpm/<new-uuid> → dataset symlink at restore time (ensure_vtpm_link runs only from vm_layout.rs:277 at daemon startup, and it looks for <new-slug>-vtpm, which does not exist). So a restored VM with a vTPM comes up with empty/fresh TPM state — the WS2016/BitLocker/measured-boot gap seen in L5.

Placement decision: fix it in the D5 restore path, as a small dedicated change, before or independent of P4 — because it is orthogonal to peer transport and already affects local/NFS restores today. The fix: at define time, rename the captured vtpm dir to the new slug (or adopt a slug-independent vtpm/ name) and call ensure_vtpm_link for the new UUID against the captured state. Capture needs no change. Not folded into P4.

§D13 — PLACEMENT: restore boot-verification (SureBackup-style) is a future phase, NOT P4.

Booting a restored VM in an isolated network to prove it actually starts + answers is valuable but orthogonal to peer transport, and it needs its own design (isolated vSwitch, heartbeat probe, teardown, scheduling). Placement: a dedicated later phase (candidate P6/P7), tracked as an open roadmap item. Not P4, not a blocker for it.

---

4. Wire additions (additive, enumerated)

Nothing here removes or reorders an existing field.

• Reinterpret PeerConnection.target_dataset semantics → destination pool label (§D5); rustdoc + validation change, wire shape unchanged.
• RepositoryTestResult.free_bytes starts being populated for peer repos (§D7) — field already exists, only the value changes from None.
• clusterd control proto: PATH_REMOTE_BACKUP_SEND, PATH_REMOTE_BACKUP_FETCH, and an extended BackupProbeResponse carrying pool free-bytes (§D1,§D7,§D8).
• New typed errors: E_BACKUP_PEER_FULL, E_BACKUP_PEER_CAPABILITY_MISSING (the §D9 half-upgrade case), joining the existing E_BACKUP_PEER_DATA_PLANE_UNAVAILABLE / E_BACKUP_PEER_UNREACHABLE.
• RepositoryStatus::DataPlanePending retires for peer repos once shipped (§D9) — stays in the enum (append-only) but a peer repo no longer parks in it.

---

5. Evidence ladder P4 code MUST clear (when it gets a GO)

Hermetic first; fleet is a separate gated event.

1. Unit: target_dataset pool-label validation (rejects /), origin-id path derivation, receive-path prefix guard (rejects anything outside <pool>/Backups/).
2. Hermetic two-node (extend the existing dev-box 2-node cluster rig): push a full + incremental from origin to peer over a real (loopback) mesh; assert the peer tree is <pool>/Backups/<origin-fpr>/synvirt/v1/..., manifest-last, sha256 per stream matches; restore-from-peer yields a byte-identical new VM, shut off, fresh UUID/MAC, origin untouched.
3. Guardrail: a hostile origin naming ../ or an absolute path is refused before the first byte; a target_dataset with / is refused at registration.
4. Failure: kill the origin mid-push → peer leaves no valid restore point; fill the peer pool → E_BACKUP_PEER_FULL, pre-empted by the capacity probe.
5. Capability gate: origin against an un-upgraded peer → create-job stays blocked with the update-your-peer message (§D9).
6. Fleet (separate GO): .11→.12 real push, guest-safety-gated, rollback bundles, .13 never touched.

---

6. Signature block

Sign each decision (initial + date) or annotate an amendment. P4 implementation has no GO until §D1–§D11 are signed. §D12 and §D13 are placement decisions — signing them records where the work lands, not that it is authorised now.

Decision	Summary	Sign / amend
§D1	Daemon-brokered transport via clusterd, not direct backupd↔backupd	SIGNED Tony 2026-07-09 — conditions §7.1
§D2	Auth/pinning inherited from clusterd, no new keys	SIGNED Tony 2026-07-09 (administrative)
§D3	Receive only under <pool>/Backups/**, plain dirs, guard unchanged	SIGNED Tony 2026-07-09 — condition §7.2
§D4	Sublayout <pool>/Backups/<origin-fpr>/synvirt/v1/<job>/<run>/…	SIGNED Tony 2026-07-09 — condition §7.2
§D5	target_dataset = peer pool label (reject /), not a path	SIGNED Tony 2026-07-09
§D6	Streaming push, manifest-last, NOT resumable in v1	SIGNED Tony 2026-07-09 — conditions §7.3
§D7	Capacity/used_bytes are peer-side stats brokered by clusterd	SIGNED Tony 2026-07-09 — conditions §7.4
§D8	Restore = pull from peer, then unchanged D5 define	SIGNED Tony 2026-07-09 — conditions §7.5
§D9	Block flips only on per-repo capability probe + Reachable status	SIGNED Tony 2026-07-09 — probe fails CLOSED
§D10	Failure modes: partial→no restore point, full→typed, rotation→one failed run	SIGNED Tony 2026-07-09 — mechanics §7.6
§D11	Rollout receivers-first; .13 never touched; deploy separately gated	SIGNED Tony 2026-07-09
§D12	vTPM restore fidelity → Phase-5 restore fix, NOT P4	LANDED 12639057 (2026-07-09)
§D13	SureBackup boot-verification → future phase, NOT P4	LOGGED as future phase

---

7. Signed conditions (2026-07-09) — part of the spec

§7.1 — Transport (§D1). (a) The implementation must state IN CODE whether Raft/control traffic shares the channel with bulk streams; if shared, a configurable per-push bandwidth throttle (sane default) plus the starvation-safety mechanism, documented as implemented. (b) clusterd restart mid-push = typed failure

• crash-honesty cleanup on BOTH sides. (c) When the fleet-equivalent run lands, report measured mesh throughput vs the 71 MB/s NFS baseline.

§7.2 — Origin identity in the UI (§D3/§D4). The UI never shows a bare fingerprint where a host name exists — resolve origin-fpr → friendly name from cluster inventory (five-second rule); the fingerprint appears in detail views only.

§7.3 — Non-resumable push hygiene (§D6). (a) RECEIVER-side sweep of incomplete run directories (boot reconcile + on demand) — the origin may never return; a half-pushed run never looks like a restore point and never leaks space silently. (b) Resumable push is a tracked item for a later phase.

§7.4 — Capacity semantics (§D7). used_bytes on the peer is scoped to the requesting origin’s subtree (<pool>/Backups/<origin-fpr>/). Free space is the pool’s REAL free — shared across origins, so it is displayed as pool free, never “yours”. The capacity limit stays a soft warning, never a block.

§7.5 — Restore ordering (§D8). Manifests fetch FIRST; the full chain is validated BEFORE any bulk stream leaves the peer (a broken chain costs zero bytes — the same invariant as local restore). A local free-space pre-check against the manifest sizes runs before pulling. An interrupted pull’s staged <pool>/synvirt-restore/<run-id> is swept by the existing staging reconcile — VERIFIED, not assumed.

§7.6 — Failure mechanics (§D10). The receiver discards an incomplete run directory on STREAM TERMINATION (drop/abort), not only at receiver boot — in addition to the §7.3 boot + on-demand sweeps; an origin that dies and never returns must not leak space on a receiver that never reboots. A permanently-lost peer gets a force-delete escape hatch on delete_repository: typed-name confirm + honest warning (“restore points become unrecoverable records”) — the soft-warning doctrine, not a hard trap. Orphaned run records stay visible in history, flagged repo-deleted.