Design spec — Backup engine (P3): jobs, runs, restore points, retention, scheduling, restore

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/specs/backup-p3-engine.md. Edit at the source, not here.

Design spec — Backup engine (P3): jobs, runs, restore points, retention, scheduling, restore

Status: CONDITIONALLY SIGNED (2026-07-01). Phase 1: GO.

Signature record:

• Phase 1 GO (contract + BackupRetentionPolicy collision fix — the fix was already sanctioned in the P2 doc as separate work). The spec itself commits first as its own docs commit, never bundled with P2’s parked doc changes.
• D1 SIGNED with conditions (all incorporated below, §2.1/§2.2/§2.9): doctrine amended in writing in docs/architecture.md + docs/BACKEND_CONVENTIONS.md (+ the CLAUDE.md §2 line and storaged rustdoc); a code-enforced prefix guardrail with a typed E_* error; zfs hold on source/anchor snapshots; a tracked consolidation item for the scattered direct zfs call sites (NOT P3 scope).
• D2 SIGNED with the additive multi-VM seam condition (§2.5); concrete shape to be confirmed when the job schema lands.
• Architecture endorsed; the four hard invariants from the signature are written into §2.3, §2.6, §2.8, §2.10.
• D3–D10 SIGNED 2026-07-01 (second signature round; conditions woven into §2.4/§2.5/§2.7/§2.8/§2.11): D3 with its own typed unsupported-type error + configurable full_every (soft warnings, never blocks); D4 catch-up = at most ONE run per job (most recent missed occurrence, flagged in the run record); D6 croner authorized (license + zero-new-advisories gate), sha2 promoted from the tree if already transitive; D7 quiesce flag VISIBLE in the restore-point list UI; D8 peer data plane → P4 confirmed; D9 poller interval configurable; D10 object-lock honesty checks (see §2.4).
• Phase 2 GO (2026-07-01): create_job real + peer hard-block + Playwright CRUD. Same bar: atomic commit, gates green, stop-and-wait.
• L5 double-gated (§2.12): on-host identity verification + explicit GO naming the target VM.
• Amendment 2026-07-07 (design directive, docs-only): reserved Backups dataset for SynVirt-peer repositories — §2.3.1 (layout proposal + the reserved-names table + product-wide name reservation) and the §2.9 guardrail extension. Implementation lands with the peer data plane project (P4); no engine code moves off this amendment. PeerConnection.target_dataset is deprecated in favor of the fixed layout (§2.3.1) — kept on the wire forever, ignored-or-validated in P4.

Audience: the agent that will execute this as a phased implementation, and the operator signing it off. Two halves:

1. What exists today (P1 skeleton + P2 repository control plane, verified against the 0.15.0 working tree on 2026-07-01 — all file:line references read directly from the tree).
2. P3 scope + architecture + phases + open decisions for the backup engine: the part that actually captures a snapshot, moves bytes into a repository, records restore points, prunes with GFS retention, schedules runs, and restores.

No VM, host, or runtime state is touched by this document or its review.

---

Part 1 — What exists today (verified)

1.1 The honest skeleton (P1) + repository control plane (P2)

One line: backup is a fully-wired metadata control plane with no engine. Every engine route returns a truthful 501 E_BACKUP_NOT_IMPLEMENTED (crates/synvirt-backupd/src/http.rs:150,174,440).

Layer	Where	State
Contract	crates/synvirt-backup-core	Types + one Error enum. Append-only wire (types.rs:14-16). No I/O.
Daemon	crates/synvirt-backupd	Sole redb owner (state.rs:5-8), UDS 0o600 at /run/synvirt/backupd.sock (config.rs:14,20). Repository CRUD + S3/NFS probes REAL; jobs/restore stubs.
Proxy	crates/daemon/src/api/backup_proxy.rs	/api/v1/backup/* → backupd 1:1; 503 E_BACKUP_UNAVAILABLE when down (:44); peer test-connection intercepted to clusterd (:307-323). Body cap 1 MiB (uds_proxy.rs:23).
Console	crates/web-ux-v2/src/views/backup/	Jobs table + KPIs built with disabled actions (“arrives in a later release”, BackupJobsTable.vue:36,146-181) + sample-data Preview mode (BackupView.vue:78-80). Repository wizard (6 steps) fully live.

Key P1 types the engine must build on (all in synvirt-backup-core/src/types.rs):

• BackupJob :483 — definition: id, name, source: String (opaque VM ref), backup_type, schedule, retention, repository_id, state, created_at, updated_at. All fields wire-required.
• JobState :182 — full lifecycle already modeled (Idle, Queued, Preparing, Running, Completed, Failed, Cancelled, Interrupted, Disabled) with #[schema(as = BackupJobState)] (:181). P1 “never drives a transition” (:169-173) — the seam for this engine. InvalidStateTransition → E_BACKUP_INVALID_STATE_TRANSITION (409) is defined and unused.
• BackupType :136 — Full, Differential, ForeverIncremental, Locked.
• Schedule :226 — { cron: String, enabled: bool }, stored verbatim, unparsed (“parsing and next-run computation are a later phase” :221-224).
• RetentionPolicy :239 — GFS {daily, weekly, monthly, yearly}, default 7/4/6/1 (:250-258). Known OpenAPI defect: it collides with synvirt_logging::RetentionPolicy; the logging shape wins in openapi.snapshot.json:34417 and BackupJob.retention’s $ref (:21081) resolves to the WRONG shape. The P2 doc mandates the fix name: BackupRetentionPolicy — never re-register the colliding name (docs/modules/backup/repository-management.md:109-111).
• RestorePoint :509 — id, job_id, repository_id, source, backup_type, size_bytes, created_at.
• Repository :373 — frozen 5-field head + #[serde(default)]-appended connection, secret_ref, status, capacity_limit_bytes, used_bytes, updated_at. used_bytes :398 exists and is never written today.
• Connections: S3Connection :303 (has unused object_lock :325, prefix :314), NfsConnection :346, PeerConnection :334 (reserved address field = the phase-2 seam).

redb store (synvirt-backupd/src/state.rs): tables jobs :25, repositories :28, restore_points :31, repo_secrets :36 (AES-256-GCM ciphertext only). Existing methods: put/get/list/delete for jobs + repositories, put_repo_secret/get_repo_secret/delete_repo_secret, put_restore_point :308, list_restore_points :331. Missing for an engine: get_restore_point, delete_restore_point, and any run-level store — put_job exists but has no HTTP caller (blocked at the 501).

1.2 Binding mandates inherited from P2

1. Peer hard-block. When create_job lands it MUST refuse a PeerHost repository target with Error::PeerDataPlaneUnavailable → E_BACKUP_PEER_DATA_PLANE_UNAVAILABLE — “a hard block, not a soft warning, because it literally cannot work” (docs/modules/backup/repository-management.md:207-212; backup-core/src/error.rs:54-59).
2. Append-only wire. Never reorder/remove serialized enum variants; new Repository/BackupJob fields #[serde(default)]; OpenAPI additions additive-only under brand-new schema names (repository-management.md:137-147).
3. Secrets write-only. Never on a read, never in logs/error details; sealed via synvirt_crypto::EncryptionKey; secret_ref is a handle only.
4. backupd is the exclusive redb owner; the daemon never opens redb.
5. GFS schema name = BackupRetentionPolicy (see 1.1).
6. English-only strings; every #[utoipa::path] sets an explicit operation_id; re-dump openapi.snapshot.json + regenerate TS after any contract touch.

1.3 The storage plane the engine stands on

The “sole zfs executor” doctrine is aspirational, not enforced. It is stated in CLAUDE.md:148 and storaged’s own docs (synvirt-storaged/src/lib.rs:3), but storaged’s HTTP surface (synvirt-storaged/src/http.rs:28-53) is pool-level only — no snapshot create/destroy, no dataset/zvol ops, no zfs send/recv, no streaming. Meanwhile the actual data-path components all run zfs directly:

• The shipped VM snapshot stack: crates/daemon/src/api/snapshots.rs (2610 lines) — “a thin shell around the zfs CLI” (:1-27). Atomic recursive zfs snapshot -r <dataset>@<name> over the per-VM dataset (:224-229), restore/clone/delete, quiesce (below), cross-host zfs send | zfs recv (:993, :1769, federation_hosts.rs:139).
• The migrator sink: crates/migrator/src/zfs_sink.rs — zfs create -s -V … -o volblocksize=64k -o compression=lz4 (:94-105), destroy, snapshot, blkdiscard zero-punching (:307).
• The per-VM layout allocator: crates/core/src/storage/vm_layout.rs (zfs create/destroy :254,288,396,543).

Quiesce already exists and is proven. snapshots.rs::try_fsfreeze :1852 does guest-ping → guest-fsfreeze-freeze via virsh qemu-agent-command (run_qemu_agent :1870), thaws after the snapshot, and falls back to crash-consistent when the agent is absent, reporting quiesced=false + quiesce_skipped_reason (:215-234, response fields :135-143). The guest-tools ISO ships qemu-ga for every guest tier.

Per-VM layout — two shapes coexist; never hardcode either. The newer root-level shape is <pool>/<slug> with the boot zvol at <pool>/<slug>/<slug>, disk N at <pool>/<slug>/<slug>-disk<N>, plus <slug>_VARS.fd, <slug>-vtpm/, <slug>.xml as files in the parent dataset (crates/core/src/storage/vm_layout.rs:1-30,180,222). Legacy installs still use <pool>/vms/<slug>/. The daemon resolves both: discover_vm_dir (daemon/src/vm_layout.rs:98,179), and the snapshot stack has resolve_vm_layout / vm_dataset / list_child_volume_datasets (snapshots.rs:1388,1376,1951). Because every persistent VM artefact is a file in the parent dataset or a child zvol, one recursive snapshot of the parent dataset captures the complete VM (disks + NVRAM + vTPM + XML mirror) atomically.

Streaming transport exists. synvirt-ipc::UdsClient::send_stream (synvirt-ipc/src/client.rs:138) carries chunked multi-GB bodies; its own rustdoc names “a zfs send stream feeding the cross-host clone data mover” (:130-137). Three production callers today, all cluster clone — none in backup. The backup proxy path is buffered + 1-MiB-capped (uds_proxy.rs:23) — fine for control, unusable for data. No backup data may ever flow through the daemon proxy.

NFS: the daemon’s persistent datastore mounts live in shared_storage.rs::mount_one :1072 (typed E_DATASTORE_NFS_CLIENT_MISSING); backupd’s probe mounts are throwaway read-only mounts under /run/synvirt/backupd/probe/<uuid> (probe/nfs.rs:14,20,46).

S3: probe/s3.rs has the correct client recipe (path-style, custom endpoint/region, WhenRequired checksums :41-42 — the SDK default breaks S3-compatible servers) but only list_buckets + create_bucket. No object PUT/GET, no multipart, nothing consumes object_lock/prefix yet.

1.4 Job-engine precedents to copy (and their gaps)

Pattern	Where	What P3 takes
Callback plane (engine calls daemon for VM-context ops)	migratord → daemon: /define-and-start, /connect-nics, /destroy-and-undefine, /resolve-zpool (synvirt-migratord/src/callback_proto.rs:16-23; server daemon/src/api/migrator_callback.rs:47-54; spawn daemon/src/main.rs:1739-1776)	The identical seam: a backupd-callback socket for snapshot/resolve/define ops
Cancellation	CancellationToken per job + immediate persisted Cancelled (migrator/src/jobs.rs:85,202-223); cooperative tokio::select! at chunk boundaries (orchestrator.rs:348-352)	Same mechanism
Progress	Progress{total_bytes, copied_bytes, throughput_bps_1m} persisted + broadcast per update, emission throttled to 64 MiB (orchestrator.rs:528-547,707)	Same fields, same throttle
Restart survival	livemotiond reconcile() bulk-marks `queued	preparing
Single-slot guard	packages ChangeRegistry::start refuses a second active change (synvirt-packages/src/changes.rs:80-84); updaterd Busy	One active run per job + a global runner semaphore
Tasks in the dock	TaskRegistry/TaskHandle{progress,succeed,fail} (synvirt-observability/src/tasks/mod.rs:26-137), daemon wrapper task_helper::start_with_actor (daemon/src/api/task_helper.rs:50-78)	Gap: no automatic sub-daemon→tasks bridge exists. Migration jobs surface only as failure events via a 5 s daemon poller (daemon/src/main.rs:1784-1858). P3 adds a poller bridge (§2.7)
Scheduling	No cron crate anywhere; in-daemon tokio::time::interval loops (observability 30 s, rollup 60 s); the one systemd timer (synvirt-cert-renew.timer, Persistent=true catch-up); sanoid = config-file + its own packaged 15-min tick	In-backupd tick + boot catch-up (§2.8)
Concurrency gap	No cross-subsystem per-VM operation lock exists (verified: migration vs snapshot vs backup can collide today)	Documented limitation; P3 serializes only its own runs (§2.9)

Store engines by subsystem: migrator = sled, livemotiond = SQLite, observability = SQLite, backupd = redb (sole redb user in the workspace).

1.5 Frontend state

• stores/backup.ts (Pinia, legacy fetchJson convention, hand-declared types) — 3 GETs; write actions for repositories only.
• BackupView.vue — Preview/sample mode when all three lists are empty (:78-80); real→console mapping already wired (:93-141) with lastRun/nextRun/sizeBytes stubbed; “New backup job” + per-row Run/Pause + Restore buttons exist, disabled with release-note tooltips.
• Reusable: WizardStepper.vue rail (the repository wizard already uses it standalone), TaskProgress.vue (generalised from the migrator’s phase bar), ActivityDock reads /api/v1/tasks with no new fetch path (ActivityDock.vue:12-14).
• Curated error copy exists for all six E_BACKUP_* codes (api/errors.ts:493-522).
• E2E: e2e/backup-repository-{s3,nfs,peer}.spec.ts — real daemon, env-gated, skip (never fail) when unconfigured; SYNVIRT_E2E_BASE required (playwright.config.ts:23-32).
• Backup surface is deliberately hardcoded English (no vue-i18n keys) — P3 matches its siblings (RepositoryWizard.vue:10-12).

---

Part 2 — P3 design

2.0 One-paragraph summary

backupd becomes a real engine following the migratord split: backupd owns job state, orchestration, and all repository I/O; the daemon serves a small callback plane that reuses the shipped snapshot stack (atomic recursive snapshot + proven quiesce-with-fallback) and, for restore, the native VM define pipeline. A backup = zfs snapshot -r of the VM’s dataset tree, then one zfs send stream per dataset (parent + each zvol) written to the repository (NFS file / S3 multipart) under a manifest that is written last and acts as the commit marker. Incrementals are zfs send -i against the job’s anchor snapshot. Restore points prune chain-wise under GFS retention. Restore always lands as a new VM (never in-place). Runs surface in the ActivityDock via a daemon-side poller bridge. Peer repositories are hard-blocked at job create, as mandated.

2.1 Process placement + the zfs question (Decision D1)

Proposed: backupd executes zfs send (backup) and zfs receive (restore) itself; snapshot create/destroy and quiesce stay in the daemon behind the callback plane.

Rationale, grounded in 1.3: storaged exposes none of the needed primitives and adding a streaming surface to it is new plumbing with no consumer precedent; every existing data-path component (daemon snapshot stack, migrator sink, core allocator) already runs zfs in-process; zfs send is read-only against an immutable snapshot; zfs receive targets a freshly-created, backup-owned dataset. Snapshot creation stays in the daemon because freeze→snapshot→thaw must be one tight in-process sequence (the shipped create handler already is exactly that, snapshots.rs:215-234) and because reusing it inherits naming validation, clone guards, and the quiesce fallback for free.

This amends the storaged doctrine explicitly, in writing (signature condition). Phase 1 lands the amendment in docs/architecture.md and docs/BACKEND_CONVENTIONS.md (plus the CLAUDE.md §2 line and the storaged rustdoc headers): storaged = control plane for storage object lifecycle (create/delete/resize/list); streaming data-plane work (zfs send/recv) executes in the owning daemon — migratord precedent, now backupd. No pretending the old doctrine holds.

Three more signature conditions bind D1:

1. Blast-radius guardrail, enforced in code (§2.9): backupd may create/destroy ONLY snapshots carrying its own prefix; it never destroys, renames, or modifies datasets or zvols outside its restore staging namespace; violations trip a typed E_BACKUP_SNAPSHOT_FORBIDDEN, never a silent no-op.
2. zfs hold on source/anchor snapshots for the duration of every send, and anchors stay held until superseded — nothing (including our own retention) can destroy a snapshot mid-stream or orphan a chain (§2.2).
3. Tracked consolidation item (TODO.md, Phase 1): funnel the 14+ scattered direct zfs invocations across daemon/migrator/core through one shared helper later. Explicitly NOT P3 scope.

Alternative (rejected unless you prefer it): route send/recv through storaged with a new streaming surface — cleaner doctrine, but adds a storaged HTTP-stream data plane, a second UDS hop for every byte, and touches R2’s “restart never touches the data plane” guarantee for no functional gain.

2.2 The daemon callback plane (new, mirrors migratord’s)

New UDS server in the daemon (default /run/synvirt/backupd-callback.sock, config key backup_callback_socket_path beside the migrator’s, daemon/src/config/mod.rs:90 precedent; spawned like daemon/src/main.rs:1739-1776). backupd is the client. Routes:

Route	Reuses	Returns
POST /resolve-vm {vm}	resolve_vm_layout / discover_vm_dir / list_child_volume_datasets	{uuid, slug, dataset, mountpoint, child_datasets[], running}
POST /snapshot-vm {vm, label, quiesce}	the shipped freeze→zfs snapshot -r→thaw path (snapshots.rs:215-234, try_fsfreeze :1852)	{snapshot, quiesced, quiesce_skipped_reason, datasets[]}
POST /destroy-snapshot {dataset, label}	snapshots.rs delete internals (recursive, clone-guarded)	204
POST /resolve-zpool {label}	same as migrator_callback.rs:114-137 (storaged registry)	{zpool}
POST /define-restored-vm {…} (Phase 8)	the migrator-proven native define pipeline (migrator_domain.rs::ProfileDomainStarter :60,104)	{domain_uuid}

Snapshot-stack internals needed by the callback get extracted into callable fns (no behavior change to the existing REST handlers — same pattern as the support-bundle extraction from the console TUI).

Backup-owned snapshots are namespaced synvirt-backup-<run-short-id> (the guardrail prefix, §2.9) and tagged with a ZFS user property (synvirt:backup-run=<run-id>, precedent DESCRIPTION_PROP snapshots.rs:58) so the snapshot UI can label them and teardown can find strays. Holds (signature condition): backupd places a zfs hold (tag synvirt-backup) on every snapshot it is actively sending and keeps the job’s anchor (the last successful run’s snapshot) held until a newer anchor supersedes it — so neither our own retention nor an operator zfs destroy can kill a snapshot mid-stream or orphan an incremental chain (zfs destroy on a held snapshot fails with EBUSY by design). Supersede order: hold new anchor → release + destroy old one. Older synvirt-backup-* snapshots of the job are destroyed after a successful run; deleting a job releases and destroys its anchor.

2.3 Restore-point format (repository layout + manifest)

<root or S3 prefix>/synvirt/v1/<job-id>/
  job.json                          # {job name, vm name/slug} for browsability
  <run-id>/
    streams/00-parent.zfs           # zfs send of the parent dataset (nvram, vtpm, xml)
    streams/01-<zvolname>.zfs       # one per child zvol
    manifest.json                   # WRITTEN LAST = commit marker

• Streams are raw zfs send output. Send flags: -Lc (large blocks + compressed — preserves on-disk zstd/lz4 so bytes over the wire ≈ allocated bytes; sparse/thin zvols stay cheap, consistent with feedback_thin_provision). Flags are recorded in the manifest; restore replays with plain zfs receive.
• Incremental streams are zfs send -Lc -i @<anchor> @<new> per dataset.
• manifest.json (versioned, "version": 1): run + job ids, vm {name, slug, uuid}, backup_type, chain_parent_run_id, quiesced + skip reason, timestamps, send flags, per-stream {dataset, role, file, bytes, sha256, incremental_from}, embedded domain XML (from the parent dataset’s mirror at snapshot time), totals. sha256 computed inline during upload — every restore point is verified-on-write by construction.
• Hard invariant (signature): manifest.json is the commit marker, written LAST, with inline sha256 per stream. The RestorePoint redb row is written ONLY after the manifest rename succeeds — a run without a readable+valid manifest is INCOMPLETE and is never listed as a restore point. The boot reconcile re-validates: any restore-point row whose manifest is missing or unparseable is dropped and its artifacts flagged for cleanup.
• A run directory without manifest.json is garbage by definition: prune and crash-cleanup delete it without ceremony.

2.3.1 Peer repositories — the reserved Backups dataset (amendment 2026-07-07)

When a SynVirt host is selected as a backup repository (PeerHost), received backup data lives under a RESERVED top-level ZFS dataset named exactly Backups (case-sensitive) on the receiving peer:

<pool>/Backups/<origin-id>/<job-or-vm>/<run-id>

The sublayout below <pool>/Backups/ is a spec proposal — the final shape is confirmed at the P4 (peer data plane) spec review. What is FIXED now:

• The dataset name is a product convention, never operator-editable — the same allowed-hardcode class as /etc/synvirt/. The repository wizard’s peer step exposes at most a POOL selector; there is no dataset field.
• Auto-create + honest adopt. The P4 data plane creates <pool>/Backups (zfs create -p) when absent. If a dataset with that name already exists and holds foreign data: typed E_* + user_message — adopt only if empty/compatible, never silently mix.
• Product-wide name reservation (pool top level). Every product surface that can create or rename a top-level dataset (storaged storage-object ops, datastore-browser create paths) must refuse the name Backups for anything that is not the backup subsystem, with a typed E_*. VM datasets are unaffected — they live under <pool>/vms/<slug> (or the root-level <pool>/<slug> layout), never at the reserved name.
• Wire stays additive. The P2 PeerConnection shape carries a free-form target_dataset field: it is deprecated by this amendment and NEVER removed. P4 ignores it or validates it — if present, it must resolve to the reserved layout above; anything else is a typed validation error, not a silent alternate location.

Reserved top-level dataset names (the single canonical table):

Name	Position	Owner	Purpose
Backups	<pool>/Backups/**	backup subsystem	backup DATA — peer-received restore points (<pool>/Backups/<origin-id>/…, P4) AND local-ZFS repository data (<pool>/Backups/local/<repo-id>/…, L5 pre-slice, 2026-07-08). backupd may zfs create ONLY the <pool>/Backups dataset itself; the subtree below is ordinary directories.
synvirt-restore	<pool>/synvirt-restore/<run-id>	backupd restore engine (§2.10, Phase 8)	restore staging receives (signed D5/P3)

Amendment 2026-07-08 (L5 pre-slice): LocalZfs is now an ADDABLE repository kind — a dataset on a pool this host manages. The wizard exposes a pool selector only; data lives under <pool>/Backups/local/<repo-id>/. The guardrail (§2.9) permits zfs create of the reserved <pool>/Backups root (paralleling <pool>/synvirt-restore); the per-repo subtree is plain directories written with tokio::fs, never datasets.

2.4 Data movers

NFS: mount read-write for the duration of the run at /run/synvirt/backupd/mounts/<repo-id> (probe pattern, probe/nfs.rs:14, minus ro), stream zfs send stdout → tokio::fs::File with a 1 MiB copy buffer (zfs_sink precedent) + sync_all, write to manifest.json.part → rename. Unmount best-effort after the run.

S3: multipart upload per stream, 64 MiB parts, reusing the probe’s client recipe verbatim (probe/s3.rs:20-42). AbortMultipartUpload on any failure or cancel. Honors S3Connection.prefix. Object lock (D10 SIGNED with honesty condition): P3 relies on bucket-default retention — but that only exists when the bucket has Object Lock enabled, which can ONLY be set at bucket CREATION. So when a repository claims immutability (locked/object_lock), the engine verifies via GetObjectLockConfiguration (at test-connection and/or first write) and surfaces the truth: if the bucket lacks object-lock/default-retention, the repository and its restore points must NOT claim immutability — soft warning + honest flag, never silent false security. Per-object Mode/RetainUntilDate params are deferred to P4. Known P2 follow-up (logged in docs/modules/backup/repository-management.md): the wizard’s create-bucket does not pass x-amz-bucket-object-lock-enabled, so wizard-created buckets can never be object-locked. The scoped E2E S3 policy will additionally need s3:GetBucketObjectLockConfiguration when this check’s evidence level lands. Prune against a locked object surfaces the repository’s immutability honestly (§2.6).

Peer: hard-blocked at create (§2.5). When the P4 data plane lifts the block, received data lands under the reserved Backups dataset — layout and name reservation in §2.3.1. LocalZfs/LinuxTarget stay non-addable (http.rs::ensure_addable_kind :190).

Repository.used_bytes is recomputed (sum of restore-point sizes) after every run and prune. capacity_limit_bytes is a soft limit per P1’s definition: a projected overrun logs a warning on the run, never blocks.

2.5 Job + run model (wire: everything additive)

• BackupJob unchanged on the wire; source = the VM name (single-VM jobs in P3 — D2 SIGNED). Appended #[serde(default)] fields: full_every: Option<u32> (force a new full after N incrementals, default 7 — Decision D3) and last_run_at: Option<DateTime<Utc>>. D2 condition (multi-VM seam, additive): source stays frozen as the primary/single ref; multi-VM later = an appended sources: Vec<String> #[serde(default)] where an empty vec means “use source” — no singular→plural rename, no wire break. Concrete shape to be confirmed with the operator when the job schema lands (Phase 1/2).
• New type BackupRun (new schema name, additive): id, job_id, state (JobState), backup_type, started_at, finished_at, bytes_total, bytes_transferred, throughput_bps, current_dataset, quiesced, restore_point_id, error {code, message}. New redb table runs; store gains put_run/get_run/list_runs(job_id)/get_restore_point/ delete_restore_point.
• RestorePoint appended fields: run_id, chain_parent: Option<RestorePointId>, quiesced: Option<bool>, manifest_path.
• State machine now enforced via a can_transition whitelist (updaterd pattern, synvirt-updaterd/src/state.rs:49-81): Idle → Queued → Preparing → Running → Completed | Failed | Cancelled; Interrupted only from the boot reconcile; Disabled ⇄ Idle by operator. Anything else → 409 E_BACKUP_INVALID_STATE_TRANSITION.
• create_job validation: name non-empty; repository exists; PeerHost repo → E_BACKUP_PEER_DATA_PLANE_UNAVAILABLE (the mandate); kind addable; VM resolvable via /resolve-vm; cron parses (§2.8); backup_type ∈ {Full, ForeverIncremental} — Differential/Locked are rejected with their own typed error E_BACKUP_TYPE_UNSUPPORTED whose user_message says the type is unsupported in this release (D3 signature: honest, not a generic 400 nor a repurposed invalid-job). full_every is a smart DEFAULT (7), operator-configurable per job, no hardcoded floor/ceiling — extreme values (e.g. 0 = never force a full) get a soft warning in logs/UI advisory copy, never a block.
• Routes (all under the existing proxy; new ones additive):
  • POST /v1/backup/jobs — becomes real (was 501).
  • PUT /v1/backup/jobs/:id — new (definition edits; rejected while a run is active).
  • POST /v1/backup/jobs/:id/run — new, manual trigger → 202 + run id; 409 E_BACKUP_RUN_ACTIVE if one is in flight.
  • GET /v1/backup/jobs/:id/runs + GET /v1/backup/runs/:id — new.
  • POST /v1/backup/jobs/:id/cancel — becomes real (cancels active run).
  • DELETE /v1/backup/restore-points/:id — new (manual prune).
  • POST /v1/backup/restore — becomes real in Phase 8.
  • Every handler gets a full #[utoipa::path] + unique operation_id (the job routes are unannotated forwarders today — backup_proxy.rs:137-154,433).
• New Error variants (append-only codes): RunNotFound (E_BACKUP_RUN_NOT_FOUND, 404), RestorePointNotFound (E_BACKUP_RESTORE_POINT_NOT_FOUND, 404), RunActive (E_BACKUP_RUN_ACTIVE, 409), SnapshotFailed (E_BACKUP_SNAPSHOT_FAILED, 502), TransferFailed (E_BACKUP_TRANSFER_FAILED, 502), RestoreFailed (E_BACKUP_RESTORE_FAILED, 502), InvalidSchedule (E_BACKUP_INVALID_SCHEDULE, 400), and the guardrail’s SnapshotForbidden (E_BACKUP_SNAPSHOT_FORBIDDEN, 500) — an internal safety trip, loud by design. Each gets curated ERROR_MAP copy.
• The BackupRetentionPolicy fix ships in Phase 1: add #[schema(as = BackupRetentionPolicy)] to the backup-core type (exactly how JobState :181 solved the same clash). Rust type name and serde wire shape are untouched; the snapshot gains the correct component and BackupJob.retention stops resolving to the logging shape. This corrects the documented defect through its sanctioned fix.

2.6 Retention (GFS, chain-aware) + prune

A chain = one full + its dependent incrementals. Pruning operates on whole chains only (an incremental without its full is garbage).

1. After every successful run (and on manual trigger), classify restore points into GFS buckets (daily/weekly/monthly/yearly, keep-counts from the job’s policy) by created_at.
2. A chain survives if any member is retained by any bucket. Hard invariant (signature): before pruning anything, retention verifies chain integrity — the chain’s full + every kept incremental’s manifest linkage must resolve, and the job’s held anchor is never a prune candidate. An incremental is never orphaned: members are only deleted together with their whole dead chain.
3. Dead chains are deleted: repo artifacts first (S3 DeleteObjects / NFS unlink), then redb rows, then used_bytes recompute.
4. locked repositories / object-locked objects: prune skips the chain and logs a warning task — immutability is the product promise, not an error.
5. full_every bounds chain length so GFS pruning can actually free space on forever-incremental jobs.

2.7 Observability: tasks, progress, events

• backupd persists run progress (bytes_transferred, 64 MiB emission throttle — migrator numbers) in redb; run list/get expose it.
• Daemon poller bridge (D9 SIGNED): a daemon task on a configurable interval (daemon config key backup_task_bridge_interval_secs, default 5 — no magic number; exact shape of the migrator failure poller, daemon/src/main.rs:1784-1858) that mirrors backupd runs into observability Tasks: new run → TaskRegistry.start (kind backup.run, target the VM, TaskSource::Scheduler for scheduled runs so the dock marks them system-created); progress → TaskHandle::progress(pct); terminal → succeed/fail. Prune and restore get their own kinds (backup.prune, backup.restore). Result: runs appear in the ActivityDock, /monitor/tasks, and the bell with zero new frontend fetch paths.
• The UI polls /api/v1/backup/jobs + runs at 5 s while any run is active (migrator store pattern, stores/migrator.ts:35); an SSE/WS run stream is explicitly deferred (polling is the shipped precedent and the dock covers ambient visibility).

2.8 Scheduler

In-backupd tokio loop, 30 s tick (observability-detector cadence):

• Parse Schedule.cron with a new dependency (croner — Decision D6; P1 stored cron verbatim precisely so a parser could land later, types.rs:221-224). Validation at create/update returns E_BACKUP_INVALID_SCHEDULE.
• A job fires when next_after(max(last_run_at, boot_grace)) <= now, the job is enabled, not Disabled, and no run is active.
• Boot catch-up (D4 SIGNED, scoped): catch-up = at most ONE run per job at boot — the most recent missed occurrence only, never a backlog replay — fired after the boot reconcile and a grace period (the Persistent=true semantics of synvirt-cert-renew.timer). The run record carries a catch_up: bool flag (appended BackupRun field, lands with the scheduler phase) so dock/history show honest provenance. Storm control (signature, hard): catch-up is capped and jittered, all operator-configurable in /etc/synvirt/backupd.toml — [scheduler] boot_grace_secs (default 60), catchup_max_runs (max jobs that may catch up per boot, default 0 = unlimited-but-serialized), catchup_jitter_secs (random 0..N enqueue jitter per job, default 300). No magic numbers: defaults live in BackupdConfig serde defaults and are documented in the module doc; the global runner semaphore (§2.9) already serializes execution.
• Startup reconcile first: any queued|preparing|running run → Interrupted (livemotiond pattern, store.rs:136-148) + orphan .part / manifest-less run-dir cleanup + stray synbk-* snapshot sweep (keep anchors).

2.9 Concurrency + safety

• One active run per job (E_BACKUP_RUN_ACTIVE) + a global runner semaphore of 1 in P3 (packages single-slot precedent) — serialized runs, no I/O storms; widening to per-repo parallelism is a later knob.
• Cancellation: CancellationToken checked at stream-chunk boundaries; on cancel: kill the zfs send child, abort the multipart / delete .part files, destroy the new snapshot (keep the anchor), persist Cancelled immediately (migrator UX precedent).
• Documented limitation (unchanged from today): there is no cross-subsystem per-VM lock — a migration and a backup of the same VM can overlap, exactly as a migration and a manual snapshot can today. The snapshot itself is atomic regardless. A workspace-wide per-VM operation lock is out of P3 scope (it belongs to the daemon, not to backup).
• Blast-radius guardrail (signature condition, enforced in code). All of backupd’s zfs invocations go through ONE guarded helper module that refuses, with typed E_BACKUP_SNAPSHOT_FORBIDDEN, anything outside this allow-list — checked on the literal argv before spawn, unit-tested against bypass:
  • zfs snapshot / zfs destroy of snapshots only, and only when the snapshot label starts with synvirt-backup- (destroying a dataset — an argument without @ — is unconditionally refused);
  • zfs hold / zfs release with tag synvirt-backup on those same snapshots;
  • zfs send (read-only) of any snapshot it holds;
  • zfs receive ONLY into the restore staging namespace <pool>/synvirt-restore/<run-id> (§2.10), and zfs destroy -r ONLY of datasets under that exact namespace (failure cleanup);
  • (P4 amendment, receiving-peer side only) dataset create/destroy ONLY under <pool>/Backups/** (§2.3.1) — the same argv-checked guard mechanism, extended when the peer data plane lands. Enforcement in code, not convention; the origin-side allow-list above is unchanged;
  • no zfs rename, no property writes outside its own snapshots, no zvol/dataset mutation anywhere else, ever. Retention is prefix-scoped snapshot deletes only. The snapshot on the LIVE VM dataset is created by the daemon callback (§2.2), so backupd never mutates a production dataset even indirectly.
• backupd stays root (it already must for mount.nfs); zfs is in the same trust domain. The callback socket is 0o600 root-owned like its migrator sibling.

2.10 Restore (Phase 8) — always a new VM

1. Operator picks a restore point + target pool (+ new name, default <vm>-restored-<n>).
2. Chain validation first (hard invariant, signature): before any byte moves, backupd resolves the full chain from the manifests (full → every incremental up to the chosen point), verifies every stream is present and every manifest parses + links; a broken chain fails fast with E_BACKUP_RESTORE_FAILED before touching the pool.
3. backupd zfs receives parent dataset + zvols into the restore staging namespace <pool>/synvirt-restore/<run-id> (the only place the guardrail lets it write, §2.9); sha256 verified against the manifest as streams flow.
4. Callback /define-restored-vm: the daemon performs the final placement — zfs rename of the staged tree to <pool>/<newslug> (metadata-only, the migrator relocation precedent migrator_domain.rs:750-770) — then defines the VM through the native pipeline (ProfileDomainStarter precedent), fresh UUID + MACs, never auto-starts.
5. Failure at any step: backupd destroys only its staging tree, typed E_BACKUP_RESTORE_FAILED, nothing half-defined, nothing outside staging touched.

Hard invariant (signature, absolute): restore is ALWAYS a new VM via the native define pipeline — never in-place, no override flag exists at any layer. Guest safety by construction: local ZFS snapshots already cover in-place rollback via the shipped snapshot feature; repository restore is recovery onto fresh metal, where “new VM” is the honest semantic.

2.11 UI (Phase 9)

• New backup job wizard — Teleport modal reusing WizardStepper (the repository wizard’s exact construction): 1 VM · 2 Repository · 3 Type & schedule · 4 Retention · 5 Review. VM picker from the inventory store; repository picker excludes peer/pending kinds with the mandated explanation; cron presets (daily 02:00 / weekly Sun / custom) + retention GFS inputs with the 7/4/6/1 defaults.
• Jobs table: Run/Pause(→Disabled)/More menus become real; state chips map BackupJobState; lastRun/nextRun/sizeBytes populate from runs + restore points (the mapping stubs at BackupView.vue:106-115 already exist).
• Runs drawer per job: TaskProgress.vue steps (Snapshot → Transfer → Verify → Prune) + bytes/throughput, cancel button (hold-to-cancel precedent available).
• Restore points card becomes real; Restore wizard: point → target pool (picker via storage store) → name → typed-name confirm (destructive-action convention) → progress. D7 signature condition: the quiesced flag is VISIBLE in the restore-point LIST itself (a “Quiesced” vs “Crash-consistent” chip per row), not buried in the manifest — the operator picking a point sees the consistency tier at a glance.
• KPIs real: Protected (VMs with an enabled job), Last 24h (runs), Repository usage (used_bytes/capacity), Restore points count, Last backup. HostSummary.last_backup_at gets its real writer.
• Preview mode now triggers only when there are no jobs AND no repositories AND no restore points — unchanged logic, it simply empties out naturally.
• Copy stays hardcoded English (sibling convention); store keeps the fetchJson + hand-declared-types convention of stores/backup.ts; vitest specs for the store actions + wizard gating + state mapping.

2.12 Test & acceptance ladder (the evidence bar)

Level	What	Gate
L1	Unit: state machine whitelist, GFS chain classifier, cron next-fire, manifest ser/de, validation (incl. the peer hard-block), store methods	cargo test per phase
L2	Integration (SYNVIRT_E2E_ZFS=1, root, self-contained): create a file-backed throwaway zpool (sparse file + zpool create), fabricate a VM-shaped dataset tree (parent files + 2 zvols with known content), run full + incremental → hermetic NFS export, corrupt-a-byte checksum test, prune, restore to a second file-backed pool, byte-compare	dedicated test binary in backupd; skips (never fails) without the env
L3	E2E S3: same run against SynBaaS (SYNVIRT_E2E_S3_* env, skip-if-unset — the P2 pattern)	env-gated
L4	Playwright: job CRUD + wizard happy path + honest-error assertions against the hermetic daemon (no data run; the 2-node harness from P2 re-used as-is)	SYNVIRT_E2E_BASE
L5	Real-VM run on the dev appliance .65 — double-gated (signature), see below	explicit GO per run
—	Offline gates on every phase: cargo fmt / clippy -D warnings / test, cargo audit (no NEW advisories — the 6 baseline ones are logged), vue-tsc, eslint --max-warnings 0, additive --dump-openapi diff vs openapi.snapshot.json	every commit

The callback plane gets its own L2-style test using the harness technique from P2 (svqa daemon + root backupd, /tmp staging).

L5 double gate (signature condition). .65 is synvirt-01, the nested dev appliance (a VM on the external lab hypervisor host, CLAUDE.md §7) — a lab host, not production. Gate 1 — on-host identity preflight, verified from the host itself at run time, all must pass or L5 aborts: management IP is 10.10.26.65; hostname + /etc/synvirt/release match the dev appliance; the node’s cluster identity fingerprint is NOT a member of the production synnet-cluster; the target is none of .11/.12/.13, the VIP .100, or the web hosts. Gate 2 — explicit GO from the operator at that phase, naming the target libvirt domain. Test VM source: .65’s own guest inventory (virsh list --all at GO time — typically the Windows or Linux test guests CLAUDE.md §7 lists), or a fresh disposable guest created from its ISO library for the run; the GO message names the domain explicitly (standing rule: every VM-touching task names domain + host).

2.13 Phases (each = atomic commit(s), all gates green, stop-and-wait honored between phases)

#	Slice	Proves
1	Contract: backup-core additive types (BackupRun, appended fields, new Error variants incl. E_BACKUP_SNAPSHOT_FORBIDDEN), #[schema(as = BackupRetentionPolicy)] fix, redb runs table + missing store methods, OpenAPI re-dump + TS regen, doctrine amendment in docs/architecture.md + docs/BACKEND_CONVENTIONS.md + CLAUDE.md + storaged rustdoc, TODO.md consolidation item	wire is ready, snapshot diff additive
2	create_job real + PUT/validation (peer hard-block, cron parse, VM resolve stub-able) + Playwright job-CRUD spec	first UI value; the mandate lands
3	Callback plane in the daemon (resolve-vm, snapshot-vm, destroy-snapshot, resolve-zpool) + snapshot-stack fn extraction + harness wiring	daemon reuse, no dup snapshot logic
4	Engine core — manual full run → NFS: runner, state machine, mover, manifest, restore point, cancel, progress, boot reconcile, task-bridge poller	the first real backup exists (L2 green)
5	S3 mover (multipart + abort) + used_bytes	L3 green (creds env)
6	Incrementals + GFS prune (anchors, chains, full_every, locked-repo skip)	space actually reclaims
7	Scheduler (croner dep, boot catch-up, TaskSource::Scheduler)	unattended operation
8	Restore-as-new-VM (+ /define-restored-vm callback)	recovery path (L2 restore leg)
9	UI: job wizard, runs UX, restore wizard, KPIs, vitest	console fully lit
10	Docs + evidence: module doc grows the engine section, CHANGELOG, L5 evidence when GO’d	closure

Phases 2–3 are order-independent; everything else is sequential.

2.14 Decisions — ALL RESOLVED (D1/D2/D5 first signature round; D3–D10 second round, both 2026-07-01)

#	Decision	Status
D1	zfs execution placement	SIGNED 2026-07-01 with conditions (§2.1: written doctrine amendment, code-enforced prefix guardrail §2.9, zfs hold §2.2, tracked consolidation item)
D2	Job source scope	SIGNED 2026-07-01, single VM in P3; condition: additive multi-VM seam, no singular→plural rename (§2.5); concrete shape confirmed at schema landing
D3	Incremental defaults	SIGNED 2026-07-01: Full + ForeverIncremental; Differential rejected with its own E_BACKUP_TYPE_UNSUPPORTED + honest user_message; full_every default 7, per-job configurable, no floor/ceiling (soft warning on extremes, never a block)
D4	Missed schedule while down	SIGNED 2026-07-01, scoped: at most ONE catch-up run per job at boot (most recent missed occurrence, never backlog replay), under the signed storm control; run record carries a catch_up flag
D5	Restore semantics	Made absolute by the signature (§2.10): new VM only, never auto-start, never in-place, no override flag
D6	New dependencies	SIGNED 2026-07-01: croner authorized (verify license + zero new advisories on add); sha2 — check the tree first and promote/unify an existing transitive version rather than introduce a duplicate
D7	Quiesce default	SIGNED 2026-07-01: fs-freeze ON default, crash-consistent fallback + honest flag; condition: the flag is VISIBLE in the restore-point LIST UI (§2.11)
D8	Peer data plane	CONFIRMED 2026-07-01: → P4; the create_job hard-block lands in Phase 2 with typed error + user_message (P2 mandate)
D9	Dock integration	SIGNED 2026-07-01: daemon poller bridge, migration-poller precedent; interval configurable, no magic number
D10	S3 object lock	SIGNED 2026-07-01 with honesty condition (§2.4): verify GetObjectLockConfiguration when immutability is claimed; never silent false security; wizard create-bucket object-lock gap logged as P2 follow-up; E2E policy needs s3:GetBucketObjectLockConfiguration later; per-object params → P4

---

Drafted + conditionally signed 2026-07-01 against the 0.15.0 working tree (HEAD 3163981b). Companion current-state doc: docs/modules/backup/repository-management.md.