Daemon split — running progress log

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/daemon-split/PROGRESS.md. Edit at the source, not here.

Daemon split — running progress log

Durable per-phase log for the feature/daemon-split overnight run. Kept in-repo so progress survives across sessions. The final, polished summary lives in REPORT.md (Phase 6).

Phase 0 — Preflight & isolation ✅

• Disk: 30 GB free on / (gate ≥ 20 GB). PASS.
• Source checkout /home/synvirt/dev: branch 0.15.0, HEAD cfd9061935ad6f22e3af26322b09928ddd439cb8 (“docs: cluster-phase2 consolidated into 0.15.0 mainline”), working tree clean. Untouched.
• Worktree: /home/synvirt/worktrees/daemon-split on branch feature/daemon-split, branched from cfd9061.
• Toolchain: rust stable 1.95.0 via /home/synvirt/.rustup + CARGO_HOME=/home/synvirt/.cargo (see /tmp/cenv.sh). The rustup shim was missing from PATH; cargo is invoked from the toolchain bin.

Baseline (Phase 0 gate) ✅ GREEN

Command: cargo test --workspace --exclude synvirt-guest-installer --exclude synvirt-guest-installer-rs

Result: 1998 passed / 0 failed / 4 ignored across 136 test binaries (unit + integration + doctests). Exit 0.

The prompt’s “~388–396 passing” estimate referred to a narrower subset; the full-workspace suite is unambiguously green at 1998.

Environment deviations (documented, not code issues)

A fresh git worktree lacks gitignored, build-time-staged vendored assets that exist only in the working tree of the main checkout. None relate to the daemon-split mission. Handling:

1. crates/live — build.rs needs assets/JetBrainsMono-{Regular,Medium}.ttf (gitignored, fetched at build time). Fixed by copying the two TTFs from /home/synvirt/dev into the worktree (read-only from dev; the files stay gitignored, never committed).
2. crates/synvirt-guest-installer — build.rs panics without 131 external SHA-256-pinned guest payloads (virtio-win drivers, qemu-ga.msi) that are never committed (CLAUDE.md §5.8) and absent here. Excluded from the baseline + the work scope.
3. crates/synvirt-guest-installer-rs — depends on (2). Excluded.

These exclusions are safe: the mission crates (daemon, synvirt-update, migrator, synvirt-iso, synvirt-core, synvirt-libvirt) and all the daemon’s transitive deps build and test green.

Phase 1 — Investigation + design doc ✅

Findings:

• Updater lives in crates/daemon/src/update_v2/ (orchestrator 874 LOC, + state/progress/error/host_info) consuming synvirt-update + synvirt-iso. Routes in crates/daemon/src/api/update_v2.rs (11 routes) + update_isos.rs (library CRUD). The orchestrator is self-contained — it does not touch AppState, the daemon activity log, or the daemon SSE bus. Post-verify resume task in daemon/src/main.rs:1022.
• Migration (host-to-host): none exists. migrateToURI / virsh migrate / virDomainMigrate appear only in comments. migrator is the unrelated VM import engine. → synvirt-migrationd is greenfield.
• IPC: no daemon↔daemon local-IPC pattern exists. Chosen: HTTP-over-UDS (axum server + hyper client), all libs already in lock.

Design captured in DESIGN.md.

Phase 2 — Scaffold ✅

Three workspace members added:

• crates/synvirt-ipc (lib) — generic HTTP-over-UDS transport (bind_uds with 0600 socket, serve over UnixListener via hyper-util, UdsClient with get_json/send/forward). 5 round-trip integration tests green (bind mode, typed GET, raw POST echo, forward adaptation, connect-failure-is-error-not-panic).
• crates/synvirt-updaterd (lib+bin) — scaffold: UpdaterdConfig (TOML + env, defaults), /healthz, SIGTERM graceful shutdown, build.rs stamping SYNVIRT_RELEASE. 3 config tests green.
• crates/synvirt-migrationd (lib+bin) — scaffold: MigrationdConfig (adds db_path), /healthz, SIGTERM, build.rs. 3 config tests green.

Logging: tracing_subscriber::fmt to stderr (systemd → journald), matching synvirt-network. English throughout.

systemd units written to the repo packaging location (iso-builder/live-build/config/includes.chroot/etc/systemd/system/), not installed:

• synvirt-updaterd.service — Type=simple, lifecycle INDEPENDENT of synvirt-daemon (no After/Requires/PartOf on it), After=local-fs.target only. Keeps CAP_SYS_ADMIN for the apply phase (mount/reboot).
• synvirt-migrationd.service — Type=simple, After=libvirtd.service ordering-only (no Requires/PartOf), no relationship with synvirt-guest-drain.service.

cargo fmt --check + cargo clippy -D warnings clean on all three new crates.

Phase 3 — Extract updater → synvirt-updaterd ✅

Behavior-preserving move (not a redesign):

• git mv’d state.rs, progress.rs, error.rs, host_info.rs, orchestrator.rs from crates/daemon/src/update_v2/ into crates/synvirt-updaterd/src/ (history preserved). Fixed module paths (crate::update_v2:: → crate::). Deleted the now-empty update_v2/mod.rs.
• Added crates/synvirt-updaterd/src/http.rs: the axum router + the OrchestratorError→envelope map, moved from the daemon’s api/update_v2.rs. The error envelope is byte-shape-identical to the daemon’s ErrorEnvelope (trace_id is null; the daemon’s middleware still stamps X-Trace-Id). set_state_force/transition_public made pub so the bin’s post-verify resume can call them.
• main.rs constructs the orchestrator (ISO library + BootEnvManager + AuditLog + RELEASE) and runs the post-verify resume (moved verbatim from the daemon’s startup task).
• Daemon side: api/update_v2.rs rewritten as a transparent proxy — the 11 #[utoipa::path] handlers keep their exact contract and forward to the updaterd UDS via synvirt_ipc::UdsClient::forward; a missing socket returns 503 E_UPDATER_UNAVAILABLE. AppState.update_orchestrator replaced with AppState.updater: UdsClient. Orchestrator construction and post-verify task removed from main.rs. mod update_v2; removed. Config gained updater_socket_path (default /run/synvirt/updaterd.sock). The update-ISO library CRUD (update_isos.rs) stays in the daemon.

OpenAPI verification (synvirt-daemon --dump-openapi, before vs after): byte-identical except one line — the logs operation description now points at synvirt_updaterd::ProgressEvent instead of the deleted crate::update_v2::ProgressEvent. No route, method, operation_id, parameter, or schema changed (the ApplyQuery/HistoryQuery params were re-pinned to parameter_in = Query so they stayed query params, not path). The committed crates/web-ux-v2/openapi.snapshot.json was already stale vs HEAD before this work (59-line pre-existing drift), so it is left untouched per scope discipline; the description-only delta is not a contract change.

Tests: updaterd 12 (state machine 7 + config 3 + host_info 2). Daemon proxy tests against a fake UDS server: 3 (forward GET, forward POST body, 503 on unreachable socket). cargo fmt/clippy clean on new + moved code (pre-existing daemon fmt/clippy deviations in untouched files left alone).

Phase 4 — synvirt-migrationd ✅

Greenfield (no host-to-host migration existed). TDD-built modules:

• model.rs — Job, JobKind (live/offline), JobState (queued→preparing→running→completed|failed|cancelled|interrupted), CreateJobRequest; FromStr/as_str/is_terminal.
• store.rs — SQLite (rusqlite, WAL + synchronous=FULL) at /var/lib/synvirt/migrationd.db: create/get/list/update_state/reconcile.
• executor.rs — MigrationExecutor async trait; MockExecutor (tests); LibvirtExecutor stub that fails fast (real migration is the documented next step — no real migration executed).
• manager.rs — JobManager: create (persists + spawns the worker), list, get, cancel, reconcile. The worker drives preparing→running→completed/failed and honours a cancel that lands mid-flight.
• http.rs — create/list/get/cancel routes + /healthz + the daemon-shape error envelope.
• main.rs — opens the store, runs startup reconcile (in-flight → interrupted, never touches a guest), builds the manager with the stubbed executor, serves the UDS.

Daemon side (additive):

• api/uds_proxy.rs — the generic UDS reverse-proxy helper extracted and shared by both surfaces (update + migration); update_v2.rs refactored to use it.
• api/migration.rs — 4 proxy routes (POST/GET /jobs, GET /jobs/{id}, POST /jobs/{id}/cancel) forwarding to the migrationd socket; 503 E_MIGRATION_UNAVAILABLE when down. Mounted in web/routes.rs, registered in openapi.rs. AppState gained migration: UdsClient; config gained migration_socket_path (default /run/synvirt/migrationd.sock).

OpenAPI delta (vs pre-split golden): purely additive — +3 paths (/api/v1/system/migration/jobs[/{id}[/cancel]]), +4 schemas (Job, JobKind, JobState, CreateJobRequest); 0 removed; the only changed existing path is still update/logs (Phase 3 description line).

Tests: migrationd 18 (model 4, store 6, manager 5, + roundtrips). cargo fmt/clippy -D warnings clean on the new crate and the new daemon files.

Phase 5 — Integration polish ✅

• Integration harness (api/uds_proxy.rs harness_tests): spawns the real synvirt-updaterd + synvirt-migrationd binaries as plain processes (kill-on-drop guards), waits for health, then round-trips through the actual daemon proxy helper: updater status (idle), migration create → list → poll get until terminal (failed, via the stub executor) → 404 for an unknown job. Sockets + DB live in a tempdir and config is pointed at an absent path, so the harness never touches /etc, /run/synvirt, or /var/lib/synvirt. It skips gracefully (no failure) if the sub-daemon binaries are not built (only under isolated -p synvirt-daemon); a full cargo test --workspace builds them first and runs it. Verified green.
• cargo fmt applied to all new + moved code (the three new crates and the touched daemon files). Pre-existing fmt deviations in untouched daemon files (cluster_control.rs, inventory.rs, main.rs:1285, migrator_domain.rs) are left alone per scope discipline.
• cargo clippy -D warnings clean on the new crates; no new clippy warnings in the touched daemon files (the daemon’s ~20 pre-existing clippy warnings live in untouched webrtc/network/console code).

Phase 6 — Report ✅

See REPORT.md.

---

Follow-up (2026-06-10): extract the real synvirt-migrator → synvirt-migratord

Phase 0 — Preflight ✅

Worktree clean @ de61e5c; reclaimed disk to 20 GB (removed worktree target/debug/incremental + /tmp junk; dev/ left untouched per the boundary). Baseline cargo test --workspace (excl. guest-installer): 2028 passed / 0 failed / 4 ignored. Host snapshots + active-jobs gate in hosts/ — .56/.65 clean (deploy), .75 FROZEN (non-terminal copying_full import job). Golden OpenAPI dumped (178 paths; migrator absent from the spec).

Phase 1 — Design ✅

DESIGN-migratord.md: reverse-UDS callback seam (the DomainStarter/ PoolResolver pipeline stays in the daemon, invoked over a 2nd socket), 1:1 wire-frozen proxy, Tasks/Events bridge by polling, compile-linked go-shim, unit + swap order.

Phase 2 — Extraction ✅ (code green)

• synvirt-migratord (new lib+bin): hosts JobManager + sled StateStore + sources/uploads/inventory + orchestrator + go-shim; RemoteDomainStarter (async) + RemotePoolResolver (sync blocking) invoke the daemon over the callback socket; serves the migrator router over the control socket.
• daemon: api/migrator_callback.rs runs the kept migrator_domain pipeline on the callback socket; api/migrator_proxy.rs is a wildcard transparent proxy of /api/v1/migrator/* (streaming; 503 E_MIGRATOR_UNAVAILABLE on socket down); the in-process construction + observability subscribe were removed and replaced with a poll-based Tasks/Events bridge. DestinationDisk gained serde; synvirt-ipc gained post_json.
• Gates: grep-gate CLEAN (zero StateStore::open / migrator-sled refs in the daemon); OpenAPI --dump-openapi byte-identical to the golden; migratord 3 unit tests; daemon harness extended — the real migratord binary round-trips GET /sources + /capabilities through the real proxy and 503s when down. fmt + clippy -D clean on new/moved code.
• systemd unit synvirt-migratord.service written to the repo (NOT installed): root + libvirt group, After=libvirtd ordering-only.