Report — migrator extraction (synvirt-migratord) + staged deploy

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/daemon-split/REPORT-migratord.md. Edit at the source, not here.

Report — migrator extraction (synvirt-migratord) + staged deploy

Branch: feature/daemon-split @ worktree /home/synvirt/worktrees/daemon-split. Follow-up run, 2026-06-10. Base HEAD at session start: de61e5c.

Outcome: green through deploy. The real synvirt-migrator import engine is extracted into synvirt-migratord; /api/v1/migrator/* is byte-identical through the daemon proxy; the split stack (daemon + updaterd + migratord) is live on .56 and .65 with every pre-existing guest PID exactly as found; .75 was FROZEN by the active-jobs gate and left untouched. Zero import/migration jobs were created — all smokes were GET-only.

---

Per-phase outcomes

Phase	Result
0 — Preflight	✅ disk reclaimed to 17–20 GB (worktree caches only; dev/ untouched), baseline 2028/0/4, host snapshots + gates
1 — Design	✅ DESIGN-migratord.md (reverse-UDS callback seam)
2 — Extraction	✅ synvirt-migratord + daemon proxy/callback; grep-gate + OpenAPI byte-diff PASS; suite 2032/0/4
3 — Polish	✅ fmt + clippy -D clean on new/moved code (folded into Phase 2 gates)
4 — Deploy	✅ .56 + .65 (FROZEN .75 skipped); all guest PIDs preserved
5 — Report	✅ this file

The coupling and the seam

The migrator’s DomainStarter/PoolResolver are implemented in the daemon (migrator_domain::{ProfileDomainStarter,RegistryPoolResolver}) against the daemon’s VM-creation pipeline (vm_profiles, libvirt_renderer, telemetry, cpu). Moving the JobManager would otherwise drag that whole pipeline along, so the faithful seam is a reverse UDS callback: the daemon keeps migrator_domain unchanged and serves it on a private callback socket; migratord implements the two traits as thin clients (RemoteDomainStarter async, RemotePoolResolver sync/blocking, fail-soft to identity). The real domain-definition code never moved — only its invocation crosses a socket.

---

Commits (this follow-up, newest first)

c138878 refactor(daemon-split): extract migrator import engine into synvirt-migratord
cde677b docs(daemon-split): migratord extraction design + host preflight snapshots

(on top of the prior split work: de61e5c … 25c24cb). No push, no merge into 0.15.0 — left for Tony’s call (note in §gaps).

Test counts

• Baseline (session start), cargo test --workspace --exclude synvirt-guest-installer --exclude synvirt-guest-installer-rs: 2028 passed / 0 failed / 4 ignored.
• Final (post-extraction), same command: 2032 passed / 0 failed / 4 ignored (+ migratord 3 config tests + 1 real-binary harness test).
• Env caveats unchanged from the prior run (guest-installer payloads, live fonts; the SVT-AV1 FFI flake did not recur this run).

OpenAPI diff (rule 5 — wire-frozen)

/api/v1/migrator/* was never in the OpenAPI (the router was mounted but never registered in the #[openapi] derive). synvirt-daemon --dump-openapi before vs after the extraction is byte-identical (178 paths). The proxy carries no #[utoipa::path] annotations, so nothing was added or removed. Zero routes/schemas changed.

Grep-gate (rule 6 — sled single ownership)

grep -rn "StateStore::open|/var/lib/synvirt/migrator|migrator::{StateStore, JobManager,SourceRegistry,UploadStore,Cipher}" crates/daemon/src → 0 matches. Only synvirt-migratord opens the migrator sled after the split.

---

Per-host deploy results

Deploy order (priority ladder, FROZEN skipped): .56 → .65. Swap order per host (sled-lock handoff): stop daemon → swap binary → start migratord (verify sled acquired) → start updaterd → start new daemon.

Host	Role	Pre daemon	Post daemon	Units (daemon/migratord/updaterd)	Status
10.10.26.56	synvirt-02, no guests	0.15.0-Fix53+build.803	0.15.0-Fix75+build.1024	active / active / active	✅ DEPLOYED
10.10.26.65	synvirt-01, WinServer2026	0.15.0-Fix74+build.1014	0.15.0-Fix75+build.1024	active / active / active	✅ DEPLOYED
10.10.26.75	synvirt-01, WS2016	0.15.0-Fix74+build.1014	(unchanged)	unchanged / not installed / not installed	⏸ FROZEN (non-terminal import job)

SEV-0 guest PID continuity (rule 1)

Host	Guest	PID before	PID after	start time before→after	Verdict
10.10.26.56	— (none)	—	—	—	n/a (no guests)
10.10.26.65	Windos Server 2026	1699966	1699966	Tue Jun 9 14:39:58 2026 → unchanged	✅ PRESERVED
10.10.26.75	Windows Server 2016	1469944	1469944	(untouched — FROZEN)	✅ PRESERVED

On .65 the stop was guest-safe by design (the running daemon was Fix74 ⇒ post-Fix70 ⇒ systemctl stop is HTTP-only; the guest is also non-autostart), and a hard PID gate ran immediately after the stop (would have aborted the swap and restarted the old daemon had 1699966 died). It passed.

Health / GET-only smokes (both deployed hosts)

• GET /api/v1/health → ok, 0.15.0-Fix75+build.1024.
• GET /api/v1/migrator/{sources,migrations} → [] (through the proxy → migratord). GET /api/v1/migrator/capabilities → {"preparer_available":false}.
• GET /api/v1/system/update/status → idle (through the proxy → updaterd).
• GET /api/v1/system/migration/jobs → 503 E_MIGRATION_UNAVAILABLE (migrationd intentionally NOT installed — rule 7; documented/expected).
• Unix-socket health on /run/synvirt/migratord.sock (/capabilities) and /run/synvirt/updaterd.sock (/healthz) OK.
• journalctl -p err for the three units after the swap → no entries.

Rollbacks

None needed. Live daemon binaries were backed up first on each host: /opt/synvirt/bin/synvirt-daemon.bak-20260610 (present on .56 and .65).

---

Morning manual checklist for Tony (you run these — I created zero jobs)

Per deployed host (.56 first, then .65):

1. Open the dashboard https://<host>/ → Migrator / Tasks views render (the proxy is transparent; the UI needs no change). Confirm the sources/jobs lists load (empty is expected).
2. Register a source (e.g. a vim_api endpoint) → it should land in /api/v1/migrator/sources. virt-v2v is not installed on these boxes, so capabilities.preparer_available is false (Windows driver-injection warning is expected; install libguestfs-tools + virtio-win to enable).
3. Run a real import end-to-end (this exercises the reverse callback: migratord → daemon define-and-start → the unchanged VM-creation pipeline). Watch the throughput/progress panel (the /migrations/stream SSE is streamed through the proxy).
4. The money shot: while the import is mid-copy, systemctl restart synvirt-daemon on that host. The job must keep running (it lives in migratord, untouched by the daemon restart) and the UI must reattach when the daemon is back. This is the whole point of the split.
5. Confirm the protected guest PID is unchanged throughout (pgrep -af qemu-system).

If anything looks wrong, rollback per host: systemctl disable --now synvirt-migratord synvirt-updaterd → cp /opt/synvirt/bin/synvirt-daemon.bak-20260610 /opt/synvirt/bin/synvirt-daemon → systemctl restart synvirt-daemon → re-check the guest PID. (The pre-split daemon ran the migrator in-process, so the backup is self-sufficient.)

---

Gaps + next steps

1. .75 is FROZEN, not deployed. It carries a non-terminal copying_full import job (id ff8ebf8c…, esx15 → atlas_replica, 84/213 GB, throughput 0 — almost certainly an orphan from a past daemon restart, but the active-jobs gate is mechanical). Decide whether to cancel/clear that job, then deploy .75 the same way (its WS2016 guest PID 1469944 must be protected; running daemon is Fix74 ⇒ stop is guest-safe).
2. synvirt-migrationd name collision. The prior session’s greenfield host-to-host migration stub (synvirt-migrationd, vMotion-style, stub executor) now sits next to synvirt-migratord (the import engine). The names are one letter apart and confusing. Recommendation: rename synvirt-migrationd → synvirt-vmotiond (or park/drop it until the real host-to-host executor is built). It is NOT installed on any host; its proxy routes return 503 E_MIGRATION_UNAVAILABLE by design.
3. Reverse callback is unvalidated by an actual import (rule: GET-only, zero jobs). The real define code is unchanged daemon-side and the wire DTOs have serde round-trips, but the end-to-end callback is exercised first by Tony’s morning import (step 3 above). A define-serialization bug would fail that import and roll back — it cannot harm an existing guest (define creates a new domain).
4. RemotePoolResolver is a blocking call on a sync trait method (resolve_zpool), made once per job before zfs runs; it fails soft to the identity mapping. Acceptable, documented.
5. No push / no merge. The branch is green and deployed from built binaries, but it is not merged into 0.15.0 — your call.
6. OpenAPI snapshot (crates/web-ux-v2/openapi.snapshot.json) remains stale vs HEAD (pre-existing; migrator was never in it). Unaffected by this work.

---

.75 — morning follow-up (2026-06-10): orphan cleared + deployed

.75 was FROZEN by the active-jobs gate (a non-terminal copying_full import job). This session proved it orphaned, terminal-ized it via the public API, cleaned its partial target, and deployed the split stack with the .65 procedure. All SEV-0 gates passed; WS2016 PID unchanged.

Safety proofs (Phase 0)

• WS2016 guest: QEMU PID 1469944, lstart Sun Jun 7 13:51:37 2026, domain 3 running.
• Running daemon true version 0.15.0-Fix74+build.1014 (via /api/v1/node/identity; the /etc/synvirt/release stamp Fix54 is stale) ⇒ post-Fix70, so systemctl stop is HTTP-only.
• Unit proof: synvirt-daemon has no ExecStop/PartOf/BindsTo; synvirt-guest-drain is ExecStart=/bin/true + RemainAfterExit=yes + ExecStop=…drain-guests, ordered After=…synvirt-daemon.service (ordering-only) ⇒ a systemctl stop synvirt-daemon cannot trigger the drain. WS2016 Autostart: disable (belt-and-suspenders).

Orphan verdict — DEAD (Phase 1)

Check	Result
job id / state	ff8ebf8c… / copying_full
throughput_bps_1m	0
updated_at vs now	2026-06-09 21:23:17 CST (~10.5 h stale)
daemon ActiveEnterTimestamp	2026-06-09 21:44:08 CST — started 21 min AFTER the job’s last update; JobManager has no resume path ⇒ not driven
transfer / virt-v2v / go-shim procs	NONE
target zvol referenced over 6 s	unchanged (10597065792) ⇒ zero writes to target
zpool iostat STO-196E blips	the running WS2016 guest’s own disk (STO-196E/vms/windows-server-2016/disk-0), sporadic — not the import

Terminal-ize + cleanup (Phase 2)

• POST /api/v1/migrator/migrations/ff8ebf8c…/cancel → HTTP 200, state cancelled, finished_at set. (Sanctioned terminal-ize; no new job.)
• Cancel left the partial target zvols. Name-guarded against the job’s own UUID under STO-196E/synvirt/migrations/ff8ebf8c…/ (not vms/, not /var/lib/synvirt/migrator/); zfs destroy -nv -r dry-run listed ONLY that subtree; then zfs destroy -r → GONE. Guest disk STO-196E/vms/windows-server-2016/disk-0 untouched (23.2 G), PID 1469944 unchanged.

Deploy (Phase 3)

• Backup /opt/synvirt/bin/synvirt-daemon.bak-20260610 (Fix74).
• Ordered swap: stop daemon → PID gate PASS (1469944 alive) → start migratord (sled acquired; adoption proof: migratord lists all 7 historical jobs incl. the now-cancelled orphan) → start updaterd (healthy) → start new daemon.
• POST PID gate PASS: 1469944 alive, lstart unchanged.
• Health/smokes: /api/v1/health Fix75+build.1024; migrator sources=1 (esx15 preserved) / migrations=7 (orphan cancelled) / capabilities preparer_available:false; update/status idle; migration/jobs 503 E_MIGRATION_UNAVAILABLE (migrationd not installed — expected). No daemon/migratord/updaterd errors from the swap (the only journal noise is pre-existing OVS vSwitch0-bond lines, unrelated).

Final fleet state

All three hosts run the 3-unit split stack at 0.15.0-Fix75+build.1024:

Host	guest PID (unchanged)	units (daemon/migratord/updaterd)
10.10.26.56	— (no guests)	active / active / active
10.10.26.65	1699966 (WinServer2026)	active / active / active
10.10.26.75	1469944 (WS2016)	active / active / active

.75 is no longer FROZEN. Rollback artifact present on every host (synvirt-daemon.bak-20260610). Zero jobs created this session; one orphan job cancelled + its partial storage reclaimed (~10 G).