Refactor roadmap — daemon split (R1 → R4)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/architecture/refactor-roadmap.md. Edit at the source, not here.

Refactor roadmap — daemon split (R1 → R4)

Multi-phase split of the monolithic synvirt-daemon into separate systemd-managed services. Each refactor extracts one domain into its own service, with the daemon becoming a thin client of it.

Transport note (2026-06): the pattern below originally specified a D-Bus interface per service. That is not what shipped. R1 ultimately runs in-process (Commit 6 removed its D-Bus path), and R2 (synvirt-storaged) ships over HTTP-over-UDS via synvirt-ipc, not D-Bus. Trust the per-phase status sections, not the pattern, for what actually exists today.

The structural goal is simple: a control-plane restart never touches the data plane. The 2026-05-05 incident (3-hour outage on synvirt-01 after systemctl restart synvirt-daemon triggered a stale boot-reapply that wiped OVS) exposed the full cost of the monolith. R1 fixed that one slice; R2-R4 finish the job.

Architectural pattern (every refactor follows this)

1. Dedicated systemd service — Type=notify, hardening directives (ProtectSystem=strict, capability bounding, NoNewPrivileges=yes), boot ordering before synvirt-daemon.service and after the resource it owns (e.g. openvswitch-switch.service for R1).
2. YAML source of truth at /etc/synvirt/<domain>.yaml, editable from a console session with vi when nothing else works. Sled cache at /var/lib/synvirt/<domain>.db/ is rebuilt from YAML on every start — never the other way.
3. D-Bus interface on the system bus (org.synnet.synvirt.<Domain>1), policy-restricted to root + the synvirt user. Read methods (Get*, List*) opt into a future read-only group for monitoring.
4. Reconciliation rules — never auto-delete operator state. The locked rule from R1: declared-in-YAML missing-in-runtime → create; runtime missing-in-YAML → log + drift signal, never delete; empty YAML + non-trivial runtime on initial start → refuse to start with a clear error.
5. Daemon thin client — the matching storage_client.rs / host_client.rs in crates/daemon/src/ (R1’s network_client.rs followed this pattern, but Commit 6 reverted network to in-process) wraps zbus::Proxy, surfaces 503 synvirt.<domain>.service_unavailable when the bus name is absent, decodes JSON envelopes that mirror the existing REST DTOs verbatim. The HTTP shape on /api/v1/<domain>/* does not change.
6. Cargo feature legacy-<domain>-direct preserves the old in-process path for emergency rollback. Default OFF after the refactor ships; removed entirely in R4.

Phase status

R1 — synvirt-network ✅ Shipped on 0.15.0

Tag: r1-deployed-20260505. Commits on 0.15.0:

38dd6ba refactor(network): extract synvirt-network service (R1)
0c6ee6a refactor(network): swap daemon handlers to D-Bus, default feature OFF (R1.1)
548c869 chore(scripts): R1 automated deploy
0e1db11 chore(scripts): support PEER=local in R1 deploy
dfbcc30 chore(scripts): honour CARGO_TARGET_DIR in R1 deploy
e9c711d fix(bmc): three latent bug patterns from R1.2 deploy
caebf99 fix(services): add synvirt-network to allowlist, forward-declare R2/R3

Domain extracted: OVS, vSwitches, firewall apply, boot reapply, the management-interface schema (apply still in installer).

In-process surface: R1 exposed a synvirt-network D-Bus interface, but Commit 6 collapsed it — the daemon now calls synvirt-network’s NetworkController directly (no D-Bus).

Daemon-side: crates/daemon/src/api/network.rs (in-process). The legacy-network-direct Cargo feature was removed by Commit 6.

Deep dive: ../modules/synvirt-network/. Migration runbook: ../operations/refactor-r1-network.md.

R2 — synvirt-storaged ✅ Shipped on 0.15.0 (commit 12606977 + follow-ups)

Domain extracted: ZFS pools, datastores, auto-import on boot, and the /api/v1/storage/* surface. The resident synvirt-storaged service owns the pool registry and the storage-object lifecycle control plane (zpool create/destroy/rename, import/adopt). Doctrine amended 2026-07-01 (backup-P3 signature): storaged is NOT the sole zfs executor — streaming data-plane work (zfs send/recv) executes in the daemon that owns the job (migrator, the daemon snapshot stack, synvirt-backupd under its prefix guardrail); see docs/BACKEND_CONVENTIONS.md §External command execution. Startup is read-only with respect to existing pools/datasets/zvols (probe + import-if-absent, never destroy/export/recreate), so a control-plane restart never touches the ZFS data plane backing running guests.

Transport: HTTP-over-UDS via synvirt-ipc on /run/synvirt/storaged.sock (not D-Bus). The daemon proxies the surface through crates/daemon/src/api/storage_client.rs + storage_proxy.rs, surfacing 503 E_STORAGE_UNAVAILABLE when the socket is absent.

Config: /etc/synvirt/storaged.toml (service config) + /etc/synvirt/storage.toml (pool registry) — TOML, not YAML. Wire DTOs + error envelope live in the synvirt-storage-types crate.

The vm-layout consolidation (/rpool/vms/<slug>/, commit 4bb5619) and its vm_layout::migrate_all legacy migration are part of this domain. ZFS module loading stays with zfs-zed.service; synvirt-storaged orders after it.

R3 — synvirt-host 📐 Planned

Domain to extract: every Settings module section that today loads in-process via the synvirt-time / synvirt-resources / synvirt-services / synvirt-firewall / synvirt-swap / synvirt-packages / synvirt-bmc / synvirt-certs crates. Also: the host-tunings reapply path (crates/daemon/src/host_tunings.rs) and the perf-reapply for offline VMs.

Anticipated D-Bus interface: org.synnet.synvirt.Host1 — one method per section (GetTime, SetTime, GetReservation, SetReservation, ListServices, RestartService, GetFirewall, ApplyFirewall, GetSwap, ApplySwap, ListPackages, RefreshPackages, UpgradePackages, GetBmc, SetBmc, TestBmc, GetCert, UploadCert, RequestLetsencrypt, RenewCert, ResetSelfSigned, Reconcile). Signals: StateChanged, PackageRefreshProgress, CertRenewProgress, ServiceStateChanged.

YAML source of truth: /etc/synvirt/host.yaml — folds the existing synvirt.toml [time], [reservation], [firewall], [swap] sections plus references to secrets.key / tls/cert.pem for cross-section reuse.

Daemon-side: crates/daemon/src/host_client.rs. Cargo feature legacy-host-direct.

Open questions to resolve before R3 starts:

• The Phase 9 synvirt-certs already runs hot-reload via RustlsConfig::reload_from_pem_file. R3 must keep that loopback intact: the daemon stays the TLS terminator (it’s the consumer of :443), but cert management is owned by R3. The reload trait stays an interface between the two.
• The EncryptionKey::ensure_machine_id heal-on-load lives in synvirt-bmc::encryption. After R3, that module is part of synvirt-host. The helper stays in place; the daemon never imports it directly anyway.

R4 — Daemon reduction + rename 📐 Planned

Domain to remove from synvirt-daemon: everything R1 / R2 / R3 extracted. The daemon’s final shape:

• HTTP / TLS termination on :443.
• PAM Basic Auth + session middleware (auth migration to session cookies happens on the same timeline; tracked in TODO.md).
• OpenAPI doc + /docs/ Swagger UI.
• WebSocket bridges: VNC console (/console/vnc/{uuid}), in-browser host shell (/api/v1/host/shell), telemetry SSE.
• Static asset serving for web-ux-v2 + web-ux legacy.
• D-Bus client glue for the three other planes.

Concrete deliverables:

1. Remove the remaining legacy-storage-direct / legacy-host-direct Cargo features (the legacy-network-direct one is already gone — Commit 6 collapsed the network D-Bus path in-process).
2. Delete the in-process plumbing each refactor preserved (the crate dependencies on synvirt-time / synvirt-firewall / etc. fall to zero).
3. Rename synvirt-daemon.service → synvirt-api.service. The placeholder is already in synvirt_services::allowlist::ALLOWLIST.
4. Rename crates/daemon → crates/api (workspace member rename
   • Cargo feature lookup updated). The binary name follows.

Anticipated risk: the web-ux-v2 dashboard makes ~25 calls to /api/v1/* paths today; none of those change. The rename is service-level (systemd unit) and crate-level (workspace member), not API-level.

Cross-phase invariants

These are enforced across every refactor; new entries inherit all of them:

1. One crate per domain. Never ship a “lite” parallel crate. The dropped R1.2 stub (parallel synvirt-certs) is the antipattern.
2. One sled handle per DB. Open in main.rs of the owning service, accept &Db from callers. sled::open() inside a service constructor is a bug. Documented on synvirt_bmc::credentials::CredentialStore::open.
3. /etc/machine-id is heal-on-load. synvirt_bmc::encryption::ensure_machine_id runs idempotently inside EncryptionKey::load; cloned templates and live-migrated VMs no longer 503 forever.
4. Cross-service paths come from the owning crate. No hardcoded paths in consuming crates.
5. tracing::warn! is invisible at default INFO level. Any startup failure that takes a panel offline must be error! with error_chain = ?e. Success paths log at info! so operators have a positive heartbeat.
6. HTTP API shapes do not change across a refactor. The dashboard never has to relearn an endpoint when a refactor ships.

See ../../MEMORY.md for the deploy-time narrative and the close-out lessons that produced these rules.