Report — Settings ▸ Services: dynamic discovery of the split daemons

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/daemon-split/REPORT-services-panel.md. Edit at the source, not here.

Report — Settings ▸ Services: dynamic discovery of the split daemons

Branch: feature/daemon-split @ worktree /home/synvirt/worktrees/daemon-split. 2026-06-10. Base HEAD at session start: 4960e87.

Outcome: green through deploy. The Services panel no longer reads a stale curated SynVirt list — it now DISCOVERS synvirt-*.{service,timer} units from systemd at request time. The four ghosts are gone; the split stack (synvirt-daemon + synvirt-migratord + synvirt-updaterd) plus network/console/cert-renew/etc. now appear with live status + memory. Only the daemon binary changed (migratord/updaterd were NOT restarted). Deployed to .56 → .65 → .75; every protected guest PID is exactly as found.

Commit: 07a93b5 feat(synvirt-services) + this report commit.

---

Registry before → after

Before (curated const ALLOWLIST)

SynVirt rows were a hardcoded list. Four had no shipped unit file and rendered as permanent “Not installed” ghosts; the real control-plane daemon was absent:

Curated SynVirt row	Reality
synvirt-api.service	never shipped (R4 future rename) → ghost
synvirt-console-gateway.service	never shipped as a unit → ghost
synvirt-storage.service	R2, never shipped → ghost
synvirt-host.service	R3, never shipped → ghost
synvirt-cert-renew.timer	real
synvirt-network.service	real
synvirt-daemon.service	MISSING from the panel
synvirt-migratord / synvirt-updaterd	MISSING (didn’t exist when the list was written)

After (dynamic discovery + curated platform)

• SynVirt rows = discovered. SystemdClient::discover_synvirt_units() unions ListUnitFiles ∪ ListUnits, filters synvirt-*.{service,timer}, dedupes. ServicesService::list() orders them (control-plane daemon first, then alphabetical) and prepends them to the curated platform deps.
• Denylist (SYNVIRT_DENYLIST): synvirt-guest-drain.service (its ExecStop runs the guest drain — a UI Stop would power down running VMs) and synvirt-self-restart.service (transient oneshot helper). Both are pattern-matched but never surfaced and are not_managed for actions.
• Platform deps stay curated (libvirtd, virtlogd, chrony, nftables, openvswitch-switch, zfs-zed, ssh, ceph.target@enterprise) so “Not installed” stays a useful signal.
• Self-protection repurposed from the never-shipped synvirt-api.service to the real API daemon synvirt-daemon.service (is_api_daemon(), also matching the future R4 rename): can’t stop/disable; restart routes through the out-of-process self-restart helper (whose shipped unit already falls back to synvirt-daemon.service). migratord/updaterd restart SOFT-WARN (never block) via impact warning codes.

Gates

• OpenAPI services schema byte-identical. Normalized --dump-openapi (local build) vs the deployed .65 daemon: 9/9 /settings/services* paths identical, every UnitStateView/RestartImpactView/ServicesListView/… schema equal, whole spec normalized-identical. Only the dataset changed.
• Suite: full workspace 2039 passed / 0 failed / 4 ignored.
• fmt + clippy -D clean on touched code. (Pre-existing crate fmt deviations in untouched error.rs/journal.rs/self_restart.rs left as found.)
• Tests added/updated: discovery/merge, ordering (daemon-first + alphabetical + dedupe), denylist suppression, edition gate, api-daemon protection, discovered-unit action set — all against a mocked systemd source.

---

Per-host deploy (only the daemon binary changed)

Procedure per host: PID snapshot → timestamped backup synvirt-daemon.bak-$(date +%Y%m%dT%H%M) (NOT overwriting last night’s bak-20260610) → install binary (rename-over-running, safe) + systemctl daemon-reload → systemctl restart synvirt-daemon (plain restart; sled-owning migratord and updaterd untouched) → PID gate → smokes.

Host	Pre ver	Post ver	units (daemon/migratord/updaterd)	ghosts	discovered rows	Status
10.10.26.56	Fix75+build.1024	Fix75+build.1027	active / active / active	NONE	11 synvirt + 7 platform = 18	✅
10.10.26.65	Fix75+build.1024	Fix75+build.1027	active / active / active	NONE	13 synvirt + 7 platform = 20	✅
10.10.26.75	Fix75+build.1024	Fix75+build.1027	active / active / active	NONE	10 synvirt + 7 platform = 17	✅

migratord/updaterd were NEVER restarted — only synvirt-daemon. Each host’s row count differs because discovery reflects that host’s actual units (e.g. .65 also has synvirt-live + synvirt-vnic-rename).

SEV-0 guest PID continuity (before / after every daemon restart)

Host	Guest	PID before	PID after	start time	Verdict
10.10.26.56	— (none)	—	—	—	n/a
10.10.26.65	Windows Server 2026	1699966	1699966	Tue Jun 9 14:39:58 2026 (unchanged)	✅ PRESERVED
10.10.26.75	Windows Server 2016	1469944	1469944	Sun Jun 7 13:51:37 2026 (unchanged)	✅ PRESERVED

Restart is guest-safe by design: the deployed daemon is Fix75 (post-Fix70 ⇒ systemctl restart is HTTP-only) and a hard PID gate ran immediately after each restart with an auto-rollback to the timestamped backup had the PID changed. It passed on both guest hosts. End-of-session snapshot reconfirmed all PIDs.

Smoke evidence (per host)

• GET /api/v1/settings/services → daemon+migratord+updaterd present, active, with memory_current_bytes > 0; the four ghosts absent; guest-drain + self-restart absent; 7 platform rows present.
• Live per-host JSON saved as evidence: docs/daemon-split/hosts/10.10.26.{56,65,75}-services.json.
• GET /api/v1/migrator/migrations through the proxy → alive (.56/.65 = 0 jobs, .75 = 7 historical jobs incl. last session’s cancelled orphan).
• journalctl -p err for the three units after each restart → no entries (.75 shows only pre-existing, unrelated OVS vSwitch0-bond noise).

---

synvirt-network “Activating” finding (.65 — investigate-only, NOT touched)

The dashboard showed synvirt-network stuck “Activating” on .65. Root cause, from systemctl status synvirt-network:

synvirt-network.service: Scheduled restart job, restart counter is at 47538.
(install)[…]: synvirt-network.service: Failed to set up mount namespacing:
              /etc/systemd/resolved.conf.d: No such file or directory
Active: activating (start-pre)

The unit’s mount-namespacing (a ReadWritePaths=/BindPaths=-style directive) references /etc/systemd/resolved.conf.d, which does not exist on .65. systemd cannot set up the namespace ⇒ ExecStartPre fails ⇒ Restart= loops (counter 47538 — long-standing, pre-dates this work). That perpetual activating (start-pre) is exactly what the panel now surfaces (active_state=activating, sub_state=auto-restart).

This is a pre-existing host bug, not caused by this change — and the dynamic panel is now correctly exposing it (the curated list showed synvirt-network too, but the value here is the operator can see it is crash-looping). The unit was deliberately NOT touched.

Suggested fix for Tony (your call, not done here): either mkdir -p /etc/systemd/resolved.conf.d on .65, or make the unit tolerate the missing dir / create it in an earlier ExecStartPre. synvirt-network on .56 is inactive and on .75 inactive — only .65 is in the crash-loop.

---

Left for Tony

1. Open the dashboard on each host → Settings ▸ Services. Confirm the split daemons render with memory, the four ghosts are gone, and the platform deps still show. The UI needs zero changes (OpenAPI byte-identical).
2. synvirt-network on .65 — apply the resolved.conf.d fix above to stop the 47k-restart loop.
3. Action policy is live but unexercised by a click: discovered synvirt units get synvirt-network’s action set; synvirt-daemon is stop/disable locked with restart via the self-restart helper; migratord/updaterd restart soft-warns. Worth a manual click-test of “restart synvirt-daemon” from the UI (it should reconnect, not brick).
4. Discovery surfaces ALL real synvirt units (console, boot-splash, cpufreq, tuned, debug-cleanup, vnic-rename, cert-renew.service, …), not just the long-running daemons. That is the intended zero-curation behavior; if any should be hidden, extend SYNVIRT_DENYLIST.
5. Branch still unmerged/unpushed — your call.