Daemon lifecycle — guests survive control-plane restarts

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/operations/daemon-lifecycle.md. Edit at the source, not here.

Daemon lifecycle — guests survive control-plane restarts

Invariant: systemctl restart synvirt-daemon (every hot deploy and every v2 update) performs zero guest lifecycle actions. Orderly guest draining happens only on a real host shutdown/reboot. This is the data-plane-independence guarantee: the management plane (the daemon) may come and go without disturbing running VMs (libvirt/QEMU).

Landed in Fix70 (2026-06-07). Supersedes the CRITICAL finding in ../cluster.md.

The three binding rules

The lifecycle is split by systemd topology, not by runtime heuristics:

1. Daemon stop (SIGTERM/SIGINT from systemctl restart, an upgrade, or a crash): the signal handler drains in-flight HTTP connections and returns. It issues no virsh shutdown/destroy. The old run_host_shutdown / stop_one / stop_sequence code paths were removed entirely.

2. Host shutdown / reboot: the dedicated unit synvirt-guest-drain.service owns the drain. It is a oneshot with ExecStart=/bin/true + RemainAfterExit=yes (so it is active (exited) for the whole uptime and does nothing at boot) and ExecStop=/opt/synvirt/bin/synvirt-daemon drain-guests. systemd runs ExecStop only when the unit stops — i.e. during the host shutdown transition. It is ordered After=libvirtd.service, so on shutdown (the reverse of start order) the drain runs before libvirtd goes down and virsh can still reach the domains.

   The unit is deliberately not coupled to synvirt-daemon.service: it carries only an After= ordering edge, never PartOf/BindsTo/Requires. systemctl restart synvirt-daemon therefore does not stop this independent unit, so a control-plane restart never drains guests. That separation is the fix.

3. Daemon start: run_boot_autostart enumerates running guests and adopts them — it logs adopted N running guests (left untouched) and starts only the enabled VMs that are not already running. This is the idempotency guarantee: rapid restarts never re-start an already-running guest.

The drain-guests subcommand

synvirt-daemon drain-guests is a short-lived process (not the daemon). It is the only guest-stopping path in the codebase. It loads /etc/synvirt/autostart.json, drains every running guest in reverse start order with bounded concurrency, logs a per-guest and a summary outcome to stderr (journald captures it under synvirt-guest-drain.service), then exits 0 (best-effort: a per-VM failure is logged, never aborts the rest).

Configuration (/etc/synvirt/autostart.json, host block)

No hardcoded values — every knob is config with a sane serde default:

Field	Default	Meaning
drain_parallelism	4	Max guests drained concurrently (clamped ≥1 at use).
force_off_on_timeout	true	Force-off a guest that ignores ACPI within its stop timeout; when false the straggler is left to the hypervisor.
drain_deadline_s	180	Overall wall-clock budget for the whole host-shutdown drain. Must stay strictly below the unit’s TimeoutStopSec (300s).

Per-VM overrides (each VM entry): stop_action (guest_shutdown | power_off, default guest_shutdown) and stop_delay_s (per-guest ACPI grace window, falls back to the host default_stop_delay_s).

⚠️ drain_deadline_s vs TimeoutStopSec. systemd’s default unit stop timeout is 90s and would SIGKILL the drain mid-flight. synvirt-guest-drain.service sets TimeoutStopSec=300, comfortably above the 180s default deadline. If you raise drain_deadline_s past ~270, raise TimeoutStopSec to match.

Outcomes and machine codes

drain-guests reports a typed outcome per guest; the warn-level ones carry a machine code:

Outcome	Code	Meaning
GracefulOff	—	Guest acknowledged ACPI and powered off within its timeout.
PoweredOff	—	Operator policy was immediate power-off.
AlreadyOff	—	Guest was already shut off — nothing to do.
ForcedOff	E_DRAIN_FORCED_OFF	Guest ignored ACPI; force-off issued per policy.
TimedOut	E_DRAIN_GUEST_TIMEOUT	Guest ignored ACPI and force-off is disabled; left to the hypervisor.

Power-seam failures surface as E_LIBVIRT (from PowerError::code()).

Verifying on a live host

# The drain unit must be active (exited) for its ExecStop to fire on shutdown:
systemctl status synvirt-guest-drain.service        # → active (exited), enabled

# A daemon restart must NOT touch guests — the journal shows adoption, never a stop:
systemctl restart synvirt-daemon
journalctl -u synvirt-daemon -b | grep -E "adopted running guests|left untouched"
# (and nothing matching "stopping autostart" / "drain: stopping" / force_off)

# The drain only fires on a real shutdown — inspect the PREVIOUS boot after a reboot:
journalctl -b -1 -u synvirt-guest-drain.service     # → "drain: ..." outcomes

Upgrading from a pre-Fix70 binary (one-time)

A host still running the old binary will drain once on the next restart — the old shutdown handler runs on the way down before the new binary is live. To upgrade without that single bounce, stage the new binary first, then systemctl kill -s SIGKILL synvirt-daemon (SIGKILL skips the old handler; the HTTP drain is skipped but the SQLite stores are WAL and recover), and let Restart=on-failure relaunch the staged new binary. Every subsequent deploy uses the normal scripts/deploy-to-host.sh path and is guest-safe.