Daemon Restart Must Never Touch Running Guests — Implementation Plan

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/plans/2026-06-06-daemon-restart-never-touches-guests.md. Edit at the source, not here.

Daemon Restart Must Never Touch Running Guests — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:test-driven-development per code task. Steps use checkbox (- [ ]) syntax.

Goal: A synvirt-daemon restart/upgrade/crash performs zero guest lifecycle actions; orderly guest drain happens only on real host shutdown via a dedicated systemd unit; daemon start adopts running guests instead of restarting them.

Architecture: Split lifecycle by systemd topology (operator decision, binding):

DAEMON stop (SIGTERM/SIGINT from systemctl restart/upgrade/crash): drain HTTP only, no VM ops. The drain call is removed from main.rs’s signal handler.
HOST shutdown/reboot: synvirt-guest-drain.service (oneshot, ExecStart=/bin/true + RemainAfterExit=yes + ExecStop=/opt/synvirt/bin/synvirt-daemon drain-guests, ordered After=libvirtd.service so its ExecStop runs before libvirtd on shutdown). Restarting the daemon never stops this unit ⇒ never drains.
DAEMON start: run_boot_autostart adopts running guests (enumerate + log) and starts only enabled VMs that are not already running (idempotent across rapid restarts).

A mockable GuestPower trait (mirrors api::vm_nic_edit::LibvirtBackend) makes the adoption + drain state machine unit-testable with a call-recording fake — no libvirt required.

Tech Stack: Rust, axum daemon, clap derive subcommand, async-trait, futures-util buffer_unordered for bounded drain parallelism, tokio::time (paused-time tests), systemd units, scripts/deploy-to-host.sh.

Grounding facts (verified):

Bug site: crates/daemon/src/main.rs:1298 (autostart::run_host_shutdown().await inside the SIGTERM handler) + crates/daemon/src/autostart/mod.rs:316 run_host_shutdown. Boot autostart mod.rs:236 starts without a running-check; spawned at main.rs:1092.
Only caller of run_host_shutdown = main.rs:1298. stop_sequence used only by run_host_shutdown + its test. Safe to remove.
Deps present in crates/daemon/Cargo.toml: async-trait (51), clap derive (52), futures-util 0.3 (90), tempfile 3 (120), tracing-subscriber env-filter (48).
Trait/mock precedent: crates/daemon/src/api/vm_nic_edit.rs:144 (#[async_trait] trait LibvirtBackend + VirshBackend + test MockLibvirt).
synvirt_libvirt::vm: list()->Result<Vec<Vm>> (147), get(&str) (162), start (298), shutdown (317), force_off (326). VmState derives Copy (95). Vm{name,uuid,state,...}.
Unit source dir: iso-builder/live-build/config/includes.chroot/etc/systemd/system/. Install copy list install.rs:1390; enable list install.rs:1569. Deploy push deploy-to-host.sh:61, apply :88.

Task 1: Config — drain knobs on `HostAutostartConfig`

Files: Modify crates/daemon/src/autostart/mod.rs (struct HostAutostartConfig ~48-75, its Default impl, tests).

Step 1: Failing test — add to mod tests:

#[test]
fn drain_defaults_are_sane() {
    let h = HostAutostartConfig::default();
    assert_eq!(h.drain_parallelism, 4);
    assert!(h.force_off_on_timeout);
    assert_eq!(h.drain_deadline_s, 180);
}

#[test]
fn partial_host_json_fills_drain_defaults() {
    // A host block missing the new fields must still get the real
    // defaults, not the field-type zero/false.
    let c: AutostartConfig =
        serde_json::from_str(r#"{"host":{"enabled":true},"vms":[]}"#).unwrap();
    assert_eq!(c.host.drain_parallelism, 4);
    assert!(c.host.force_off_on_timeout);
    assert_eq!(c.host.drain_deadline_s, 180);
}

Step 2: cargo test -p synvirt-daemon drain_defaults_are_sane → FAIL (no field).
Step 3: Implement — add module-level default fns + three fields:

fn default_drain_parallelism() -> u32 { 4 }
fn default_force_off_on_timeout() -> bool { true }
fn default_drain_deadline_s() -> u32 { 180 }

Add to HostAutostartConfig (before the closing brace):

    /// Max guests drained concurrently on host shutdown (≥1; clamped at use).
    #[serde(default = "default_drain_parallelism")]
    pub drain_parallelism: u32,
    /// Force-off a guest that ignores ACPI within its stop timeout.
    /// When false the straggler is left to the hypervisor.
    #[serde(default = "default_force_off_on_timeout")]
    pub force_off_on_timeout: bool,
    /// Overall wall-clock budget for the whole host-shutdown drain
    /// (seconds). Must stay below synvirt-guest-drain.service's
    /// `TimeoutStopSec` so systemd never SIGKILLs the drain mid-flight.
    #[serde(default = "default_drain_deadline_s")]
    pub drain_deadline_s: u32,

Update the manual Default for HostAutostartConfig body to set the three fields to default_drain_parallelism(), default_force_off_on_timeout(), default_drain_deadline_s().

Step 4: cargo test -p synvirt-daemon autostart:: → the two new tests PASS, existing config tests still PASS.
Step 5: Commit feat(synvirt-daemon): add drain parallelism/deadline/force-off-policy config

Task 2: `GuestPower` seam + recording fake

Files: Create crates/daemon/src/autostart/power.rs. Modify crates/daemon/src/autostart/mod.rs (add pub mod power; near top, after the module rustdoc/uses).

Step 1: Create power.rs with the trait, GuestRef, PowerError (+code()), LibvirtGuestPower real impl, and a #[cfg(test)] pub(crate) mod test_support with RecordingPower (records calls, tracks max concurrency, models state). Full content:

//! Mockable seam over the VM power operations the autostart adoption
//! and host-shutdown drain paths need. The real impl shells to `virsh`
//! via `synvirt-libvirt`; tests substitute a recording fake so the
//! adoption/idempotency and drain state machine are verifiable without
//! a libvirt install — the pattern `api::vm_nic_edit::LibvirtBackend`
//! established for hotplug.

use async_trait::async_trait;
use synvirt_libvirt::vm::VmState;
use thiserror::Error;

/// A guest as seen by [`GuestPower::list`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct GuestRef {
    /// libvirt domain name.
    pub name: String,
    /// Current power state.
    pub state: VmState,
}

/// Failure from the power seam. Wraps the underlying libvirt error text
/// so callers can log it without depending on `synvirt-libvirt::Error`.
#[derive(Debug, Error)]
pub enum PowerError {
    /// A `virsh` operation failed.
    #[error("libvirt operation failed: {0}")]
    Libvirt(String),
}

impl PowerError {
    /// Machine-readable discriminant.
    pub fn code(&self) -> &'static str {
        match self {
            PowerError::Libvirt(_) => "E_LIBVIRT",
        }
    }
}

/// VM power operations used by autostart adoption and the guest drain.
#[async_trait]
pub trait GuestPower: Send + Sync {
    /// List all defined guests with their current state.
    async fn list(&self) -> Result<Vec<GuestRef>, PowerError>;
    /// Current state of one guest.
    async fn state(&self, name: &str) -> Result<VmState, PowerError>;
    /// Start (or resume) a guest.
    async fn start(&self, name: &str) -> Result<(), PowerError>;
    /// Request an ACPI graceful shutdown (returns immediately).
    async fn shutdown(&self, name: &str) -> Result<(), PowerError>;
    /// Force the guest off (destroy).
    async fn force_off(&self, name: &str) -> Result<(), PowerError>;
}

/// Production [`GuestPower`] backed by `synvirt-libvirt`'s `virsh` wrapper.
pub struct LibvirtGuestPower;

#[async_trait]
impl GuestPower for LibvirtGuestPower {
    async fn list(&self) -> Result<Vec<GuestRef>, PowerError> {
        let vms = synvirt_libvirt::vm::list()
            .await
            .map_err(|e| PowerError::Libvirt(e.to_string()))?;
        Ok(vms
            .into_iter()
            .map(|v| GuestRef {
                name: v.name,
                state: v.state,
            })
            .collect())
    }

    async fn state(&self, name: &str) -> Result<VmState, PowerError> {
        synvirt_libvirt::vm::get(name)
            .await
            .map(|v| v.state)
            .map_err(|e| PowerError::Libvirt(e.to_string()))
    }

    async fn start(&self, name: &str) -> Result<(), PowerError> {
        synvirt_libvirt::vm::start(name)
            .await
            .map(|_| ())
            .map_err(|e| PowerError::Libvirt(e.to_string()))
    }

    async fn shutdown(&self, name: &str) -> Result<(), PowerError> {
        synvirt_libvirt::vm::shutdown(name)
            .await
            .map_err(|e| PowerError::Libvirt(e.to_string()))
    }

    async fn force_off(&self, name: &str) -> Result<(), PowerError> {
        synvirt_libvirt::vm::force_off(name)
            .await
            .map_err(|e| PowerError::Libvirt(e.to_string()))
    }
}

#[cfg(test)]
pub(crate) mod test_support {
    use super::*;
    use std::collections::HashMap;
    use std::sync::atomic::{AtomicUsize, Ordering};
    use std::sync::Mutex;
    use std::time::Duration;

    /// Records every mutating call, tracks live concurrency, and models
    /// guest state transitions so the drain/adoption logic can be
    /// exercised without libvirt.
    pub struct RecordingPower {
        states: Mutex<HashMap<String, VmState>>,
        calls: Mutex<Vec<String>>,
        cur: AtomicUsize,
        max: AtomicUsize,
        /// When true, `shutdown` flips the guest to Shutoff (ACPI
        /// honored). When false it stays Running (drain must time out).
        shutdown_completes: bool,
        /// Artificial per-`shutdown` delay so concurrency is observable.
        op_delay: Duration,
    }

    impl RecordingPower {
        pub fn new(states: &[(&str, VmState)], shutdown_completes: bool) -> Self {
            Self {
                states: Mutex::new(states.iter().map(|(n, s)| (n.to_string(), *s)).collect()),
                calls: Mutex::new(Vec::new()),
                cur: AtomicUsize::new(0),
                max: AtomicUsize::new(0),
                shutdown_completes,
                op_delay: Duration::ZERO,
            }
        }
        pub fn with_delay(mut self, d: Duration) -> Self {
            self.op_delay = d;
            self
        }
        pub fn calls_for(&self, verb: &str) -> usize {
            self.calls
                .lock()
                .unwrap()
                .iter()
                .filter(|c| c.starts_with(verb))
                .count()
        }
        pub fn max_concurrency(&self) -> usize {
            self.max.load(Ordering::SeqCst)
        }
        fn record(&self, c: String) {
            self.calls.lock().unwrap().push(c);
        }
    }

    #[async_trait]
    impl GuestPower for RecordingPower {
        async fn list(&self) -> Result<Vec<GuestRef>, PowerError> {
            self.record("list".into());
            Ok(self
                .states
                .lock()
                .unwrap()
                .iter()
                .map(|(n, s)| GuestRef {
                    name: n.clone(),
                    state: *s,
                })
                .collect())
        }
        async fn state(&self, name: &str) -> Result<VmState, PowerError> {
            Ok(self
                .states
                .lock()
                .unwrap()
                .get(name)
                .copied()
                .unwrap_or(VmState::Unknown))
        }
        async fn start(&self, name: &str) -> Result<(), PowerError> {
            self.record(format!("start:{name}"));
            self.states
                .lock()
                .unwrap()
                .insert(name.to_string(), VmState::Running);
            Ok(())
        }
        async fn shutdown(&self, name: &str) -> Result<(), PowerError> {
            self.record(format!("shutdown:{name}"));
            let now = self.cur.fetch_add(1, Ordering::SeqCst) + 1;
            self.max.fetch_max(now, Ordering::SeqCst);
            if !self.op_delay.is_zero() {
                tokio::time::sleep(self.op_delay).await;
            }
            self.cur.fetch_sub(1, Ordering::SeqCst);
            if self.shutdown_completes {
                self.states
                    .lock()
                    .unwrap()
                    .insert(name.to_string(), VmState::Shutoff);
            }
            Ok(())
        }
        async fn force_off(&self, name: &str) -> Result<(), PowerError> {
            self.record(format!("force_off:{name}"));
            self.states
                .lock()
                .unwrap()
                .insert(name.to_string(), VmState::Shutoff);
            Ok(())
        }
    }
}

Step 2: add pub mod power; to mod.rs. cargo build -p synvirt-daemon → compiles (trait unused yet ⇒ allow; it is pub so no dead-code warning).
Step 3: Commit feat(synvirt-daemon): add GuestPower seam + recording fake for lifecycle tests

Task 3: `drain` module — the only guest-stopping path

Files: Create crates/daemon/src/autostart/drain.rs. Modify mod.rs (pub mod drain;). Remove from mod.rs: run_host_shutdown, stop_one, stop_sequence, consts SHUTDOWN_BUDGET, STOP_POLL_INTERVAL, and the stop_sequence_is_reverse_of_start test.

Step 1: Create drain.rs (full content):

//! Host-shutdown guest drain. This is the ONLY guest-stopping path in
//! the daemon. It runs as the `synvirt-daemon drain-guests` subcommand,
//! wired to `synvirt-guest-drain.service`'s `ExecStop` — i.e. only on a
//! real host shutdown/reboot, never on a daemon restart. See
//! `docs/operations/daemon-lifecycle.md` for the binding semantics.

use std::time::Duration;

use futures_util::stream::{self, StreamExt};
use synvirt_libvirt::vm::VmState;
use tracing::{info, instrument, warn};

use super::power::{GuestPower, LibvirtGuestPower};
use super::{AutostartConfig, StopAction};

/// Poll cadence while waiting for a guest to actually power off.
const STOP_POLL_INTERVAL: Duration = Duration::from_millis(500);

/// Per-guest drain outcome. Notable cases carry a machine code.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DrainOutcome {
    /// Guest was already shut off — nothing to do.
    AlreadyOff,
    /// Operator policy was immediate power-off.
    PoweredOff,
    /// Guest acknowledged ACPI and powered off within its timeout.
    GracefulOff,
    /// Guest ignored ACPI; force-off issued per policy.
    ForcedOff,
    /// Guest ignored ACPI and force-off is disabled; left to the hypervisor.
    TimedOut,
}

impl DrainOutcome {
    /// Machine-readable code for the warn-level outcomes.
    pub fn code(self) -> Option<&'static str> {
        match self {
            DrainOutcome::ForcedOff => Some("E_DRAIN_FORCED_OFF"),
            DrainOutcome::TimedOut => Some("E_DRAIN_GUEST_TIMEOUT"),
            _ => None,
        }
    }
}

/// Order running guests for shutdown: reverse of the autostart start
/// order (dependencies started first stop last); guests without an
/// autostart entry sort last among themselves by name. Pure.
pub(crate) fn drain_order(cfg: &AutostartConfig, running: &[String]) -> Vec<String> {
    let key = |name: &str| -> (u32, String) {
        match cfg.vms.iter().find(|e| e.vm == name) {
            Some(e) => (e.start_order, name.to_string()),
            None => (u32::MAX, name.to_string()),
        }
    };
    let mut v: Vec<String> = running.to_vec();
    v.sort_by(|a, b| key(a).cmp(&key(b)));
    v.reverse();
    v
}

/// Resolve the effective stop action + timeout for a guest (per-VM
/// override else host default).
fn resolve(cfg: &AutostartConfig, vm: &str) -> (StopAction, u32) {
    let entry = cfg.vms.iter().find(|e| e.vm == vm);
    let action = entry
        .and_then(|e| e.stop_action)
        .unwrap_or(cfg.host.default_stop_action);
    let delay = entry
        .and_then(|e| e.stop_delay_s)
        .unwrap_or(cfg.host.default_stop_delay_s);
    (action, delay)
}

/// Drain one guest per its resolved policy, bounded by both its own
/// timeout and the overall `deadline`.
async fn drain_one<P: GuestPower>(
    power: &P,
    cfg: &AutostartConfig,
    vm: &str,
    deadline: tokio::time::Instant,
) -> DrainOutcome {
    match power.state(vm).await {
        Ok(VmState::Running) => {}
        Ok(_) => return DrainOutcome::AlreadyOff,
        Err(e) => {
            warn!(vm, code = e.code(), error = %e, "drain: cannot read state; skipping");
            return DrainOutcome::AlreadyOff;
        }
    }

    let (action, delay_s) = resolve(cfg, vm);

    if action == StopAction::PowerOff {
        if let Err(e) = power.force_off(vm).await {
            warn!(vm, code = e.code(), error = %e, "drain: power-off failed");
        }
        info!(vm, "drain: powered off (operator policy)");
        return DrainOutcome::PoweredOff;
    }

    // Graceful ACPI shutdown, then poll for power-off.
    if let Err(e) = power.shutdown(vm).await {
        warn!(vm, code = e.code(), error = %e, "drain: ACPI request failed; forcing off");
        let _ = power.force_off(vm).await;
        return DrainOutcome::ForcedOff;
    }

    let vm_deadline =
        (tokio::time::Instant::now() + Duration::from_secs(u64::from(delay_s))).min(deadline);
    loop {
        if matches!(power.state(vm).await, Ok(VmState::Shutoff)) {
            info!(vm, "drain: guest shut down gracefully");
            return DrainOutcome::GracefulOff;
        }
        if tokio::time::Instant::now() >= vm_deadline {
            if cfg.host.force_off_on_timeout {
                warn!(vm, code = "E_DRAIN_FORCED_OFF", "drain: ACPI ignored within timeout; forcing off");
                let _ = power.force_off(vm).await;
                return DrainOutcome::ForcedOff;
            }
            warn!(vm, code = "E_DRAIN_GUEST_TIMEOUT", "drain: ACPI ignored within timeout; force-off disabled, leaving to hypervisor");
            return DrainOutcome::TimedOut;
        }
        tokio::time::sleep(STOP_POLL_INTERVAL).await;
    }
}

/// Drain every running guest, in reverse-start order, with bounded
/// concurrency, bounded by the overall deadline. Best-effort: a per-VM
/// failure is logged and never aborts the rest.
#[instrument(skip_all)]
pub(crate) async fn drain_with<P: GuestPower>(
    power: &P,
    cfg: &AutostartConfig,
) -> Vec<(String, DrainOutcome)> {
    let guests = match power.list().await {
        Ok(g) => g,
        Err(e) => {
            warn!(code = e.code(), error = %e, "drain: cannot list guests; nothing drained");
            return Vec::new();
        }
    };
    let running: Vec<String> = guests
        .into_iter()
        .filter(|g| g.state == VmState::Running)
        .map(|g| g.name)
        .collect();
    if running.is_empty() {
        info!("drain: no running guests");
        return Vec::new();
    }
    let order = drain_order(cfg, &running);
    let parallelism = (cfg.host.drain_parallelism.max(1)) as usize;
    let deadline = tokio::time::Instant::now()
        + Duration::from_secs(u64::from(cfg.host.drain_deadline_s.max(1)));
    info!(
        count = order.len(),
        parallelism,
        deadline_s = cfg.host.drain_deadline_s,
        "drain: stopping running guests on host shutdown"
    );

    stream::iter(order.into_iter())
        .map(|vm| async move {
            let outcome = drain_one(power, cfg, &vm, deadline).await;
            (vm, outcome)
        })
        .buffer_unordered(parallelism)
        .collect()
        .await
}

/// Entry point for the `synvirt-daemon drain-guests` subcommand. Loads
/// the autostart config, drains every running guest, logs a per-VM and
/// summary outcome, then returns. Always succeeds (best-effort).
pub async fn run_guest_drain() {
    let cfg = super::load();
    let outcomes = drain_with(&LibvirtGuestPower, &cfg).await;
    for (vm, outcome) in &outcomes {
        match outcome.code() {
            Some(code) => warn!(vm = %vm, outcome = ?outcome, code, "drain: guest outcome"),
            None => info!(vm = %vm, outcome = ?outcome, "drain: guest outcome"),
        }
    }
    let count = |o: DrainOutcome| outcomes.iter().filter(|(_, x)| *x == o).count();
    info!(
        total = outcomes.len(),
        graceful = count(DrainOutcome::GracefulOff),
        powered_off = count(DrainOutcome::PoweredOff),
        forced_off = count(DrainOutcome::ForcedOff),
        timed_out = count(DrainOutcome::TimedOut),
        already_off = count(DrainOutcome::AlreadyOff),
        "drain: complete"
    );
}

#[cfg(test)]
mod tests {
    use super::super::power::test_support::RecordingPower;
    use super::*;
    use crate::autostart::{AutostartConfig, HostAutostartConfig, StopAction, VmAutostartEntry};
    use synvirt_libvirt::vm::VmState;

    fn entry(vm: &str, order: u32) -> VmAutostartEntry {
        VmAutostartEntry {
            vm: vm.to_string(),
            enabled: true,
            start_order: order,
            start_delay_s: None,
            stop_action: None,
            stop_delay_s: Some(1), // 1s graceful window; paused-time advances it
            wait_for_heartbeat: false,
        }
    }

    fn cfg(host: HostAutostartConfig, vms: Vec<VmAutostartEntry>) -> AutostartConfig {
        AutostartConfig { host, vms }
    }

    #[test]
    fn drain_order_is_reverse_start_then_unconfigured_by_name() {
        let c = cfg(
            HostAutostartConfig::default(),
            vec![entry("db", 10), entry("web", 20)],
        );
        let order = drain_order(&c, &["web".into(), "db".into(), "scratch".into()]);
        // start order db(10) < web(20) < scratch(MAX); reversed:
        assert_eq!(order, vec!["scratch", "web", "db"]);
    }

    #[tokio::test(start_paused = true)]
    async fn graceful_shutdown_completes() {
        let p = RecordingPower::new(&[("win", VmState::Running)], true);
        let c = cfg(HostAutostartConfig::default(), vec![entry("win", 0)]);
        let out = drain_with(&p, &c).await;
        assert_eq!(out, vec![("win".to_string(), DrainOutcome::GracefulOff)]);
        assert_eq!(p.calls_for("shutdown"), 1);
        assert_eq!(p.calls_for("force_off"), 0);
    }

    #[tokio::test(start_paused = true)]
    async fn timeout_forces_off_when_policy_on() {
        let p = RecordingPower::new(&[("win", VmState::Running)], false); // ACPI ignored
        let mut h = HostAutostartConfig::default();
        h.force_off_on_timeout = true;
        let out = drain_with(&p, &cfg(h, vec![entry("win", 0)])).await;
        assert_eq!(out, vec![("win".to_string(), DrainOutcome::ForcedOff)]);
        assert_eq!(p.calls_for("shutdown"), 1);
        assert_eq!(p.calls_for("force_off"), 1);
        assert_eq!(DrainOutcome::ForcedOff.code(), Some("E_DRAIN_FORCED_OFF"));
    }

    #[tokio::test(start_paused = true)]
    async fn timeout_leaves_running_when_policy_off() {
        let p = RecordingPower::new(&[("win", VmState::Running)], false);
        let mut h = HostAutostartConfig::default();
        h.force_off_on_timeout = false;
        let out = drain_with(&p, &cfg(h, vec![entry("win", 0)])).await;
        assert_eq!(out, vec![("win".to_string(), DrainOutcome::TimedOut)]);
        assert_eq!(p.calls_for("force_off"), 0);
        assert_eq!(DrainOutcome::TimedOut.code(), Some("E_DRAIN_GUEST_TIMEOUT"));
    }

    #[tokio::test(start_paused = true)]
    async fn power_off_action_is_immediate() {
        let p = RecordingPower::new(&[("win", VmState::Running)], true);
        let mut h = HostAutostartConfig::default();
        h.default_stop_action = StopAction::PowerOff;
        let out = drain_with(&p, &cfg(h, vec![entry("win", 0)])).await;
        assert_eq!(out, vec![("win".to_string(), DrainOutcome::PoweredOff)]);
        assert_eq!(p.calls_for("shutdown"), 0);
        assert_eq!(p.calls_for("force_off"), 1);
    }

    #[tokio::test(start_paused = true)]
    async fn already_off_guest_is_untouched() {
        let p = RecordingPower::new(&[("win", VmState::Shutoff)], true);
        let c = cfg(HostAutostartConfig::default(), vec![entry("win", 0)]);
        let out = drain_with(&p, &c).await;
        // Shutoff guest is filtered out of `running`, so it never enters
        // the drain at all → empty result, no ops.
        assert!(out.is_empty());
        assert_eq!(p.calls_for("shutdown"), 0);
        assert_eq!(p.calls_for("force_off"), 0);
    }

    #[tokio::test(start_paused = true)]
    async fn parallelism_is_bounded() {
        let states: Vec<(&str, VmState)> =
            ["a", "b", "c", "d", "e", "f"].iter().map(|n| (*n, VmState::Running)).collect();
        let p = RecordingPower::new(&states, true).with_delay(Duration::from_millis(50));
        let mut h = HostAutostartConfig::default();
        h.drain_parallelism = 2;
        let vms = ["a", "b", "c", "d", "e", "f"]
            .iter()
            .enumerate()
            .map(|(i, n)| entry(n, i as u32))
            .collect();
        let out = drain_with(&p, &cfg(h, vms)).await;
        assert_eq!(out.len(), 6);
        assert!(out.iter().all(|(_, o)| *o == DrainOutcome::GracefulOff));
        assert!(p.max_concurrency() <= 2, "max concurrency was {}", p.max_concurrency());
    }
}

Step 2: Wire module — add pub mod drain; to mod.rs. Remove run_host_shutdown, stop_one, stop_sequence, the SHUTDOWN_BUDGET/STOP_POLL_INTERVAL consts, and the stop_sequence_is_reverse_of_start test from mod.rs (callers updated in Task 5). Keep start_sequence, warnings, persistence, HTTP, wait_for_heartbeat, HEARTBEAT_*.
Step 3: cargo test -p synvirt-daemon autostart::drain → all 7 drain tests PASS.
Step 4: Commit feat(synvirt-daemon): host-shutdown guest drain (parallel, deadline, force-off policy)

Task 4: Boot autostart adopts running guests (idempotent)

Files: Modify crates/daemon/src/autostart/mod.rs (run_boot_autostart + new helpers + tests).

Step 1: Failing tests (add to mod tests):

#[test]
fn boot_start_plan_skips_running_and_keeps_order() {
    let c = cfg(vec![entry("web", true, 20), entry("db", true, 10), entry("app", true, 30)]);
    let plan: Vec<String> = boot_start_plan(&c, &["db".into()]).into_iter().map(|e| e.vm).collect();
    assert_eq!(plan, vec!["web", "app"]); // db already running → skipped
}

#[test]
fn boot_start_plan_idempotent_when_all_running() {
    let c = cfg(vec![entry("a", true, 1), entry("b", true, 2)]);
    assert!(boot_start_plan(&c, &["a".into(), "b".into()]).is_empty());
}

#[test]
fn boot_start_plan_excludes_disabled() {
    let c = cfg(vec![entry("a", true, 1), entry("d", false, 2)]);
    let plan: Vec<String> = boot_start_plan(&c, &[]).into_iter().map(|e| e.vm).collect();
    assert_eq!(plan, vec!["a"]);
}

#[tokio::test]
async fn adopt_returns_only_running() {
    use crate::autostart::power::test_support::RecordingPower;
    use synvirt_libvirt::vm::VmState;
    let p = RecordingPower::new(
        &[("A", VmState::Running), ("B", VmState::Shutoff), ("C", VmState::Running)],
        true,
    );
    let mut got = adopt_running_guests(&p).await;
    got.sort();
    assert_eq!(got, vec!["A".to_string(), "C".to_string()]);
}

#[tokio::test]
async fn boot_autostart_adopts_running_never_stops_and_is_idempotent() {
    use crate::autostart::power::test_support::RecordingPower;
    use synvirt_libvirt::vm::VmState;
    let tmp = tempfile::tempdir().unwrap();
    let tools = AgentState::bootstrap(synvirt_guest_tools::AgentConfig {
        socket_dir: tmp.path().join("sock"),
        db_path: tmp.path().join("gt.db"),
    })
    .await
    .unwrap();
    let mut host = HostAutostartConfig::default();
    host.enabled = true;
    host.default_start_delay_s = 0;
    let vms = vec![entry("A", true, 0), entry("B", true, 1), entry("C", true, 2)];
    let c = AutostartConfig { host, vms };

    // A and C already running, B is off → only B should start.
    let p = RecordingPower::new(
        &[("A", VmState::Running), ("B", VmState::Shutoff), ("C", VmState::Running)],
        true,
    );
    run_boot_autostart_with(&p, &tools, c.clone()).await;
    assert_eq!(p.calls_for("start"), 1, "only the stopped VM starts");
    assert_eq!(p.calls_for("shutdown"), 0, "boot autostart never stops a guest");
    assert_eq!(p.calls_for("force_off"), 0, "boot autostart never force-offs a guest");

    // Rapid restart: everything now running → zero starts (idempotent).
    let p2 = RecordingPower::new(
        &[("A", VmState::Running), ("B", VmState::Running), ("C", VmState::Running)],
        true,
    );
    run_boot_autostart_with(&p2, &tools, c).await;
    assert_eq!(p2.calls_for("start"), 0);
}

Step 2: cargo test -p synvirt-daemon autostart::tests::boot_ → FAIL (fns missing).
Step 3: Implement in mod.rs. Replace run_boot_autostart with:

/// Pure: VMs to actually start on boot = the enabled start sequence
/// minus those already running. This is the idempotency + "never
/// restart an adopted guest" guarantee, expressed without I/O.
pub(crate) fn boot_start_plan(cfg: &AutostartConfig, running: &[String]) -> Vec<VmAutostartEntry> {
    start_sequence(cfg)
        .into_iter()
        .filter(|e| !running.iter().any(|n| n == &e.vm))
        .collect()
}

/// Enumerate running guests on daemon start and log the adopted count.
/// Adoption proper (telemetry CID map, guest-tools listeners) is done
/// by the daemon's existing `telemetry::bootstrap_from_libvirt` +
/// guest-tools `reconcile_listeners`; this records the inventory the
/// autostart skip-list needs and surfaces it in the log.
pub(crate) async fn adopt_running_guests<P: power::GuestPower>(power: &P) -> Vec<String> {
    match power.list().await {
        Ok(guests) => {
            let running: Vec<String> = guests
                .into_iter()
                .filter(|g| g.state == synvirt_libvirt::vm::VmState::Running)
                .map(|g| g.name)
                .collect();
            info!(count = running.len(), guests = ?running, "autostart: adopted running guests (left untouched)");
            running
        }
        Err(e) => {
            warn!(code = e.code(), error = %e, "autostart: could not enumerate running guests for adoption");
            Vec::new()
        }
    }
}

/// Start the enabled VMs in order on daemon boot, adopting any guest
/// that is already running (it is left untouched, never restarted).
pub async fn run_boot_autostart(tools: AgentState) {
    let cfg = load();
    run_boot_autostart_with(&power::LibvirtGuestPower, &tools, cfg).await
}

/// Testable core of [`run_boot_autostart`] with an injected power seam
/// and config.
pub(crate) async fn run_boot_autostart_with<P: power::GuestPower>(
    power: &P,
    tools: &AgentState,
    cfg: AutostartConfig,
) {
    let running = adopt_running_guests(power).await;
    if !cfg.host.enabled {
        info!("autostart: host master switch off; not auto-starting any VM");
        return;
    }
    let plan = boot_start_plan(&cfg, &running);
    if plan.is_empty() {
        info!("autostart: nothing to start (no enabled VMs, or all already running)");
        return;
    }
    info!(count = plan.len(), "autostart: starting VMs in order");
    for e in plan {
        let delay = e.start_delay_s.unwrap_or(cfg.host.default_start_delay_s);
        if delay > 0 {
            tokio::time::sleep(Duration::from_secs(u64::from(delay))).await;
        }
        match power.start(&e.vm).await {
            Ok(_) => {
                info!(vm = %e.vm, order = e.start_order, "autostart: started");
                if e.wait_for_heartbeat {
                    wait_for_heartbeat(tools, &e.vm).await;
                }
            }
            Err(err) => warn!(vm = %e.vm, code = err.code(), error = %err, "autostart: failed to start; continuing"),
        }
    }
    info!("autostart: boot sequence complete");
}

Keep wait_for_heartbeat as-is (uses synvirt_libvirt::vm::get).

Step 4: cargo test -p synvirt-daemon autostart:: → all PASS (drain + adoption + config).
Step 5: Commit feat(synvirt-daemon): boot autostart adopts running guests, idempotent across restarts

Task 5: Wire `main.rs` — no drain on daemon stop + `drain-guests` subcommand

Files: Modify crates/daemon/src/main.rs (CLI ~224-249, signal handler ~1293-1299, autostart spawn comment ~1086-1092).

Step 1: CLI subcommand. In the Cli struct add field:

    #[command(subcommand)]
    command: Option<Command>,

After the struct add:

#[derive(clap::Subcommand)]
enum Command {
    /// Gracefully drain all running guests, then exit. Wired to
    /// `synvirt-guest-drain.service`'s ExecStop: runs on host
    /// shutdown/reboot, never on a daemon restart.
    DrainGuests,
}

Step 2: Dispatch. Immediately after the if cli.dump_openapi { … } block (before the rustls provider install) add:

    if let Some(Command::DrainGuests) = cli.command {
        // Short-lived ExecStop process (not the daemon): log to stderr
        // so journald captures the per-guest drain outcomes.
        tracing_subscriber::fmt()
            .with_env_filter(
                EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")),
            )
            .with_writer(std::io::stderr)
            .init();
        autostart::drain::run_guest_drain().await;
        return Ok(());
    }

Step 3: Signal handler. Replace the body after the signal select! (the tracing::info!("Shutdown signal received; stopping autostart VMs…") + autostart::run_host_shutdown().await; lines) with:

        tracing::info!(
            "Shutdown signal received; draining HTTP connections. Running guests are left \
             untouched — host-shutdown guest drain is synvirt-guest-drain.service's job, not \
             the daemon's restart path."
        );
        shutdown_handle.graceful_shutdown(Some(Duration::from_secs(30)));

Step 4: Update the autostart spawn comment (~1086) to mention adoption: “…start operator-enabled VMs that are not already running (running guests are adopted, never restarted)…”. No code change to the spawn line itself.
Step 5: cargo build -p synvirt-daemon → compiles. cargo run -p synvirt-daemon -- drain-guests on the dev box (no libvirt) → logs “drain: cannot list guests; nothing drained” or “no running guests”, exits 0.
Step 6: Commit fix(synvirt-daemon): daemon restart no longer drains guests; add drain-guests subcommand

Task 6: `synvirt-guest-drain.service` unit + install + deploy wiring

Files: Create iso-builder/live-build/config/includes.chroot/etc/systemd/system/synvirt-guest-drain.service. Modify crates/daemon/src/api/install.rs (copy list ~1390, enable list ~1569). Modify scripts/deploy-to-host.sh (push ~76, apply ~91).

Step 1: Create the unit:

[Unit]
Description=SynVirt guest drain on host shutdown
Documentation=https://synvirt.mx
# Guest draining is a HOST-shutdown concern, never a daemon-restart one.
# This unit owns it via systemd topology, not runtime heuristics:
#   * ExecStart=/bin/true + RemainAfterExit=yes -> the unit is "active
#     (exited)" for the whole uptime and does nothing at boot.
#   * ExecStop runs the real drain. systemd invokes ExecStop only when
#     the unit STOPS — i.e. on host shutdown/reboot. `systemctl restart
#     synvirt-daemon` does not stop this independent unit, so a
#     control-plane restart never drains guests.
# Ordering: After=libvirtd.service means that on shutdown (the reverse
# of start order) our ExecStop runs BEFORE libvirtd goes down, so
# `virsh shutdown`/`destroy` still reach the running domains. Default
# dependencies add the implicit Before=shutdown.target that schedules
# the stop during the shutdown transition.
After=libvirtd.service synvirt-network.service synvirt-daemon.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
# Graceful per-guest drain honoring /etc/synvirt/autostart.json (per-VM
# stop action + timeout, parallelism, force-off-on-timeout policy,
# overall deadline). The drain self-bounds by `drain_deadline_s`
# (default 180s); TimeoutStopSec below is the systemd backstop and must
# stay strictly above any configured deadline.
ExecStop=/opt/synvirt/bin/synvirt-daemon drain-guests
TimeoutStopSec=300
# Same resource shape as the other SynVirt planes.
CPUWeight=900
OOMScoreAdjust=-900

[Install]
WantedBy=multi-user.target

Step 2: install.rs copy list — add "synvirt-guest-drain.service", to the array at install.rs:1390.
Step 3: install.rs enable list — add "synvirt-guest-drain.service", to the array at install.rs:1569 (after synvirt-daemon.service), with a short comment that it must be enabled so the unit is active at boot and its ExecStop fires on shutdown. The existing if unit.starts_with("synvirt-") copy-existence check + symlink fallback already handle it.
Step 4: deploy push — in stage_push after the synvirt-daemon.service scp, add:

    $SCP "iso-builder/live-build/config/includes.chroot/etc/systemd/system/synvirt-guest-drain.service" \
         "$USER@$HOST:/tmp/synvirt-guest-drain.service.new"

Step 5: deploy apply — in stage_apply’s $SSH 'set -e …' block, before systemctl daemon-reload, add:

        install -m 0644 /tmp/synvirt-guest-drain.service.new /etc/systemd/system/synvirt-guest-drain.service

and after systemctl daemon-reload add:

        systemctl enable --now synvirt-guest-drain.service || true

(--now runs /bin/true so the unit becomes active immediately on an already-running host, arming its ExecStop for the next shutdown.)

Step 6: cargo build -p synvirt-daemon (install.rs compiles). bash -n scripts/deploy-to-host.sh → syntax OK. Verify the unit file is valid: systemd-analyze verify iso-builder/.../synvirt-guest-drain.service if available (else visual check).
Step 7: Commit feat(synvirt-daemon): synvirt-guest-drain.service — host-shutdown-only guest drain unit

Task 7: Docs — lifecycle semantics + deploy note + OpenAPI snapshot

Files: Create docs/operations/daemon-lifecycle.md. Modify docs/cluster.md (supersede the CRITICAL finding with the resolution + cross-link). Modify CHANGELOG.md. Regenerate crates/web-ux-v2/openapi.snapshot.json + src/api/generated/schemas.ts.

Step 1: docs/operations/daemon-lifecycle.md — document the 3 binding rules, the unit topology (with the reverse-order shutdown reasoning), the config reference (drain_parallelism, force_off_on_timeout, drain_deadline_s, per-VM stop_action/stop_delay_s), the typed outcomes/codes (E_DRAIN_FORCED_OFF, E_DRAIN_GUEST_TIMEOUT, E_ADOPT_DOMAIN_FAILED/E_LIBVIRT), and the start-time adoption behavior (“adopted N running guests”).
Step 2: docs/cluster.md — under the CRITICAL finding, append a “RESOLVED (Fix N)” note: daemon stop no longer drains; drain moved to synvirt-guest-drain.service; hot deploy is now guest-safe (one last bounce only when upgrading from a pre-fix binary, since the old binary’s shutdown handler still drains on the way down). Cross-link docs/operations/daemon-lifecycle.md.
Step 3: OpenAPI — cargo run -p synvirt-daemon -- --dump-openapi > crates/web-ux-v2/openapi.snapshot.json; then cd crates/web-ux-v2 && npm run generate:api:from-file. The three new HostAutostartConfig fields appear as optional in the schema (additive; no UI work). Run npm run lint -- --max-warnings 0 and npx vue-tsc --noEmit (or the repo’s typecheck script) to confirm green.
Step 4: CHANGELOG — add an entry under the 0.15.0 line describing the lifecycle fix.
Step 5: Commit docs(synvirt-daemon): daemon lifecycle semantics + cluster.md resolution + openapi snapshot

Task 8: Whole-workspace gate + version bump

Files: iso-builder/VERSION (Fix N bump — code change affects distributable artifacts per feedback_version_bump).

Step 1: cargo fmt -p synvirt-daemon then cargo fmt --check.
Step 2: cargo clippy -p synvirt-daemon -- -D warnings → clean (no new warnings).
Step 3: cargo test -p synvirt-daemon → green (new tests + pre-existing; document any pre-existing failures unrelated to this change).
Step 4: Bump iso-builder/VERSION Fix N (e.g. 0.15.0-FixNN). Do NOT build an ISO (per feedback_iso_build_explicit). Commit chore(release): bump Fix NN — daemon guest-safe restart.

Task 9: Field proof (the bar that failed last time)

Memory reference_daemon_restart_bounces_guests: warn before deploying to a host with running guests. The first deploy still bounces once — the old running binary’s shutdown handler drains on the way down. The new behavior is proven on restarts AFTER the new binary is live.

Step 1: Build + deploy to BOTH .65 and .75 via scripts/deploy-to-host.sh hot (per feedback_deploy_targets). On .75 record Win2016 boot_id + uptime BEFORE deploy.
Step 2: After deploy (restart #1 — old binary may bounce Win2016 one last time, expected), run systemctl restart synvirt-daemon on .75 three times. After each, confirm Win2016 boot_id unchanged and uptime strictly increasing (no reboot), and the console/agent session survives. Confirm synvirt-guest-drain.service is active (exited) and journalctl -u synvirt-daemon shows “adopted N running guests” on each start, with no “stopping autostart VMs”.
Step 3: On .65 (no critical VMs): reboot. After boot, confirm journalctl -b -u synvirt-guest-drain.service shows the ExecStop drain fired during the previous shutdown (check journalctl -b -1 -u synvirt-guest-drain.service), and the daemon logged adoption on the new boot.
Step 4: Update memory reference_daemon_restart_bounces_guests → superseded/resolved, pointing at docs/operations/daemon-lifecycle.md.

Self-review (spec coverage)

DAEMON stop = zero guest impact → Task 5 (remove drain from handler). ✓
HOST shutdown drain via dedicated unit, not daemon stop → Task 3 + Task 6. ✓
Distinction by systemd topology → Task 6 unit (ExecStop + RemainAfterExit + After=libvirtd). ✓
Startup adoption + idempotent autostart (only not-running+enabled) → Task 4. ✓
New host-shutdown drain path (subcommand + oneshot unit, ordered before libvirtd, not on restart) → Task 3/5/6. ✓
Config (no hardcodes): per-VM timeout (stop_delay_s), parallelism, force-off policy, overall deadline → Task 1 + Task 3. ✓
Typed errors/events + INFO logs (“adopted N”, per-VM drain outcome) → Task 2/3/4 (PowerError::code, DrainOutcome::code, adoption log). ✓
Docs (3 rules, unit topology, config ref, deploy note) → Task 7. ✓
Verification: no-VM-ops-on-stop (recording seam), adoption idempotency, autostart-only-not-running, drain graceful/timeout/force-off/parallelism → Tasks 3-4 tests. Field proof → Task 9. ✓
fmt/clippy clean, pre-existing tests green → Task 8. ✓

Daemon Restart Must Never Touch Running Guests — Implementation Plan

Daemon Restart Must Never Touch Running Guests — Implementation Plan

Task 1: Config — drain knobs on HostAutostartConfig

Task 2: GuestPower seam + recording fake

Task 3: drain module — the only guest-stopping path

Task 4: Boot autostart adopts running guests (idempotent)

Task 5: Wire main.rs — no drain on daemon stop + drain-guests subcommand

Task 6: synvirt-guest-drain.service unit + install + deploy wiring

Task 7: Docs — lifecycle semantics + deploy note + OpenAPI snapshot

Task 8: Whole-workspace gate + version bump

Task 9: Field proof (the bar that failed last time)

Self-review (spec coverage)

Task 1: Config — drain knobs on `HostAutostartConfig`

Task 2: `GuestPower` seam + recording fake

Task 3: `drain` module — the only guest-stopping path

Task 5: Wire `main.rs` — no drain on daemon stop + `drain-guests` subcommand

Task 6: `synvirt-guest-drain.service` unit + install + deploy wiring