Federation First — manage standalone hosts together

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/plans/2026-06-07-federation-first.md. Edit at the source, not here.

Federation First — manage standalone hosts together

Operator re-sequence (2026-06-07): hosts are ADDED + MANAGED TOGETHER while standalone (vCenter-style tree + cross-host VM ops) BEFORE “Create cluster” (Raft), which becomes a later promotion of an already-federated group. Raft (Task B) parks behind closed Gate A.

Binding decisions (carry forward)

• Modes: standalone → federated (≥1 paired peer, NO Raft) → clustered (later).
• Federation = symmetric 2c pairing trust. mesh_peers IS the managed-host list (display_name set on add).
• Group symmetry via sync_group_peers (transitive, over the Gate A Membership engine).
• LISTENER POLICY: mesh listener in ALL modes — tier-1 pairing always, +membership/federation when federated, +Raft when clustered; config kill-switch cluster.mesh.enabled. Documented deviation from Gate A’s zero-new-listeners (see docs/cluster.md). Update the daemon standalone regression assert accordingly.
• Remote authz: a pinned peer daemon (mesh SPKI, tier-2) IS the authorization; pairing approval is the operator consent.
• Tree root = “Datacenter” group node (configurable label; cosmetic).
• English, typed E_*, no hardcodes, no unwrap/expect in lib, soft warnings, local commits only.

F1 — federation backbone (library) — DONE (commit 828075a)

mesh::federation (tier-2 Federation service over FederationProvider; fetch_inventory + remote_vm_power clients; sync_group_peers) + mesh_peers.display_name (migration 0005) + E_CLUSTER_FEDERATION_FAILED. 131 cluster tests; clippy clean; single rustls.

F2 — FederationRuntime (library + harness) — DONE (bc333e2 refactor, 5ccb5a6 runtime+harness)

Landed: serve_federation_mesh (mode-scaled mounting; shared mesh_acceptor

• serve_router factored out of serve_mesh, behaviour-identical — 8-scenario harness green). FederationRuntime (load/start, add_host, on_paired admission, hosts/mode/forget, remote_inventory/remote_vm_power) + group-symmetry via a periodic reconcile (cluster.federation_resync_ms, since a transitive add has no Raft event). cluster.mesh_enabled kill-switch. 8-test harness (symmetric add, transitive symmetry, federation RPC E2E + tier-2 denial, pairing-only, restart re-pin, reject + pending-TTL, forget fails-closed, unreachable soft-fail); 139 cluster tests; 20/20 repeat-stable; clippy clean; single rustls. As-built note: the runtime mounts the provider in every mode and leans on the tier-2 gate for standalone safety (no re-bind on transition); serve_federation_mesh(provider: None) is the strict pairing-only path (harness-proven), available to the daemon. add_host takes a mesh address + operator-confirmed fingerprint — the management-plane probe→fingerprint step is F3 (HTTP node/identity), keeping F2 collision-free + harness-testable.

F2 original sub-steps (for reference)

A Raft-less sibling of ClusterRuntime, collision-free (synvirt-cluster only):

1. Mode-aware mesh server. serve_mesh is Raft-coupled; add a serve_federation_mesh(listener, identity, registry, pairing, provider?) that mounts tier-1 pairing always + membership + (when a provider is given) federation — NO Raft/write-proxy. Factor the shared accept-loop + MeshPeer
   • Tier2Gate plumbing out of serve_mesh so both reuse it (do NOT fork it).
2. FederationRuntime: holds identity / config / MeshPeerStore / PeerRegistry / Arc<dyn FederationProvider> / bind address. start() binds the listener per the listener policy; add_host(address, port?, display_name) = probe (probe_peer to learn mesh_port from identity) → pair_with → on approval persist pin + set_display_name + sync_group_peers; on_paired admission (responder side) = persist + sync_group_peers; hosts() lists mesh_peers; forget(fingerprint) local-removes; shutdown().
3. Harness (reuse the runtime.rs multi-node pattern, MULTINODE-serialized): two-host add (approve/reject/pending-TTL/channel-binding); transitive group-symmetry (.65 pairs .80 ⇒ .75 pins .80) — the direct proof; restart re-pins from mesh_peers; sync_group_peers fixpoint; federation RPC E2E over real mTLS tier-2 (GetInventory + VmPower via a mock provider) + tier-2 rejection of an unpinned caller. Raft::wait-free here (no Raft) — use bounded metric-free condition polls or readiness channels; zero bare sleeps.

F3 — daemon wiring (the heavy slice; Gate: 393 daemon suite) — NEXT

Binding operator decisions (2026-06-07):

• vms.rs collision AVOIDED BY ARCHITECTURE. F3 mounts the federated VM-power routes in a NEW module (api/federation_hosts.rs) that calls the SAME service layer vms.rs uses — the contested file is never edited. The daemon FederationProvider::vm_power likewise routes through that shared service layer (the local GuestPower seam), so cross-host power ops reuse the zero-regression local path. Ground this while writing the provider.
• en_us.json: F4 is BLOCKED until the parallel console session commits its WIP (or moves it to a worktree). F4 does not start over a dirty en_us.json.

NON-COLLIDING daemon files only — confirmed clean by the 2026-06-07 survey: main.rs, api/health.rs (AppState), api/inventory.rs, api/cluster.rs, web/routes.rs, NEW api/hosts.rs, NEW api/federation_hosts.rs, NEW api/vm_power.rs. Only vms.rs (+ datastores/handlers.rs, storage_bootstrap.rs, several web-ux files) carry parallel-session edits. 🔴 STOP-AND-FLAG if any non-colliding file above gains uncommitted edits.

Grounded wiring (survey 2026-06-07 — file:line)

• AppState = api/health.rs Inner (cluster fields at :206–215: cluster_discovery, node_identity, cluster_config). ADD pub federation: Option<Arc<synvirt_cluster::FederationRuntime>> and pub pairing_approver: Arc<crate::api::federation_hosts::HttpPairingApprover>.
• Construct in main.rs: identity via build_local_identity/ NodeIdentity::load_or_create(config.cluster.node_key_path) (:64–96); cluster setup block at :695–697; AppState built :887–928. Build the HttpPairingApprover + DaemonFederationProvider then FederationRuntime::load(identity, config.cluster.clone(), config.bind_address, approver, Some(provider)) guarded by mesh_enabled; store both in AppState.
• Shutdown hook: the SIGTERM/SIGINT task at main.rs:1288–1325 (after graceful_shutdown, :1324) — call federation.shutdown().await to close the mesh listener. (No supervised-subsystem framework exists; the discovery pattern at :146–172 is “spawn loop + log, no restart” — FederationRuntime already owns its task lifecycle, so just call shutdown().)
• vm_power seam (NEW api/vm_power.rs): mirror the four vms.rs handlers (:309–342 start, :359–399 shutdown, :504–540 force_off, :557–583 reboot) EXACTLY — Client::system() → c.{start_vm,shutdown_vm,force_off_vm, reboot_vm}(&name) + telemetry CID register/unregister (state.cid_map) + emit(DaemonEvent::VmStateChanged) + activity::hooks::vm(..,actor) + task_helper::{start,settle}. Signature apply_power(&AppState, name, action, actor: &str) -> Result<(), ApiError>. vms.rs keeps its inline copy until the parallel session frees it; a FOLLOW-UP dedups vms.rs onto this seam. vm id = libvirt domain name (the lib layer is name-based, synvirt-libvirt/src/ vm.rs:298–340).
• DaemonFederationProvider (NEW api/federation_hosts.rs): holds AppState (or a weak handle to avoid a cycle — AppState would hold the runtime which holds the provider which holds AppState; use Weak<Inner> or pass the needed sub-handles). inventory_json = serialize the local HostNode via a NEW pub api::inventory::build_local_host(&state) extracted from the handler (inventory.rs:183, collectors collect_vms/pools/networks :227/:249/:275). vm_power = vm_power::apply_power(.., "federation").map_err(|e| e.to_string()).
• HttpPairingApprover (NEW): impl PairingApprover — decide() records the PairingRequest in a Mutex<HashMap<fp, (request, oneshot::Sender<Decision>)>>
  • awaits the operator (a drop-guard removes the entry on completion OR cancellation so the PairingService pending-TTL at pairing_svc.rs:245 cleans up without a leak). pending()/approve(fp)/reject(fp,reason) drive the channel. (PairingService does NOT expose its internal pending map — the approver IS the pending registry.)
• HTTP (mount in web/routes.rs vms_api, Basic-Auth, near the cluster routes :397–419; node/identity stays public :91–97): POST /api/v1/hosts/add {address, port?, display_name?} (HTTP probe node/identity → fingerprint card → runtime.add_host(mesh_addr, fp, name)), GET /api/v1/hosts, DELETE /api/v1/hosts/{fingerprint}; GET /api/v1/cluster/pairing/pending, POST .../{fingerprint}/{approve,reject}.
• Inventory aggregation (inventory.rs): federated mode → group-root that aggregates peers via runtime.remote_inventory(fp); per-host degraded/unreachable + freshness on the typed E_CLUSTER_PEER_UNREACHABLE; NEVER fail the whole document. Standalone shape unchanged.
• cluster/status (cluster.rs:458–501 already injects mode): extend with federated mode + peer count (so the frontend flips on one field).
• OpenAPI: regenerate the snapshot — synvirt-daemon --dump-openapi > crates/web-ux-v2/openapi.snapshot.json (main.rs:254–259); additive.

Slice order (each commit keeps the daemon coherent + 393 green)

1. vm_power.rs seam + inventory::build_local_host + DaemonFederationProvider
   • HttpPairingApprover + AppState fields + main.rs embed + shutdown hook + standalone listener assert (mesh pairing-only with no peers; nothing when mesh_enabled=false; federation/membership/Raft RPCs unreachable/denied).
2. hosts + pairing HTTP (api/hosts.rs) + routes.
3. inventory aggregation + cluster/status mode/peer-count + OpenAPI regen.
4. federated power routes (api/federation_hosts.rs HTTP) + two-daemon integration harness (add→approve→federated→aggregate→remote power op via the real seam against a TEST domain; reject/TTL/forget).

Gate F3 (hard)

Full daemon suite (393 baseline — note survey counted 388 #[test]+#[tokio:: test]; reconcile the exact number on run) re-run, no shortcut; the documented pre-existing libvirt_renderer::tests::windows_11_with_vbs_overlay_emits_smm_ and_secure_boot failure is the ONLY allowed exception. Standalone regression updated (listener pairing-only / none). Restart paths ZERO VM lifecycle ops (autostart/drain.rs GuestPower seam, RecordingPower fake — Fix70 holds with the runtime embedded). Cluster 139 + new green; fmt/clippy clean on touched crates; single rustls; no unwrap/expect in lib.

F4 — frontend (Gate: vitest + vue-tsc + eslint)

🔴 STOP-AND-FLAG: en_us.json has the parallel session’s uncommitted edits. Resolve (their commit lands / file isolated) before adding i18n keys, OR get operator direction. Then:

• Add-host enabled E2E: discovered/manual probe → identity/fingerprint card → confirm → pending approval on the responder (bell + Settings) → approve → host on BOTH sides.
• Tree morph v1: standalone root → “Datacenter” group root with N host children → real VMs/pools/networks (inventory sidebar); footer shows mode.
• Cross-host VM power ops from the tree/VM list (host-scoped).

F5 — deploy + field demo (the DoD that matters)

Hardened deploy to .65 AND .75 (BOTH stages — daemon + web). From .65’s GUI: Add host → probe .75 → confirm fingerprint → approve on .75 → both GUIs show the Datacenter tree with both hosts’ real inventory. Power-op proof on a THROWAWAY test VM created on .65. 🔴 NEVER touch the Win2016 VM on .75 without explicit operator instruction.

Backlog (explicitly OUT)

Remote console; create-VM-on-remote; folders; mutual remove (DELETE is local-forget only); cluster promotion (Raft, parked behind Gate A).

Parallel-session collision map (2026-06-07)

Active parallel session (console/VNC + network/storage work) has UNCOMMITTED edits to: daemon/src/api/{console.rs,vms.rs,datastores/handlers.rs}, daemon/src/storage_bootstrap.rs, web-ux-v2/src/{api/network.ts, components/network/VSwitchPanel.vue,composables/useVncSession.ts, console/vncSession.ts,stores/networking.ts,i18n/en_us.json}. F3 avoids vms.rs (route VM ops through a non-colliding seam or wait); F4 is blocked on en_us.json. Stage explicit paths only; never git add -A; stash@{0} + network-panel WIP untouched.

---

Decisions + progress (2026-06-07)

F3 slice 1 — DONE (commit 95a3192)

FederationRuntime embedded in synvirtd (mesh per listener policy + mesh_enabled kill-switch); api/vm_power.rs seam (mirrors vms.rs, which stays untouched); DaemonFederationProvider + HttpPairingApprover; AppState federation/pairing_approver (OnceLock); post-serve shutdown hook; standalone listener assert. Daemon suite 394 pass / 1 documented pre-existing fail (windows_11_with_vbs_overlay). Slices 2-4 next.

clusterd extraction (C1–C4) — PARKED

Federation ships EMBEDDED in synvirtd (F3→F4→F5). The synvirt-clusterd service extraction is deferred until after F5, NOT cancelled. Extraction- ready price paid now: all cluster-domain HTTP handlers live in the new modules and reach the runtime through ONE ClusterControl facade (slice 2) — the extraction swaps the facade impl (direct → UDS client) without touching handlers or URLs. Cluster-domain tables (mesh_peers, discovered_hosts, pairing state) stay cleanly separable.

Supervisor deviation — accepted

The daemon has no supervised-subsystem framework (panic-contain + restart- backoff); long-lived tasks are spawn-and-log. FederationRuntime owns its own task lifecycle and is closed by the post-serve shutdown hook. The generic supervisor is backlog. Documented in main.rs + docs/cluster.md.

Management VIP — decision

Floating management VIP is APPROVED as a feature, scheduled for the CLUSTER promotion phase (Raft): leader-owned, applied/removed via the OVS reconciler desired-state path, GARP announce, all over unicast mTLS. No VIP in federation mode (every federated GUI already shows the whole group). Classic VRRP is REJECTED for this product: VRRP advertisements are multicast (224.0.0.18) and the .65↔.75 path already proved multicast does not traverse (the mDNS finding) → both nodes would go MASTER = duplicate management IP; VRRPv3 also lacks authentication.

F3 — DONE (daemon wiring complete)

Slices all green + committed, daemon suite 396 pass / 1 documented pre-existing fail (windows_11_with_vbs_overlay, Fix61):

• slice 1 (95a3192): embed FederationRuntime + DaemonFederationProvider + api/vm_power seam + HttpPairingApprover + standalone listener assert.
• slice 2 (d2bfe82): hosts + pairing HTTP behind the ClusterControl facade.
• slice 3 (aa1078f): federated inventory aggregation + status mode/peer_count.
• slice 4 (027fc12): cross-host VM power routes + facade integration test (add→approve→federated→aggregate→remote power via the facade).
• closeout (4fb3df2): utoipa on the 7 endpoints + additive OpenAPI regen (165→172 paths); (76d2866) docs/cluster.md federation API reference. vms.rs/en_us.json/foreign WIP + stash@{0} untouched throughout; explicit- path commits. synvirt-cluster unchanged (139 tests still green).

F4 — NEXT, still GATED on en_us.json

Add-host UI (probe → fingerprint card → confirm → pending-approval bell + Settings → approve → host on both sides), “Datacenter” tree morph, cross-host power ops. The OpenAPI contract is ready (generate:api:from-file). 🔴 Do NOT start over the parallel console session’s uncommitted en_us.json — wait for it to land/isolate.

F5 — the .65/.75 demo

Deploy daemon+web to both; from .65’s GUI add .75 → approve on .75 → Datacenter tree on both → power-cycle a THROWAWAY VM on .65 (Win2016 on .75 untouched without explicit instruction).

F4 — frontend — DONE (visible payload), power-ops/forget cut to F5-prep

en_us.json was clean at session start (precondition met). Committed:

• 1820621: API client regen from the F3 snapshot.
• 1fc872a (A): data layer — api/hosts.ts, federated mode, HostNode.reachable, 5 federation E_* codes, store federation actions.
• 2d90ac4 (B): AddHostCard (probe→confirm→pair) + PendingApprovals (approve/ reject) in Settings → Cluster; vitest green.
• cb16292 (C): THE MORPH — SidebarInventory re-roots to the Datacenter group with per-host children + degraded rendering; vitest green; vite build green.
• 0606228: docs/cluster.md F4 status + the F5 operator checklist. Cut to F5-prep (acceptable per the cut line): cross-host VM power-op UI (the F3 route + api/hosts.hostVmPower exist) + forget-host confirm UI (store forgetManagedHost exists). vue-tsc + vite build green; the only test failures are pre-existing (untouched Button/Tag/ISO/Wizard unit tests + Playwright e2e). vms.rs/en_us.json/foreign WIP + stash@{0} untouched; explicit-path commits.

F5 — NEXT: the .65/.75 live demo

Per the operator checklist now in docs/cluster.md. Add .75 from .65’s GUI → approve on .75 → Datacenter tree on both → (power-op proof once that UI lands, on a THROWAWAY VM; Win2016 untouched).