Daemon Wiring — Embedded Raft, Cluster Lifecycle APIs, Creation Wizard, GUI Morph

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/plans/2026-06-07-daemon-cluster-wiring.md. Edit at the source, not here.

Daemon Wiring — Embedded Raft, Cluster Lifecycle APIs, Creation Wizard, GUI Morph

Execution plan for the integration session. Foundation already landed (commit on the former feature/cluster-phase2 branch, since consolidated into 0.15.0): cluster.mesh_port config + DEFAULT_MESH_PORT/TXT_MESH_PORT constants + additive Beacon.mesh_port. RISK #1 decided: fallback to a config-driven mesh port (see docs/cluster.md → “Mesh port (deviation from single-port)”).

Binding decisions (carry forward verbatim):

• STANDALONE = RAFT DORMANT. Cluster subsystem starts only if initialized state exists in synvirt.db OR an explicit init/join API call creates it. Standalone binds zero new listeners. This is THE safety property.
• Mesh on cluster.mesh_port (fallback); :443/axum-server untouched. serve_mesh presents the ed25519 cert directly (no SNI/dual-cert).
• TREE ROOT = CLUSTER in cluster mode (no Datacenter modeling here).
• Supervised-subsystem model: panic contained, restart with backoff, leadership relinquished before subsystem restart, bounded leadership- transfer attempt on graceful shutdown.
• English; typed E_*; no unwrap()/expect() in lib; instrumented public async fns; no hardcodes; soft warnings only (2-node allowed + warned); local commits, explicit paths, never push; leave the network-panel WIP untouched.

Survey findings (grounded — do not re-discover):

• Serving: crates/daemon/src/main.rs:1366 axum_server::bind_rustls(addr, tls_config).handle(handle).serve(app.into_make_service()). Port config.https_port; bind config.bind_address. Shutdown handler main.rs:1291-1325 (Fix70: HTTP drain only). Hook leadership-relinquish there.
• AppState = Arc<Inner> (crates/daemon/src/api/health.rs:218). Existing cluster fields: cluster_discovery: Option<Arc<synvirt_cluster::DiscoveryService>>, node_identity: synvirt_cluster::types::LocalNodeIdentity, cluster_config: synvirt_cluster::config::ClusterConfig. Add cluster_runtime: Option<Arc<synvirt_cluster::ClusterRuntime>>. Constructed main.rs:887-928; cluster setup at main.rs:695-697. synvirt.db opened main.rs:119 (ClusterDb). Identity loaded main.rs:77 (NodeIdentity::load_or_create(config.cluster.node_key_path)).
• Cluster HTTP handlers: crates/daemon/src/api/cluster.rs; routes mounted crates/daemon/src/web/routes.rs:397-419. node_identity is unauthenticated (:97); cluster_status + the rest under the Basic-Auth vms_api router. Add new /v1/cluster/* there.
• synvirt-cluster runtime-assembly APIs:
  • raft::store::RaftStore::open(path, busy_ms) -> Result<Self, StorageError<u64>>; .split() -> (LogStore, StateMachine). Migrations run on open. NodeId=u64, Node=BasicNode, TypeConfig=raft::types::TypeConfig.
  • No RaftConfig -> openraft::Config helper exists — AUTHOR one: map heartbeat_interval_ms→heartbeat_interval, election bounds, snapshot_log_threshold→SnapshotPolicy::LogsSinceLast, then .validate().
  • No is_initialized() primitive — use statemachine::ClusterStore::new(Arc<ClusterDb>).cluster_id() (empty ⇒ uninitialized) and/or list_members(). initialize must apply ClusterCommand::InitializeCluster{cluster_id} so this signal is set.
  • raft::driver::spawn_transition_driver<R: VipReconciler>(node_id, raft.metrics(), Arc<SingletonSupervisor>, Arc<VipController<R>>, mpsc::UnboundedSender<LeadershipEvent>) -> JoinHandle. LeadershipEvent{code,node_id,term}.
  • singleton::SingletonSupervisor::new(vec![]) (no real singletons yet); singleton::NoopSingleton::new(name).
  • vip::VipController::new(VipConfig, R: VipReconciler); vip::VipReconciler{apply_vip,remove_vip}. AUTHOR a NoopVipReconciler (recording) — disabled VipConfig makes on_leadership a no-op anyway.
  • mesh::serve_mesh(listener, Arc<NodeIdentity>, PeerRegistry, Raft<TypeConfig>, node_id, PairingService); mesh::GrpcNetworkFactory::new(identity, registry, MeshTransportConfig, FaultPolicy::new(), node_id); mesh::{PeerRegistry, PeerInfo}; mesh::pairing_svc::{PairingService, PairingApprover, PairingDecision, PairingRequest, pair_with, PairOutcome}; mesh::write_proxy::ProxyClient; mesh::MemberDirectory::from_bootstrap(...).
  • Single-node init: raft.initialize(BTreeSet::from([node_id])); learners raft.add_learner(id, BasicNode::default(), true); raft.change_membership(BTreeSet, false). Addressing is via PeerRegistry, not BasicNode.addr.
  • node id: derive a stable u64 from the ed25519 fingerprint (first 16 hex → u64, non-zero) so init + join both get deterministic unique ids.
  • Beacon TXT mesh_port done (mport); when the runtime is active it must announce its real mesh_port (override Beacon::from_identity’s 0).
• #![forbid(unsafe_code)] + #![warn(missing_docs)] in synvirt-cluster lib — rustdoc every pub.

---

Task A — synvirt-cluster::runtime (library, harness-tested)

Create crates/synvirt-cluster/src/runtime.rs; pub mod runtime; + pub use runtime::ClusterRuntime; in lib.rs.

ClusterRuntime holds: identity: Arc<NodeIdentity>, node_id: u64, config: ClusterConfig, db: Arc<ClusterDb>, cluster_store: ClusterStore, bind_address: String, approver: Arc<dyn PairingApprover>, and Mutex<Option<Active>>. Active: raft, registry, _mesh_task: JoinHandle, _driver_task: JoinHandle, _ev_rx, mesh_addr.

• fn openraft_config(&RaftConfig, cluster_name) -> Result<Arc<openraft::Config>> (the missing mapping).
• fn node_id_from_fingerprint(fp: &str) -> u64.
• async fn load(identity, config, bind_address, approver) -> Self — opens ClusterDb; does NOT bind anything. is_initialized() = cluster_store.cluster_id()? != "". If initialized, activate().
• async fn activate(&self) -> Result<()> — bind TcpListener {bind_address}:{mesh_port} (mesh_port 0 ⇒ ephemeral for tests); read actual port; build store/factory/raft; insert self into registry (mesh_addr); spawn serve_mesh; build SingletonSupervisor::new(vec![]) + VipController::new(config.vip, NoopVipReconciler) + spawn_transition_driver. Idempotent (already-active ⇒ Ok).
• async fn initialize(&self, cluster_name) -> Result<()> — already-initialized ⇒ Err(E_CLUSTER_*) (idempotent-safe); else activate(), raft.initialize({node_id}), client_write(InitializeCluster{cluster_id}), client_write(AddMember{node_id, mesh_addr}).
• async fn join(&self, address, mesh_port_opt) -> Result<...> — JOINER side: activate() (dormant→learner-ready), pair_with(identity, node_id_str, self_mesh_addr, target_mesh_addr, expected_fp, mesh_cfg); seed MemberDirectory/registry from bootstrap_members; the RESPONDER (leader) side runs add_learner+change_membership (its API handler, Task B). Joiner waits for catch-up via raft.wait(...).metrics(applied >= ...).
• fn status(&self) -> ClusterStatusView — from raft.metrics() + cluster_store.list_members(): leader, term, membership revision, quorum, read_only (current_leader.is_none()), last_applied index, warnings (2-node soft warning). Reuse status::{ClusterStatus, RaftView} if they fit.
• async fn submit(&self, ClusterCommand) -> Result<ClusterResponse> — via ProxyClient (write-on-any-node).
• async fn shutdown(&self) — best-effort bounded leadership transfer, abort mesh+driver tasks.
• Tests (mirror raft/harness.rs gRPC build): dormant→mesh NOT bound; initialize→single-node leader + status; shutdown→closed; reload-from-disk (drop + load same db)→reactivates with state; two in-process runtimes→full join (probe→approve→add_learner→change_membership→catch-up→both status agree), reject, pending-TTL, 2-node warning, follower submit == leader-direct. Raft::wait only; repeat-stable.

Task B — daemon wiring (thin)

• AppState.cluster_runtime: Option<Arc<ClusterRuntime>>; construct in main.rs (dormant ClusterRuntime::load) before router build; the HTTP PairingApprover impl lives here (pending map keyed by fingerprint, TTL from cluster.pairing.pending_ttl_secs, surfaced via the pending/approve/reject APIs).
• Routes (web/routes.rs, Basic-Auth tier): POST /v1/cluster/initialize, POST /v1/cluster/join, GET /v1/cluster/pairing/pending, POST /v1/cluster/pairing/{fingerprint}/approve|reject; extend GET /v1/cluster/status with the live Raft view; extend GET /v1/node/identity with mesh_port.
• Cluster-scoped writes: standalone = today’s local path; initialized = route through runtime.submit() (flip on AFTER init only).
• Inventory: cluster mode populates the cluster root (members as hosts, served_by/leader).
• Shutdown hook (main.rs:1324): if let Some(rt) = state.cluster_runtime { rt.shutdown().await } before HTTP drain.
• Firewall: open/close mesh_port with the subsystem lifecycle if daemon-managed; else surface in the join unreachable hint.

Task C — frontend (whole, next turn per budget-cut-line; do NOT half-build)

• “Create cluster” wizard from /cluster standalone → name → confirm → initialize → view flips to cluster mode (members=1, leader=self, 1-node quorum note).
• “Add host”: discovered/probe → fingerprint + identity card → operator confirm (wired to approver APIs, both directions: outgoing join + incoming pending via bell/Settings) → join progress (pairing→membership→catch-up) → member in tree + list. Activate the Pair/Join buttons in initialized mode.
• THE MORPH: SidebarInventory re-roots host→cluster on mode flip (consumes the cluster-root inventory). 2-node soft warning in /cluster + Settings. Morph lands on the INVENTORY sidebar (cluster mode effectively requires it — note in docs); legacy ignores it.

---

Tests / DoD (re-supplied, fallback-adapted)

• Mesh listener lifecycle: standalone→NOT bound (assert); initialize→bound; shutdown→closed; restart-with-state→rebound; subsystem-restart→rebound.
• Daemon integration regression: standalone boot → no cluster tasks, no mesh listener, all existing behavior intact. Full daemon suite (393) re-run this session — no compile-only shortcut.
• initialize→single-node leader; status reflects leader/term/membership-rev/quorum; cluster-scoped writes flow through Raft.
• Two in-process daemons: full join via HTTP APIs (probe learns mesh_port → pair w/ HTTP approver → add_learner → change_membership → catch-up → both status agree); reject; pending-TTL expiry; 2-node soft warning; ProxyClient follower write == leader result.
• Restart-with-cluster-state: subsystem resumes (vote/term/log/applied recovered) + ZERO VM lifecycle ops (Fix70 property, assert via GuestPower seam).
• Frontend vitest: wizard happy + already-initialized typed error; add-host states; morph re-root; incoming pending-approval surface.
• Manual operator checklist + REVERT procedure (cluster reset → standalone) in docs, before the two-node field session.
• Baselines: cluster (current + new); daemon 393 + new; frontend; fmt/clippy clean on touched crates; single-rustls intact; only the documented pre-existing exceptions, no new ones.
• DoD: standalone behaviorally unchanged AND zero new listeners (suite-proven); harness initialize→1-node, join→2-node+warning, write-on-any-node, restart resumes, morph from real documents. docs/cluster.md updated (done: deviation record). Next step: two-node formation on .65/.75 with rollback plan, then real VIP/OVS + folders + TUI.

---

Progress

A1 — DONE (commit 1affabe, synvirt-cluster::runtime)

Dormancy (cluster_id() predicate → no Raft/tasks/listener), initialize (single-node, InitializeCluster+AddMember), mesh-listener lifecycle (activate binds cluster.mesh_port; shutdown graceful-closes; load resumes), status() (live Raft view + 2-node warning), restart-resume from disk, RaftConfig→openraft::Config mapping, node_id_from_fingerprint, NoopVipReconciler. 4 runtime tests; 116 cluster tests green; clippy clean.

A2 — NEXT (exact sub-steps; completes Gate A; library-only, NO daemon files)

1. pairing_svc address refinement (mesh/pairing_svc.rs): the responder must retain the joiner’s mesh address. In complete(), store the joiner’s hello.address alongside the PairedPeer (extend paired to a struct { peer: PairedPeer, address: String }; expose via paired_peers()). Change PairOutcome.bootstrap_members from Vec<(String, u64)> to a Vec<BootstrapMember{ fingerprint, node_id, address }> (keep the address from pb::MemberIdentity). Update the 2c test pairing_approved_returns_bootstrap_set_that_authorizes_leader to build MemberDirectory::from_bootstrap(outcome.bootstrap_members.iter().map(|m| (m.fingerprint.clone(), m.node_id))).
2. runtime.join(responder_mesh_addr, expected_fp) (JOINER): activate (dormant→learner-ready, NOT initialized); pair_with(identity, node_id.to_string(), self_mesh_addr, responder_mesh_addr, expected_fp, mesh_cfg); seed the Active registry from outcome.bootstrap_members (insert each PeerInfo{node_id, fingerprint, address}) so it can dial + authorize the leader; wait for catch-up via raft.wait().metrics(|m| m.current_leader.is_some() && last_applied>=…). Persist cluster_id so a restart resumes (it will once the leader’s AddMember replicates; or persist locally on join).
3. runtime.admit_paired_peers() (RESPONDER/leader): for each paired_peers() entry not yet a member, node_id = node_id_from_fingerprint(fp); registry.insert(PeerInfo{node_id, fp, address}); raft.add_learner(node_id, BasicNode::default(), true); client_write(AddMember{node_id_str, address}); change_membership(voters ∪ {node_id}, false). Idempotent (skip existing members; only if leader). In Task B the approve API calls this after approval.
4. ProxyClient in Active (store it): ProxyClient::new(raft, identity, registry, mesh_cfg, config.proxy.redirect_budget). runtime.submit(cmd) delegates to it (write-on-any-node).
5. Tests (two in-process runtimes, real mTLS gRPC, Raft::wait only, repeat-stable): A initializes; B joins (approve) → admit on A → both status agree (2 voters, 2-node warning, same leader); reject path (B.join → Err PairRejected); pending-TTL path (blocking approver + short ttl → Err); follower-submit: B.submit(cmd) == A.client_write(cmd) result (revision via leader). Restart-resume on a 1-node already covered (A1).
6. Gate A: cluster suite (116 + new) green; fmt/clippy clean; committed. THEN Task B (daemon) may begin — not before.

A2 — DONE except restart-of-both (commit e02436e)

Sub-steps 1–5 landed: pairing-address refinement (PairedMember + event sink on_paired; PairOutcome.bootstrap_members = Vec<BootstrapMember{ fingerprint,node_id,address}>); runtime.join (joiner: activate → pair → seed registry → wait-for-voter); runtime.admit (responder/leader, event-driven: register → add_learner → AddMember → change_membership); ProxyClient in Active + runtime.submit. 8 runtime tests (join forms {1,2} + both views agree + 2-node warning; reject; pending-TTL; follower- submit == leader-direct). 120 cluster tests; 20/20 repeat-stable; clippy clean. Channel-binding through join is covered by the existing 2c test (pairing_channel_binding_mismatch_is_rejected — join uses pair_with, which enforces it in begin).

Gate-A-final — NEXT (the ONE remaining Gate A item; Task B stays LOCKED)

Restart-resume of BOTH nodes after a join. Blocked by two gaps:

1. Peer-identity persistence (node-local, NOT replicated). On resume, activate rebuilds the registry with only self; cluster_members carries node_id+address but NOT the ed25519 fingerprint (the SPKI pin), so the registry cannot re-pin peers. Do NOT add the fingerprint to ClusterCommand::AddMember (wire-frozen, golden-vector discipline). Instead add a node-local mesh_peers(node_id, fingerprint, address) table (new ClusterDb migration; NOT a replicated cluster_* table), written by join (the bootstrap members) and admit (the joiner), read by activate to seed the registry before Raft::new. This is local state, so it is safe outside consensus.
2. Stable mesh port across restarts. Production uses the fixed cluster.mesh_port; the restart test must reuse a fixed port pair (bind- probe two free ports, set mesh_port to each) — ephemeral 0 changes the port on reload and strands the persisted peer addresses. Then the test: A.init + B.join → shutdown both → reload both (SAME identities + SAME mesh ports) → both resume, re-pin peers from mesh_peers, re-elect, status reconverges to {1,2}. Repeat-stable. THEN Gate A is fully closed and Task B (daemon wiring) may begin.

---

GATE A — CLOSED ✅ (commits 3370d93, d08c5e4)

The full runtime is library-only, harness-proven, 20× repeat-stable:

• Two-table trust model. cluster_members (replicated, WHO) + mesh_peers (node-local SQLite, NEVER replicated, HOW this node verifies + dials each peer via its ed25519 SPKI pin). Trust anchors stay off the Raft channel they authenticate.
• Member identity-sync (append-only proto: MemberIdentitySet + service Membership.GetMemberIdentities). A member PULLS a later joiner’s pin from a peer it already trusts (the mirror of pairing’s bootstrap set) — once per membership revision + on resume, from ANY pinned peer. Closes the availability defect (sequential-join cluster dying when the founder fails).
• Restart-resume for one-follower / both-of-2 / all-three cold boot; founder-crash survivors keep quorum; address-refresh; pin-completeness proven directly (B holds C). All Raft::wait-driven, zero bare sleeps.
• Fix: mesh_peers.node_id stored as the i64 bit pattern (fingerprint- derived u64 exceed i64::MAX → SQLITE out-of-range otherwise).
• Baselines: 127 cluster tests; clippy -D warnings clean; single rustls 0.23.38; no unwrap/expect in lib.

NEXT — direction change (operator, 2026-06-07): FEDERATION FIRST

Task B (daemon cluster wiring) is SUPERSEDED as the next session. The roadmap is re-sequenced: standalone hosts are added and managed together first (pairing-based add-host, multi-host inventory tree, cross-host VM operations) BEFORE any cluster creation. Raft parks behind closed Gate A and returns later as “promote a federated group to a cluster.” mesh_peers

• the identity-sync built here are direct prerequisites of federation (group symmetry). The operator supplies the federation spec next session.

Task B’s recorded sub-steps (ClusterRuntime into synvirtd, lifecycle HTTP APIs, HTTP PairingApprover, ProxyClient write-flip, inventory cluster-root) remain valid for whenever cluster creation returns — not deleted, just deferred behind federation.