Skip to content
SYNNET DevHub

SynVirt Cluster Control Plane

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/cluster.md. Edit at the source, not here.

SynVirt Cluster Control Plane

Section titled “SynVirt Cluster Control Plane”

Leaderless install, Raft-elected runtime leader, advisory peer discovery. Every node runs the same synvirt-daemon; there is no dedicated master and no external consensus dependency (no etcd, ZooKeeper, corosync, or sidecars).

This document is the canonical reference for the cluster architecture, the discovery trust model, and the failure scenarios the design must handle. It is written against the phased delivery plan below — each section marks what is implemented today versus planned, so the doc never overstates the running system.

Current state (2026-06-22) — extracted control plane, re-IP’d fleet, live 3-node Raft

Section titled “Current state (2026-06-22) — extracted control plane, re-IP’d fleet, live 3-node Raft”

This is the authoritative running-system snapshot; it supersedes the phase table below for “what is live today”. The dated phase/NIGHT_RUN logs are kept on purpose as historical record.

  • Control plane extracted to its own daemon. The cluster/federation control plane now runs as synvirt-clusterd (its own systemd service), no longer embedded in synvirt-daemon. It is thin: it owns the Raft consensus, the mesh listener (:7443), pairing, federation, discovery, and the cluster SQLite DB (/var/lib/synvirt/synvirt.db) — and holds zero libvirt/zfs/network handles. Every node-local effect (VM power, dataset recv, define-VM, local inventory, PAM auth) routes BACK to synvirt-daemon over a reverse FederationProvider channel (HTTP-over-UDS). Invariant: a clusterd restart / crash / election can never touch a running guest (Fix70) — proven live.
  • Mesh advertise = management IP. synvirt-clusterd advertises and binds its mesh on the host’s ipMGMT address, read from /etc/synvirt/network.yaml (any floating VIP alias on that interface is excluded). The earlier “first non-loopback IPv4” heuristic stranded the mesh on a storage NIC / DHCP uplink and degraded the aggregated cloud view on a management-IP change; that is fixed. synvirt-network now re-homes clusterd automatically (systemctl try-restart synvirt-clusterd) when it detects the ipMGMT address changed, so a deliberate re-IP keeps the cloud view working without manual steps.
  • Live fleet: a real 3-node Raft cluster. Three hosts (management IPs .11 / .12 / .13) are federated and then promoted into a Raft cluster (cluster_id set, mode: cluster). Proven live: all three agree on one elected leader, 3 reachable voters, quorum + writes-allowed, the replicated log advances; leader-kill re-election proven (stop the leader’s clusterd → the survivors re-elect under quorum 2/3, writes stay allowed, guests untouched); the killed node rejoins as a follower with no spurious re-election. Form: POST /api/v1/cluster/initialize {"name":…} on the initiator → POST /api/v1/hosts/{fingerprint}/promote per peer from the leader. Revert a node: POST /api/v1/cluster/leave.
  • Pending: the leader-pinned cluster VIP — the management/cloud entry follows the Raft leader, retiring the separate VRRP keepalived election — is the remaining dedicated slice. See Design positioning next.

Design positioning & gap analysis

Section titled “Design positioning & gap analysis”

Where SynVirt’s cluster/cloud control plane sits relative to the two dominant management-plane designs in the market. Described by behaviour — SynVirt names no third-party products in-repo; infra building blocks (Raft, etcd, ZooKeeper, corosync) are technical terms and fair to reference.

vs. a single external management appliance — one dedicated VM/server that owns the inventory and is a hard dependency to operate the estate. SynVirt has no such appliance and no single point of failure: the management UI (web-ux-v2) and the aggregated cloud/datacenter view are served by every synvirt-daemon; any host is a valid entry point. Nothing extra to install, patch, back up, or recover out-of-band. Gap closed by design (it is an advantage, not parity).

vs. a consensus-coordinated hyperconverged control plane — a ZooKeeper/etcd-class consensus electing per-service leaders, a management service embedded on every node, and an elected leader that hosts the cluster VIP. SynVirt matches the model, with the consensus embedded (openraft in-process — no external ZooKeeper/etcd ensemble to deploy): proven 3-node, leader-elected, quorum-fenced; management plane embedded per-node; cloud entry is (will be) a leader-held VIP. One gap remains: the cluster VIP is still a separate VRRP/keepalived election that can disagree with the Raft leader (the split-brain class — observed once when VRRP multicast did not cross the L2/OVS isolation and two hosts both claimed the VIP). Closing it = C5: drive the VIP from the Raft leader via OVS, term/lease-fenced, with the VIP intent stored only in Raft — one source of truth for “who leads” and “who holds the VIP”.

vs. a separate scale-out multi-cluster manager — a dedicated 1-or-3-VM appliance that aggregates many independent clusters. SynVirt embeds the multi-host cloud/datacenter view in the daemons themselves (the cloud entry’s central-admin passthrough proxies every byte to the owning host), so a single cloud needs no separate appliance — its HA is the cluster’s HA. A dedicated multi-cloud role (one pane over several independent clusters) is a future option, not a current need.

Logs / observability — operational logs are per-node (journald + the activity / audit stores), surfaced across the fleet through the cloud passthrough; the replicated Raft log carries only cluster state (config, membership, placements), never operational logs (flooding consensus with logs is an anti-pattern the reference HCI design also avoids). Gaps to close, mirroring those designs’ on-demand collection + external forwarding: a fleet-wide log-collect bundle (gather every member’s logs into one archive — today only a per-host diagnostics bundle exists) and remote-syslog forwarding to an external SIEM. SynVirt deliberately does not embed a central log lake.

Capability gap table

Section titled “Capability gap table”
Capability Reference design (behaviour) SynVirt today Gap / roadmap
Management UI availability embedded on every node, no SPOF ✅ web-ux-v2 on every synvirt-daemon closed (advantage)
Cluster consensus ZooKeeper/etcd-class, quorum, leader-elected ✅ openraft, 3-node proven, quorum-fenced, embedded (no external ensemble) closed
Aggregated multi-host view per-node + management-plane aggregation ✅ cloud view via central-admin passthrough closed
Cluster/cloud entry VIP leader-held, follows consensus ⚠️ separate VRRP/keepalived (can disagree with the Raft leader) C5 — Raft-leader OVS VIP, term/lease-fenced, intent in Raft only
Management-IP change resilience (varies; often L2-bound) ✅ mesh re-homes on ipMGMT change automatically closed
Fleet log collection on-demand bundle across all nodes ⚠️ per-host diagnostics bundle only fleet-wide log collect
External log forwarding remote syslog / SIEM ❌ none rsyslog → SIEM (Settings)
Multi-cluster management separate scale-out manager appliance embedded single-cloud only dedicated multi-cloud role (future)

Dev deploy log

Section titled “Dev deploy log”

2026-06-07 — frontend deploy gap closed. Two “I don’t see the changes” episodes traced to committed != built != deployed: the daemon serves a static web-ux-v2 bundle and nothing rebuilt/refreshed it. Fixed in scripts/deploy-to-host.sh: the web-ux-v2 stage now (1) runs npm run build itself before packaging — a stale-dist/ deploy is impossible; (2) writes dist/.build-stamp (git rev + UTC + index asset hash) at build time; (3) post-deploy, curls the target and asserts the served index-*.js hash == the freshly built one (MATCH or exit 1). The daemon-only apply stage prints a loud NOTICE when crates/web-ux-v2 sources are newer than the stamp, pointing at the web-ux-v2 stage. Proven E2E against .65: build → deploy → verified: 10.10.26.65 serves index-BivKfpNX.js (MATCH).

2026-06-06 — visible slice → .65 + .75 (feature/cluster-phase2, daemon build.901-dirty, web-ux-v2 240 assets, via scripts/deploy-to-host.sh hot). Both nodes: daemon active; GET /api/v1/node/identity, /cluster/status (mode=standalone, quorum held), /inventory, and /cluster/discovery/config all return expected shapes. .75 inventory showed the real host: Windows Server 2016 running, pool Gold, vmnics ens10f0/f1 up + f2/f3 down — the inventory aggregator is correct on a live host. Cross-node manual probe validated both ways (.65↔.75): reachable=true + version + fingerprint (matching each node’s node/identity) + cluster_state — the probe-enrichment path works over the real network. Cross-node mDNS discovery is empty both ways — the expected nested-ESXi L2 multicast limitation (the manual probe is the supported path there). Operator A/B URLs: https://10.10.26.65/ and https://10.10.26.75/, append ?sidebar=inventory (vs ?sidebar=legacy) for the parity check; see the operator click-test checklist below.

🔴 CRITICAL FINDING — control-plane restart bounces guests. Restarting synvirt-daemon on .75 stopped the running Windows Server 2016 guest, which the daemon’s own autostart then restarted — i.e. a systemctl restart synvirt-daemon (every hot-deploy / update) reboots running guests. Root cause: the daemon’s graceful-shutdown handler stops autostart VMs ("Shutdown signal received; stopping autostart VMs" → autostart::stop), violating data-plane independence. The guest recovered via autostart and is running, but it was interrupted — the Part-1 “guest uninterrupted” criterion is NOT met by current daemon behavior. This is a pre-existing daemon-lifecycle bug (not cluster code) the deploy surfaced for the first time. Fix (separate, needs operator sign-off on host-shutdown semantics): the management daemon must not stop guests on its own restart; host-shutdown guest draining belongs to libvirt-guests.service. Until fixed, every daemon deploy/update bounces running guests.

✅ RESOLVED (Fix70, 2026-06-07) — daemon restart never touches running guests; hot deploy is guest-safe. The lifecycle was split by systemd topology: (1) daemon stop drains HTTP only — the SIGTERM/SIGINT handler no longer performs any VM op (run_host_shutdown/stop_sequence removed); (2) host shutdown drains guests via the dedicated synvirt-guest-drain.service (oneshot, ExecStart=/bin/true + RemainAfterExit=yes + ExecStop=… drain-guests, ordered After=libvirtd.service so the drain runs before libvirtd stops) — it is NOT coupled to synvirt-daemon (After= ordering only, no PartOf/BindsTo/Requires), so a daemon restart never triggers it; (3) daemon start adopts running guests (logs adopted N running guests) and starts only enabled VMs that are not already running (idempotent). Field proof on .75: the running Windows Server 2016 guest (libvirt domain Id 2, QEMU PID 1382893) kept the same PID and ever-increasing uptime across the SIGKILL deploy and 3 consecutive systemctl restart synvirt-daemon — journal showed only adopted running guests … left untouched, zero shutdown/force-off. One last bounce only ever occurs when upgrading from a pre-fix binary (its old handler still drains on the way down); the SIGKILL deploy procedure avoids even that. Full semantics: docs/operations/daemon-lifecycle.md.

Delivery status

Section titled “Delivery status”

The synvirt-cluster crate is being built in phases.

2026-06-09 — feature/cluster-phase2 consolidated into 0.15.0. The branch was fast-forwarded into the 0.15.0 mainline (tip 5e250b4) and then retired; all phase work below now lives on 0.15.0. The feature/cluster-phase2 branch carried Phase 2 through the federation backbone (2026-06-06 → 2026-06-09); the dated merge narratives further down keep its name on purpose, as accurate historical record.

Phase Scope Status
1 Crate skeleton, typed E_* catalog, cluster.* config schema, node-local SQLite store (synvirt.db) + discovered_hosts, advisory mDNS discovery, manual peer reachability probe, node-lifecycle state machine. ✅ Implemented (merged to 0.15.0) + daemon wiring + web-ux-v2 sidebar//cluster view
2 (logic) ed25519 node identity, replicated state machine (ClusterCommand + deterministic apply over cluster_*), pairing handshake, quorum + witness stub, leader-singleton supervisor, VIP intent controller, status model. ✅ Implemented — 0.15.0 mainline, library-only (58→63 tests)
3a step 1 openraft pinned =0.9.24 (storage-v2 + serde); TypeConfig over ClusterCommand; versioned postcard log-entry codec; raft_log / raft_meta / raft_snapshot schema (migration 0003). ✅ Implemented — 0.15.0 mainline
3a step 2 SQLite RaftLogStorage + RaftStateMachine adapters (raft::store): single-writer actor (WAL + synchronous=FULL + busy_timeout), durability-ordered vote/append, applied-index in the same txn as cluster_* mutations. Snapshots (logical cluster_* export, atomic install) — part of this step (the state machine does not compile without snapshot support and the suite exercises it). openraft::testing::Suite::test_all conformance gate GREEN. Restart/no-double-apply, snapshot round-trip + divergent install, vote/log-state durability, and golden-vector wire-format tests. ✅ Implemented — 0.15.0 mainline (69 tests)
3a step 3 In-process Raft network (raft::net_inproc) with per-pair fault injection (drop/delay/partition) behind the RaftNetworkFactory boundary; leadership transition driver (raft::driver); openraft→E_CLUSTER_* error mapping (raft::error_map); 3-node harness (raft::harness) running all 8 scenarios via Raft::wait()/metrics condition-waits, 20/20 repeat-stable. ✅ Implemented — 0.15.0 mainline (80 tests)
transport — mesh security mTLS mesh identity (mesh::identity_tls + mesh::authz): self-signed-ed25519 cert per node, SPKI public-key pinning custom rustls verifiers (signature delegated to aws-lc-rs), two trust tiers, channel binding, bootstrap auth set. Verified with real mutual-mTLS handshakes. ✅ Implemented — 0.15.0 mainline (91 tests)
transport step 2a — gRPC deps + wire tonic =0.14.6 (TLS off — TLS is ours), tonic-prost =0.14.6, prost =0.14.3; vendored protoc; the full synvirt.cluster.v1 proto wire contract (Raft + pairing + write-proxy), frozen; mesh::wire versioned postcard envelope codec. Single-rustls graph verified. ✅ Implemented — 0.15.0 mainline (94 tests)
transport step 2b — gRPC services + dual harness gRPC Raft service + server over manual tokio-rustls (eager handshake → MeshPeer via Connected → serve_with_incoming); per-peer SPKI-pinned client (connect_with_connector_lazy + TokioIo); mesh::grpc_net::GrpcNetworkFactory on the phase-3a boundary; shared raft::fault::FaultPolicy; the 8 scenarios over in_proc + grpc_loopback, 20/20 repeat-stable; transport tests (authz, pinning, garbage→INVALID_ARGUMENT). ✅ Implemented — 0.15.0 mainline (101 tests)
transport step 2c — pairing + write-proxy (library + harness) Pairing service (tier-1) + write-proxy (tier-2) over the frozen protos; the trust-tier inversion realised — the shared listener accepts any valid self-signed ed25519 peer at the TLS accept (tier-1) and tier-2 membership is the primary structural gate via mesh::interceptor::Tier2Gate/Tier2Interceptor (a tier-2 service is mountable only through the gate; RaftService retrofitted onto it — one mechanism, not two). PairingService over the existing pairing.rs handshake (channel binding + async PairingApprover, configured pending-TTL, no auto-accept) returning the bootstrap auth set. WriteProxyService + ProxyClient apply-or-forward with a config redirect budget and typed E_CLUSTER_* (NoLeader retryable, WriteProxyFailed + event). Services exported as mountable Routes; daemon compiles with no mounting. ✅ Implemented — 0.15.0 mainline (111 tests, 10/10 repeat-stable; E2E over real mTLS gRPC)
cluster runtime (Gate A) — library + harness synvirt-cluster::runtime::ClusterRuntime: dormant-by-default (a standalone node binds zero listeners), initialize (single-node Raft), join (probe→pair→admit→catch-up), submit (ProxyClient write-on-any-node), status, restart-resume. Two-table trust model (cluster_members replicated WHO + mesh_peers node-local HOW) + member identity-sync (append-only Membership.GetMemberIdentities, pull-from-any-pinned-peer) so sequential joins converge to complete pins. Harness: two-runtime join (approve/reject/TTL/channel-binding), follower==leader-direct, pin-completeness, founder-crash availability, restart one-follower/both-of-2/all-three cold boot. ✅ Implemented — 0.15.0 mainline (127 tests, runtime 20/20 repeat-stable; commits 1affabe/e02436e/3370d93/d08c5e4)
transport step 3 — daemon wiring Embed the Raft handle in the daemon and mount the mesh services on the daemon’s own port; cluster init/join HTTP APIs over the ProxyClient + an HTTP PairingApprover; the cluster-creation wizard + the GUI morph. Real VipReconciler over synvirt-network OVS. ⏸️ Superseded by federation-first (2026-06-07) — cluster creation parks behind closed Gate A; returns later as “promote a federated group to a cluster”. Sub-steps preserved, deferred.
federation backbone — library mesh::federation: tier-2 Federation service (append-only proto — GetInventory + VmPower) over a daemon-supplied FederationProvider; pinned fetch_inventory + remote_vm_power clients; sync_group_peers (transitive group-symmetry over the Gate A Membership engine). mesh_peers.display_name (migration 0005). E_CLUSTER_FEDERATION_FAILED. ✅ Implemented — 0.15.0 mainline (131 tests; commit 828075a)
federation runtime + daemon + GUI (next) FederationRuntime (Raft-less, mode-aware mesh server: tier-1 pairing always + federation/membership when federated; on_paired ⇒ persist + sync_group_peers; add-host = probe + pair_with). Daemon: listener policy (mesh in ALL modes + cluster.mesh.enabled kill-switch), HTTP POST/GET/DELETE /api/v1/hosts + pairing pending/approve/reject (HTTP PairingApprover), inventory aggregation over the federation RPC into the group-root tree, host-scoped VM-op routing. Frontend: Add-host enabled E2E, “Datacenter” tree morph, cross-host power ops. ⏳ Next (see docs/superpowers/plans/2026-06-07-federation-first.md)
visible slice — backend seam GET /api/v1/node/identity (public; beacon fields), GET /api/v1/cluster/status (standalone-safe + mode), GET /api/v1/inventory (the host↔cluster inventory document). Frontend data layer: api/inventory.ts + api/cluster.ts status/identity + stores/inventory.ts (+ parity test) + stores/cluster.ts status/mode. ✅ Implemented — 0.15.0 mainline
visible slice — UI surfaces Probe enrichment (target node/identity); discovery config GET (toggles read-only). Settings → Cluster tab (discovery + manual-peer, deep-link); ClusterBell “another host exists” notification (one per fingerprint, Review deep-link, Dismiss persists); inventory-driven sidebar (SidebarInventory + AppSidebar toggle, default legacy) with vSwitch-uplink parity. ✅ Implemented — 0.15.0 mainline (awaiting operator parity sign-off)
visible slice — sign-off follow-up After operator A/B parity sign-off on .65/.75 (?sidebar=inventory): flip the sidebar default to inventory + remove the legacy Sidebar.vue. Ship the discovery-config PUT (live subsystem stop/start + node-config persistence) so the panel toggles go live. ⏳ Next
4 TUI cluster screen + quorum-wait; diagnostics bundle section. (web-ux-v2 sidebar + /cluster view + Settings tab land in the visible slice above.) 📐 Planned

Mesh port (deviation from single-port)

Section titled “Mesh port (deviation from single-port)”

Deviation (operator-approved, 2026-06-07). The node-to-node mesh (Raft + pairing + write-proxy gRPC over mTLS) binds its own TCP port (cluster.mesh_port, default 7443) instead of sharing the daemon’s HTTPS :443. :443 and the axum-server accept loop are not touched. Single binary / single port remains the product identity — the multiplex is a planned dedicated session, not abandoned.

Why. The daemon serves HTTPS via axum_server::bind_rustls(...).serve(...), which owns the accept loop and integrates two things a single-port multiplex would have to re-implement by hand: synvirt-certs hot-reload (RustlsConfig::reload_from_pem_file) and the Fix70 graceful_shutdown HTTP-drain handle. On top of that the multiplex has a dual-cert tension: browsers must receive the synvirt-certs server cert (CA/Let’s Encrypt) while mesh peers’ RaftDialer/ProxyClient pin the node’s ed25519 SPKI, so one listener must present different server certs by SNI (ResolvesServerCert: ed25519 for SNI synvirt.cluster, the synvirt-certs cert otherwise) plus optional-client-cert + cert-presence dispatch. Feasible, but the integration surgery is disproportionate for the wiring session — and serve_mesh on its own port presents the ed25519 cert directly, so the dual-cert/SNI problem dissolves.

Binding conditions (held).

  • cluster.mesh_port config, default constants::DEFAULT_MESH_PORT (7443); no hardcoded port at any call site.
  • The mesh listener exists only when the cluster subsystem is active — a standalone node binds zero new listeners (stronger than the original single-port plan). initialize/join start it; graceful daemon shutdown closes it; a supervised-subsystem restart rebinds it.
  • Membership/BasicNode records + the PeerRegistry carry the mesh address (host:mesh_port). GET /api/v1/node/identity and the mDNS beacon TXT carry mesh_port (additive optional field, mport; 0 = standalone / no mesh) so probe/join learn the target’s mesh port before pairing.
  • Firewall: if the node firewall is daemon-managed (Settings → Firewall), the mesh port is opened/closed with the subsystem lifecycle; otherwise the port requirement is documented and surfaced in the join-flow E_CLUSTER_PEER_UNREACHABLE hint.

Single-port multiplex — follow-up design (dedicated session). Replace axum_server::bind_rustls with a manual tokio-rustls accept loop that: (1) resolves the server cert by SNI (ed25519 for synvirt.cluster, the synvirt-certs cert otherwise) via ResolvesServerCert; (2) requests but does not require a client cert; (3) after the handshake, dispatches by client- cert presence — present → the serve_mesh tonic stack (MeshPeer in extensions, 2c tier rules); absent → the axum HTTP/UI app with today’s session auth. The loop must re-implement cert hot-reload (swap the Arc<ServerConfig> under an ArcSwap) and the graceful_shutdown drain that Fix70 relies on. The mesh then moves back onto :443 and cluster.mesh_port is retired.

Inventory document (GET /api/v1/inventory)

Section titled “Inventory document (GET /api/v1/inventory)”

ONE versioned, typed tree the dashboard renders directly — the seam that lets the GUI morph from host-centric to cluster-centric without a second app. Top level: schema (=1), mode (standalone|cluster), served_by (local node id), leader (null standalone), root.

  • Standalone root {kind:"host", id, name, address, model, version, uptime_seconds, vms, storage_pools, networks} — vms {running, total, items[]} (each {id, name, power_state, host_id}), storage_pools[] {name, health, usable_bytes, usable_free_bytes, host_id}, networks {vswitches[], vmnics[]} (vmnics carry live link_up + speed_mbps). Exactly the figures today’s sidebar renders.
  • Cluster root {kind:"cluster", id, name, hosts[], storage_pools[], networks} — hosts[] each shaped like a standalone host node. Defined now (types + serializers + fixtures); assembled once membership is replicated (a later phase).
  • Every VM / pool / network item carries its owning host_id (the local host while standalone) so cluster aggregation composes with no schema break. Additive evolution only; the frontend tolerates unknown fields. Subsystem failures degrade to empty groups (soft), never a 500.

Operator click-test checklist (dev nodes .65 / .75)

Section titled “Operator click-test checklist (dev nodes .65 / .75)”

The inventory sidebar ships behind a toggle (default legacy) precisely so this A/B parity check is done in a real browser with instant rollback.

  1. Sidebar parity. Open the dashboard, note the legacy sidebar. Append ?sidebar=inventory to the URL (persists to localStorage['synvirt.ui.sidebar']). Compare section by section against legacy: VM list + per-VM power icon + the “N on” running count; storage pools name + used/usable figures; Networks → each vSwitch’s uplink NICs with link dot + speed; selection highlight + breadcrumb; the Cluster badge count; the footer Standalone/Cluster badge. They must be identical. Roll back with ?sidebar=legacy (or clear the localStorage key).
  2. Settings → Cluster. Discovery toggles render On/Off (read-only, “Configured in node configuration”); discovered hosts list shows hostname/address/version/state/short-fingerprint/last-seen; Dismiss + Forget + “Rescan now” work; Pair/Join are disabled with “Not available in this build yet.”; manual probe against the other dev node returns a card with version + fingerprint + cluster state (or a soft detail line if the identity could not be read).
  3. Notification. With a second node or test beacon on the same L2: the bell shows exactly one entry per host (an IP change does not add a second); “Review” lands on Settings → Cluster with the host highlighted; “Dismiss” survives a reload. A host in another cluster shows “Belongs to another cluster” with no Review.

Exact next step. After the operator signs off on the parity A/B: flip the AppSidebar default to inventory and remove the legacy Sidebar.vue; ship the discovery-config PUT (live subsystem stop/start + node-config persistence) so the panel toggles go live. Then transport 2c (below), daemon Raft wiring, and the cluster-creation wizard.

Trust-tier inversion (transport 2c — implemented)

Section titled “Trust-tier inversion (transport 2c — implemented)”

The master spec forbids new listeners, so pairing shares the mesh listener with Raft + write-proxy — yet pairing must be reachable by nodes that are NOT yet members. Consequence:

  • The shared listener’s TLS-layer acceptance becomes tier-1 (any peer with a valid self-signed ed25519 cert; the handshake signature check stays delegated to aws-lc-rs, never skipped). The current RegistryAuthorizer at TLS accept must therefore relax to tier-1 on the shared listener.
  • Tier-2 (membership, incl. the bootstrap auth set) becomes the PRIMARY gate for Raft + write-proxy — enforced structurally by a shared tonic interceptor that reads MeshPeer from request extensions and returns PERMISSION_DENIED (typed E_MESH_PEER_UNTRUSTED) for non-members, NOT ad-hoc per-handler checks. A tier-2 service is mountable ONLY through Tier2Gate::guard — wiring one without the interceptor is impossible by construction. The old RaftService per-request check (2b) was REMOVED and retrofitted onto this one mechanism — no second check.
  • REQUIRED test: a valid-ed25519 non-member completes pairing RPCs but gets PERMISSION_DENIED on every Raft + proxy RPC.

Transport 2c — as built. (1) mesh::interceptor::Tier2Gate + Tier2Interceptor (a tonic Interceptor reading MeshPeer) wrap Raft + WriteProxy via InterceptedService; the shared-listener TLS authorizer relaxed to AnyEd25519Authorizer (tier-1) in serve_raft/serve_mesh. (2) mesh::pairing PairingService impl of the generated pairing_server::Pairing (Begin/Complete) wrapping the existing pairing.rs crypto (make_challenge/verify_challenge/verify_confirm) over the frozen PairHelloMsg/PairChallengeMsg/PairConfirmMsg/ PairResult/MemberIdentity; service-level channel binding (in-band claimed SPKI == TLS peer SPKI → E_CLUSTER_PAIR_REJECTED); async PairingApprover trait (approve/reject by fingerprint, pending TTL from config, one pending per fingerprint, no auto-accept); Complete returns the member identity set → joiner persists it as its initial tier-2 set (bootstrap-auth from 2b wired for real). (3) mesh::proxy WriteProxyService (SubmitCommand+request_id / GetLeaderStatus / GetLeaderHealth, all tier-2) + a proxy client: apply-if-leader else forward to the leader (ForwardToLeader hint) within a configured redirect budget → E_CLUSTER_WRITE_PROXY_FAILED+event / E_CLUSTER_NO_LEADER (retryable). (4) Harness E2E over the gRPC harness: pairing approve/reject/TTL/channel-binding/no-auto-accept/bootstrap-authorizes-leader; the tier-inversion test; proxy follower-submit==leader-direct / stale-hint redirect / no-leader / budget-exhausted / removed-member; all via Raft::wait()/metric deadlines, repeat-stable. Export as mountable Routes (no daemon mounting in the diff). Consensus membership change (add_learner → change_membership) is the join-flow’s job in the daemon session — already proven by harness scenario 8.

Tech debt (pre-existing — separate cleanup session, NOT in-band)

Section titled “Tech debt (pre-existing — separate cleanup session, NOT in-band)”

These predate the cluster work and are explicitly out of scope for cluster commits; listed so they are tracked, not silently inherited:

  • Frontend test isoLibrary.spec.ts > deleteIso removes the entry on success fails (last touched by 909366d); unrelated to cluster.
  • vue-tsc errors in Dropdown.vue, SegmentedControl.vue, and stores/vmHardware.ts (private-name exports) — the dashboard build’s strict typecheck flags them; untouched by cluster work.
  • Stale clippy warnings in synvirt-webrtc / synvirt-migrator / daemon (never constructed / never used) — pre-existing; the cluster commits add no new ones.

The 8 harness scenarios (all green + 20/20 stable): 1 cold bootstrap → single leader + VIP/singleton on the leader only; 2 deterministic replication (identical cluster_* at equal applied index); 3 leader crash → re-election + VIP/singleton move; 4 minority partition → read-only + write rejected (no isolated-leader step-down asserted, per the GARP/fencing caveat); 5 heal → quorum restored, converge, eventual single VIP holder; 6 restart from disk → recover vote/term/log/applied, no double-apply; 7 snapshot catch-up past log purge → install_snapshot; 8 joint-consensus membership grow {1}→{1,2}→{1,2,3} then remove a voter, cluster_members converges with openraft membership.

Snapshot reordering note. Snapshots are part of the storage adapter (step 2), not a later step: openraft’s RaftStateMachine does not compile without RaftSnapshotBuilder + install support, and Suite::test_all exercises the snapshot paths.

Mesh trust model (transport phase — implemented)

Section titled “Mesh trust model (transport phase — implemented)”
  • Identity = ed25519, envelope = self-signed TLS. Each node mints a self-signed X.509 cert from its existing ed25519 identity key (mesh::identity_tls::node_cert). The cert has no CA, no name checks, no expiry policy and may be regenerated every boot — because trust is SPKI public-key pinning of the ed25519 key, the durable identity. The SPKI fingerprint derived from a peer cert is byte-identical to the discovery / pairing NodeIdentity::fingerprint (SHA-256 of the 32-byte public key).
  • Custom rustls verifiers, both directions. PinnedVerifier delegates the TLS handshake signature check to aws-lc-rs’s standard routines (skipping that would be a vulnerability, not a shortcut) and makes only the acceptance decision: extract the ed25519 SPKI → fingerprint → PeerAuthorizer.
  • Two tiers. Tier 1 (pairing): AnyEd25519Authorizer accepts any valid ed25519 peer; trust is the Hello/Challenge/Confirm handshake + operator confirm, with channel binding (verify_channel_binding) requiring the in-band claimed identity to equal the TLS peer SPKI (mismatch → E_CLUSTER_PAIR_REJECTED). Tier 2 (Raft + write-proxy): MembershipAuthorizer requires the peer SPKI to resolve to a current member; unknown/removed → E_MESH_PEER_UNTRUSTED.
  • Bootstrap auth set. A freshly paired joiner seeds its MemberDirectory from the member identity set received at pairing, so it authorizes the leader’s incoming Raft RPCs before replicated cluster_members is applied; the replicated table then supersedes it.
  • Verified by real in-memory mutual-mTLS handshakes (authorized peers succeed; unknown client/server SPKI rejected; pairing tier accepts any ed25519).

Transport step 2a (done). tonic =0.14.6 / tonic-prost / prost added with tonic’s TLS OFF (TLS is ours); single rustls graph verified (cargo tree -i rustls → only v0.23.38); vendored protoc for reproducible builds; the full synvirt.cluster.v1 wire contract frozen; mesh::wire versioned postcard envelope codec.

Transport step 2b (done) — gRPC transport architecture.

  • Server (mesh::server): MeshConn wraps tokio_rustls::server::TlsStream and impls tonic’s Connected with ConnectInfo = MeshPeer { node_id, spki_fingerprint, remote_addr }. The accept loop does the eager tokio-rustls handshake with identity_tls::server_config, extracts the peer cert via tls.get_ref().1.peer_certificates() → cert_fingerprint, and only then yields into serve_with_incoming (a failed handshake / certless client never reaches hyper). RaftService reads MeshPeer from request extensions and re-checks tier-2 membership per request (defense in depth); decode failure → INVALID_ARGUMENT. raft_service() returns a mountable RaftServer for the daemon.
  • Client (mesh::client): connect_with_connector_lazy with a tower connector (TCP dial + tokio-rustls handshake, stream in hyper_util::rt::TokioIo). Each peer gets a client_config pinned to ONLY that node’s SPKI (SingleSpkiAuthorizer) — per-target pinning, so a compromised member cannot impersonate another’s address. Channels cached per peer; timeouts from MeshTransportConfig; failures → Unreachable.
  • mesh::grpc_net::GrpcNetworkFactory implements the phase-3a boundary 1:1; each RPC consults the shared raft::fault::FaultPolicy first (identical partition/heal to in_proc). The 8 harness scenarios run over both transports (Net::{InProc,Grpc}), gRPC nodes on ephemeral 127.0.0.1 listeners (identity+port reused across restart so scenario 6 works), 20/20 repeat-stable.

Transport step 2c — done. mesh::pairing_svc pairing service over tier-1 (Begin/Complete, channel binding via verify_channel_binding, async PairingApprover + pending-TTL, no auto-accept) returning the bootstrap auth set; mesh::write_proxy write-proxy (SubmitCommand with request_id, GetLeaderStatus/GetLeaderHealth) + a ProxyClient with a configured redirect budget → E_CLUSTER_WRITE_PROXY_FAILED / E_CLUSTER_NO_LEADER; tier-2 enforced structurally by mesh::interceptor::Tier2Gate. Library + harness only — 111 tests, 10/10 repeat-stable, daemon compiles with no mounting.

Transport step 3 (next session). Embed the Raft handle in the daemon and mount the mesh Routes on the daemon port; cluster init/join HTTP APIs over the ProxyClient + an HTTP PairingApprover; cluster-creation wizard + GUI morph. The real VipReconciler over synvirt-network’s OVS desired state; E_CLUSTER_SNAPSHOT_INSTALLED. Sidebar default flip + legacy removal + discovery PUT ride along after the operator A/B sign-off.

Why phased. The original spec assumed an existing mTLS gRPC mesh and an existing pairing/ed25519 join flow. Neither exists in the codebase yet (synvirt-node-agent is a stub; there is no tonic, openraft, or pairing module). A cluster cannot authenticate peers without that transport, so it is built first in phase 2. Phase 1 delivers everything that is genuinely independent of it — most importantly the advisory discovery subsystem, which by design must work in standalone mode before any cluster or trust relationship exists.

Two-table trust model + member identity-sync (Gate A — implemented)

Section titled “Two-table trust model + member identity-sync (Gate A — implemented)”

The persistent membership picture is split across two tables in synvirt.db with a deliberate, security-driven boundary:

Table Replicated? Answers Columns
cluster_members ✅ via Raft WHO is a member node_id, address, …
mesh_peers ❌ node-local, never on Raft HOW this node verifies + dials each peer fingerprint (PK, the ed25519 SPKI pin), node_id, address, updated_at

Why pins are node-local. The Raft channel is mTLS authenticated by the very SPKI pins a node would need. Replicating pins over Raft is therefore circular — you’d need the pin to receive the pin. So trust anchors never travel over the channel they authenticate. AddMember stays wire-frozen (it carries no fingerprint); pins live in mesh_peers, populated out of band of consensus.

How pins propagate. Two mechanisms, both rooted in operator approval at pairing (the sole trust origin):

  1. Pairing bootstrap set — when a node joins, the responder returns the full current member identity set; the joiner seeds its registry + pins. This covers the joiner learning existing members.
  2. Member identity-sync (append-only proto Membership.GetMemberIdentities, tier-2) — the mirror, covering existing members learning a later joiner. A node that observes a member in applied cluster_members (or the Raft voter set) without a pin pulls the full identity set from a peer it has already pinned — over that already-authenticated channel, so it is not circular. Pull-from-ANY-pinned-peer (the gap-holder may itself be the leader); triggered once per membership revision and on activate() resume; no hot loop.

The defect this closes. Without (2), a sequentially-formed cluster (A → B joins → C joins) leaves B without C’s pin: B can neither dial C nor authorize C’s inbound (tier-2 rejects an unknown SPKI). If the founder A then fails, B and C cannot assemble a majority — an availability defect, not just a missing test. With the sync, every node converges to complete pins, so the founder can fail and a full cold boot still elects + replicates.

Residual window. Between admitting a joiner and existing members first syncing its pin, a founder failure could briefly strand the new peer. The sync fires on the membership-change application (i.e. immediately, as part of normal replication) and retries each revision, so the window is the few hundred ms of one replication round.

Restart resilience. On load() a node re-pins every peer from mesh_peers with zero re-pairing and zero discovery, then the sync heals any gap. Storage note: mesh_peers.node_id holds the i64 bit pattern of the fingerprint-derived u64 (which can exceed i64::MAX).

Federation — manage standalone hosts together (no Raft)

Section titled “Federation — manage standalone hosts together (no Raft)”

Modes: standalone → federated (≥1 paired peer, no Raft) → clustered (Raft; a later promotion of an already-federated group — paired peers admit without re-pairing).

Federation is symmetric trust. A federated group is a set of hosts that have pairwise-approved each other via the 2c pairing handshake (operator confirms the fingerprint on the responder). mesh_peers is the managed-host list (display_name carries the operator label). There is no leader and no replicated state — each host calls its peers directly.

Group symmetry. sync_group_peers reuses the Gate A identity-sync engine to a transitive fixpoint: every member pulls every peer’s known set over an already-pinned channel, so adding a host to one member propagates its pin to all (.65 pairs .80 ⇒ .75 learns .80). Same trust rationale as Gate A — pairing approval is the sole origin; the sync only redistributes already-minted pins. It runs on pairing completion (direct + the adder’s transitive view) and on a periodic reconcile (cluster.federation_resync_ms, 5 s default): a transitive add has no Raft membership event to notify the uninvolved members (the Gate A trigger), so a federated host re-pulls peers on a timer to converge .75’s view of .80.

As-built mode-scaling. serve_federation_mesh is the composable mesh builder: tier-1 pairing is ALWAYS mounted; Membership + Federation (tier-2) mount only when a FederationProvider is supplied (provider: None ⇒ a strict pairing-only mesh — proven by harness). FederationRuntime is constructed with the daemon’s provider, so it mounts the tier-2 services in every mode and relies on the tier-2 gate for standalone safety: a standalone host has no pins, so every federation/membership RPC is denied — behaviourally identical to not mounting them, without a re-bind on the standalone→federated transition. (A daemon that prefers strict not-mounted-until-federated can pass provider: None while standalone and re-start on the first add.)

Remote authorization. For federation RPCs (GetInventory, VmPower) a pinned peer daemon — proven by its mesh SPKI at tier-2 — IS the authorization. The operator’s consent was given once, at pairing approval; no per-call re-confirmation. Federation RPCs are tier-2 gated exactly like Raft RPCs.

Listener policy (deviation from “standalone binds zero new listeners”). Gate A kept the mesh listener bound only when a Raft subsystem was active. To add a host, a standalone host must be reachable — so the mesh listener now runs in all modes, with the service set scaling by mode:

Mode Mesh services mounted
standalone tier-1 pairing only (inert without operator approval)
federated + membership + federation (tier-2)
clustered + Raft + write-proxy (tier-2)

A config kill-switch cluster.mesh.enabled disables the listener entirely. Rationale: a host that cannot be reached cannot be managed; tier-1 pairing alone exposes nothing without operator approval, and tier-2 services require a pin. The Gate A “zero new listeners” daemon regression assert is updated to “standalone binds only the tier-1 pairing listener (or none when cluster.mesh.enabled = false)”.

Federated tree root. A vSphere-style group node, default label “Datacenter” (configurable, cosmetic). Multi-cluster modeling stays in VirtDCM.

Architecture (target)

Section titled “Architecture (target)”
  • Every node runs synvirt-daemon. After join, nodes form an embedded Raft group using openraft. The Raft leader is the temporary cluster coordinator; any node stays usable from UI/API. Followers proxy writes to the leader; reads may be served locally with staleness metadata.
  • Raft transport rides the existing mTLS gRPC mesh (phase 2) — another gRPC service on the existing daemon port. No new listener, no sidecar, no unauthenticated Raft endpoint.
  • Raft data and the replicated cluster store live in the node-local SQLite database /var/lib/synvirt/synvirt.db (raft_log, raft_meta, replicated cluster_* tables — phase 2). The node-local discovered_hosts table (phase 1) lives in the same database but is never replicated.
  • The replicated store owns only cluster-scoped desired state (identity, members, roles, labels, auth/users, API tokens, storage pool definitions, logical networks, VM inventory, VM placement, events). Node-scoped physical configuration (NICs, OVS bridges, internal ports, uplinks, management interface) stays in the per-node network.yaml (network schema v1) and is never replicated.
  • Each node’s reconcilers consume local node YAML plus the slice of cluster desired state relevant to that node. Raft apply only mutates replicated SQLite state; side effects are performed by reconcilers observing applied desired state.

Storage decision (phase 1)

Section titled “Storage decision (phase 1)”

The spec specifies SQLite at /var/lib/synvirt/synvirt.db. The codebase core uses sled for most per-feature state, but rusqlite is already a workspace dependency (used by synvirt-observability and guest-tools telemetry), and a relational store is the natural fit for the Raft log + replicated tables. We therefore introduce synvirt.db as specified. Phase 1 creates only discovered_hosts there (migration 0001_discovered_hosts.sql); phases 2+ add raft_log, raft_meta, and the replicated cluster_* tables via additional numbered migrations. Migrations are idempotent (CREATE TABLE IF NOT EXISTS) and never break standalone mode.

Peer discovery (advisory only) — phase 1

Section titled “Peer discovery (advisory only) — phase 1”

Discovery answers exactly one question: “another SynVirt host exists.” It is advisory end to end:

  • It never authenticates, never authorizes, and never auto-joins.
  • Beacons are unauthenticated by nature. The ed25519 (phase 1: TLS- derived) fingerprint in a beacon exists only so an operator can visually confirm a host during the real, trusted join path (pairing + mTLS). Beacon content never influences an authorization decision.
  • The trusted path into a cluster is pairing + mTLS — the single source of trust. Discovery is a convenience layer on top, nothing more.

Wire model

Section titled “Wire model”
  • Service type: _synvirt._tcp.local. (DNS-SD, RFC 6763).
  • Transport: pure-Rust mDNS via the mdns-sd crate, in-process inside the daemon. No avahi, no external mDNS daemon, no sidecar.
  • The mDNS multicast group/port (224.0.0.251:5353) are RFC protocol constants, not configuration. Everything else (intervals, TTL, interface, enable flags) is configuration.
  • Announce TXT metadata: node id (nid), hostname (host), daemon version (ver), cluster id (cid, empty if standalone), cluster state (cst: standalone | member | degraded), identity fingerprint (fp), API port (port).
  • Announce/browse bind to the management interface only (resolved through the network model). Discovery is never bound to non-management interfaces by default.

Defensive parsing

Section titled “Defensive parsing”

Beacons are untrusted input. The decoder (discovery::beacon::Beacon::from_txt):

  • requires the fingerprint (the dedup key); a beacon without one is E_CLUSTER_BEACON_MALFORMED and is logged and dropped — never panicked on;
  • tolerates unknown TXT keys (forward compatibility);
  • degrades an unparseable cluster state to standalone (the most conservative reading — an advisory beacon must never be upgraded to “member” by accident);
  • rejects a non-numeric port as malformed rather than guessing.

discovered_hosts store

Section titled “discovered_hosts store”

Node-local table keyed by identity fingerprint, not IP:

field meaning
fingerprint primary key; stable across IP changes
node_id, hostname, version, cluster_id, cluster_state, api_port last-announced values
addresses JSON array of textual management addresses last seen
first_seen, last_seen RFC-3339 UTC
dismissed operator dismissed this fingerprint; suppress notifications until the fingerprint changes; persisted

Entries expire after a configurable TTL without announcements (expire_older_than), at which point a HostLost notification fires.

Configuration (cluster.discovery)

Section titled “Configuration (cluster.discovery)”
cluster.discovery.enabled            (default true)
cluster.discovery.announce           (default true)
cluster.discovery.browse             (default true)
cluster.discovery.announce_interval_secs (default 60)
cluster.discovery.host_ttl_secs      (default 300)
cluster.discovery.interface_override (default "" → resolved mgmt iface)

Disabling discovery in config opens no sockets: DiscoveryService::with_mdns returns E_DISCOVERY_DISABLED without constructing the mDNS daemon, and a running service’s stop() closes the socket and aborts the browse loop.

Typed errors

Section titled “Typed errors”

E_DISCOVERY_SOCKET_FAILED, E_DISCOVERY_DISABLED, E_DISCOVERY_INTERFACE_UNAVAILABLE, E_CLUSTER_BEACON_MALFORMED. Discovery failure is never fatal to the daemon — every condition is logged as a coded warning and the daemon continues.

“Another host exists” notification — phase 1 model

Section titled ““Another host exists” notification — phase 1 model”

(The dashboard bell and TUI surfaces land in phase 4; the crate already produces the notification stream and enforces the semantics below.)

  • When a new standalone host first appears, exactly one notification fires per fingerprint (HostDiscovered).
  • Dedup is by fingerprint. An IP change updates the same row and does not re-notify.
  • Dismissing a host sets the persisted dismissed flag; that fingerprint never re-notifies — including across daemon restarts — unless the fingerprint itself changes.
  • Version skew between the local and discovered host is a soft flag on the notification, never a block.
  • A host that belongs to a different cluster is flagged different_cluster → the UI renders “Belongs to another cluster” with no join action.
  • Notification content: hostname, address, version, short fingerprint.

Manual peer add (why it exists) — phase 1 probe

Section titled “Manual peer add (why it exists) — phase 1 probe”

Multicast does not traverse routed segments. A management network on a routed (L3-separated) segment will never be discovered via mDNS. Manual peer add is therefore the supported mechanism, not an optional extra.

Phase 1 implements the reachability half: probe_peer(host, port, timeout) performs a bounded TCP connect to the target’s API port and returns reachable. The daemon_version / fingerprint / cluster_state fields require the authenticated pairing probe endpoint and are populated in phase 2. The only hard error on this path is an empty target address (an operation that literally cannot proceed); everything else returns E_CLUSTER_PEER_UNREACHABLE (retryable) or E_CLUSTER_PEER_PROBE_FAILED.

Failure scenarios

Section titled “Failure scenarios”

The scenarios below are the design contract. Items marked (phase N) are validated by the phase that implements the mechanism; phase 1 ships the discovery-related guarantees (scenario 8) and the data-plane independence stance.

1. Leader crash (phase 2/3)

Section titled “1. Leader crash (phase 2/3)”

A follower detects leader loss; Raft elects a new leader; the management VIP moves to the new leader through the OVS reconciler and a GARP is announced; dashboard/TUI show the new leader. Target recovery time is under 10 seconds (cluster.raft.target_recovery_secs, configurable).

2. Network partition (phase 2/3)

Section titled “2. Network partition (phase 2/3)”

The majority side continues. The minority side becomes read-only for cluster-scoped writes and must not hold the VIP. Existing VMs keep running (the data plane does not depend on the control plane); local node APIs and local VM console access continue. Cluster writes on the minority fail with E_CLUSTER_READ_ONLY / E_CLUSTER_QUORUM_LOST.

3. Node restore (phase 2/3)

Section titled “3. Node restore (phase 2/3)”

A restored node starts in Restoring/Degraded, validates its identity, detects stale logs, and catches up via snapshot + log replay. It returns to Member only after state-machine sync, and becomes leadership-eligible only after its membership is committed.

4. Full cluster cold boot (phase 2/3)

Section titled “4. Full cluster cold boot (phase 2/3)”

Nodes start without assuming leadership. A quorum-wait state is visible (TUI quorum-wait screen). Once a majority forms, a leader is elected, the VIP is applied by the leader, and the APIs become writable.

5. 2-node cluster (allowed, with warnings)

Section titled “5. 2-node cluster (allowed, with warnings)”

A 2-node cluster is allowed but surfaces a soft warning everywhere (API, dashboard, TUI, and this doc). A witness/tiebreaker trait (ClusterWitnessProvider / NoopWitnessProvider) and cluster.witness config stub exist but the witness is not implemented; the absence yields the soft E_CLUSTER_WITNESS_NOT_CONFIGURED warning, never a hard block. Risk: with two nodes, losing either loses quorum; the cluster goes read-only until a node returns. Run three nodes for production.

6. Version skew (soft)

Section titled “6. Version skew (soft)”

Warning only — never a hard block. An incompatible protocol produces a typed error only when an operation is actually unsafe.

7. Clock skew (soft)

Section titled “7. Clock skew (soft)”

Warning only. Raft does not depend on wall-clock correctness for safety (it uses logical terms + randomized election timeouts). UI/TUI surface a warning where skew detection exists.

8. Discovery unavailable or filtered (phase 1 — implemented)

Section titled “8. Discovery unavailable or filtered (phase 1 — implemented)”

Multicast blocked, or management on a routed L3 segment: discovery silently finds nothing and the daemon and cluster remain fully functional. Manual peer add (Settings → Cluster, phase 4 UI; the probe API underneath, phase 1) is the documented path. All E_DISCOVERY_* conditions are logged as warnings, never fatal.

Invariants (non-negotiable)

Section titled “Invariants (non-negotiable)”
  • Data plane independence. If quorum is lost, existing VMs keep running. Only cluster-scoped writes become unavailable; local node APIs, local consoles, and node-local observability continue.
  • No raw ip addr for the VIP. VIP add/remove goes through the OVS/network reconciler desired-state path only. No raw ip addr add/del, no shell side-channel, no second interface-mutation path.
  • Discovery is advisory. It never joins, pairs, or trusts anything by itself. Pairing + mTLS is the only trusted path into a cluster.
  • Config is the source of truth. No port, VIP, timeout, retry interval, election setting, TTL, or path is hardcoded; defaults live in synvirt_cluster::constants and are overridable via cluster.*. A required-but-missing value returns an explicit typed E_* error.
  • No unwrap() / expect() in runtime/library code. Every fallible path returns ClusterError.

Crate map (crates/synvirt-cluster)

Section titled “Crate map (crates/synvirt-cluster)”
Module Responsibility
error Typed E_CLUSTER_* / E_DISCOVERY_* catalog + code() / is_retryable().
events Event codes (E_CLUSTER_HOST_DISCOVERED, …).
config cluster.* schema with defaults + feature-gated validation.
constants RFC protocol constants + tunable defaults.
types NodeLifecycle, WireClusterState, DiscoveredHost, LocalNodeIdentity, PeerProbeResult.
db synvirt.db opener + migration runner.
discovery::beacon Beacon TXT encode/decode (defensive).
discovery::store discovered_hosts repository (dedup, dismiss, TTL).
discovery::transport DiscoveryTransport trait + mdns-sd impl + mock.
discovery DiscoveryService lifecycle + notification model.
probe Manual peer reachability probe.

Phase 2+ adds: raft (openraft storage + network), statemachine (replicated commands + apply), membership, transport_grpc, vip, singleton, status.

Federation HTTP API (F3 — embedded in synvirtd)

Section titled “Federation HTTP API (F3 — embedded in synvirtd)”

All cluster-domain handlers reach the runtime through one ClusterControl facade (extraction-ready: a future synvirt-clusterd swaps a UDS impl in without changing handlers or URLs). Basic-Auth, except node/identity (public). 503 E_FEDERATION_DISABLED when cluster.mesh_enabled = false.

Method Path Purpose
POST /api/v1/hosts/add {address, mesh_port?, fingerprint, display_name?} — pair (operator-confirmed fingerprint), background; returns 202 pending
GET /api/v1/hosts managed-host list (mesh_peers + display_name)
DELETE /api/v1/hosts/{fingerprint} local forget (mutual remove = backlog)
POST /api/v1/hosts/{fingerprint}/vms/{vm_id}/{action} cross-host VM power (start/shutdown/force-off/reboot), target-qualified
GET /api/v1/cluster/pairing/pending pairings awaiting the operator’s fingerprint confirmation
POST /api/v1/cluster/pairing/{fingerprint}/{approve,reject} the HTTP PairingApprover
GET /api/v1/inventory federated mode ⇒ “Datacenter” group root aggregating peers (unreachable ⇒ reachable:false, never fails the doc)
GET /api/v1/cluster/status adds mode (federated) + peer_count

Standalone-assert (updated for the listener deviation): with no peers the daemon binds the mesh listener pairing-only (federation/membership RPCs tier-2-denied — no pins); cluster.mesh_enabled = false binds nothing. The daemon’s vm_power seam mirrors api::vms exactly (telemetry CID + event + activity + task_helper); cross-host ops are attributed to actor federation. Restart leaves running guests untouched (Fix70 / the synvirt-guest-drain seam); the mesh closes only post-serve.

F4 — frontend (add-host + Datacenter morph)

Section titled “F4 — frontend (add-host + Datacenter morph)”

Implemented (web-ux-v2, against the F3 OpenAPI; #0566C1 + dark theme; reuses Card/Button/Tag/Modal/toast):

  • API client regenerated from the F3 snapshot; api/hosts.ts typed client; ClusterStatus.peer_count; HostNode.reachable; 5 federation E_* codes.
  • Add-host flow (Settings → Cluster): AddHostCard probes a target, shows its identity for the operator to confirm the fingerprint, then POST /hosts/add (background pairing).
  • Approval surface: PendingApprovals lists incoming requests with approve/reject (polled); the cluster store’s badge counts them.
  • THE MORPH: SidebarInventory re-roots to a “Datacenter” group with N host children (each expands to its VMs + a pools·vSwitches summary); an unreachable peer renders degraded (offline dot), tree intact; flat sections hide when grouped; footer reads “Datacenter”. Off inventory.grouped.
  • Tests: add-host probe→confirm→pairing; incoming pairing approve; morph re-roots + degraded host; standalone unchanged. vue-tsc + vite build green.

Cut to F5-prep (per the F4 budget cut line — power ops are the acceptable cut): cross-host VM power ops (the F3 POST /hosts/{fp}/vms/{vm}/{action} route + api/hosts.hostVmPower exist; needs a store wrapper + per-VM buttons on the tree) and forget-host (DELETE wired in the store as forgetManagedHost; needs a confirm action on the host row).

F5 — operator checklist (.65 / .75 live demo)

Section titled “F5 — operator checklist (.65 / .75 live demo)”
  1. Deploy daemon and web to .65 and .75 (scripts/deploy-to-host.sh; the web stage self-builds + hash-verifies).
  2. On .65 GUI → Settings → Cluster → Add host: enter 10.10.26.75 → Probe → confirm the fingerprint matches .75’s node/identity → Add host (shows “pending”).
  3. On .75 GUI → Settings → Cluster → Pending host requests → Approve.
  4. Both GUIs (sidebar ?sidebar=inventory): the tree morphs to Datacenter with both hosts; expand .75 → its real VMs/pools/networks aggregated. GET /api/v1/hosts agrees on both sides; cluster/status mode=federated.
  5. (When the power-ops UI lands) create a throwaway VM on .65 and power-cycle it cross-host from the tree. 🔴 NEVER touch the Win2016 VM on .75 without explicit operator instruction.
  6. (When forget lands) Forget .75 from .65 → it leaves .65’s list.

F5 demo log (2026-06-07, .65 / .75) — deploy ✅, Fix70 ✅, BLOCKER found

Section titled “F5 demo log (2026-06-07, .65 / .75) — deploy ✅, Fix70 ✅, BLOCKER found”

Deploy (both hosts, hot stage, no reboot): .65 and .75 both upgraded to 0.15.0-Fix72+build.944 with the embedded federation control plane. Mesh listener bound on 0.0.0.0:7443 (pairing-only, standalone) on both; /hosts → 200, cluster/status mode=standalone peer_count=0.

Fix70 verified LIVE (the headline safety result): .75 runs the production Windows Server 2016 guest. Across the daemon restart (deploy), the guest kept the SAME QEMU PID 1469944 with ever-increasing uptime (36081 → 36672 → 36854 s) — zero bounce. The management-daemon restart never touched the guest.

🔴 BLOCKER (the live demo surfaced it; the auto-approver tests could not): operator-confirmed add-host fails. From .65: POST /hosts/add → pending, but the pairing never completes — .65 logs E_CLUSTER_PAIR_REJECTED: pairing rejected: Timeout expired (target .75:7443); .75’s pending list is empty; approve → 404. TCP :7443 is open both ways (not firewall).

Root cause — timeout inversion in the operator-approval flow. The responder’s pairing complete() blocks on PairingApprover::decide() (the HTTP approver awaits the operator, up to pairing.pending_ttl = 300 s, config.rs:292), but the JOINER’s mesh client RPC timeout is MeshTransportConfig::rpc_timeout = 5 s (mesh/config.rs:33). The joiner gives up after 5 s — long before a human can approve. F2/F3 harnesses used an auto-approver (decide returns instantly), so they never hit it.

Fix (synvirt-cluster; NOT hot-patched on live hardware per the demo cut line): the operator-approval pairing must not be a single blocking RPC under the 5 s budget. Options, cleanest last: (a) give the pairing complete client a long timeout (≥ pending_ttl) — one mesh connection held open for the approval window; smallest change. (b) joiner polls: complete returns pending immediately and the joiner retries (short RPCs) until approved/rejected/TTL — no long-held stream. (c) decouple: begin records the pending + returns; approval is fully out-of-band; a separate finalize RPC runs post-approval.

State after: clean — 0 managed hosts on both sides (the failed add persisted nothing); Win2016 untouched. No revert needed. Both daemons healthy.

NIGHT_RUN log (2026-06-07 → overnight)

Section titled “NIGHT_RUN log (2026-06-07 → overnight)”

Autonomous session: Phase 1 (fix operator-approved pairing via polling + close F5) → Phase 2 (cluster-create wizard, harness-validated) → Phase 3 (Datacenter drag-and-drop). Each phase committed green before the next.

Phase 1 ✅ — pairing fix + F5 live demo closed (2026-06-07 overnight)

Section titled “Phase 1 ✅ — pairing fix + F5 live demo closed (2026-06-07 overnight)”

Backend fix (2bcede0): operator-approved pairing is now poll-based — Begin returns immediately and spawns the decision; Complete reports PENDING/APPROVED/REJECTED; pair_with polls with short RPCs (400ms / 360s). Append-only proto (PairStatus + PairResult.status/detail). Regression guard added: delayed-approve + delayed-reject under a 600ms rpc_timeout a blocking Begin would have failed — the gap the auto-approver harness hid. Cluster suite 141/141; fmt + clippy clean.

Second fix the demo surfaced (17d689c): the mesh advertised its wildcard bind (0.0.0.0:7443), which peers stored and could not dial — the federated peer rendered degraded. Now the daemon advertises the detected management IP when the bind is a wildcard.

F5 live demo — .65 / .75, both on 0.15.0-Fix72+build.948 (hot stage):

  • Fix70 verified again, live: the production Windows Server 2016 guest on .75 kept QEMU PID 1469944 across every daemon restart (uptime climbed 37895 → 39076 s). The management plane never bounced the guest.
  • Add → approve → federated: .65 added .75 (operator confirmed the fingerprint); .75’s pending list showed the request; approve on .75 landed inside the poll window; both /hosts agreed with dialable addresses (10.10.26.75:7443 / 10.10.26.65:7443). cluster/status mode=federated, peer_count=1 on both.
  • Datacenter tree, both sides: /inventory re-rooted to the Datacenter group; .65’s tree showed .75’s node reachable with its real Windows Server 2016 (running) + pool, and vice-versa.
  • Cross-host power: from .75, started then force-off’d a throwaway VM on .65 (204/204; .65 went running → shut off). Win2016 never touched (PID 1469944, still running). Throwaway undefined after.
  • Revert verified: forget both directions → both hosts mode=standalone, empty /hosts. Federation then re-established for the morning (both federated, peer_count=1).

Next operator step (supervised): form a real Raft cluster on .65/.75 via the Phase 2 “Create cluster” action (persistent new state — see Phase 2 below).

Phase 2 — DESIGN FORK (documented, not forced unattended)

Section titled “Phase 2 — DESIGN FORK (documented, not forced unattended)”

Goal: a “Create cluster” action that promotes an already-federated group into a Raft cluster. Architecture survey (this session) found a genuine fork that blocks live wiring; per the overnight autonomy rules this is documented for Tony’s decision rather than forced on the proven, deployed federation plane.

The fork. FederationRuntime and ClusterRuntime are DISJOINT structs:

  • FederationRuntime (federation_runtime.rs) — what the daemon embeds today. Holds provider: Option<Arc<dyn FederationProvider>> and the federation surface (add_host, remote_inventory, remote_vm_power, federated mode). NO Raft. This is what F1–F5 delivered and what is live on .65/.75.
  • ClusterRuntime (runtime.rs) — holds store: ClusterStore + the openraft handle and the full create/join/admit/status path (13 multi-node tests, green). NO FederationProvider — it cannot serve add_host / remote inventory / cross-host power.

So the daemon cannot simply “add create_cluster” to the embedded runtime:

  • Swapping FederationRuntime → ClusterRuntime would LOSE the federation surface (add-host, Datacenter tree, cross-host power) that is deployed and proven.
  • The create-cluster LOGIC is already done + tested in ClusterRuntime; the gap is purely that the two runtimes are not unified, and only one can own the single mesh listener.

Also: mesh_peers pins (the federation trust) are reusable for RESTART (re-pin on activate) but NOT for cluster membership bootstrap — admission is event-driven off pairing events, so a federated peer must re-pair to be admitted as a Raft voter (runtime.rs join → pair_with → add_learner → change_membership).

Recommended path (one session, supervised — it touches a deployed plane): Unify into ONE runtime the daemon embeds, dormant-Raft by default:

  1. Give ClusterRuntime the federation surface: add the provider field + add_host + the tier-2 federation service (remote_inventory / vm_power) + mode() (“standalone” → “federated” once mesh_peers non-empty → “cluster” once Raft initialized). Its activate() already serves the mesh; mount the federation provider on the same listener (mode-scaling already exists).
  2. Daemon embeds ClusterRuntime instead of FederationRuntime; the EmbeddedClusterControl facade keeps all federation methods (now backed by the unified runtime) and gains create_cluster(name) → initialize(name) and cluster_status() → ClusterRuntime::status().
  3. “Create cluster” semantics from a federated group: initialize on the initiator, then drive each federated peer to re-pair into the cluster (reusing the pinned trust for the dial, so the operator does NOT re-confirm) → admit promotes each to voter. This is the one genuinely-new bit of logic (a “convert federation to cluster” flow) and deserves its own harness test alongside the existing initialize/join/admit suite.
  4. Then the HTTP route (POST /api/v1/cluster/create) + the frontend wizard land on a real backend, harness-validated, before any live .65/.75 formation.

A half-built wizard (UI that always errors) was deliberately NOT shipped — it would violate the “no half-built feature between phases” rule. The frontend “Create cluster” surface lands with the unified backend.

Next operator step: approve the runtime unification (option above) for a supervised session; create-cluster wiring + wizard follow it.

NIGHT_RUN summary (morning hand-off)

Section titled “NIGHT_RUN summary (morning hand-off)”
Phase Status Commits
1 — pairing fix + F5 live ✅ DONE 2bcede0, 17d689c, 5f28db3
2 — cluster-create wizard ⏸ DESIGN FORK (documented) f0b6506
3 — Datacenter drag-and-drop ✅ DONE 292dd6a

Phase 1 ✅ — operator-approved pairing made poll-based (Begin no longer blocks on the operator; Complete polls PENDING/APPROVED/REJECTED), closing the 5s-rpc-vs-300s-approval inversion the live demo found. A second demo-surfaced bug fixed: the mesh advertised its wildcard bind (0.0.0.0) so peers stored an undialable address — now advertises the management IP. F5 closed live on .65/.75: add → approve → federated Datacenter tree with .75’s real Win2016 → cross-host power on a throwaway → revert. Fix70 held throughout (Win2016 PID 1469944 never bounced across ~6 daemon restarts). Cluster 141/141, daemon builds, frontend green.

Phase 2 ⏸ — genuine design fork (NOT forced unattended on the deployed plane). FederationRuntime (what the daemon embeds; owns add-host + remote inventory + cross-host power) and ClusterRuntime (owns Raft create/join/admit; 13 green tests) are disjoint structs — ClusterRuntime has no federation provider. Wiring “create cluster” requires UNIFYING them (recommended: give ClusterRuntime the FederationProvider so the daemon embeds one dormant-Raft runtime; then facade create_cluster + HTTP + wizard). A half-built wizard was deliberately not shipped. Full breakdown above under “Phase 2 — DESIGN FORK”.

Phase 3 ✅ — drag a VM onto another host in the Datacenter tree → honest “move not available yet (live migration is planned work)” dialog; no faked relocate, no invented backend contract. Same-host drop is a no-op. vitest green.

Live state: .65 + .75 on 0.15.0-Fix72+build.948, federated (peer_count=1), Datacenter tree live on both — ready for a GUI walk-through and the supervised Phase 2 runtime-unification decision.

Exact next operator steps (in order):

  1. Decide on the Phase 2 runtime unification (the recommended option) → a supervised session wires create-cluster + the wizard, then forms a real Raft cluster on .65/.75 with a rollback plan (forget → standalone is proven).
  2. Build the live-migration engine so Phase 3’s drag-and-drop routes to a real cross-host move (today it honestly says “not yet”).
  3. Backlog unchanged: folders, mutual host-remove, remote console, clusterd extraction (C1–C4, parked).

Runtime unification — IN PROGRESS on branch feature/cluster-unification

Section titled “Runtime unification — IN PROGRESS on branch feature/cluster-unification”

Worktree: /home/synvirt/synvirt-cluster-unification (off 4ac5af6). The console session’s 3 cargo-fmt files stay in the main checkout, untouched — this branch is clean. .65/.75 were NOT redeployed; they remain federated-only.

Sub-slice 1 ✅ committed (8b5b6e4): serve_mesh_unified — one mesh serve path whose service set scales with mode (pairing always; membership when federated OR clustered; federation when a provider is given; Raft + write-proxy when a raft handle is given). serve_mesh/serve_federation_mesh are now thin wrappers, so the existing 141 tests prove parity by construction. This is the listener-level foundation: one listener can serve federation AND cluster.

Remaining sub-slices (precise, mechanical — execute in order, gate each):

Sub-slice 2 — ClusterRuntime absorbs the federation plane (the crux):

  • Add provider: Option<Arc<dyn FederationProvider>> to ClusterRuntime (struct + load() arg).
  • Make the mesh ALWAYS serve: load() always binds the listener and serves via serve_mesh_unified(.., pairing, provider, raft_layer), where raft_layer is Some((raft, node_id)) only when is_initialized(). Split Active’s Raft bits (raft, proxy, driver_task, _events) into an Option<RaftLayer> inside Active; registry/mesh_addr/mesh_task/admit_task/sync_task stay mandatory (the federation plane is always up). Update every with_active / raft accessor for the optional layer.
  • Port verbatim from FederationRuntime (they touch only peers/registry/mesh_cfg, all present on ClusterRuntime): add_host, hosts, forget, mode, remote_inventory, remote_vm_power, converge_group, persist_pin, the group-symmetry sync_task.
  • mode(): standalone (no peers, no raft) → federated (peers, no raft) → cluster (raft active).
  • Admit routing in the on-paired task: raft layer present AND leader → Raft admit (existing admit); else → federation admit (persist_pin + converge).
  • Port FederationRuntime’s 8 tests onto ClusterRuntime as the federation PARITY suite; then retire FederationRuntime (or keep a deprecated type alias).

Sub-slice 3 — initialize on the running mesh + promote-without-re-pairing:

  • KEY design note: tonic’s router is static — you cannot add the Raft service to an already-serving server. So initialize(name) REBUILDS the Active with the raft layer: abort the current mesh_task, build Raft, re-serve via serve_mesh_unified(.., provider, Some(raft)) on a freshly-bound listener (brief, local re-bind; federated peers re-pin from mesh_peers, no re-pairing). Then do_initialize as today.
  • promote_federated_host(fingerprint): leader-only; the peer is already pinned in mesh_peers + the registry, so this is admit MINUS the pairing handshake — add_learner(peer_id) → client_write(AddMember) → change_membership.

Sub-slice 4 — daemon embed swap + facade + HTTP + status + OpenAPI:

  • main.rs: embed ClusterRuntime::load(.., mesh_bind, approver, Some(provider)) instead of FederationRuntime (the mesh_bind mgmt-IP fix from 17d689c carries).
  • cluster_control.rs: EmbeddedClusterControl wraps ClusterRuntime; keep all federation facade methods (now backed by ClusterRuntime); add create_cluster(name) → initialize, promote_host(fp) → promote_federated_host, cluster_status() → ClusterRuntime::status().
  • HTTP behind the facade: POST /api/v1/cluster/initialize {name}; POST promote-federated-host; extend cluster/status with the live Raft view (leader/term/membership_revision/quorum/read_only/2-node warning). VIP stays on the no-op reconciler. OpenAPI regen (additive).

Gate (run in the worktree before any merge): daemon 388 (vbs_overlay the one documented exception); federation PARITY tests green (dormant ClusterRuntime ≡ deployed FederationRuntime for add-host / /hosts / federated /inventory / cross-host power); standalone unchanged; Fix70 (restart with cluster state = ZERO VM lifecycle ops via the GuestPower seam); two-in-process-daemon harness (federate → initialize → promote peer without re-pairing → membership {1,2} + 2-node warning → write-on-any-node via proxy → restart resumes); cluster 141 + new green; fmt/clippy clean; single rustls; no unwrap()/expect() in lib.

Attended next steps (operator, after the gate is green): review + merge feature/cluster-unification into feature/cluster-phase2 → redeploy the unified build to .65/.75 (hot stage, Win2016 PID check) → form the .65↔.75 cluster from the GUI (after the wizard lands) with the proven forget → standalone rollback. Console session: move to its own worktree (git worktree add ../synvirt-console <branch>) so the main checkout stops being contended (root cause of the cargo-fmt collisions).

Runtime unification — COMPLETE on feature/cluster-unification (gate green)

Section titled “Runtime unification — COMPLETE on feature/cluster-unification (gate green)”

All four sub-slices landed; the daemon embeds ONE runtime (ClusterRuntime, both planes, Raft dormant by default) behind the unchanged ClusterControl facade.

Sub-slice Commit
1 — serve_mesh_unified (listener foundation) 8b5b6e4
2+3 — ClusterRuntime owns both planes (optional Raft layer) + initialize/promote 2ce502b
4 — daemon embed swap + HTTP + status + OpenAPI eb7d913

What changed. Active gained an Option<RaftLayer>; load() always serves the federation mesh (gated by mesh_enabled) and brings up Raft only when clustered. initialize()/join() shut the federation mesh and re-serve WITH the Raft layer (tonic’s router is static — reconstruct, not mutate). The federation surface (add_host/hosts/forget/remote_inventory/remote_vm_power/mode) is ported verbatim; on_paired routes by mode (Raft admit when clustered, federation pin+converge when federated). promote_federated_host(fp) admits an already-pinned peer with NO re-pairing; enter_cluster_mode() re-serves a federated peer with Raft so it can be admitted as a learner. The daemon facade gained create_cluster/promote_host/cluster_status; HTTP added POST /api/v1/cluster/initialize + POST /api/v1/hosts/{fp}/promote, and cluster/status now carries the live Raft view (3-way mode + leader/term/ membership_revision/read_only/voters; warnings when clustered).

Gate result (run in the worktree):

  • cluster 143/143 green, repeat-stable; fmt + clippy -D warnings clean on the touched crates; single rustls (no new deps); no unwrap/expect in lib.
  • Federation parity proven: federation_parity_add_inventory_power_forget (add-host / inventory / cross-host power / forget with Raft = None) + the facade E2E facade_add_approve_aggregate_and_remote_power (now through the swapped runtime).
  • Standalone unchanged semantics: standalone_serves_mesh_but_no_raft (mode standalone, no Raft) — the always-on mesh is the intended unification, not a regression.
  • Promote-without-repairing harness: promote_federated_peer_without_repairing (federate → initialize → enter_cluster_mode + promote → membership {1,2} + 2-node warning; B’s cluster id arrives by replication, not pairing). Write-on-any-node-via-proxy + restart-resumes covered by the existing follower_submit_matches_leader_direct + restart_both_in_2node_cluster.
  • Fix70 holds by construction: the unification touches only the mesh lifecycle; the daemon’s guest-drain seam is unchanged.
  • Daemon suite: 395 pass; the two failures are the documented vbs_overlay exception + an environment statvfs check (see disk note below) — neither is a code regression from this branch.

⚠️ Environment note (flag for the operator): the dev box root FS is 100% full (0 bytes avail) — df -P / = 100%. This fails the unrelated statvfs_bytes_returns_nonzero_for_root test and blocked npm install in the worktree, so schemas.ts was NOT regenerated (the openapi.snapshot.json contract IS updated, additively). Free space before the wizard session.

Attended next steps (operator):

  1. Review + fast-forward merge feature/cluster-unification (5 commits: 8b5b6e4, 2ce502b, eb7d913 + the two docs commits) into feature/cluster-phase2 (clean FF — nothing committed there since 4ac5af6).
  2. Free disk space; regen schemas.ts from the updated snapshot.
  3. Redeploy the unified build to .65/.75 (hot stage; record Win2016 PID before/after — Fix70 must hold).
  4. Form the .65↔.75 cluster from the GUI once the create-cluster + drag-to- promote wizard lands (next session), with the proven forget → standalone rollback.
  5. Console session → its own worktree (git worktree add ../synvirt-console <branch>) so the main checkout stops being contended (root cause of the recurring cargo-fmt collisions).

Unification MERGED to feature/cluster-phase2 (2026-06-08)

Section titled “Unification MERGED to feature/cluster-phase2 (2026-06-08)”

feature/cluster-unification (5 commits) merged via 3f8c636 (no-ff, conflict-free — no overlap with the console session’s 8937f01 or the 3 uncommitted cargo-fmt files). Post-merge gate re-run in the main checkout:

  • cluster 143/143 green; fmt + clippy -D warnings clean on synvirt-cluster.
  • daemon 396 pass / 1 fail = the documented vbs_overlay exception only; statvfs_bytes_returns_nonzero_for_root now PASSES (disk freed).
  • schemas.ts regenerated from the merged OpenAPI (+cluster/initialize, +hosts/{fingerprint}/promote); vue-tsc clean.

Disk: reclaimed the worktree target (12.4 G) + stale incremental → root FS 0 → 15 GB free (84% used). Standing watch item: target/ regrows; rerun cargo clean between heavy sessions.

Attended next steps (operator):

  1. Redeploy the unified build to .65/.75 (hot stage; record Win2016 domain Id + QEMU PID before/after — Fix70 must hold; the unified runtime serves the same federation plane, so the live federation on .65/.75 is unaffected by the swap, but verify the PID continuity anyway).
  2. Form the .65↔.75 cluster from the GUI once the create-cluster + drag-to-promote wizard lands (next session): POST /api/v1/cluster/initialize on one node, then POST /api/v1/hosts/{fp}/promote for the federated peer (no re-pairing). Rollback: forget → standalone (proven).
  3. Console session → its own worktree (git worktree add ../synvirt-console <branch>) so the main checkout stops being contended — the 3 cargo-fmt files + the network WIP riding the shared tree is the recurring hazard.

Self-pin bug — diagnosed live + fixed (2026-06-08)

Section titled “Self-pin bug — diagnosed live + fixed (2026-06-08)”

Symptom: operator added a host in the GUI + approved, “nothing visible happened.” Live read-only diagnosis on .65/.75 (F5 build Fix72+build.948; the unified build was NOT deployed):

  • .65 clean (peer_count=1, correct 2-host Datacenter tree).
  • .75 had a SELF-PIN: mesh_peers listed its own fingerprint (739a…@10.10.26.75:7443), peer_count=2, the tree showed .75 twice (local + a phantom remote peer that was itself).

Cause (two layers):

  • CASE A (why “nothing happened”): the operator re-added .65 from .75’s GUI, but .65 was ALREADY a federated peer (overnight) — a no-op for the federation, with no UI feedback. Backlog UI fix: adding an already-managed host should say “this host is already managed”, not sit silent.
  • CASE B (the anomaly): add_host persists every bootstrap_member of the peer WITHOUT filtering self. Once the peer already knows us (group symmetry), its bootstrap set includes US, so we pin our own id. The bug existed in BOTH the deployed F5 FederationRuntime AND the merged unified ClusterRuntime (ported verbatim); the parity tests never re-added an already-federated peer.

Live recovery (done, out-of-band, single mutation): DELETE /api/v1/hosts/<.75-self-fp> on .75 only. Result verified: .75 peer_count=1, two-host tree, no ghost; .65 unchanged; Win2016 PID 1469944 unchanged, still running (no daemon restart). The fix sticks because sync_group_peers filters self, so converge never re-creates it.

Code fix (7361b30): skip self in add_host’s bootstrap-persist loop in both runtimes (if m.node_id == self.node_id { continue; }) — one rule, consistent with sync_group_peers. Regression test re_adding_a_federated_peer_does_not_self_pin in both suites (asserts a node never persists its own fingerprint + peer_count stays 1); proven to fail without the fix and pass with it. cluster 145/145; fmt + clippy clean; daemon builds. Ships with the next attended unified-build deploy (with the Win2016 PID-continuity check) — NOT redeployed this turn.

Replay-on-restart investigation (2026-06-08) — framed bug NOT reproducible

Section titled “Replay-on-restart investigation (2026-06-08) — framed bug NOT reproducible”

The leave-cluster session flagged a suspected “openraft replay-on-restart” bug: live .65 /cluster/status reported applied_index:0 while the SQLite store persisted last_applied:5. A focused investigation disproves a reload-replay defect:

  • Reproduce, faithful daemon path (ClusterRuntime::load, SAME identity+port across reload — the real restart), single-node: applied_index resumes (reload_resumes_applied_index_single_node). PASSES.
  • Reproduce, the exact .65 shape — a 2-voter cluster reloaded with the OTHER voter ABSENT (no quorum): applied_index resumes from persisted state even WITHOUT quorum (reload_resumes_applied_index_without_quorum). PASSES. A leaderless node does not forget what it applied.

So the rebuilt engine reads last_applied back correctly; the existing restart_resumes_from_persisted_state test missed nothing about replay — it just (a) reloaded with a FRESH identity (node_id mismatch) and (b) never asserted last_applied_index. The two new tests close that gap and assert the invariant holds.

Real cause of the live applied:0: the .65 raft_meta showed committed=5, last_applied=5, last_membership=log_id 4 (configs [[.65]], ONE voter), yet the raft_log carried an UNCOMMITTED entry 6 = a 2-voter membership change adding .75. .75 ran the old build (no Raft RPC), so that membership change could never replicate/commit — it sat uncommitted, the leader spun forever retrying AppendEntries to an unreachable voter, and the engine was wedged (not a reload-replay defect). That state is gone — leave-cluster returned .65 to clean standalone.

This is a PROMOTE/cluster-FORMATION safety gap, not a replay gap: a node was added to the Raft voter set when it could never catch up. Hardening target for the attended formation session: promote must add-learner → confirm catch-up → change_membership, and refuse/guard promoting an unreachable or RPC-incompatible node (so an uncommittable membership change is never created).

Revised next steps: merge (leave-cluster + these invariant tests) → feature/cluster-phase2 → GUI Leave button → attended .65↔.75 formation, with the promote-path hardening above (and an openraft-maturity read) as that session’s pre-flight.

Promote hardening (2026-06-08) — never add an uncommittable voter

Section titled “Promote hardening (2026-06-08) — never add an uncommittable voter”

The .65↔.75 formation safety gap above is now closed in admit_peer (runtime.rs). A peer NEVER enters the voter set until it is admitted as a learner AND confirmed caught up, and an unreachable / Raft-incompatible target is refused BEFORE any membership mutation — no raft_log entry is written for a bad target, so the leader can never be wedged spinning on a non-committable voter change. Two layers:

Layer A — Raft-capability pre-check (mesh::federation::probe_promote_eligibility). Before any membership change, the leader probes the target’s Raft RPC by calling Raft::Vote with a deliberately undecodable envelope. It is side-effect-free: the payload fails the wire codec (→ INVALID_ARGUMENT) and never reaches the state machine. The decision is purely “does the peer answer the Raft RPC” — there is no daemon-version number and no version-floor constant (a build number would be brittle to forks/custom builds; capability is the truth):

Probe result Meaning Typed error
transport failure (down / TLS / deadline) unreachable E_CLUSTER_PROMOTE_UNREACHABLE
Unimplemented peer serves federation but NOT Raft (old / federation-only build) — the exact .75 case E_CLUSTER_PROMOTE_INCOMPATIBLE
InvalidArgument / Ok the Raft service is present eligible → Layer B

Probing federation (GetInventory) would be wrong: an old build serves federation but not Raft, so it would slip past Layer A and only be caught slowly by the Layer B timeout — that was the original WIP bug, now fixed.

Layer B — bounded catch-up around the membership change. add_learner is issued non-blocking (blocking=false), then a raft.wait(deadline) polls the learner’s replication-matched index until it reaches the leader’s last log index; only then is the change to VOTER appended. On deadline the learner is removed (change_membership(original_voters, false)) and the promote returns E_CLUSTER_PROMOTE_TIMEOUT. Why non-blocking: a blocking=true add_learner wrapped in tokio::time::timeout cannot be cancelled cleanly — openraft keeps the command in flight, so the rollback change_membership deadlocks behind it (the WIP’s blocking-add_learner Layer B hung indefinitely; verified). Non- blocking frees the core, makes the catch-up wait cancellable, and lets the rollback commit promptly. The deadline is raft.promote_catch_up_secs (config, default 20s) — the only tunable; no hardcode.

Tests (runtime.rs, all green): promote_refuses_unreachable_peer_no_membership_change (UNREACHABLE), promote_refuses_incompatible_peer_no_membership_change (the .65↔.75 shape → INCOMPATIBLE at Layer A, not via the Layer B timeout), promote_times_out_when_learner_cannot_catch_up (Layer B → TIMEOUT, rollback, no wedge), promote_wedges_without_the_guard (proves that with both guards disabled the same promote never returns — the wedge), and the existing promote_federated_peer_without_repairing happy path stays green via the guarded path. Each refusal asserts no membership entry, voters unchanged, leader healthy. The Layer A probe is injectable in tests (set_promote_probe, #[cfg(test)]) so Layer B can be exercised in isolation; production always uses the live mesh probe.

Next step: merge feature/cluster-leave → feature/cluster-phase2 → GUI Leave button → attended .65↔.75 formation from the clean base (deploy the unified + hardened build to .75 with the Win2016 PID-1469944 continuity check, initialize on .65, promote .75 → membership {1,2}).

Merge + GUI Leave button (2026-06-08)

Section titled “Merge + GUI Leave button (2026-06-08)”

feature/cluster-leave (leave-cluster + replay tests + promote hardening) was fast-forward merged into feature/cluster-phase2 (clean — phase2 was a strict ancestor; no conflicts). Post-merge gate green: cluster suite 155/0, fmt + clippy clean on synvirt-cluster, single rustls 0.23.38, daemon 398 pass / 2 fail (only the two documented pre-existing fails: libvirt_renderer vbs_overlay, update_v2 statvfs_bytes).

GUI Leave button (web-ux-v2): the cluster root node in the inventory tree (SidebarInventory.vue, data-testid="cluster-leave", hover-revealed, shown only on a Raft-cluster root — NOT on a federated Datacenter root) → a confirm dialog → clusterStore.leave() → POST /api/v1/cluster/leave → inv.refresh() so the tree morphs back to the host root (standalone). The confirm uses the primary variant (plain Are-you-sure), NOT the PIN-gated danger variant: leaving is destructive to cluster MEMBERSHIP but reversible and guest-safe (it is the rollback affordance), so the PIN gate stays reserved for data-loss actions (delete, force-off). E_CLUSTER_LEAVE_FAILED is mapped in the ERROR_MAP and surfaced as operator copy on failure. New API leaveCluster() + store action leave(). Tests: SidebarInventoryLeave.spec.ts (4) — shows on cluster root only, hidden on the Datacenter root, gated-then-calls-then-re-roots, typed-error path; vue-tsc + vite build clean; the 14 pre-existing multiline-html-element-content-newline warnings in SidebarInventory.vue are unchanged (HEAD==working), no new lint problems.

EXACT NEXT STEP — the attended .65↔.75 formation (operator session):

  1. Deploy the unified+hardened build to .75 — hot stage, NO reboot — with the Win2016 PID-1469944 continuity check (the guest must stay up; the daemon restart no longer bounces guests, Fix70). The deploy worktree’s deploy-to-host.sh lost its +x bit AND has no node_modules for the web stage — either restore both, or deploy from the main checkout post-merge.
  2. Verify federation parity between .65 and .75 (probe both ways reachable).
  3. From .65’s GUI: initialize → promote .75. The promote is now safe — a non-Raft / unreachable peer is refused (INCOMPATIBLE/UNREACHABLE) instead of wedging .65 as it did the first time — so on success membership becomes {1,2} with the 2-node tiebreak warning. The GUI Leave button is the visual rollback if anything looks wrong.

Federation GUI — root cause of “no morph / no cross-host control” (2026-06-08)

Section titled “Federation GUI — root cause of “no morph / no cross-host control” (2026-06-08)”

API success was never the bar; the GUI is the product. The operator reported add-host “succeeds” but the tree never morphs and peer VMs can’t be controlled. Root cause, found by reconnaissance: every federation affordance — the Datacenter/multi-host tree morph, cross-host VM power, and the Leave action — lives ONLY in SidebarInventory.vue, which shipped behind ?sidebar=inventory with AppSidebar.vue defaulting to the legacy Sidebar.vue. On the default GUI the operator saw none of it. (Live recon also found .65/.75 running Fix72 builds OLDER than this branch, and a live self-pin residue on .75: /hosts listed .75’s own fingerprint, mode=federated peer_count=2 — pre-fix build.948 artefact, to be cleaned in the attended session; .75 left untouched.)

Fix (this session):

  • AppSidebar.vue default flipped to inventory (commit ba3a52d); legacy kept as ?sidebar=legacy escape hatch (persisted) for one release; Sidebar.vue not removed yet (removal follows once the full federation E2E is green).
  • Deployed HOT to .65 (web-ux-v2 stage; daemon untouched, .65 guestless); served bundle hash-verified.
  • Playwright E2E proves it in a real browser against live .65 (e2e/federation.spec.ts, commit b8603be, 2/2 green): no-flag → inventory sidebar + standalone host tree; ?sidebar=legacy → legacy sidebar. data-testid="app-sidebar-inventory" added as the DOM discriminator. API success is no longer accepted as proof — Playwright E2E against a real daemon is the bar from here.

Still to prove in the browser (needs a 2nd real node — NOT done this session): add-host join (Settings → Cluster → AddHostCard.vue), the morph firing on real federation, and cross-host VM power. These need two real hypervisor nodes, and cross-host CONTROL needs a real throwaway VM on the PEER. Safe options are constrained: the dev box is not a SynVirt hypervisor (libvirtd inactive, no /rpool/vms), and .75 is off-limits (production Win2016). A 2nd bare daemon can federate for join+morph but cannot host a controllable VM cleanly. Recommended: fold the full add-host → morph → cross-host-control Playwright E2E into the attended .65↔.75 session (both real nodes; throwaway VM created on .65 and power-cycled from .75’s GUI; .75’s Win2016 PID 1469944 protected) — OR stand up a dedicated throwaway SynVirt VM-node on .65 in a focused setup. Next: that 2-node federation E2E, then the cluster (Raft) Create/promote wizard.

Federation GUI — PROVEN end-to-end in the browser (2026-06-09)

Section titled “Federation GUI — PROVEN end-to-end in the browser (2026-06-09)”

The full add-host → morph → cross-host-control flow now passes in Playwright against the real .65↔.75 pair. The deeper “doesn’t join” cause beyond the sidebar default was diagnosed and fixed:

  • Backend federates fine — but slowly/async (hosts.rs add_host returns 202 pending; the pairing runs in the background, even on the credential path). The nodes also ran stale Fix72 builds (.65 build.972, .75 build.948 with a live self-pin). Fixed by deploying the branch (build.978) to BOTH nodes and forget-cleaning .75’s self-pin → both clean standalone on the same build.
  • False-success (AddHostCard.vue + store): the credential path fired toast.success("Host added") on the 202, before the join completed, and the managed list never re-polled — “shows success but didn’t join”. Fixed: store.waitForManagedHost polls until the peer ACTUALLY federates (bounded by CLUSTER_ADD_CONFIRM_TIMEOUT_MS), success only then, typed failure on timeout; approval path stays honestly pending.

Playwright suite (real daemons, browser-observable, repeat-stable): e2e/federation.spec.ts (inventory sidebar is the default + legacy escape hatch), e2e/federation-join.spec.ts (credential add-host → peer federates on BOTH sides → tree morphs to Datacenter with both hosts), e2e/federation-xhost.spec.ts (Start a peer VM from the driver GUI → it powers on the peer). Unit: AddHostCard success-only-after-federate + no-false-success.

Deploy: branch daemon+web HOT to .65 (guestless) and .75 — .75’s Win2016 PID 1469944 verified same + uptime climbing across the daemon restart (Fix70 guest-drain held); web stage is a static swap (no restart). Both nodes left clean standalone; the disposable test VM was created/destroyed only on guestless .65.

Standing rule: API success is NOT proof — a Playwright E2E against a real daemon is the bar. Legacy Sidebar.vue removal is now a safe follow-up (the default-flip parity is browser-proven). Next: the cluster (Raft) Create/promote wizard, then the attended .65↔.75 Raft formation.

Cluster wizard — Phases 1–5 LANDED + browser-proven; Phase 6 blocked by daemon Raft wiring (2026-06-09)

Section titled “Cluster wizard — Phases 1–5 LANDED + browser-proven; Phase 6 blocked by daemon Raft wiring (2026-06-09)”

The cluster (Raft) Create / promote / Leave GUI wizard is built and proven in a real browser against the real .65↔.75 pair (build.989 on both). The bar held: a flow is “done” only when a Playwright spec asserts it in a real browser DOM against a live daemon — never API 200.

Landed:

  • Phase 1 — nested inventory shape. Backend (398c9ae, api/inventory.rs): mode=cluster emits a Datacenter root that NESTS the Raft cluster (clustered:true leaf + members, each aggregated over the federation RPC) ALONGSIDE loose federated-but-not-clustered hosts. ClusterNode.clusters (additive). Crash fix 2d6cb26: ClusterNode is self-referential, so #[schema(no_recursion)] on clusters is REQUIRED — without it utoipa’s OpenAPI generation recurses forever at startup → SIGSEGV before the listener binds (caught on the .65 deploy; .65 guestless, .75 untouched). Frontend types + nested SidebarInventory render (129e840, 8296812): Datacenter → Cluster node (name + quorum r/t · rev N status line + soft-warning badge from clusterStore.status) → members; loose hosts flat. DatacenterHostRow extracted (same dc-host/dc-vm/pw-* ids). Federation E2E stays green (federated mode is flat, clusters empty) — re-verified 4/4 on build.989.
  • Phase 2 — Create cluster (797c596). “Create cluster” action on the Datacenter root → name dialog → clusterStore.createCluster → POST /cluster/initialize → poll status until mode==cluster BEFORE success (no false success) → tree morphs to the Cluster node.
  • Phase 3 — promote (84bab9d). A loose host carries “Add to cluster” (dc-promote-*) and is draggable onto the Cluster node (drop target) → POST /api/v1/hosts/{fp}/promote → confirm membership grew before success. The daemon handler now surfaces the DISTINCT typed code (E_CLUSTER_PROMOTE_UNREACHABLE/_INCOMPATIBLE/_TIMEOUT) instead of flattening to _FAILED; ERROR_MAP renders each as copy.
  • Phase 4 — Leave (landed with the render). The Leave action lives on the Cluster node (cluster-leave) → confirm → POST /cluster/leave → tree morphs back. E_CLUSTER_LEAVE_FAILED copy.
  • Phase 5 — E2E (real .65↔.75): cluster-create.spec.ts (Create → nested node appears + daemon mode==cluster → Leave → morphs back; repeat-stable 2/2), cluster-promote.spec.ts (promote → the daemon’s typed refusal reaches the browser as copy, no false success, no forever-spinner), SidebarInventoryPromote component test (each typed copy + no-false-success + no-forever-spinner, deterministic). All self-cleaning to standalone. Win2016 PID 1469944 never power-opped, continuous across every deploy + run.

✅ Phase 6 RESOLVED — serve openraft from boot, uninitialized (the “cluster of one” model). The dormant-Raft design was replaced (commit 10b5447, feature/cluster-phase2). EVERY node now builds and serves its openraft instance on the mesh listener from boot, in all modes; a standalone/federated node is simply uninitialized (no membership) — it answers RaftClient.vote()/append_entries so the leader’s capability-probe sees it COMPATIBLE (the wire codec rejects the deliberately-undecodable probe → InvalidArgument, no longer Unimplemented) and can add_learner → catch-up → change_membership it, yet it elects no leader and grows no log while it has no peers. “Clustered” is derived from is_initialized() (the persisted cluster_id), NOT from the presence of the Raft handle — being joinable is not being clustered. runtime::activate() always builds the Raft layer; the clustered flag now selects only the sync task (identity-sync vs federation-symmetry). mesh::server is unchanged (it already mounts the Raft service when a handle is present). The promote-hardening guards still hold against a genuinely Raft-less peer (an old FederationRuntime build) — that is now the only _INCOMPATIBLE shape.

Tests: 156/156 cluster (incl. uninitialized_node_serves_raft_rpc_and_stays_quiescent proving the probe sees COMPATIBLE + term stays 0 / no leader / no log with no peers; the INCOMPATIBLE/TIMEOUT/wedge guards rewritten to use a federation-only peer); daemon suite 399 pass / 2 pre-existing fails (vbs_overlay Haswell-no-smap, statvfs_bytes).

Live-verified .65↔.75 (both on the same Phase-A build, sha256-identical): standalone- regression gate green on .65 (journal “Raft RPC served uninitialized; joinable, not clustered” + openraft Learner, no churn, listeners unchanged :80/:443/:7443). Then the full real cycle over the cluster API: federate → initialize (“prod”) → promote .75 returns 204 (was E_CLUSTER_PROMOTE_INCOMPATIBLE) → both nodes agree mode=cluster, leader=.65, term=1, voters=2, rev=7, quorum, 2-node warning; /inventory renders Datacenter → prod{.65, .75} both reachable → leave on both → clean standalone, 0 hosts. Win2016 (PID 1469944) never bounced across the SIGKILL deploy, the formation, or the teardown (data-plane independence / Fix70 confirmed live; .75 was pre-Fix70 so the upgrade used a SIGKILL deploy — old daemon’s SIGTERM guest-drain handler never ran).

✅ Phase D — GUI-PROVEN in a real browser (the standing bar closed). Both nodes re-deployed to the same build (build.996, sha256-identical) — daemon (with the serializer fix below) + the web-ux-v2 wizard bundle (index-DWUoFVeI.js, hash-verified served on both). .75 is post-Fix70 now so a normal systemctl restart is guest-safe (verified: Win2016 PID 1469944 unchanged across the deploy). Playwright vs the live .65↔.75, asserting browser DOM (not API): cluster-create.spec (Create “e2e-lab” from the GUI → Datacenter morphs to a Cluster node → daemon mode==cluster → Leave → morphs back) and cluster-promote.spec (rewritten from the wedge-era refusal to the happy path: promote the federated peer → cluster node grows to “2 nodes” + the 2-node quorum warning surfaces + daemon voters==2) — both GREEN + repeat-stable (4/4 ×2), self-cleaning to standalone. Restart-resilience exercised LIVE: with the cluster formed, restarting the follower .75 daemon → both nodes still agree mode=cluster/voters=2/rev=7 (.75 resumed membership from disk) and Win2016 PID 1469944 unchanged (a clustered follower restart is guest-safe). Follower-write-proxy has no exposed live API trigger → stays harness-proven (follower_submit_matches_leader_direct). Nodes left clean (standalone, 0 hosts).

✅ Serializer fix landed (commit 9993278). api/cluster.rs::cluster_status no longer hardcodes members: &[] / reads the static boot cluster_id: it feeds synvirt_cluster::status::build the LIVE members + cluster_id from the embedded runtime, with reachable derived from the live quorum signal (a pure, unit-tested cluster_status_value() helper, +3 daemon tests). /node/identity overlays live cluster_id/cluster_state from the runtime instead of the boot snapshot. So a clustered node now reports clustered everywhere (no more standalone/0/"" while clustered).

Backlog (unchanged): legacy Sidebar.vue removal; real OVS VIP (unicast, leader-owned); remove-member drag (needs a backend change_membership remove path); folders; synvirt-clusterd extraction; NTP reachability on .75 (durable clock fix — UDP 123 to upstreams is blocked, see [[reference_host_deploy_gotchas]]). Also: the daemon log line “cluster control plane embedded (mesh listening; Raft dormant)” is now inaccurate wording (Raft is served, uninitialized) — reword; the Input component mirrors data-testid onto both its wrapper and the native input (E2E must scope by role).

Deploy/infra notes (2026-06-09): .65 + .75 are installed-to-disk (ext4, persistent — hot deploys survive reboot; no live-ISO reversion). The /etc/synvirt/release stamp is written only by the installer/updater, NOT by deploy-to-host.sh, so it reads stale after a hot deploy (the binary’s true version is /node/identity). .75’s clock was ~5 months ahead (chronyd couldn’t reach NTP upstreams — UDP 123 blocked) and was set correct manually; the mesh verifier ignores cert validity windows (SPKI pinning), so the skew never affected mTLS. Both nodes left on the same Phase-A build, STANDALONE + clean; Win2016 on .75 untouched.