SynVirt Night Shift — 2026-06-11 — Morning Report

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/night/20260611/NIGHT-REPORT.md. Edit at the source, not here.

SynVirt Night Shift — 2026-06-11 — Morning Report

Exec summary (≤10 lines)

1. Both of Tony’s direct asks SHIPPED + proven live: duplicate-hostname guard on Cloud add-host (409 + UI), and N clusters (wizard founds on ANY federated host via the entry proxy; daemon groups peers by membership into N leaves).
2. ⭐ Daemon optimized, profile-driven, 3 perf: commits: auth-cache killed the PAM-per-request serialization — cluster_status p50 494→0.6 ms (rps 16→8991), storage 496→27 ms, vms 506→87 ms, inventory 525→256 ms. Zero API/behavior change; tests green.
3. UI polling consolidated: idle Summary traffic ~26→~13 req/min, hidden tab near-zero.
4. Network: a real 4th DHCP source (Debian ifupdown per-NIC dhcpcd) found and killed; enslaved uplinks now end addressless + DHCP-free, persistent. ipStorage configured through the UI; 9.35 Gbit/s on the test subnet.
5. Shared storage SHIPPED + proven on .65: software iSCSI initiator (discover→login→LUN) + NFS datastore wizard (create→mounted→delete), both UI-driven E2E green.
6. Cluster: Guardian (HA auto-restart), Smart Balance config, Cluster Health (Raft + fleet observability) SHIPPED; replicated config proven REPLICATING leader→follower live; 2-node cluster formed + Cluster Health green across all 3 entry points.
7. VM OS profile catalog 39→52 (rhel-8/10, ubuntu-20.04/26.04, debian-11, freebsd-14, opnsense, fortigate-vm, …), wizard-visible.
8. 🔴 ONE SEV: a buggy intermediate of the DHCP fix took .56’s uplink LINK down → host isolated; fixed in-code; .56 needs one reboot (docs/night/…/INCIDENT-56.md). Its guest kept running throughout.
9. synvirt-migrationd → synvirt-livemotiond rename done (Alias keeps the old unit name resolving).
10. DO NOT MERGE — morning review decides. Branch feature/nightshift-20260611, worktree clean.

⭐ Perf table — see PERF.md (3 columns: ORIGINAL → POST-OPT → END-OF-NIGHT)

Headline (same .56 workload, the honest win): cluster_status p50 494→0.6 ms; storage 496→27 ms; vms 506→87 ms; inventory 525→256 ms; RSS 82→79 MiB. END-OF-NIGHT is the .65 cluster leader (cross-host + cluster loaded) — its inventory p50 ≈2.9 s is the cross-mesh aggregation cost, OPTIMIZE-BACKLOG #1, not a workload regression.

NAMING TABLE (Tony’s veto) — see NAMING.md

Live Motion (synvirt-livemotiond), Guardian, Smart Balance, Cluster Health, Placement Groups/Rules (not shipped — cut from the bottom), Aggressiveness 1–5, Manual/Assisted/Automatic, Maintenance Mode (kept), Service Ports (API path renamed off vmkernels).

perf: commits (per-commit gains)

• perf(synvirt-daemon): verified-credentials cache in front of PAM — cluster_status p50 494→0.6 ms, rps 16→8991 (measured, .56).
• perf(synvirt-daemon): parallel inventory aggregation — inventory p50 525→256 ms (local host + peers concurrent).
• perf(synvirt-libvirt): bounded-concurrent per-VM reads in vm::list — vms p50 506→87 ms; keeps flat as VM count grows (ceil(N/6) batches).

Guardian / Smart Balance / Cluster Health — E2E matrix

#	Item	Result
1	3-node cluster healthy	PARTIAL — .56 isolated by the SEV; a real 2-node cluster (.65 leader + .75) formed via the wizard’s REMOTE-SEED path + verified in Cluster Health (leader/follower, both reachable, quorum at-risk[correct], lag 0, services replicated) across .65/.75/VIP.
2	Enable Guardian+Smart Balance via UI	DONE — replicated config written on the leader, READ BACK identical on the follower (Raft replication proven); Guardian Services panel screenshotted.
3	Hard-failure restart	NOT RUN — blocked: only .65 can mount the shared NFS datastore (.56 down, .75 has no nfs-common/no storage leg), so a cross-host restart has no second NFS-capable target. Guardian LOGIC is unit-tested (priority order, shared-domicile eligibility, OFD double-start-guard branch). Morning: stand up a 2nd NFS-capable host.
4	False positive (lock honored)	NOT RUN — same single-NFS-host block; the possibly-alive branch is unit-covered.
5	Leader-change visibility	DONE — Cluster Health reports the live leader/term; survived the promote.
6	Smart Balance imbalance	NOT RUN — engine is config-only this build (Live Motion executor still stubbed); knobs replicate + validate.
7	Anti-affinity rule	NOT SHIPPED — Placement Rules cut from the bottom.
8	Maintenance Mode	NOT SHIPPED tonight (blocked on Live Motion evacuation).
9	PID 1469944/2396345 on .75	HELD — protected guest QEMU 2396345 identical before/after every deploy + the cluster ops + the bench restart (the brief’s 1469944 was stale; corrected in RECON.md).
10	Full suites green	cargo: daemon 423 + cluster 156 + network 274 green. Playwright: see the run log; new specs (hostname-guard, ipstorage, shared-storage, cluster-form) green live.

Storage stack status

Software iSCSI initiator (open-iscsi managed by the daemon, iface bound to ipStorage, CHAP, sessions/devices) + NFS datastores (v4.1+, reconcile on boot, OFD-safe) — both UI-driven and green on .65. Lab targets on the console (LIO night-target + night-chap, NFS /srv/night-test-nfs). .75 cannot mount NFS (no nfs-common; apt has no candidate under the host’s +5-month clock skew) — host prereq for the morning.

Network audit + DHCP fix proof + test addressing

Enslaved uplinks must be L2-only. Found FOUR DHCP sources; the live offender on the fleet was Debian ifupdown (ifup@<nic>.service per-NIC dhcpcd masters), which no dhcpcd.conf/unit-reload/SIGTERM stops. Fix: dhcpcd denyinterfaces (restart, not reload) + NM unmanaged + ifupdown stanza→manual rewrite + per-process kill + L3 flush, all behind the management-readiness gate, NEVER touching link state (the SEV lesson). Test addressing (user input via UI): ipStorage subnet 10.99.26.0/24, console .1, .65 .3 (Service Port), .56 .2; zero route overlap. ping+iperf3 console↔.65 = 9.35 Gbit/s.

Profile catalog status — 39 → 52

Added rhel-8/rhel-10, rocky-8, alma-8, oracle-8/9, ubuntu-20.04/26.04, debian-11, opensuse-tumbleweed, freebsd-14, opnsense, fortigate-vm. firewall appliances ship single-queue virtio + offloads-off (NET_OFFLOADS_DISABLED). All resolve through the wizard’s cascade (data -driven, zero UI change); unit-tested.

Bugs fixed (with test names)

• NUL bytes in network.ts/NicCard.vue → \0 escapes (tooling unbroken).
• webrtc encode_decode missing codec field (workspace test floor).
• pgrep -x qemu-system-x86_64 15-char truncation (guest checks were vacuous) → pgrep -f '[q]emu-system'.
• iSCSI adapter “enabled” now means SynVirt OWNS the IQN (a stock distro initiator IQN no longer reads as managed; was an auth-failure trap).
• (cargo) dhcp_deny unit tests: marker-block idempotency, ifupdown stanza rewrite, NM/dhcpcd round-trips.

API changes — see API-CHANGES.md

2 breaking (live-motion path, service-ports path; zero external consumers, UI moved same-commit), the rest additive; OpenAPI snapshot + TS types regenerated.

OPTIMIZE-BACKLOG top items — see OPTIMIZE-BACKLOG.md

#1 event-invalidated peer-inventory cache (the .65 2.9 s inventory), #2 PAM cold-burst pre-warm, #3 persistent virsh session, #4 cargo machete/bloat/tree -d, #5 release-profile experiment.

Remaining competitive gaps (short, prioritized — trademarks allowed here)

• Live Motion executor still stubbed → no live VM migration, so Smart Balance Automatic + Maintenance-Mode evacuation can’t run (the gating dependency for half the cluster matrix). HIGHEST priority.
• Placement Groups/Rules: scheduler honors them but no UI/CRUD shipped.
• DRS-class live rebalancing depends on Live Motion above.

BLOCKED items — see BLOCKED.md

Guardian failover drill (single NFS-capable host live), Maintenance Mode + Placement Rules (Live Motion), full 3-node matrix (.56 down).

Decisions Tony must make

1. Reboot .56 to recover it (INCIDENT-56.md) — or inspect first.
2. Veto/approve the naming table (NAMING.md).
3. The vmkernel internal-identifier sweep in synvirt-network (414 pre-existing lint hits) — approve the mechanical rename or keep vmkernel as locked internal vocabulary? Product surfaces are already clean.
4. Live Motion executor is the unlock for the rest of the cluster roadmap — prioritize it next?

Final fleet state at hand-off

• .56: ISOLATED (SEV) — reboot to recover (INCIDENT-56.md). Guest Win2016-Clone-56 was running throughout.
• .65 (synvirt-03) + .75 (synvirt-01): build.1095 sha 02bc6b18, federated (the night-cluster was formed + verified + then DISSOLVED for clean hand-off; both report mode=federated, members=0). .75’s protected guest QEMU 2396345 held across every op.
• Console lab fully torn down: LIO clearconfig, NFS export removed, /srv/night-test-* gone, 10.99.26.1 removed from ens192, iperf3 off; .65’s 10.99.26.3 storage leg removed; iSCSI sessions logged out.
• Branch feature/nightshift-20260611, worktree clean, NOT merged.

Live E2E results (per round, against the real fleet)

1. Duplicate-hostname guard — PASS (UI rejection toast + API 409 + rename-then-accept), .65 renamed→collide→back, fleet restored.
2. ipStorage Service Port via UI on .56 — the flow drove the real Set-up modal + Apply; the no-DHCP host assertions ran — and SURFACED the SEV (the link-down bug) which is now fixed. Re-verification pending .56’s reboot.
3. Shared storage on .65 — PASS both: software iSCSI enable→discover→login→LUN-in-devices, and NFS wizard create→mounted→delete.
4. N-cluster formation via the wizard’s REMOTE-SEED — PASS to founding (night-cluster created on .65 from .75’s UI through the proxy); 2nd-node join completed via the leader’s promote; Cluster Health verified green (2 reachable, quorum at-risk, lag 0, services replicated) across .65/.75/VIP. UI gap found: the entry host isn’t offered as a promote candidate in its own wizard (documented).