INCIDENT — .56 isolated (02:4x), recovery = one reboot

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/night/20260611/INCIDENT-56.md. Edit at the source, not here.

INCIDENT — .56 isolated (02:4x), recovery = one reboot

What happened

The Phase-3 E2E re-applied ipStorage on .56’s vSwitch0 through the UI. The brand-new ifupdown neutralization in dhcp_deny then ran systemctl stop ifup@eno1/eno2 — whose ifdown takes the PHYSICAL link admin-down. vSwitch0’s only two uplinks went down; the management Service Port (ipMGMT 10.10.26.56, riding the bridge) lost all egress. SSH/API/ping dead since.

Impact

• .56 management unreachable (console box has no path; no documented BMC for synvirt-02).
• Guest Win2016-Clone-56 (QEMU 132218) keeps RUNNING on the host — compute untouched, only the host’s network egress is down. The guest was night-collateral-class (a test clone), not the protected .75 VM.
• The 3-node cluster rounds were re-planned around .65/.75 (.56 was to be the seed).

Root cause

The destructive teardown ran AFTER the management-readiness gate had passed (the gate ran at mutation time; the damage came from a unit stop whose side effect — link down — the code did not account for). Classic action-at-a-distance: “stop the DHCP unit” ≠ “stop DHCP”; ifupdown units own the link itself.

Fix (committed + deployed to .65/.75 the same hour)

fix(synvirt-network): SEV — never take an enslaved uplink's LINK down: the teardown now kills only the per-NIC dhcpcd process, strips L3, and explicitly pins the link UP, in both teardown branches. The boot-time persistence stays purely declarative (stanza rewrite to manual).

Recovery procedure for .56 (morning)

1. Power-cycle / reboot the host (front panel or BMC if one exists).
2. On boot, synvirt-network.service reapplies the OVS plan, which issues SetLinkUp on every enslaved uplink → management returns on ipMGMT 10.10.26.56.
3. Verify: ip -4 -br addr show eno1 eno2 (both should be ADDRESS-FREE — the night’s intended end-state), ip -br addr show ipStorage (10.99.26.2/24), dashboard reachable, guest still running.
4. The deployed build on .56 is 1094 (f32b6c64…) — pre-link-fix. Hot deploy the final night build to it once reachable (SKIP_BUILD=1 SSHPASS=1234 HOST=10.10.26.56 scripts/deploy-to-host.sh hot from the night worktree).

Hard lesson (engraved)

Network teardown steps must be expressed in terms of the EXACT state they may change (address, route, process) — never via service units whose stop semantics bundle link state. The night’s no-trespass rule for uplink links is now in code, not convention.