STO-196E (.11 "Gold Virt01") degraded-pool replace plan — AWAITING GO

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/operations/incident-2026-07-10-sto196e-replace.md. Edit at the source, not here.

STO-196E (.11 “Gold Virt01”) degraded-pool replace plan — AWAITING GO

Written 2026-07-10 (post-deploy work order, Arm 1). READ-ONLY diagnostic done; NO zpool command runs without Tony’s GO on this document.

Diagnosis (evidence, all read-only)

Pool: STO-196E, raidz1-0, 4 × 447.1 GiB SATA SSD (Micron MTFDDAK480T*), DEGRADED — member guid 12850117987818464447 UNAVAIL (“was /dev/sde1”, label missing/invalid). 0 read/write/cksum errors on the three live members; last scrub 2026-06-14 clean.
The “missing” disk is physically present and healthy: it is sdd (by-id ata-MTFDDAK480TGA-1BC1ZABDA_240646FFDABA, wwn 0x500a075146ffdaba). Its ZFS label reads pool STO-196E, pool_guid 2240768861121222181, vdev guid 12850117987818464447 — the exact UNAVAIL member, frozen at txg 554070 vs pool txg 703135 (~149 k txgs behind). Unmounted, not in use by anything.
Device names rotated across a boot: the boot SATADOM (ata-SATADOM-SV_3SE_…, 29.8 G) now owns the sde name the member had at import time. Drop window: after the clean scrub of 2026-06-14, at or before the 2026-07-08 19:12 boot. The label being this stale means the pool ran without it long enough that a full resilver, not a DTL catch-up, is the only honest re-add.
SMART: all four data disks + the SATADOM PASSED; 0 reallocated, 0 pending/uncorrectable across the board; 1 005–1 266 power-on hours. sdd shows nothing that explains an electrical failure — consistent with a transient enumeration/controller hiccup on this nested host, not a dying disk.
Replacement viability: sdd is byte-identical in size to the live members (480 103 981 056 B) — it IS the replacement. Nothing to buy. (Fallback spec if it ever re-fails mid-resilver: any enterprise SATA SSD ≥ 480 103 981 056 bytes — Micron 5100/5300 PRO 480 GB class.)

Blast radius while degraded (why this is top priority)

raidz1 with one member out = zero remaining redundancy. A second member failure loses the pool, which today holds:

Guests (all of .11’s disk-bearing VMs): Windows Server 2016 (RUNNING, the production guest), synnet.siem, eqweqweqw, PremierTest, test-windows11, test-windowsserver2022, windows-2022 (empty zvol).
Repo: “L5 Local Gold” (local_zfs, ~14.8 GB of WS2016 restore points) — ON this pool, so it dies with it. Off-pool copies that survive: “L5 NFS to .12” (WS2016) and “Peer .12 — vDisk” (synnet.siem), both physically on .12’s storage pool.
Pool alloc is ~97 GB of 1.3 TB usable — small, which keeps the resilver short.

Resilver estimate

raidz1 resilver scans allocated data only (~97 GB) and reconstructs the missing column (~24 GB written to sdd). Observed sequential read on these SSDs ≈ 135 MB/s per stream under contention (measured today hashing two 200 G zvols in parallel), 250 MB/s+ uncontended. Estimate: 7–20 min; budget 30 min. Honest statement: for that whole window the pool remains at zero redundancy — a second member failure during resilver kills it. That risk exists RIGHT NOW every minute; the resilver window does not add exposure, it ends it.

The plan (execute only after GO, in this order)

Step 0 — fresh OFF-HOST restore points (the product protects its own fleet before surgery), via the public API:

# WS2016 → NFS on .12 (off-host)
curl -sk -u root:1234 -X POST https://10.10.26.11/api/v1/backup/jobs/f47e1050-daa9-4488-ab80-028469bf7211/run
# synnet.siem → Peer on .12 (off-host)
curl -sk -u root:1234 -X POST https://10.10.26.11/api/v1/backup/jobs/f911f6f9-c3bf-4b66-9e54-05ad6705c9f2/run
# WS2016 → Local Gold (on-pool; still run it so the local chain is current)
curl -sk -u root:1234 -X POST https://10.10.26.11/api/v1/backup/jobs/5ce6fbac-ab46-4189-b40b-a43ca8e0aeca/run

Gate: all three runs completed + restore points registered (GET /api/v1/backup/restore-points). The two off-host points are the survival guarantee.

Step 1 — pre-checks (read-only):

zpool status STO-196E          # 3 members ONLINE, 0/0/0 errors, no resilver in progress
smartctl -H /dev/sda /dev/sdb /dev/sdc /dev/sdd   # all PASSED
lsof /dev/sdd1                 # empty (disk free)

Abort here if any live member shows new errors — that changes the game (the plan assumes three clean sources).

Step 2 — the replace (the ONLY mutating command):

zpool replace STO-196E 12850117987818464447 /dev/disk/by-id/ata-MTFDDAK480TGA-1BC1ZABDA_240646FFDABA-part1

Notes: by-id path (never sdX — the names already rotated once). ZFS will see its own stale label on the target; if it refuses with “is part of pool STO-196E”, the documented same-disk re-add is -f on the same command — expected and safe BECAUSE the guid matches the hole it fills. No partitioning needed (sdd1 already exists, correct size, BF00).

Step 3 — watch (read-only):

watch -n 30 zpool status STO-196E   # resilver % + scan speed

Guests stay up; no service restarts; I/O impact acceptable (SSDs, ~97 GB).

Step 4 — post-checks:

zpool status -v STO-196E   # 4/4 ONLINE, "resilvered … with 0 errors", DEGRADED gone
zpool scrub STO-196E       # full verify pass now that redundancy is back

Then: dashboard alarm clears (observability disk detector), note the event, update memory.

Abort criteria (during resilver)

A live member (sda/sdb/sdc) FAULTS or accumulates read/cksum errors mid-resilver: do NOT issue further zpool commands. If the pool suspends, STOP — the off-host restore points from Step 0 are the recovery path (restore WS2016 from NFS, synnet.siem from peer, onto .12 or a rebuilt pool). This is the “second failure” scenario; no command un-does it.
The NEW disk (sdd) errors during resilver: zpool detach STO-196E <sdd-part1> returns the pool to today’s degraded-but-stable state. Then buy the fallback disk (spec above) and re-run from Step 1.
Resilver stalls at 0% >10 min with no I/O: collect zpool status, dmesg | tail, journal — report, change nothing.

Signature

GO to execute Steps 0–4: GO given 2026-07-10 (Tony, work-order block “ARM 1 — SURGERY GO”)

Execution record — 2026-07-11 UTC (2026-07-10 20:06 local)

Executed as written, Steps 0–4. All times UTC.

01:51:12 Baseline: pool DEGRADED (guid 12850117987818464447 UNAVAIL), 3 live members 0/0/0; qemu PID 3034 (WS2016, started Jul 8 19:14:44) — sole guest process.
Step 0 (gate) 01:51–02:03: all three jobs run via the public API, sequentially, each completed with a registered restore point:
- NFS → .12: run 1f3c7fa0, restore point 8576a3d5 (quiesced)
- Peer → .12: run 0a577915, restore point 432e0a92
- Local Gold: run e150eaff, restore point 39156bee (quiesced)
Off-host verification on .12: both manifests present (/storage/l5-nfs-export/... and /storage/Backups/<fp>/synvirt/v1/...); sha256 spot-check of one stream per point — 01-disk-0.zfs and 01-synnet-siem.zfs both match their manifest digests exactly. Gate PASSED.
Step 1 02:05: pre-checks green — 3 members ONLINE 0/0/0, SMART PASSED ×4 (0 realloc / 0 uncorrectable), by-id → sdd1, lsof empty, zdb -l confirms vdev guid 12850117987818464447 @ txg 554070.
Step 2 02:05–02:06 — one documented deviation: plain replace AND replace -f were both refused: this OpenZFS build treats “is part of active pool” as must be manually repaired (-f only overrides exported/foreign-pool labels). The manual repair ZFS demands is zpool labelclear. Identity re-verified a third time (by-id → sdd1 + guid match), then zpool labelclear -f <by-id>-part1 + the runbook’s exact replace command → accepted at 02:06:24. The labelclear is a strict subset of what the authorized replace writes to that same disk; no other device touched.
Step 3: resilver ran 98 s (02:06:24→02:08:02), 112G scanned, 104G issued at ~1.16–1.19 GB/s, 26.4G resilvered, 0 errors. Mid-resilver SMART sweep clean ×4; qemu PID 3034 untouched; live members 0/0/0 throughout. No abort criteria fired.
Step 4: pool ONLINE 4/4 (new member listed by by-id path, as intended). Scrub 02:08:41→02:10:27 (106 s): repaired 0B with 0 errors. Observability: 0 active issues (degraded-pool alarm cleared).
02:11: one more manual Local Gold run (2d7618f3) → restore point a31ad163 — first post-surgery point on the healed pool.
Guest safety: qemu PID 3034 identical before / during / after (same lstart Jul 8 19:14:44); zero VM operations issued.