Observability, Storage & Networking — E2E redesign (Phase 1 audit + locked decisions)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/architecture/observability-storage-networking-redesign.md. Edit at the source, not here.

Observability, Storage & Networking — E2E redesign (Phase 1 audit + locked decisions)

Status: Phase 1 (audit) complete. Phase 2 (domain model) drafted below. Owner sign-off received 2026-06-28 (green light on §2). Phases 3–9 (persistence, collectors, API, UI, alarm engine, testing) are unblocked and execute phase-by-phase per §6, each verifying before hand-off. One operational gate remains: no SEV-0 fleet deploy of the Phase-4 collectors to the production hosts without a separate explicit go (Fix70 guest-safety).

Scope of this iteration (per the brief): per-host observability for local ZFS pools + zvols, storage devices (HDD/SATA/SAS/NVMe), SMART/NVMe health, storage adapters, iSCSI, and all networking (vSwitches, bonds, VLANs, host/VM interfaces). Deferred (next domain, same taxonomy): cluster-wide aggregation, DRBD9/LINSTOR replication health, and Ceph. These are explicitly out of scope and must not be stubbed falsely.

This document is the Phase-1 deliverable: a real map of what exists locally, the gaps versus the redesign concept, and the architecture decisions locked in before any persistence is touched. It prioritizes extending the existing architecture, never duplicating it.

---

1. Audit summary — what already exists (grounded in code)

1.1 Storage (synvirt-storaged + synvirt-storage/synvirt-zfs + daemon proxy + v2 UI)

Implemented: ZFS pool CRUD + vdev topology tree (storage_proxy.rs, PoolTopology.vue); per-disk SMART/NVMe health via smartctl -j (handlers.rs::enrich_disk_from_smartctl — temperature, wear/life %, NVMe vs SCSI vs SATA parsing); iSCSI software initiator end-to-end (open-iscsi: discovery, CHAP, login/logout, session enumeration, persistent node.startup); NFS v4.1+ shared datastores; pool snapshots + sanoid policy; pool events (scrub/resilver/vdev state from zpool events); 5-minute capacity-history sampler already writing the observability DB (storage_bootstrap::collect_capacity_samples).

Partial / missing: disk-replace wizard (entirely missing — no preflight, no typed-confirm, no staged resilver task, no live-guest safety check); persistent serial/WWN identity ledger; storage-adapter detail (driver/firmware/queue-depth/resets are frontend stubs awaiting backend); performance latency is stubbed to 0 in read_pool_iostat; no per-vdev / per-dataset / per-zvol / per-VM I/O breakdown; no queue-depth metric.

1.2 Networking (synvirt-network + daemon api/network.rs + v2 UI)

Implemented: the full NIC → bond → uplink → vSwitch → VLAN/network model in the spec layer (spec/mod.rs); network.yaml as the single source of truth (no netplan); OVS reconcile with the R1–R4 safety rules; four bond modes (SinglePort / ActiveStandby / LACP / LoadBalance); three canonical vmkernels (ipMGMT / ipStorage / ipMigration); surgical IP-layer direct-apply (direct_apply.rs) with single-interface capture/restore rollback; the management-NIC dual-signal guard (schema marker + live host_probe); uplink L2-only enforcement; the 3-column topology UI (VSwitchPanel.vue).

Missing — the entire telemetry surface: no per-interface counters (rx/tx/errors/dropped/carrier), no per-bond state (member status, failovers, imbalance, LACP partner), no per-VLAN counters, no OVSDB stats monitor, no ethtool netlink, no LLDP (grep lldp = 0). Network observability today is diagnostics-only (doctor.rs, reconcile warnings). Guest rx/tx exists only via the VM telemetry side-channel.

1.3 Observability + persistence — the §0.B-critical findings

A single SQLite DB at /var/lib/synvirt/observability.db, WAL on, FK on, opened and exclusively written by the daemon process through one Arc<Mutex<Connection>> (synvirt-observability/src/db/mod.rs; opened in daemon/src/main.rs). Tables today (versioned migrations 0001–0005): issues_catalog, alarms, tasks, events, edit_settings_audit, storage_capacity_samples, vm_disk_samples. A detector registry runs on a fixed cadence: 7 real detectors (CPU / memory / load / swap / NTP drift / DNS / log-rate) + 24 catalog-registered stub detectors awaiting subsystem suppliers. Retention is age-based purge only (events 30 d, tasks 90 d, resolved alarms 180 d, capacity 30 d) — no rollups. The Monitor UI (/monitor/{issues,tasks,events,audit,logs}) is wired.

1.4 API / frontend / streaming / apply-model

utoipa-first OpenAPI → openapi.snapshot.json → generated schemas.ts is the contract pipeline. Live updates are a single SSE stream on :443 (/api/v1/events, broadcast channel, 15 s keep-alive) fanned out by frontend composables — no second listener, no proxy. A contextual second-level sidebar pattern already exists for Monitor (MONITOR_NAV in SidebarInventory.vue). Network apply is direct-apply (no pending/confirm/abort). ~13 design-system components exist; ~12 from the redesign list are missing (HealthBadge, MetricCard, TimeRangeSelector, EntityLink, TaskProgress, IssuePanel, AlarmPanel, EventTimeline, TopologyNode, ResourceInspector, FilterBar, LiveIndicator, DataFreshnessIndicator).

---

2. §0.B — LOCKED DECISIONS (resolve before touching persistence)

These were the three decisions the brief required to be locked with the owner before writing any schema. The audit resolved them by discovery of the already-shipped architecture; the locked choice is “extend what exists”.

2.1 Observability DB ownership — LOCKED: single daemon writer, domain daemons feed it over the existing UDS proxy

The brief offered (a) dedicated DB with one collector writer in the main daemon, (b) per-daemon tables aggregated at the API, (c) collector as a daemon concern fed by storaged/network. The implemented reality is already (a)+(c) combined, and it is the correct, lowest-risk choice — we keep it:

• /var/lib/synvirt/observability.db is owned and written only by the daemon (one Arc<Mutex<Connection>>, WAL). synvirt-storaged, synvirt-network, synvirt-migratord, synvirt-livemotiond and synvirt-clusterd never open it (verified: grep is clean).
• New per-domain sample tables (network, storage) are written by daemon-side sampler tasks that pull the raw counters from the domain daemon over the existing synvirt-ipc UDS proxy — exactly how the capacity-history sampler already works (storage_bootstrap::collect_capacity_samples fetches storaged’s batch and the daemon persists it).
• This honours §24.0 in substance: the collection of network counters (OVSDB monitor, rtnetlink, ethtool) lives in synvirt-network (the network’s owner) and storage counters in synvirt-storaged; only the single-writer persistence is centralized in the daemon. No cross-process SQLite writer (the fragility §0.B warned about). Consistent with §0.7 (single binary / single port, no new listeners), §0.8 (clusterd stays thin).

2.2 Relation to existing DBs — LOCKED: extend, do not rename or collide

• Do not rename synvirt.db — it is the cluster store owned by synvirt-clusterd; the daemon must never open it (enforced by a guard comment in main.rs).
• No collision with synvirt-backupd — it owns a separate redb store at /var/lib/synvirt/backupd/.
• The observability redesign extends observability.db via versioned migrations (0006+). No migration shim against any other DB. The existing activity.db (sled, VM-lifecycle log) stays separate for now; bridging it into events is a deferred follow-up, not part of this iteration.

2.3 Network apply model — LOCKED: direct-apply stays, no staging reintroduced

• The pending/confirm/abort staging flow was removed on purpose. The §19 wizard and the §8 apply paths must not reintroduce it.
• Apply is synchronous; the response carries the outcome (applied_non_management / confirmed_management / rolled_back / apply_failed). Automatic rollback is a safety net only, fired solely when a management-plane (ipMGMT) change loses gateway connectivity (60 s watchdog + ICMP probe).
• Allowed addition (read-only, non-staging): a reconcile-preview endpoint that returns the OVS plan actions that would fire (dry-run of the plan layer) so the wizard can show “what will change”. This is a preview, not a commit gate.

---

3. Cross-cutting invariants carried into every later phase

• Guest safety (Fix70), SEV-0. No daemon/collector restart may touch a running guest’s QEMU PID. New collectors must be add-only background tasks that degrade to “no data” on failure; never block the API; never kill/respawn a VM. The disk-replace + resilver flow must verify running VMs on the pool and proceed without bouncing them.
• No hardcoded restrictions / soft warnings, except genuinely destructive ops (disk replace, management-network change, pool create/destroy) which block in preflight on real data-loss/lockout risk. Ineligible disks in a replace show the reason, never hidden.
• UTC for persistence + history; monotonic clock for the real inter-sample interval (never assume exactly 5 s). Local time only in the UI.
• Single binary / single port (§0.7): all live updates ride the existing :443 SSE stream. No new listeners, no proxies.
• OpenAPI-first (§28): every new endpoint defined in the spec first; frontend consumes generated types; re-dump + typecheck before closing a phase.
• English-only output; no internal version/build identifiers in code, commits, logs, or docs — describe behaviour and invariants.

---

4. Phase 2 — proposed domain model (design only; no code yet)

Map the brief’s taxonomy (§4) onto the existing schema plus the minimal new tables. Everything is an additive migration on observability.db.

Taxonomy	Backing store	Action
Metric (time-series delta)	NEW network_samples, NEW storage_samples; existing storage_capacity_samples, vm_disk_samples	add the two delta-based sample tables (§26 column lists) + the rollup tables in §5
Event (immutable fact)	existing events	extend sources: ZFS events, vdev/device state, link up/down, bond member change, iSCSI login/logout, scrub/resilver lifecycle
Task (async op + stages)	existing tasks (+ NEW task_steps)	add task_steps; wire TaskRegistry::start/settle across all long-running ops (today it is only partial)
Issue (correlated condition)	existing issues_catalog + NEW issue_occurrences	add occurrence/dedup tracking; states Open/Acknowledged/Silenced/Resolved/Reopened
Alarm (rule instance)	existing alarms + NEW alarm_rules, NEW alarm_instances	add the rule engine tables (§13): a rule → an instance per entity, instance updates an Issue without duplicating
Audit record	existing edit_settings_audit + synvirt-time::AuditStore (with source_ip, already shipped)	wire emission everywhere; redact secrets (CHAP/keys)
Structured log	synvirt-logging (journal/file viewer, already shipped)	bridge ERROR-rate into a detector signal; no duplication of the log store
collector_health (§27)	NEW collector_health	per-collector last-success, cycle duration, samples in/dropped, queue depth, write time, clock skew, source-unavailable
entity_relationships (§12.3 correlation)	NEW entity_relationships	VM ↔ zvol ↔ pool ↔ device, vNIC ↔ port ↔ vSwitch ↔ uplink — powers “write latency rose after device X reported media errors”

Delta computation (§24.8): kernel/OVS counters are cumulative. Each sample reads current, diffs against previous, detects reset/wrap, marks gaps, stores both absolute and delta + a derived rate using the real monotonic interval, never a hardcoded 5 s. VLAN attribution (§24.7): one counting point per flow; access ports attribute to their VLAN; trunks use OpenFlow per-VLAN counters; never sum layers as if additive; document the methodology.

---

5. Retention + rollups (§25) — the one real persistence gap

Today retention is blind age-based purge with no rollups, so raw 5 s/30 s samples grow the DB linearly. The redesign adds:

• Raw 5 s: 48 h · rollup 1 min: 30 d · rollup 15 min: 13 mo · rollup 1 h: optional long-term. Issues/alarms/tasks/audit/critical-events: separate, much longer retention.
• Each rollup row keeps min/max/avg/sum(for deltas)/P95(when enough data)/sample_count/gap_count/reset_count. Rollups run off the API thread, batched, never one transaction per metric; backpressure + dropped-sample accounting surfaced via collector_health.

---

6. Phased execution plan + the STOP gate

One phase at a time; verify (tests/lint/typecheck/build); hand off; stop.

Phase	What	Gate
1 — Audit	this document	✅ done
2 — Domain model	§4 above (design)	✅ drafted
3 — Persistence	migrations 0006+ (network_samples, storage_samples, rollups, task_steps, alarm_rules/alarm_instances, issue_occurrences, collector_health, entity_relationships), batch writer, retention+rollup loops	UNBLOCKED (owner sign-off 2026-06-28) — proceed; touches the single-writer persistence path, so verify (tests/build) before hand-off
4 — Collectors	storage (zpool iostat -l latency, SMART deltas, adapters, iSCSI) in synvirt-storaged; network (OVSDB monitor, rtnetlink, ethtool, bond/LACP, LLDP, per-VLAN OpenFlow) in synvirt-network; daemon-side samplers pull + persist	UNBLOCKED to build + lab-verify (sign-off 2026-06-28); FLEET DEPLOY GATED — SEV-0 Fix70: collectors live in the data-plane daemons; a separate explicit go is required before deploying to the production hosts
5 — Backend API	read models + commands + streaming over :443, OpenAPI-first	after 3–4
6 — Storage UI	contextual sidebar, Overview, Pools, Devices, Adapters, iSCSI, Performance, Issues, Tasks/Events, disk-replace wizard	after 5
7 — Networking UI	contextual sidebar, Overview, vSwitch topology+inspector, Networks/VLANs, NICs, Bonds, Host ifaces, Performance, wizard (direct-apply)	after 5
8 — Alarm engine	rules, hysteresis, dedup, correlation, auto-resolution, antiflooding (root-issue + related impacts, not one-per-zvol)	after 3
9 — Testing & hardening	unit (delta/reset/wrap, rollup, retention, dedup, eligibility, VLAN attribution, redaction, RBAC), integration, Playwright (never destructive against .11/.12/.13 or the .75 canary), DB growth, failure injection	last

Why the stop here is deliberate

Phase 3 is exactly the line the brief’s §0.B draws (“do not create schema or migrations until the ownership decision is resolved with the owner”). The ownership decision is now resolved and documented (§2), but it was resolved by me under the delegated “prioritize local” authority, not signed off by the owner. Phases 3–4 also enter SEV-0 Fix70 territory (collectors inside the data-plane daemons; their deploy restarts must preserve live guests). Building that overnight without sign-off risks both wrong-direction code and the .75 canary. So Phase 1+2 ship as a reviewable map + locked decisions; 3→9 proceed once the owner approves §2.

UPDATE 2026-06-28 — §2 approved by the owner (green light). The sign-off gate is lifted. Execution proceeds phase-by-phase per the table above; each phase verifies (tests/lint/typecheck/build) and hands off before the next. The one remaining gate is operational, not architectural: no SEV-0 fleet deploy of the Phase-4 collectors to the production hosts without a separate explicit go (the collectors run inside the data-plane daemons, whose deploy restarts must preserve live guests — Fix70).

---

7. Highest-value first work once unblocked (recommended order)

1. Wire the 24 stub detectors to real suppliers (storage pool health, network uplink/VLAN/MTU, cert expiry, kernel panic, VM heartbeat) — fills Issues/Alarms with actionable signal using the registry that already exists.
2. Wire TaskRegistry + edit_settings_audit emission across all operations — the tables are ready and empty; this lights up Tasks + Audit.
3. Un-stub storage latency (zpool iostat -l) and add the network OVSDB + rtnetlink + ethtool collectors → the first real per-interface / per-vdev metrics.
4. Disk-replace wizard (preflight + typed-confirm + staged resilver task, guest-safe) — the single biggest missing storage workflow.
5. Rollups + collector_health — before raw samples bloat the DB.

Deferred and clearly marked as next domain: cluster-wide aggregation, DRBD9/LINSTOR, Ceph.