Storage redesign — Phase 4→6 implementation blueprint (collectors → read API → UI)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/architecture/storage-redesign-phase4to6-blueprint.md. Edit at the source, not here.

Storage redesign — Phase 4→6 implementation blueprint (collectors → read API → UI)

Status: code-grounded plan, ready to execute. Derived from a 3-front audit of the real tree (storage backend, observability persistence post-Phase-3, web-ux-v2 frontend). Parent spec: observability-storage-networking-redesign.md (§6 phase table, §7 priority order). Phase 3 persistence blueprint: observability-redesign-phase3-blueprint.md.

0. The pivotal finding — Phase 3 already shipped the whole persistence layer

Migrations 0006–0012 are live. storage_samples, network_samples, the 1-min/15-min rollup tables, collector_health, entity_relationships, task_steps, alarm_rules, alarm_instances, issue_occurrences all exist with their batch writers (insert_storage_samples @ synvirt-observability/src/db/storage_samples.rs:156, upsert_collector_health @ db/collector_health.rs:124). The rollup scheduler already runs (daemon/src/main.rs:1518, defined :1952) and reads storage_samples.

But nothing feeds the raw tables in production — the only insert_storage_samples caller is a test. So Phase 4 is purely the collector + sampler halves; no new migration, no new writer, no new listener. Once the raw table is fed, rollups light up with zero extra wiring.

1. Invariants carried into every slice

• SEV-0 Fix70. Collectors live inside synvirt-storaged (data plane). They must be add-only background tasks that degrade to “no data” on failure, never block the API, never touch a guest QEMU PID, never run a destructive zpool/zfs/qemu-img op. Read verbs only: zpool list/status/iostat/events, zfs list/get, smartctl -A, /proc, /sys. synvirt-storaged is guest-safe by construction today (zero background tasks, no zfs op on SIGTERM) — preserve that.
• Fleet deploy of Phase-4 collectors is GATED (separate explicit go). Build + lab-verify only until then.
• Single writer. Only the daemon’s Arc<Database> (Mutex<Connection>) writes observability.db. synvirt-storaged emits wire DTOs over UDS; the daemon-side sampler persists. The capacity sampler is the exact precedent (daemon/src/api/storage.rs:204 ← storaged handlers.rs:262).
• Deltas computed in the collector/sampler, not the DB. cumulative→delta, 64-bit wrap, backwards-reset, missed-tick gap, real monotonic interval (never a hardcoded 5 s). Per-entity prior-tick state lives in the sampler task.
• UTC persistence (now_iso8601() is crate-private; rows in one batch share one instant). Local time only in the UI.
• Single :443 live surface. Tasks/alarms ride the observability EventBus → WS /api/v1/observability/stream. The frontend reads the shared useObservabilityStore (mounted once in AppShell). Do NOT open a new socket. Live storage metrics either poll the new read API or fold a new frame type into the existing WS.
• OpenAPI-first. Every new endpoint = #[utoipa::path] with an explicit operation_id; re-dump openapi.snapshot.json (synvirt-daemon --dump-openapi) and npm run generate:api:from-file before closing the frontend slice.
• No 1h rollup tier exists (only 1-min + 15-min). 15-min p95 is always NULL by design. Phase 5 read models must not assume otherwise.
• English-only product surfaces; no third-party product names; no version literals.

2. Slice plan + dependency graph

Slice	What	Crate(s)	Depends on
4A	Storage I/O collector in storaged: zpool iostat -l -p -H -v → per-vdev/disk latency+iops+qd; IoSampleWire DTO; GET /v1/storage/io-samples; un-stub read_pool_iostat latency	synvirt-storaged, synvirt-storage-types	—
4B	Daemon storage sampler: delta/wrap/reset/gap/monotonic-interval engine; insert_storage_samples + upsert_collector_health; wire in main.rs	synvirt-daemon, synvirt-observability (reuse)	4A (DTO)
4C	SMART/device/adapter health collector + serial/WWN identity ledger; adapter driver/firmware/queue-depth/resets backend	synvirt-storaged, synvirt-storage-types, synvirt-daemon (sampler)	4A (pattern)
4D	zpool events → events table (cursor tail); wire 3 stubbed storage detectors (orphan disk / old snapshot / consolidation) to real storaged suppliers	synvirt-storaged, synvirt-daemon	—
5A	Read API: storage metrics/rollup history endpoints (time-ranged), collector_health freshness endpoint; OpenAPI re-dump	synvirt-daemon	4A/4B (shapes)
5B	Read models for the rich device/adapter detail + per-vdev/zvol breakdown surfaced through storage detail API	synvirt-daemon	4C
6A	The 13 design-system primitives (HealthBadge, MetricCard, TimeRangeSelector, EntityLink, TaskProgress, IssuePanel, AlarmPanel, EventTimeline, TopologyNode, ResourceInspector, FilterBar, LiveIndicator, DataFreshnessIndicator) + vitest	web-ux-v2	—
6B	Storage contextual sidebar (STORAGE_NAV) + route tree (Overview/Pools/Devices/Adapters/iSCSI/Performance/Issues/Tasks-Events)	web-ux-v2	6A
6C	Overview + Performance views (live + historical, TimeRangeSelector + charts off 5A)	web-ux-v2	6A,6B,5A
6D	Devices + Adapters views (PoolDisksTable + TopologyNode + ResourceInspector; SMART trend; driver/firmware/qd)	web-ux-v2	6A,6B,5B
6E	Disk-replace wizard: preflight (eligibility+reasons) → typed-confirm → staged resilver task (guest-safe) → live task_steps progress; storaged zpool replace op	synvirt-storaged, synvirt-daemon, web-ux-v2	6A,6B

Execution waves (parallel within a wave where no file overlap):

• Wave 1: 4A ∥ 6A
• Wave 2: 4B, 4C, 4D (after 4A)
• Wave 3: 5A ∥ 5B (after Phase 4) → OpenAPI re-dump + FE type regen
• Wave 4: 6B → 6C ∥ 6D, then 6E

3. Per-slice contract (grounded anchors)

4A — storaged I/O collector

• Add IoSampleWire to synvirt-storage-types/src/pool.rs (alongside PerfSample @ :80): per-target raw cumulative counters + identity: { storage_target, storage_type: 'vdev'|'zvol'|'disk', entity_id, read_ops_abs, write_ops_abs, read_bytes_abs, write_bytes_abs, read_latency_ms, write_latency_ms, queue_depth?, utilization_pct? }. Maps 1:1 to StorageSampleInsert columns (storage_samples.rs:24-83).
• New synvirt-storaged/src/perf.rs: collect_io_samples(&State) -> Vec<IoSampleWire> running zpool iostat -l -p -H -v <pool> 1 1 per pool (the -l wait columns → latency ms; -v → per-vdev + per-leaf-disk rows). Bounded timeout. Read-only.
• New route in synvirt-storaged/src/http.rs:23 block: GET /v1/storage/io-samples → handlers::io_samples, modeled byte-for-byte on capacity_samples (handlers.rs:262).
• Un-stub read_pool_iostat (handlers.rs:902-943, latency_us:0 @ :941): parse the -l total_wait columns into the on-demand PerfSample too.
• Verify: cargo build/clippy -p synvirt-storaged -p synvirt-storage-types; unit-test the iostat -l -v parser against captured fixture lines.

4B — daemon storage sampler (the new delta engine)

• Clone spawn_capacity_sampler (daemon/src/api/storage.rs:204-246) → spawn_storage_io_sampler. Loop: state.storage.get_json::<Vec<IoSampleWire>>("/v1/storage/io-samples") (best-effort, empty on transport error), then per entity_id compute delta/rate vs the prior tick held in a HashMap<String, PrevTick>:
  • delta = cur - prev if cur >= prev else has_reset=1 (or has_wrap if cur small & prev near u64::MAX), delta=NULL;
  • interval_secs = real monotonic elapsed (Instant), never hardcoded;
  • *_iops = delta_ops / interval, *_throughput_bps = delta_bytes / interval;
  • has_gap=1 when interval ≫ expected cadence. Build Vec<StorageSampleInsert>, db.insert_storage_samples(&batch), then db.upsert_collector_health(&now_health_check("storage_io", …)) (collector_health.rs:252), then db.purge_storage_samples_older_than(retention.raw_samples_secs).
• Wire in daemon/src/main.rs:1487 if let Some(obs) block, after spawn_capacity_sampler (:1488) and before spawn_rollup_scheduler (:1518).
• Verify: unit tests for the delta/reset/wrap/gap/interval math (table-driven). Build + clippy daemon. Lab: run daemon against a host with a pool, confirm storage_samples and collector_health populate and storage_samples_1min_rollup fills within ~1 min.

4C — SMART/device/adapter health + identity ledger

• storaged: slower-cadence (5–15 min) collector emitting storage_type:'disk'|'adapter' samples. Disk: smartctl -j -A <path> deltas (reuse enrich_disk_smart handlers.rs:716-781 field parsing). Adapter: driver/firmware from /sys/class/{scsi_host,nvme}/…, nvme id-ctrl, lspci -vmm; queue-depth from /sys/block/<dev>/device/queue_depth; resets from /sys/class/scsi_host/*.
• Serial/WWN identity ledger (the entity_id stability dependency, audit §1.1): a sled store mapping (serial|wwn) → stable entity_id so disk samples survive rename/reorder. Owner: storaged (it already owns disk enumeration). Add fields to StorageDeviceView/AdapterView for driver/firmware/qd/resets.
• daemon sampler persists via the same insert_storage_samples path (own slower interval).
• Verify: parser unit tests; build/clippy; lab-confirm disk + adapter rows land with stable entity_id across a storaged restart.

4D — zpool events → events table + detector wiring

• storaged: expose a cursor-based GET /v1/storage/event-cursor over zpool events -H (it has a monotonic sequence) so the daemon tails new rows; daemon writes them into the events table (the immutable-fact store), classified ok/warn/info (pretty_event_title handlers.rs:984).
• Wire the 3 stubbed storage detectors — VmConsolidationNeeded, VmSnapshotOld, VmOrphanDisk (main.rs:2214-2218, stub closures :2306-2351) — to real suppliers::* helpers hitting new storaged endpoints (snapshot-chain depth, oldest-snap age, orphan-disk dataset scan). Pattern: replace Arc::new(|| Box::pin(async { Ok(Vec::new()) })) with a suppliers-delegating closure (Slice-B form main.rs:2164-2168); callback aliases issues/callbacks.rs. Fail-soft.
• Verify: build/clippy; lab-confirm an injected orphan dataset surfaces an Issue and a scrub start/finish lands as an Event row.

5A — metric/rollup read API

• New #[utoipa::path] handlers under /api/v1/observability/storage/* (or /api/v1/storage/{name}/performance/history): time-ranged read of raw storage_samples
  • *_1min_rollup + *_15min_rollup (list_storage_samples storage_samples.rs:220 is the DB reader to extend). Auto-pick the tier from the range (raw ≤48 h, else rollup).
• GET /api/v1/observability/collectors/health from list_collector_health (collector_health.rs:187) → backs DataFreshnessIndicator/LiveIndicator.
• Explicit operation_ids; re-dump openapi.snapshot.json.
• Verify: build/clippy; --dump-openapi diff clean; handler unit/integration test.

5B — rich device/adapter/breakdown read models

• Surface 4C’s device (SMART trend + driver/firmware/qd/resets) and per-vdev/zvol breakdown through the storage detail API (extend PoolDetailView or add endpoints).
• Verify: build/clippy; OpenAPI re-dump; shape tests.

6A — design-system primitives (fork sources from the FE audit)

Build under web-ux-v2/src/components/ui/ (no design-system/ dir exists): HealthBadge ← observability/SeverityBadge.vue; MetricCard ← storage/PoolKpiRow.vue

• ui/Sparkline.vue; TimeRangeSelector ← new (pair w/ useTelemetryHistory.ts); EntityLink ← new (RouterLink + vocab map); TaskProgress ← migrator/PhaseProgressBar.vue; IssuePanel ← vm-detail/tabs/observability/VmIssuesPanel.vue; AlarmPanel ← activity/ActivityDock.vue; EventTimeline ← observability/EventTimelineEntry.vue; TopologyNode ← storage/PoolTopology.vue tile; ResourceInspector ← activity/ActivityDetailDrawer.vue; FilterBar ← vms/VmFilterBar.vue + activity/dockFilters.ts; LiveIndicator ← migrator/LastUpdatePulse.vue; DataFreshnessIndicator ← inline updatedLabel in StorageDetailView.vue:114-122.

• Verify: npm run typecheck + vitest per primitive; render in /playground.

6B — storage sidebar + routes

• Add STORAGE_NAV: ReadonlyArray<NavRow> (shape SidebarInventory.vue:106-114) + collapsible <li> block copied from :1188-1260, gated on isActive('/storage').
• Routes in router/routes.ts declared before /storage/:name (:150): /storage (Overview), /storage/pools, /storage/devices, /storage/adapters, /storage/iscsi, /storage/performance, /storage/issues, /storage/events.

6C / 6D — views

• 6C Overview: pool-health + capacity rollup cards (HealthBadge/MetricCard), top movers. Performance: TimeRangeSelector + inline-SVG charts (PoolPerfChart/UsageHistoryCard idiom) off 5A; live tail via shared useObservabilityStore, freshness via DataFreshnessIndicator.
• 6D Devices: PoolDisksTable rows + forked TopologyNode + ResourceInspector drawer + SMART trend; Adapters: driver/firmware/qd/resets table off 5B. Reuse setActiveHostScope cross-host scope.

6E — disk-replace wizard

• Backend: storaged zpool replace op behind a preflight (eligible spares with per-disk reasons when ineligible — never hide), staged as a task with task_steps progress. Guest-safe: enumerate running VMs on the pool and proceed WITHOUT bouncing them (Fix70). Destructive → typed-name confirm.
• UI: CreatePoolWizard/WizardShell scaffold + ConfirmDialog typed gate + live TaskProgress off the task’s task_steps.
• Note: the destructive replace path needs a real spare disk to E2E — lab-verify the preflight + task machinery + UI; flag the live replace as hardware-gated.

4. Verification & integration gates

• Each backend slice: cargo build + cargo clippy -- -D warnings on touched crates, judged diff-vs-baseline (workspace clippy/fmt not green at baseline); add unit tests for new parse/delta logic.
• After Phase 4/5: synvirt-daemon --dump-openapi > openapi.snapshot.json → npm run generate:api:from-file → frontend consumes new types.
• Frontend slices: npm run typecheck + vitest for new components (vitest baseline not fully green — judge by diff).
• No fleet deploy until explicit go (Phase-4 collectors are SEV-0/Fix70). No ISO build without explicit go.