Activity Subsystem — Phase-0 Architecture Note

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/architecture/activity-subsystem.md. Edit at the source, not here.

Activity Subsystem — Phase-0 Architecture Note

Scope: SynVirt activity subsystem = the docked Recent Tasks + Alarms panel (web-ux-v2) plus the backend that feeds it (observability tasks, the alarm/issue engine, the live stream, and the Cloud per-host fan-out).

Status: Phase-0 (findings + locked decision + slice plan). No code yet.

Governing constraint (from spec): Do not invent parallel infrastructure — extend what exists. The repo already ships an observability tasks table + TaskRegistry, the new alarm_rules / alarm_instances / issue_occurrences tables, a 40-detector registry, an SSE/WebSocket event stream, a Monitor UI, and a Cloud HTTP host-proxy fan-out. This note locks extend, not replace.

---

1. Phase-0 findings against the spec’s 5 questions

Q1 — Where do “tasks” (async jobs) live today, and is the model rich enough?

A real async-task subsystem already exists in crates/synvirt-observability:

• Model: TaskRow (crates/synvirt-observability/src/db/tasks.rs:74-110) — 16 fields: id (UUID), kind, target_type, target_id, target_name, state, progress_pct (0-100), started_at, ended_at, initiated_by_user, initiated_by_source, parent_task_id, result_json, error_code, error_msg, correlation_id.
• Enums: TaskState (Queued/Running/Success/Error/Cancelled, tasks.rs:15-42); TaskSource (Ui/Api/Scheduler/System/Cli, tasks.rs:44-72).
• Lifecycle facade: TaskRegistry (src/tasks/mod.rs:18-89) with start/get/list; TaskHandle (src/tasks/mod.rs:93-170) with progress(pct), succeed(result_json), fail(code, msg), and an auto-cancel-on-drop guard.
• Daemon entry points: crates/daemon/src/api/task_helper.rs:32-92 — start(), start_with_actor(), settle(task, result).
• Schema: tasks table (migrations/0001_initial.sql:54-70) + task_steps table (migrations/0008_task_steps_alarms_issues.sql:12-26, FK task_id CASCADE).
• Live producers: ~30 task kinds across 5 domains already emit tasks — VM power/lifecycle (vm_power.rs, vms.rs), snapshots (snapshots.rs), VM edit (vm_edit/*.rs), network (network.rs), storage (storage_proxy.rs), update (update_v2.rs), templates (templates.rs).

Verdict: rich enough for basic tracking (id/kind/target/state/progress/timestamps/initiator/error/ correlation/parent). It is missing the cluster + typed-target dimensions (see Q3) and a few cancel/history affordances (see §6).

Q2 — Where do “alarms” live, and is there a real engine or just a table?

There is a real, deduplicating alarm path plus a newer, not-yet-wired rule engine schema:

• Active path (wired): DetectorRegistry.tick_once() (src/issues/mod.rs:78-216) runs each detector and calls AlarmRegistry.upsert() (src/alarms/mod.rs:30-68), which wraps Database::upsert_alarm() (src/db/alarms.rs:150-214). Dedup is pessimistic — the partial unique index alarms_active_unique on (issue_code, target_type, target_id) WHERE state IN ('triggered','acknowledged') guarantees exactly one active row per condition; a second insert finds the existing row and bumps last_triggered_at + count. auto_resolve_stale() (alarms.rs:309-322) resolves alarms idle past auto_resolve_secs (default 172800s = 2 days).
• Rule-engine schema (NOT wired): migrations/0008_task_steps_alarms_issues.sql adds alarm_rules, alarm_instances, issue_occurrences — present in the schema but the detector tick still upserts to the legacy alarms table. No threshold/compound/state-change rule evaluator exists yet.
• Catalog: every detector self-registers a CatalogEntry (src/db/issues.rs) on boot.

Verdict: the engine is real but split — a working legacy dedup path and an unwired issue_occurrences rule schema. Phase-0 decision: build the lifecycle engine onto the existing alarms_active_unique semantics, migrating toward issue_occurrences as the occurrence ledger (see §6, backend slice B2). Do not invent a third path.

Q3 — What’s missing to make tasks/alarms cluster-aware and clickable?

• No origin-node attribution. TaskRow has no origin_host/origin_node; alarms carry a target but no host_id. FED host-scoped views work via /api/v1/hosts/{id}/observability/* proxying, but the rows themselves cannot say which cluster member produced them.
• No typed clickable target. target_type/target_id are bare strings; there is no Target{type,id} to safe-generate an href.
• No result schema in list. result_json round-trips in export but the list response does not deserialize it back to callers.

Q4 — Is there a live stream, and does it already multiplex tasks + alarms + events?

Yes. /api/v1/observability/stream (canonical alias of /api/v1/events/stream, observability/mod.rs:19, events_ws.rs) is a single multiplexed WebSocket: one tokio::sync::broadcast channel (EventBus, src/events/mod.rs:19-35) carries tasks + alarms + events, classified at wire time into {type: event|task_update|alarm|lag_warning|pong, payload} (events_ws.rs:184-188). Clients send a SubscribeFilter (severity[], target_id, categories[], events_ws.rs:50-54) applied client-side. SQLite is the source of truth; the broadcast is lossy (slow subscribers drop, get a lag_warning).

Verdict: the single-stream requirement is already satisfied in-process. Multi-host is the gap.

Q5 — How does the Cloud GUI reach a peer’s tasks/alarms today, and what’s missing?

Today it reaches peer observability only over the HTTP host-proxy: ANY /api/v1/hosts/{fingerprint}/proxy/* (api/hosts.rs:453-590) forwards REST/SSE to the peer’s :15443 with the Authorization header, 120s read-timeout, no whole-request timeout. Host-scoped reads land on /api/v1/hosts/{id}/observability/{events|tasks|issues} (observability/host_scoped.rs:34-68). GET /api/v1/cloud/health (cloud_health.rs:142-299) already rolls up per-host/per-cluster state by walking the inventory tree (build_inventory_document) over the same federation GetInventory RPC.

Missing: there is no /api/v1/hosts/{fp}/observability/stream — the Cloud GUI can only poll peer REST or relay SSE over the proxy; there is no single-WS-per-peer push, and no server-side “follow peer X’s observability” primitive. Observability is in-process per daemon and is deliberately not replicated through Raft (spec forbids routing task/alarm data through Raft — the current architecture already honors this).

---

2. Locked decision — EXTEND the observability + Cloud-proxy infra (no parallel stack)

We extend the existing observability task/alarm/stream infrastructure and the Cloud host-proxy fan-out. We do not stand up a new job store, a new alarm bus, a new socket, or a new UI shell.

Concretely, the activity subsystem is built by extending these exact pieces:

Concern	Reuse / extend (exact symbol)
Task model + lifecycle	synvirt-observability::db::tasks::TaskRow (tasks.rs:74-110), TaskRegistry (tasks/mod.rs:18-89), TaskHandle (tasks/mod.rs:93-170)
Task creation from daemon	daemon::api::task_helper::start_with_actor() (task_helper.rs:32-92)
Alarm dedup + lifecycle	Database::upsert_alarm() (alarms.rs:150-214), AlarmRegistry::upsert() (alarms/mod.rs:30-68), spawn_auto_resolve() (alarms.rs:309-322), partial index alarms_active_unique
Alarm rule ledger (to wire)	migrations/0008 tables alarm_rules, alarm_instances, issue_occurrences
Detector signal sourcing	DetectorRegistry::tick_once() (issues/mod.rs:78-216), 8 suppliers (observability/suppliers.rs:1-685), Detector trait + 8 callback types (issues/callbacks.rs)
Live multiplexed stream	EventBus (events/mod.rs:19-35), /api/v1/observability/stream (events_ws.rs), SubscribeFilter (events_ws.rs:50-54)
REST surface	GET /api/v1/{tasks,alarms,issues,events} + canonical /api/v1/observability/* + host-scoped + vm-scoped (observability/mod.rs:51-141)
Cluster/Cloud fan-out	HTTP host-proxy ANY /api/v1/hosts/{fp}/proxy/* (hosts.rs:453-590), /api/v1/cloud/health rollup (cloud_health.rs:142-299), PeerRegistry::members() (mesh/peers.rs:28-85)
Frontend store + stream	useObservabilityStore() (stores/observability.ts:389-773), useObservabilityStream() (composables/useObservabilityStream.ts:80-182)
Frontend UI primitives	AppShell.vue:236-338, Card.vue, Tag.vue, SummaryCards.vue, SeverityBadge.vue, table pattern TasksView.vue:284-353, filters IssuesView.vue:56-89, drawer EventsView.vue:275-310

---

3. Per-node ownership + on-demand request/response fan-out (NO Raft)

Ownership model. Each node owns and persists its own tasks/alarms/events in its local /var/lib/synvirt/observability.db (SQLite WAL, FKs on). The EventBus broadcast is in-process and local. Tasks and alarms are never routed through Raft — Raft stays reserved for cluster control plane state. This matches the spec constraint and the current architecture (observability is local + HTTP proxy only; nothing observability-shaped is Raft-replicated).

Fan-out model. Aggregation is on-demand request/response, not replication:

• The Cloud/aggregating node enumerates reachable owners via PeerRegistry::members() (mesh/peers.rs:28-85, cluster voters) and, where loose federation applies, the advisory DiscoveredHostsStore (mDNS, discovery/mod.rs:75-150).
• For each owner, the aggregator pulls observability over the existing HTTP host-proxy (/api/v1/hosts/{fp}/proxy/api/v1/observability/...) — REST snapshot now, and a new /api/v1/hosts/{fp}/observability/stream WS relay later (backend slice B3). Rollup counts piggyback on /api/v1/cloud/health (cloud_health.rs) extended with task.queued/running/error + alarm-by-severity tallies.
• Reads use the “fresh” tight-timeout, non-blocking pattern from cluster_control_client.rs:21-114 (snapshot cache for sync reads; fresh async reads with a short timeout; no block_in_place). A slow/dead peer degrades to its last snapshot, never stalls the aggregator.

Standalone = single-localhost-peer parity. A standalone host is modeled as a one-member peer set whose only owner is localhost. The same panel, the same store, the same stream, the same fan-out code path run — the peer list is [self]. No standalone-specific branch. This keeps standalone and cluster on one code path and makes the panel work identically with zero cluster wiring.

---

4. Enumerated real alarm sources (found, not fabricated)

The detector registry wires 40 detectors (daemon/src/main.rs:2049-2346): 7 real-signal, 30 supplier-wired, 3 deferred stubs. Backed by 8 real suppliers (observability/suppliers.rs). The conditions that actually exist in code:

Host — OS / kernel (real-signal, read /proc or subprocess):

1. Host CPU utilization threshold breached — host_cpu.rs (/proc/stat)
2. Host memory utilization threshold breached — host_mem.rs (/proc/meminfo)
3. Host load average threshold breached — host_load.rs (/proc/loadavg)
4. Swap space used threshold breached — host_swap.rs (/proc/swaps + sysctl)
5. NTP clock drift out of range — host_ntp_drift.rs (chrony stats)
6. DNS query to default gateway fails — host_dns.rs (DNS socket test)
7. Daemon error-log rate exceeds threshold — host_log_rate.rs (placeholder counter; tracing wiring pending)
8. Kernel panic detected in recent log — host_kernel.rs (journalctl -k)

Host — storage (supplier: storaged capacity-samples / drill-down): 9. Storage pool usage warn — host_pool_usage.rs 10. Storage pool usage critical — host_pool_usage.rs 11. Storage pool degraded — host_storage.rs (parses zpool status) 12. Storage pool scrub overdue — host_storage.rs (last_scrub timestamp) 13. Storage pool resilver active — host_storage.rs (scrub_state=‘resilvering’)

Host — network (supplier: in-process NetworkController bridge_status): 14. OVS bridge missing from live config — host_network.rs 15. OVS uplink degraded (bond members down) — host_network.rs 16. OVS uplink lost (no members UP) — host_network.rs 17. VLAN trunk mode mismatch — host_network.rs 18. MTU mismatch between uplink and bridge — host_network.rs

Host — TLS cert (supplier: synvirt-certs parser on cert.pem): 19. TLS certificate expiry warning — host_cert.rs 20. TLS certificate expiry critical — host_cert.rs

Host — hardware / BMC (supplier: ipmitool sdr/sel, cached): 21. BMC fan critical — host_hardware.rs 22. BMC temperature critical — host_hardware.rs 23. BMC power supply failed — host_hardware.rs 24. BMC RAID battery degraded — host_hardware.rs 25. IPMI SEL capacity full — host_hardware.rs

VM — liveness / agent (supplier: guest-tools supervisor): 26. VM heartbeat stale (no guest agent seen recently) — vm_heartbeat.rs 27. VM guest agent not running — vm_guest_agent.rs

VM — resources (supplier: vm_telemetry / guest-tools; note many telemetry fields currently 0): 28. VM CPU utilization high — vm_resources.rs (cpu_pct) 29. VM CPU ready percentage high — vm_resources.rs 30. VM memory utilization high — vm_resources.rs (mem_unused_pct) 31. VM disk read latency high — vm_resources.rs (disk_read_await_ms) 32. VM disk write latency high — vm_resources.rs (disk_write_await_ms) 33. VM disk usage high — vm_resources.rs (max_fs_used_pct) 34. VM drivers/virtio-win outdated — vm_resources.rs (virtio_win_age_days) 35. VM boot failure detected — vm_resources.rs (last_boot_failure_at) 36. VM migration aborted — vm_resources.rs (last_migration_failure_at) 37. VM snapshot delta size large — vm_resources.rs (largest_snapshot_delta_bytes)

VM — deferred stubs (return empty Vec, awaiting storaged endpoints): 38. VM consolidation needed (snapshot chain depth) — vm_consolidation.rs (STUB) 39. VM snapshot age exceeds threshold — vm_snapshot_old.rs (STUB) 40. VM orphan disk dataset detected — vm_orphan.rs (STUB)

These 40 are the whole truth of today’s alarm conditions. No additional conditions are invented in this note. (Known not-yet-built detectors — cluster/Raft peer loss, VIP/keepalived failover, backup job failure, update rollback, cgroup pressure, inode exhaustion, uplink latency/packet-loss — are listed under §6 gaps, NOT counted as existing sources.)

---

5. Guest-safety boundary (out-of-band, daemon-restart-safe)

The activity subsystem must never destabilize a running guest. The existing design already enforces this and the extension preserves it:

• Out-of-band signal collection. Detectors read host facts (/proc, subprocesses) and out-of-band guest telemetry (vsock / guest-tools supervisor last_seen, agent_connected). Activity tracking never touches the guest’s CPU/devices in-band; the worst case is a stale or neutral telemetry value (vm_telemetry fields default to 0/neutral), never a guest stall.
• Fail-soft suppliers. Every supplier catches errors and returns an empty/neutral snapshot (unreachable storaged, missing ipmitool, absent BMC, timeout) — no false alarms, no cascade onto a guest (suppliers.rs, fail-soft principle).
• Daemon-restart-safe persistence. SQLite WAL is the source of truth; the EventBus broadcast is lossy and reconstructible from the DB. A daemon restart loses no task/alarm history. This pairs with the established guest-drain invariant — daemon restart must not bounce guests (drain is owned by synvirt-guest-drain.service; daemon stop is HTTP-only; start adopts). The activity subsystem holds no guest-affecting state across restarts and adds nothing that could.
• TaskHandle drop guard. A daemon panic/shutdown mid-task auto-cancels the task row (no perpetual “running” zombie) without sending any signal to the guest.
• Tier-2 fan-out, not Raft. Cross-host observability rides peer-authenticated HTTP proxy / mesh RPC, never Raft replication — a busy activity feed cannot back-pressure the control plane or a guest scheduler.

---

6. Prioritized GAP → implementation plan

Each slice is independently buildable and unit-testable. Fleet deploy (.11/.12/.13 lockstep) and the cluster Playwright E2E are GATED on explicit owner go — not part of any slice’s “done.”

Backend slices

B1 — Task-model generalization (typed target / origin-node / progress / cancel). P1.

• Add origin_node (cluster member attribution) and a typed target_href/Target{type,id} hint to TaskRow + serialization (tasks.rs:74-110); thread origin_node through task_helper::start_with_actor(). Add a created_by_system distinction so scheduler/background tasks separate from operator intervention.
• Expose POST /api/v1/tasks/{id}/cancel (cancellation exists at DB layer cancel_task() + on TaskHandle drop, but has no REST surface).
• Surface result_json (typed Result{success|error{code,msg,context}}) in the list response, not just export.
• Add missing task kinds (already-found gaps): vm.snapshot.restore, vm.snapshot.clone (snapshots.rs), vm.nic.edit (vm_nic_edit.rs), storage.pool.snapshot_policy (storage_proxy.rs).
• Tests: TaskRow round-trip incl. new fields; cancel state transition; kind-emission per handler.

B2 — Alarm dedup + lifecycle ENGINE on issue_occurrences. P1.

• Wire the unwired migration-0008 ledger: detector tick records an occurrence in issue_occurrences and maintains one active alarm_instances row, preserving today’s alarms_active_unique (issue_code, target_type, target_id) dedup semantics (alarms.rs:150-214). Keep auto_resolve_stale() behavior.
• Add origin_node/host_id to alarm rows for cluster context.
• Tests: N ticks of the same condition → 1 active instance + N occurrences + bumped count; idle past auto_resolve_secs → resolved; distinct targets → distinct instances.

B3 — Cluster fan-out (request/response, no Raft). P2.

• Add GET /api/v1/hosts/{fp}/observability/stream as a WS relay over the existing host-proxy (hosts.rs:453-590) so the Cloud GUI gets one WS per peer instead of polling. Aggregator enumerates via PeerRegistry::members() (+ advisory DiscoveredHostsStore), reads with the cluster_control_client.rs fresh-timeout pattern.
• Extend /api/v1/cloud/health (cloud_health.rs:142-299) with task.queued/running/error_pct and alarm-by-severity rollup over the same GetInventory RPC.
• Standalone parity: peer set = [self] (localhost), identical path.
• Tests: rollup math over 2 fake peers; dead-peer degrades to snapshot without stalling.

B4 — History pagination + DB-layer filters. P2.

• Push kind_prefix (vm.*), since, until, parent_task_id into the DB query (today they are post-filters in memory over MAX_LIMIT*2). Add composite indices (target_type, started_at DESC) and (kind, started_at DESC). Return parent/group structure in the list response (parent_task_id exists but list() doesn’t return group structure).
• Tests: paginated cursor stability; kind-prefix at DB layer; group structure shape.

B5 — Ack / clear / silence API. P3.

• Beyond existing POST /api/v1/issues/{id}/{acknowledge,resolve,reset}: add bulk POST /api/v1/alarms/batch-acknowledge, bulk-resolve-by-target/code/severity, and silence-until (silence_reason, silenced_until) — none exist today.
• Tests: bulk ack idempotency; silence window suppresses re-trigger then expires.

Frontend slices

F1 — Bottom-docked two-tab panel shell. P1.

• Add a fourth dock region (bottom slot) to AppShell.vue:236-338 (today only topbar/sidebar/main). Build a collapsible/resizable panel with two tabs (Recent Tasks, Recent Alarms), height toggle, tab + expand/collapse state persisted to localStorage (NotificationBell pattern, NotificationBell.vue:35-39). Reuse Card.vue (flush), Tag.vue, SeverityBadge.vue, SummaryCards.vue.
• Default filters: tasks = running+queued; alarms = triggered.
• Tests (vitest): panel mount/collapse/persist; default-filter selection.

F2 — Store on the single stream (push handlers). P1.

• Extend useObservabilityStore() (stores/observability.ts:389-773) with pushTask() and pushAlarm() (only pushEvent exists) so useObservabilityStream() (useObservabilityStream.ts:80-182, already mounted at shell) folds live task_update / alarm frames into the visible tables. Add “Recent N” loaders (limit rows for compact display). Reuse TaskRow/AlarmRow types as-is (observability.ts:53-90).
• Tests (vitest): WS frame → table mutation; recent-N truncation; de-dup of live vs loaded rows.

F3 — History / sort / filter / acknowledge in the dock. P2.

• Reuse the TasksView.vue:284-353 table column rendering (Kind/Target/State/Progress/Started), IssuesView.vue:56-89 filters, and EventsView.vue:275-310 detail drawer adapted for task/alarm detail. Individual-row acknowledge/resolve/reset via existing store.acknowledgeAlarm/resolveAlarm/resetAlarm (IssuesView.vue:192-220). Expand-to-full link → /monitor/tasks or /monitor/issues. No bulk multi-select in the dock (too crowded).
• Optional Server column when cluster scope present (depends on B1/B2 origin_node).
• Tests (vitest): sort/filter reducers; ack action dispatch; drawer open with row payload.

Gated on owner (NOT slice-complete criteria)

• Fleet deploy to .11/.12/.13 in lockstep (await explicit “dale”).
• Cluster Playwright E2E (multi-host fan-out, live panel) — run after owner go.

---

Appendix — reuse vs gap tally

• Reusable pieces (extend, don’t build): task model/registry/handle, task_helper, alarm upsert/registry/auto-resolve + alarms_active_unique index, detector registry + 8 suppliers + 8 callback types, EventBus single-stream + SubscribeFilter, full REST surface (global/host/vm-scoped) + events WS, HTTP host-proxy + /cloud/health rollup + PeerRegistry, and the web-ux-v2 store/stream/UI primitives — ~30 named symbols across backend + frontend.
• Net-new (gaps to close): origin_node + typed target on tasks/alarms; cancel REST; result in list; 4 missing task kinds; wire issue_occurrences rule ledger; per-host observability WS relay; DB-layer history filters + composite indices; bulk/silence ack API; AppShell bottom dock + pushTask/pushAlarm store handlers — 9 backend + 3 frontend themes, mapped to slices B1-B5 / F1-F3.
• Existing alarm sources enumerated: 40 detectors (37 active across host CPU/mem/load/swap/ ntp/dns/log-rate/kernel + storage×5 + network×5 + cert×2 + BMC×5 + VM liveness×2 + VM resources×10; 3 deferred stubs: consolidation, snapshot-age, orphan-disk).