Night-run report — observability + VM Edit Settings (2026-05-09)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/NIGHT_RUN_REPORT.md. Edit at the source, not here.

Night-run report — observability + VM Edit Settings (2026-05-09)

Branch: feature/observability
Base: 0.15.0 (6fee23d)
HEAD: f42d428
Total observability commits: 4 (7f54c4e, ad912b3, cacdefc, f42d428). Two interleaved WebRTC commits from the parallel agent also live in the chain (bf24201, f93f017); they were not authored by this run.

Commits since 0.15.0

f42d428 feat(web-ux-v2): observability surface + VM Edit Settings modal
f93f017 fix(synvirt-webrtc): two host candidates per session for Chromium UDP/443 filter
cacdefc feat(synvirt-observability): expand detector catalog to 30+ codes
ad912b3 feat(synvirt-observability): example /etc/synvirt/observability.yaml
7f54c4e feat(synvirt-observability): scaffold crate with SQLite persistence and detector framework
bf24201 feat(console-webrtc): per-session FPS + bitrate knobs and quality dropdown

Coverage

Issues & Alarms — ✅ shipped

New crate synvirt-observability registered in the workspace.
30+ detector codes covering host CPU / memory / load / swap, ZFS pool usage / degraded / scrub / resilver, OVS bridge / uplink / VLAN / MTU, NTP / DNS / log error rate / kernel panic / cert expiry, BMC fan / temp / PSU / battery / IPMI SEL, and the full VM-side surface (heartbeat / agent / consolidation / snapshots / CPU / CPU ready / memory / disk latency / disk usage / drivers / tools / boot / migration / orphan disks).
Full alarm state machine: triggered → acknowledged → resolved with auto-resolve scan, Acknowledge (note + user), and Reset to Green operator transitions wired through the registry → EventBus so every transition lands as an event.
Catalog list in config::ALL_DETECTOR_CODES matches the example /etc/synvirt/observability.yaml.

Tasks & Events — ✅ shipped (crate side)

TaskRegistry + TaskHandle façade with safe drop semantics (cancel-on-drop). Helper builders for task_started, succeeded, failed events.
EventBus over tokio::sync::broadcast with persisted history in events. Retention purger spawnable from the daemon.
WebSocket subscriber + REST shape documented in docs/OBSERVABILITY.md. Daemon REST and WS handlers are the remaining Priority-2 work — flagged below.

Edit Settings — ✅ UI shipped, ⚠️ backend pending

EditSettingsModal.vue with three tabs (Virtual Hardware / VM Options / Advanced Parameters), Save / Cancel footer, dirty diff detection, editability-matrix consumption, and PATCH wire format.
Full panels: CpuPanel, MemoryPanel, DisksPanel. Per-VM Networks, CD/DVD, and Controllers panels are placeholders with the data already plumbed and copy explaining the next iteration.
VM Options tab summarises boot order, firmware, secure boot, video.
Advanced Parameters tab with whitelist-aware key/value editor; whitelist evaluation lives in synvirt_observability::config::EditSettingsConfig::allows_advanced_key.
All hot-plug rules, the matrix, and the audit table are documented in docs/EDIT_SETTINGS.md. The daemon-side PATCH /api/v1/vms/{uuid}/settings and the editability GET are Priority-2 work, see below.

Tests Playwright — ⚠️ not implemented

Skipped per the night-run triage. Contracts the suites would assert are documented in docs/OBSERVABILITY.md and docs/EDIT_SETTINGS.md so the suites can be authored once the daemon endpoints land.

Migrations DB

0001_initial.sql — issues_catalog, alarms (partial unique on active rows), tasks, events, edit_settings_audit, plus schema_version.

Build & lint gates

cargo check -p synvirt-observability: PASS.
cargo test -p synvirt-observability: PASS (3 unit tests).
cargo clippy -p synvirt-observability --tests -- -D warnings: PASS.
npx vue-tsc --noEmit (web-ux-v2): PASS.
cargo check --workspace: PASS after the parallel agent’s bf24201 commit landed the missing synvirt_webrtc::QualityOverride type. Was failing at the start of the night-run, see docs/HANDOFF_WEBRTC.md.
Workspace clippy / fmt / audit not run on the full workspace — the WebRTC area is parallel-agent territory and still in flux.

Outstanding Priority-2 work

These are intentionally deferred so the parallel WebRTC work and this observability work can both land cleanly without stepping on each other’s daemon-side changes.

Daemon REST & WS routes. The dashboard already calls /api/v1/observability/{alarms,tasks,events} and /api/v1/observability/events/stream; the daemon needs a new crates/daemon/src/api/observability/ module. Until then the store fails soft and surfaces a non-fatal warning toast — UI still renders empty.
VM Edit Settings backend. GET /settings, GET /settings/editability, and PATCH /settings need to land in crates/daemon/src/api/vm_settings.rs (next to the existing vm_hardware.rs). The contract is fully documented in docs/EDIT_SETTINGS.md.
TaskRegistry plug-in. Wire vm.start (and over time the rest of the lifecycle handlers) through TaskRegistry::start so operator actions surface in /monitor/tasks with progress and Events.
Per-host scoped views. /hosts/:id/issues|tasks|events and /vms/:uuid/issues|tasks|events. Trivial routing additions once the global views are validated against real data.
Playwright E2E suites. tests/e2e/observability/ — issues-alarms.spec.ts, tasks-events.spec.ts, vm-edit-settings.spec.ts per the original spec.

Reserved-area notes

The original spec called the WebRTC area off-limits. The parallel agent active during the run committed several WebRTC commits on top of the observability branch:

bf24201, c4b9611, d9b76ad, f93f017 — WebRTC quality knobs, host-candidate fix, e2e test fixture.

These were left intact. Branch hygiene was restored twice during the run by:

git branch -f feature/observability HEAD immediately after each observability commit landed on the parallel branch.
git branch -f feature/console-enterprise-night-batch <their_tip> to rewind their branch back off observability commits.

feature/observability now contains the four observability commits plus the parallel agent’s WebRTC commits as parents (cherry-picked forward). The observability files do not touch any reserved-area file.

Branch state at the end of the run

feature/observability                f42d428  ← my work + interleaved WebRTC parents
feature/console-enterprise-night-batch d9b76ad ← parallel agent's tip
0.15.0                               6fee23d

Decisions worth re-reviewing

SQLite over sled for observability. The crate’s API is query-heavy (filters by severity / target / time window / category / since_id), and SQLite indexes / partial unique constraints are a much better fit than sled prefix scans. Existing service crates use sled; this is the first SynVirt feature to default to SQLite.
Callback-driven detectors for everything that touches daemon state. Each detector accepts a boxed-future closure that the daemon supplies (pool health, OVS bridge status, BMC sensors, VM telemetry, …). This keeps synvirt-observability independent from synvirt-libvirt / synvirt-network / the daemon, but pushes some plumbing onto the daemon. The alternative — a thick D-Bus client surface — was deferred to keep the night-run focused on the observability data model and the dashboard surface.
Edit Settings hot-plug matrix in the daemon, not the client. The dashboard reads editability from the daemon and renders the resulting locked / unlocked state. The client never reasons about power state directly, so a future hot-plug rule update only touches the daemon.
Lossy event broadcast. The websocket fan-out drops messages on subscribers that are too slow; the SQLite events table is the source of truth and the client uses since_id to backfill.

Smoke tests REST/WS (2026-05-10)

Daemon REST + WebSocket endpoints landed on top of the night-run work in commits 90538e1, 6384f1e, 77f295f. Manual smoke run on the dev box (Debian 13, no /etc/synvirt/observability.yaml, no /etc/pam.d/synvirt-daemon) — confirms the wiring is alive.

Boot

$ /home/synvirt/dev/target/debug/synvirt-daemon --dev --port 18443
…
WARN  synvirt.daemon.observability: observability config missing or invalid;
      falling back to compiled defaults, error: E_OBS_CONFIG_READ:
      failed to read configuration at /etc/synvirt/observability.yaml:
      No such file or directory (os error 2)
INFO  synvirt_observability::db: migration applied, version: 1
INFO  synvirt.daemon.observability: observability surface ready

/var/lib/synvirt/observability.db is created with WAL + foreign keys + the 0001_initial schema (issues_catalog, alarms, tasks, events, edit_settings_audit, schema_version).

Route mount confirmation

PAM auth on this box defaults to login (the synvirt-daemon PAM service is not installed); without valid root credentials each endpoint correctly returns 401:

$ curl -sk -i https://127.0.0.1:18443/api/v1/issues             | head -1
HTTP/2 401
$ curl -sk -i https://127.0.0.1:18443/api/v1/alarms             | head -1
HTTP/2 401
$ curl -sk -i https://127.0.0.1:18443/api/v1/tasks              | head -1
HTTP/2 401
$ curl -sk -i https://127.0.0.1:18443/api/v1/tasks/some-id      | head -1
HTTP/2 401
$ curl -sk -i https://127.0.0.1:18443/api/v1/events/list?limit=10 | head -1
HTTP/2 401
$ curl -sk -i -H "Connection: Upgrade" -H "Upgrade: websocket" \
       -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGVzdA==" \
       https://127.0.0.1:18443/api/v1/events/stream             | head -1
HTTP/2 401

POST endpoints behave the same:

$ curl -sk -i -X POST https://127.0.0.1:18443/api/v1/issues/abc/acknowledge \
       -H "content-type: application/json" -d '{"note":"x"}'    | head -1
HTTP/2 401

Behavioural smoke

PAM on this dev box does not have a valid root password configured, so the positive happy path is exercised through crates/daemon/tests/observability_smoke.rs instead — three end-to-end tests (alarm lifecycle, task lifecycle + bus events, event publish + filter):

$ cargo test -p synvirt-daemon --test observability_smoke

running 3 tests
test events_publish_persist_and_filter ... ok
test alarm_lifecycle_end_to_end ... ok
test task_lifecycle_emits_events ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Assumed-paths log

Path collisions and gaps surfaced during the wiring; flagging here so they get a second pass:

/api/v1/events is owned by the legacy SSE handler. The observability list lives at /api/v1/events/list and the websocket at /api/v1/events/stream. UI store + composable updated to match.
TaskRegistry list filters on since, until, kind prefix: the crate’s list_tasks only supports (target_type, target_id, state, limit). The daemon over-fetches up to MAX_LIMIT * 2 and filters in-process. Asumido, validar después: extend TaskRegistry::list itself with the wider filter shape so the SQLite indexes can do the work.
Subtask discovery for GET /v1/tasks/{id} similarly over-fetches and filters by parent_task_id. Asumido: a dedicated list_subtasks(parent_id) belongs in the crate.
Alarm since_id cursor is not exposed on AlarmRegistry::list; the websocket reconnect path therefore cannot replay alarms before the cursor (events do replay via EventRow.id, which is what the dashboard primarily consumes).
WebSocket pong is documented in the protocol but the current implementation is a no-op — axum already handles WebSocket-level ping/pong, and the dashboard relies on those, not the JSON-level pong frame. Asumido that this is acceptable until the dashboard actively uses the application-level keep-alive.
DetectorRegistry tick loop is not spawned. Detectors all require host-side callbacks (pool health, BMC sensors, VM telemetry) the daemon does not expose yet. Wired in a follow-up.

What to pick up next

Land the daemon-side crates/daemon/src/api/observability/ module (REST + WS) — straightforward axum work given the contracts in docs/OBSERVABILITY.md.
Land crates/daemon/src/api/vm_settings.rs with the editability matrix builder and the PATCH applier (plus rollback on partial failure). Audit rows go into edit_settings_audit via the observability database handle.
Wire one or two daemon lifecycle handlers (vm.start, vm.snapshot.create) through TaskRegistry::start end-to-end so the dashboard’s /monitor/tasks shows live progress against a real backend.
Author the Playwright suites, anchored on the contracts the daemon now fulfils.