Observability

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/OBSERVABILITY.md. Edit at the source, not here.

Observability

The synvirt-observability crate owns SynVirt’s host-wide operational picture: issues catalog, alarms, tracked tasks, an append-only events log, and a forensic VM Edit Settings audit trail. The dashboard’s /monitor/* routes consume this surface; operators use it to answer the canonical “what’s wrong?” and “what changed?” questions without grepping logs.

Persistence

A single SQLite database at /var/lib/synvirt/observability.db, opened with journal_mode=WAL and foreign_keys=ON. Five tables:

• issues_catalog — static catalog of every detector code shipped by the daemon. Detectors register themselves on boot.
• alarms — durable state machine. Active rows have a partial unique on (issue_code, target_type, target_id) so detectors can upsert idempotently.
• tasks — every long-running daemon operation tracked end-to-end with progress, source, and structured outcome.
• events — append-only audit log, fed by tasks, alarms, and edit-settings audit hooks.
• edit_settings_audit — one row per field changed in a VM Edit Settings PATCH operation.

Migrations live under crates/synvirt-observability/src/db/migrations/.

Detector catalog

All detector codes are stable identifiers. Each detector’s title, description, default severity, and target type land in issues_catalog on registration. Operators turn detectors on / off in /etc/synvirt/observability.yaml; thresholds live there too.

Host detectors (resources / storage / network / infra / hardware):

Code	Source	Default severity
E_HOST_CPU_HIGH	/proc/stat delta	Warning / Critical
E_HOST_MEM_HIGH	/proc/meminfo MemAvailable	Warning / Critical
E_HOST_LOAD_HIGH	/proc/loadavg × num_cpus × factor	Warning
E_HOST_SWAP_USED	/proc/meminfo Swap	Warning
E_POOL_USAGE_WARN / _CRIT	daemon callback	Warning / Critical
E_POOL_DEGRADED	daemon callback	Critical
E_POOL_SCRUB_OVERDUE	daemon callback	Info
E_POOL_RESILVER_ACTIVE	daemon callback	Info
E_OVS_BRIDGE_MISSING	synvirt-network callback	Warning
E_NETWORK_UPLINK_DEGRADED / _LOST	bond status callback	Warning / Critical
E_VLAN_TRUNK_MISMATCH	bond status callback	Warning
E_MTU_MISMATCH	bond status callback	Warning
E_NTP_DRIFT	chronyc tracking	Warning / Critical
E_DNS_UNREACHABLE	resolver probe	Critical
E_DAEMON_LOG_ERROR_RATE	daemon log counter callback	Warning
E_KERNEL_PANIC_SEEN	journalctl callback	Critical
E_CERT_EXPIRY_WARN / _CRIT	synvirt-certs callback	Warning / Critical
E_HOST_HW_FAN / _TEMP / _PSU / _BATTERY	BMC callback	Critical / Warning
E_HOST_IPMI_SEL_FULL	BMC callback	Warning

VM detectors (state / resources / integrity):

Code	Source	Default severity
E_VM_HEARTBEAT_STALE / E_VM_GUEST_AGENT_DOWN	telemetry callback	Warning
E_VM_CONSOLIDATION_NEEDED	snapshot-depth callback	Warning
E_VM_SNAPSHOT_OLD / _LARGE	snapshot callbacks	Info / Warning
E_VM_ORPHAN_DISK	datastore scanner callback	Info
E_VM_CPU_HIGH / _READY_HIGH	telemetry callback	Warning / Critical
E_VM_MEM_HIGH	telemetry callback	Warning
E_VM_DISK_LATENCY / _USAGE_HIGH	telemetry callback	Warning
E_VM_DRIVERS_OUTDATED / _TOOLS_NOT_RUNNING	telemetry callback	Info
E_VM_BOOT_FAILED / _MIGRATION_ABORTED	telemetry callback	Critical / Warning

Alarm lifecycle

                     detector fires
                          │
                          ▼
              ┌──────► triggered ──────┐
              │           │            │
              │  detector │ operator   │
              │  silent   │ ack        │
              │  >  N s   │            │
              │           ▼            │
              │      acknowledged      │
              │           │            │
              │           │ Reset to   │
              └──── auto- │ Green      │
                   resolve│            │
                          ▼            │
                       resolved ◄──────┘

auto_resolve_secs defaults to 48 hours and is overridable per code in alarms.per_code_overrides.

Tasks

TaskRegistry::start(create) returns a TaskHandle. Callers update progress via handle.progress(pct) and finalize with succeed or fail. Dropping the handle without finalizing marks the task cancelled — that’s the safety net for panics or early returns.

Every transition emits a matching event onto the shared EventBus. The persisted history lives in tasks and is purged after the configured retention window.

Events

Append-only. Severity / source / category are part of the row. The websocket route at /api/v1/observability/events/stream broadcasts new rows; subscribers can resume from a since_id cursor on reconnect, with a polling fallback when the websocket is unreachable for more than 5 seconds.

VM Edit Settings

PATCH /api/v1/vms/{uuid}/settings accepts a partial diff and generates a parent task plus one subtask per field changed. Hot-plug rules:

• VM powered off: every field is editable.
• VM running: only fields whose editability.editable=true may be changed. CPU / memory hot-add depend on the VM’s hot-add toggle and the guest’s support; disks add / remove permitted on virtio buses; CD/DVD media-swap always allowed; controllers, firmware, and BIOS-level options blocked.

Every successful field write emits a row into edit_settings_audit with the old and new values, the operator, and the parent task id.

Configuration

The canonical file is /etc/synvirt/observability.yaml. The example is shipped at crates/synvirt-observability/observability.example.yaml and covers every section: detectors, alarms, retention, edit-settings whitelist, websocket limits.

Operator surfaces

Three monitor routes ship in the dashboard:

• /monitor/issues — operator-grade Issues + Triggered Alarms view with Acknowledge and Reset to Green actions.
• /monitor/tasks — paginated task list with live progress.
• /monitor/events — append-only event log with severity / category filters.

The EditSettingsModal component is wired into the VM detail view (/vms/:uuid) and opened from the per-VM action menu.