Syslog & System Logs — design spec

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/specs/syslog-system-logs.md. Edit at the source, not here.

Syslog & System Logs — design spec

Date: 2026-06-27 Status: proposed — awaiting owner (Tony) review BEFORE any build Scope: Task 5 — remote syslog, local log files, in-UI log viewer, per-subsystem log levels, rotation, support/log bundle, audit→syslog forwarding Risk class: logging-critical (an SIEM the customer’s compliance depends on). Spec first; do not implement from this document unreviewed.

---

1. Problem statement

SynVirt’s logs live in systemd’s journal and a handful of ad-hoc files. An operator can tail one service’s journal from the Settings → Services panel, and can export a one-shot diagnostics bundle from the Console TUI. That is the whole story. There is:

• no remote syslog — nothing ships logs to a customer SIEM, which is a hard requirement for any regulated environment;
• no in-dashboard log viewer beyond a single-unit journal tail (no search, no follow, no severity filter, no per-file download);
• no per-subsystem runtime log-level control — verbosity is fixed at process start by RUST_LOG;
• no managed rotation/retention policy SynVirt owns (journald’s defaults apply, unmanaged);
• no per-VM QEMU log capture surfaced anywhere;
• no audit→syslog forwarding — the structured AuditStore never leaves the box.

The goal is a first-class logging subsystem that an operator configures from the dashboard: point logs at one or more SIEM targets, read and search any log in the UI, dial a noisy subsystem up to debug without a restart, set retention, and hand support a complete bundle.

---

2. What exists today (cited)

2.1 Diagnostics bundle (the precedent to reuse)

crates/console/src/debug/bundle.rs::generate() assembles a .tar.gz (flate2 + tar) at /var/cache/synvirt/bundles/synvirt-bundle-<ts>.tar.gz with a .sha256 companion. It collects ~20 artifacts via three helpers — append_file() (read a path), append_cmd() (run a command, capture stdout+stderr), append_shell() (bash -c): version + uname, /proc/cpuinfo, /proc/meminfo, /etc/synvirt/network.yaml (+ last-good + backups list), systemctl status synvirt-network, journalctl -u synvirt-network -n 2000, ovs-vsctl show, ip -d addr, ip route, resolvectl status, uplink sysctls, the legacy netplan slot, zpool status, systemctl --failed, journalctl -k -n 2000, journalctl -u synvirt-daemon -n 2000, journalctl -u libvirtd -n 2000. crates/console/src/screens/usb_export.rs reuses the same generate() and writes the bundle to a mounted USB device. This is the citation the support-bundle phase builds on — do not re-invent it.

2.2 Journald tail (the only in-UI log surface today)

GET /api/v1/settings/services/{unit}/journal?lines=<n> → crates/daemon/src/settings/api/services.rs::journal → synvirt-services::journal::tail shells out to journalctl --no-pager --output=json --unit=<unit> --lines=<n> and parse_journalctl_json maps each line to a JournalEntry (__REALTIME_TIMESTAMP → ms, PRIORITY → syslog 0–7, MESSAGE, _PID). DEFAULT_LINES = 200, MAX_LINES = 1000, returned as SettingsEnvelope<JournalTailView>. It is a one-shot pull — no follow/stream, single unit, no search, no download. The product already parses syslog priority (0=emerg … 7=debug) here; that mapping is the seed for severity filtering and RFC 5424 PRI computation.

2.3 Daemon tracing

crates/daemon/src/main.rs initializes tracing_subscriber with fmt::layer().pretty() + EnvFilter::try_from_default_env() falling back to "info" (reads RUST_LOG). Output is stdout, captured by systemd into the journal. Every public async fn carries #[tracing::instrument(skip_all)]. The level is fixed at startup — there is no runtime reload handle today.

2.4 Log file locations

crates/core/src/constants.rs: LOG_DIR_LINUX = "/var/log/synvirt". Known files: /var/log/synvirt/update-audit.jsonl (append-only JSONL from synvirt-update), the in-VM guest agent’s /var/log/synvirt-agent/agent.log, and the sled audit store at /var/lib/synvirt/settings-audit.db/. Daemon, libvirt, and synvirt-network logs are journal-only (no dedicated files). Per-VM QEMU stdout/stderr is not captured by SynVirt today — libvirt writes its own /var/log/libvirt/qemu/<domain>.log, but no SynVirt code reads or surfaces it (crates/daemon/src/console/recommend.rs only suggests the serial console after a crash). Net-new surface.

2.5 SSE streaming pattern (for tail/follow)

crates/daemon/src/api/events.rs::events_sse (GET /api/v1/events) streams DaemonEvent JSON over a tokio::sync::broadcast channel (capacity 64) wrapped in BroadcastStream, with a 15 s keep-alive and a lagged-event marker. crates/daemon/src/iso_detection/api.rs is a second instance of the same pattern. A follow/tail log viewer reuses this mechanism verbatim.

2.6 No syslog anywhere

grep for rsyslog / RFC5424 / :514 returns nothing but the priority-parsing comment in synvirt-services. Remote syslog is fully net-new.

2.7 Conventions

ApiError (api/error.rs, E_* codes, trace_id), settings router Router::new().route(...).merge(crate::settings::api::router()) (web/routes.rs), SettingsEnvelope<T>.

---

3. Proposed design

3.1 Crate placement — new synvirt-logging

One crate owns the logging domain (one-crate-per-domain). It is not a separate systemd service in v1 — it loads in-process inside the daemon (like synvirt-firewall), because the data it needs (the daemon’s own tracing reload handle, the broadcast bus) lives there. R3/R4 may later lift the forwarder into a sub-daemon if it must survive daemon restarts; flagged as an open question.

synvirt-logging/
  src/
    lib.rs          LoggingService façade
    sources.rs      LogSource registry (journal units, files, per-VM)
    journal.rs      reuse/extend synvirt-services journal reader (follow mode)
    files.rs        bounded file readers (tail, search, ranged download)
    levels.rs       per-subsystem level model + the reload bridge
    syslog/         remote forwarder
      mod.rs        SyslogForwarder: target set, per-target severity filter
      rfc5424.rs    RFC 5424 framing (PRI/version/timestamp/host/app/msg)
      transport.rs  UDP / TCP / TLS senders with backoff
    rotation.rs     size/time rotation + retention + gzip for SynVirt-owned files
    bundle.rs       thin re-export wrapper over console::debug::bundle (shared)
    config.rs       persisted config (see 3.7)
    error.rs        one thiserror enum, E_* discriminants

Persisted config: a YAML at /etc/synvirt/logging.yaml (the project’s “editable source of truth” convention, like network.yaml), cached in sled /var/lib/synvirt/logging.db/ for fast reads. The bundle helper should be promoted out of crates/console into a shared location (or synvirt-logging depends on a small extracted bundle crate) so both the TUI and the daemon call one implementation — do not fork generate().

3.2 Log source model

/// Anything the viewer/forwarder can read. The registry enumerates
/// these so the UI renders a source picker and the forwarder knows what
/// to ship.
pub enum LogSource {
    /// A systemd unit, read via journalctl (existing path).
    Journal { unit: String },          // synvirt-daemon, libvirtd, synvirt-network, …
    /// A SynVirt-owned file under /var/log/synvirt or elsewhere.
    File { path: PathBuf, label: String },  // update-audit.jsonl, installer log
    /// Per-VM QEMU stdout/stderr (net-new capture, see 3.6).
    VmQemu { domain: String },
    /// The structured audit store (3.5 forwarding).
    Audit,
}

The registry is seeded from a known set (daemon, libvirt, synvirt-network, synvirt-storaged, the data-plane sub-daemons, installer log, per-VM QEMU) and is data-driven so a new sub-daemon’s unit appears without a code change (mirror synvirt-services::allowlist’s discover-synvirt-*-units approach).

3.3 In-UI log viewer

• Pull (search/range/download): GET /api/v1/logs/{source}? with lines, since, until, min_severity, grep. For journal sources this extends the existing journalctl --output=json call with --since/--until/--grep/--priority; for files it is a bounded tail + in-process filter (cap bytes scanned, never load an unbounded file).
• Follow (tail -f): GET /api/v1/logs/{source}/stream SSE, reusing the events_sse broadcast pattern — a per-source journalctl -f --output=json (or tail -f) feeds a broadcast channel; the SSE handler filters by min_severity/grep server-side.
• Download: GET /api/v1/logs/{source}/download streams the raw source (ranged for files; journalctl dump for units) with Content-Disposition.

Severity filter and timestamps reuse the existing 0–7 priority mapping in synvirt-services. The UI must obey FRONTEND_PRINCIPLES (relative times with absolute tooltip, no raw ISO timestamps in the primary view).

3.4 Per-subsystem runtime log levels

Swap EnvFilter::try_from_default_env() in main.rs for a tracing_subscriber::reload::Layer wrapping the EnvFilter, storing the reload::Handle on AppState. Then:

GET  /api/v1/logs/levels        current effective filter, per-target
PUT  /api/v1/logs/levels        e.g. { "synvirt_daemon::api::migrator": "debug", "default": "info" }

PUT rebuilds the EnvFilter and calls handle.reload(...) — no restart. Levels persist to logging.yaml so they survive a restart, but are soft: an invalid target is a warning, not a 400 that drops the whole change. Sub-daemons (storaged, migratord, …) each expose the same reload over their synvirt-ipc UDS so the daemon can fan a level change out to the whole plane.

3.5 Remote syslog forwarder

pub struct SyslogTarget {
    pub id: String,
    pub host: String,
    pub port: u16,                 // default 514 udp/tcp, 6514 tls
    pub transport: Transport,      // Udp | Tcp | Tls
    pub min_severity: Severity,    // per-target filter (0..=7)
    pub format: SyslogFormat,      // Rfc5424 (default) | Rfc3164
    pub sources: SourceSelector,   // all | a subset of LogSource
    pub tls: Option<TlsConfig>,    // CA bundle, optional client cert
    pub enabled: bool,
}

The forwarder subscribes to a unified event stream (the daemon broadcast bus for DaemonEvent, plus a journal follower per selected unit, plus audit records) and for each record above a target’s min_severity frames an RFC 5424 message (<PRI>1 TIMESTAMP HOST APP PROCID MSGID [SD] MSG, PRI = facility×8 + severity, facility local0 default) and sends it over the target’s transport. TCP/TLS use octet-counted framing (RFC 6587); UDP is fire-and-forget. A bounded in-memory queue per target with exponential backoff prevents a dead SIEM from blocking the daemon; overflow drops oldest with a counter surfaced in the target’s health. Multiple targets each get an independent filter and queue.

3.6 Per-VM QEMU logs (net-new)

libvirt already writes /var/log/libvirt/qemu/<domain>.log. v1 simply reads it as a LogSource::VmQemu { domain } (resolve domain → file), giving the viewer and forwarder per-VM logs with no QEMU reconfiguration. A later phase can add an explicit <console>/<serial> log channel in the libvirt_renderer if the default file proves insufficient. Keep v1 read-only over the existing file to stay low-risk.

3.7 Rotation & retention

journald already rotates its own store; SynVirt does not manage that (out of scope, but expose journald’s SystemMaxUse as a read-only display + a single managed knob if Tony wants it). SynVirt does own rotation for its own files (/var/log/synvirt/*, per-VM capture copies if any): rotation.rs rotates by size or age, keeps N generations, gzips rotated files, deletes past retention. Policy in logging.yaml; defaults are conservative and soft-warned when an operator sets a value that would exhaust the disk, never hard-blocked.

3.8 Audit → syslog forwarding

synvirt-time::AuditStore records (actor, endpoint, outcome, before/after) map cleanly to structured-data RFC 5424 messages (facility local1, the security audit channel a SIEM expects). The forwarder treats LogSource::Audit like any other source: enable it on a target, every audit write is shipped. This is the single most valuable compliance feature and depends only on §3.5 + the existing store.

3.9 API surface

# Viewer
GET /api/v1/logs/sources                 registry
GET /api/v1/logs/{source}                pull (lines/since/until/min_severity/grep)
GET /api/v1/logs/{source}/stream         SSE follow
GET /api/v1/logs/{source}/download       raw download

# Levels
GET /api/v1/logs/levels
PUT /api/v1/logs/levels

# Remote syslog
GET    /api/v1/logs/syslog/targets
POST   /api/v1/logs/syslog/targets
PUT    /api/v1/logs/syslog/targets/{id}
DELETE /api/v1/logs/syslog/targets/{id}
POST   /api/v1/logs/syslog/targets/{id}/test     send a test message, report ack/error
GET    /api/v1/logs/syslog/targets/{id}/health   queue depth, last-send, drops

# Rotation / retention
GET/PUT /api/v1/logs/retention

# Support bundle (reuses console::debug::bundle::generate)
POST /api/v1/logs/bundle                 build, return id
GET  /api/v1/logs/bundle/{id}/download

Error codes: E_LOG_SOURCE_NOT_FOUND, E_LOG_RANGE_TOO_LARGE, E_SYSLOG_TARGET_EXISTS, E_SYSLOG_TARGET_NOT_FOUND, E_SYSLOG_UNREACHABLE, E_SYSLOG_TLS, E_LOG_LEVEL_INVALID (warning, not hard fail).

---

4. Phased rollout

• Phase 0 — viewer over what exists (safe). Generalize the journal reader to a LogSource registry, add search/since/until/severity to the pull endpoint, add per-file reads for the known /var/log/synvirt/* files, and the support-bundle endpoint that wraps the existing bundle::generate(). Pure read; no daemon-behavior change. Recommended first build.
• Phase 1 — follow + per-VM. SSE tail/follow over the broadcast pattern; LogSource::VmQemu reading the libvirt qemu log.
• Phase 2 — runtime levels. tracing_subscriber::reload handle on AppState; GET/PUT /logs/levels; fan-out to sub-daemons.
• Phase 3 — remote syslog. RFC 5424 framing + UDP/TCP/TLS transports
  • per-target filter/queue/health + test endpoint.
• Phase 4 — audit forwarding + retention. LogSource::Audit on targets; SynVirt-owned file rotation/retention.

---

5. Risks & open questions

1. Backpressure. A dead or slow SIEM must never stall the daemon. Per-target bounded queue + drop-oldest + backoff is mandatory; confirm the drop policy with Tony (drop vs disk-spool).
2. In-process vs sub-daemon forwarder. If the daemon restarts mid- incident, an in-process forwarder misses events during the bounce. Acceptable for v1 (journal is the durable backstop), but a synvirt-logd sub-daemon over synvirt-ipc may be warranted later — open decision.
3. TLS trust. Reuse synvirt-certs/rustls for client TLS rather than a second TLS stack; decide CA-pinning vs system trust for SIEM endpoints.
4. Sensitive data in logs. Forwarding audit before/after snapshots off-box may leak secrets (e.g. a config containing a key). Add a redaction pass before egress; flag as a hard requirement — this is the kind of data-egress gap that must not ship silently.
5. Runtime level reload + instrument. Confirm the reload::Layer composes with fmt::layer().pretty() without losing the existing #[instrument] spans.
6. Per-VM log location coupling. Reading /var/log/libvirt/qemu/<domain>.log couples to libvirt’s path; guard with existence checks and degrade to a warning if libvirt logging is off.
7. Bundle ownership. bundle::generate() lives in crates/console. Promoting it to a shared crate touches the TUI build — coordinate so both callers stay on one implementation (no fork).

---

6. What’s safe to build first

Phase 0. It only reads: turn the single-unit journal tail into a LogSource registry, add search/time-range/severity filtering and per-file reads for the existing /var/log/synvirt/* files, and wire the support-bundle endpoint straight onto the already-shipping console::debug::bundle::generate(). No egress, no daemon-behavior change, no new long-running tasks — yet it immediately gives operators a real in-dashboard log viewer and a one-click support bundle. The behavior-changing and compliance-sensitive pieces (runtime level reload, remote forwarding, audit egress with redaction) come after the read path is proven.