Observability / Storage / Networking Redesign — Phase 3 Implementation Blueprint

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/architecture/observability-redesign-phase3-blueprint.md. Edit at the source, not here.

Observability / Storage / Networking Redesign — Phase 3 Implementation Blueprint

Status: first-slice blueprint (owner-approved 2026-06-28). Companion spec: observability-storage-networking-redesign.md. Scope of this doc: the FIRST shippable slices — Phase 3 persistence, §7-1 detector wiring, §7-2 task/audit emission. Phase 4 collectors are scoped here but gated (see §4). This is a code-grounded plan: every file path is absolute, every DDL block is copy-paste ready, every call site carries a file:line.

The persistence layer is a single SQLite database at /var/lib/synvirt/observability.db, opened with WAL + foreign_keys=ON + synchronous=NORMAL (crates/synvirt-observability/src/db/mod.rs:44-58), wrapped in one Arc<Database> over a Mutex<rusqlite::Connection> — the daemon is the sole writer. Migrations are a const SCRIPTS: &[(i64, &str)] array gated by a schema_version table and applied once via execute_batch (mod.rs:119-172). All timestamps are TEXT RFC 3339 UTC via now_iso8601() (mod.rs:176). Batch inserts follow capacity.rs:72-101 (one txn, prepare once, loop, commit once, all rows share one now_iso8601()).

---

1. Phase 3 — Persistence

1.1 Design decision: wide tables, not tall EAV

Two sample-table shapes were drafted during analysis: a wide shape (one column per counter/rate, e.g. rx_bytes_abs, tx_bytes_abs, has_reset) and a tall EAV shape (direction, metric, value rows). This blueprint adopts the wide shape because it (a) matches the existing storage_capacity_samples / vm_disk_samples pattern that the dashboard already reads, (b) makes rate + anomaly columns first-class instead of join-derived, and (c) makes rollups a flat MIN/MAX/AVG/SUM over named columns rather than a pivot. The tall shape is rejected: it bloats row count by ~8x per interface and complicates P95/rollup.

1.2 Migration registry — crates/synvirt-observability/src/db/mod.rs

Add four entries to the SCRIPTS array (after the version-5 line at mod.rs:128):

const SCRIPTS: &[(i64, &str)] = &[
    (1, include_str!("migrations/0001_initial.sql")),
    (2, include_str!("migrations/0002_observability_e2e.sql")),
    (3, include_str!("migrations/0003_storage_capacity_history.sql")),
    (4, include_str!("migrations/0004_vm_disk_history.sql")),
    (5, include_str!("migrations/0005_vm_disk_per_target.sql")),
    (6, include_str!("migrations/0006_network_storage_samples.sql")),
    (7, include_str!("migrations/0007_rollup_tables.sql")),
    (8, include_str!("migrations/0008_task_steps_alarms_issues.sql")),
    (9, include_str!("migrations/0009_collector_health_entity_relationships.sql")),
];

No other change to run_migrations() — the schema_version gate and include_str! macro already handle idempotency and compile-time embedding. cargo build -p synvirt-observability is the only compilation step; there is no separate migration runner.

1.3 DDL — migrations/0006_network_storage_samples.sql (NEW, verbatim)

-- 0006_network_storage_samples — raw network + storage delta time series.
--
-- Append-only raw samples written by the daemon's network + storage delta
-- samplers (Phase 4 collectors) at a fixed cadence (default 5s/30s). Each row
-- carries the cumulative absolute counters (for wrap/reset detection), the
-- per-interval deltas, the derived rates, the real monotonic interval, and the
-- has_reset/has_wrap/has_gap anomaly flags (redesign §24.8). Rows are keyed by a
-- stable entity_id so a rename does not split the series. Raw retention is 48h
-- (redesign §5); the rollup tables (0007) carry the long tail.
--
-- Idempotency: gated on schema_version by run_migrations; runs once.

CREATE TABLE IF NOT EXISTS network_samples (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    sampled_at      TEXT    NOT NULL,            -- RFC 3339, shared per tick
    interface_name  TEXT    NOT NULL,            -- kernel/OVS name
    interface_type  TEXT    NOT NULL CHECK (interface_type IN
                        ('physical','bond','vlan','vswitch_port','guest_nic')),
    entity_id       TEXT    NOT NULL,            -- stable id (survives rename)
    rx_bytes_abs    INTEGER NOT NULL,
    rx_packets_abs  INTEGER NOT NULL,
    rx_errors_abs   INTEGER NOT NULL,
    rx_dropped_abs  INTEGER NOT NULL,
    tx_bytes_abs    INTEGER NOT NULL,
    tx_packets_abs  INTEGER NOT NULL,
    tx_errors_abs   INTEGER NOT NULL,
    tx_dropped_abs  INTEGER NOT NULL,
    rx_bytes_delta  INTEGER,                     -- NULL on first sample / after gap
    rx_packets_delta INTEGER,
    tx_bytes_delta  INTEGER,
    tx_packets_delta INTEGER,
    rx_bytes_rate_bps REAL,
    tx_bytes_rate_bps REAL,
    rx_pps          REAL,
    tx_pps          REAL,
    interval_secs   REAL,                        -- real monotonic interval
    has_reset       INTEGER NOT NULL DEFAULT 0,  -- counter went backwards
    has_wrap        INTEGER NOT NULL DEFAULT 0,  -- 64-bit wrap detected
    has_gap         INTEGER NOT NULL DEFAULT 0,  -- missed tick(s)
    collector_name  TEXT    NOT NULL,
    sample_sequence INTEGER NOT NULL             -- per-collector monotonic seq
);

CREATE INDEX IF NOT EXISTS network_samples_interface_time_idx
    ON network_samples(interface_name, sampled_at);
CREATE INDEX IF NOT EXISTS network_samples_entity_time_idx
    ON network_samples(entity_id, sampled_at);
CREATE INDEX IF NOT EXISTS network_samples_type_time_idx
    ON network_samples(interface_type, sampled_at);

CREATE TABLE IF NOT EXISTS storage_samples (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    sampled_at      TEXT    NOT NULL,
    storage_target  TEXT    NOT NULL,            -- vdev path / zvol / disk / adapter id
    storage_type    TEXT    NOT NULL CHECK (storage_type IN
                        ('vdev','zvol','disk','adapter')),
    entity_id       TEXT    NOT NULL,
    read_ops_abs    INTEGER NOT NULL,
    write_ops_abs   INTEGER NOT NULL,
    read_bytes_abs  INTEGER NOT NULL,
    write_bytes_abs INTEGER NOT NULL,
    read_ops_delta  INTEGER,
    write_ops_delta INTEGER,
    read_bytes_delta INTEGER,
    write_bytes_delta INTEGER,
    read_latency_ms REAL,
    write_latency_ms REAL,
    p95_read_latency_ms REAL,
    p95_write_latency_ms REAL,
    read_iops       REAL,
    write_iops      REAL,
    read_throughput_bps REAL,
    write_throughput_bps REAL,
    interval_secs   REAL,
    queue_depth     INTEGER,
    utilization_pct REAL,
    has_reset       INTEGER NOT NULL DEFAULT 0,
    has_wrap        INTEGER NOT NULL DEFAULT 0,
    has_gap         INTEGER NOT NULL DEFAULT 0,
    is_healthy      INTEGER NOT NULL DEFAULT 1,
    error_rate_pct  REAL,
    collector_name  TEXT    NOT NULL,
    sample_sequence INTEGER NOT NULL
);

CREATE INDEX IF NOT EXISTS storage_samples_target_time_idx
    ON storage_samples(storage_target, sampled_at);
CREATE INDEX IF NOT EXISTS storage_samples_entity_time_idx
    ON storage_samples(entity_id, sampled_at);
-- Partial index for the "show me broken disks" query.
CREATE INDEX IF NOT EXISTS storage_samples_health_idx
    ON storage_samples(is_healthy, sampled_at) WHERE is_healthy = 0;

1.4 DDL — migrations/0007_rollup_tables.sql (NEW, verbatim)

-- 0007_rollup_tables — 1-minute and 15-minute rollups for network + storage.
--
-- The rollup scheduler (daemon, 60s tick) aggregates raw network_samples /
-- storage_samples into 1-min buckets, then 1-min into 15-min buckets, keyed by
-- floor(sampled_at) to the bucket and grouped by entity. Retention (redesign §5):
-- raw 48h, 1-min 30d, 15-min 13mo. The UNIQUE index per (bucket, entity, ...)
-- makes the rollup an idempotent INSERT OR REPLACE re-aggregation of the window.
--
-- Idempotency: gated on schema_version by run_migrations; runs once.

CREATE TABLE IF NOT EXISTS network_samples_1min_rollup (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    rollup_time_utc TEXT    NOT NULL,            -- floor(sampled_at) to minute
    interface_name  TEXT    NOT NULL,
    interface_type  TEXT    NOT NULL,
    entity_id       TEXT    NOT NULL,
    rx_bps_min REAL, rx_bps_max REAL, rx_bps_avg REAL, rx_bps_p95 REAL,
    tx_bps_min REAL, tx_bps_max REAL, tx_bps_avg REAL, tx_bps_p95 REAL,
    rx_pps_min REAL, rx_pps_max REAL, rx_pps_avg REAL,
    tx_pps_min REAL, tx_pps_max REAL, tx_pps_avg REAL,
    rx_bytes_total   INTEGER, tx_bytes_total   INTEGER,
    rx_packets_total INTEGER, tx_packets_total INTEGER,
    sample_count INTEGER NOT NULL,
    gap_count    INTEGER NOT NULL DEFAULT 0,
    reset_count  INTEGER NOT NULL DEFAULT 0
);
CREATE UNIQUE INDEX IF NOT EXISTS network_1min_rollup_uq
    ON network_samples_1min_rollup(rollup_time_utc, interface_name, entity_id);
CREATE INDEX IF NOT EXISTS network_1min_rollup_entity_idx
    ON network_samples_1min_rollup(entity_id, rollup_time_utc);

CREATE TABLE IF NOT EXISTS network_samples_15min_rollup (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    rollup_time_utc TEXT    NOT NULL,            -- floor to 15-min bucket
    interface_name  TEXT    NOT NULL,
    interface_type  TEXT    NOT NULL,
    entity_id       TEXT    NOT NULL,
    rx_bps_min REAL, rx_bps_max REAL, rx_bps_avg REAL, rx_bps_p95 REAL,
    tx_bps_min REAL, tx_bps_max REAL, tx_bps_avg REAL, tx_bps_p95 REAL,
    rx_pps_min REAL, rx_pps_max REAL, rx_pps_avg REAL,
    tx_pps_min REAL, tx_pps_max REAL, tx_pps_avg REAL,
    rx_bytes_total   INTEGER, tx_bytes_total   INTEGER,
    rx_packets_total INTEGER, tx_packets_total INTEGER,
    sample_count INTEGER NOT NULL,
    gap_count    INTEGER NOT NULL DEFAULT 0,
    reset_count  INTEGER NOT NULL DEFAULT 0
);
CREATE UNIQUE INDEX IF NOT EXISTS network_15min_rollup_uq
    ON network_samples_15min_rollup(rollup_time_utc, interface_name, entity_id);
CREATE INDEX IF NOT EXISTS network_15min_rollup_entity_idx
    ON network_samples_15min_rollup(entity_id, rollup_time_utc);

CREATE TABLE IF NOT EXISTS storage_samples_1min_rollup (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    rollup_time_utc TEXT    NOT NULL,
    storage_target  TEXT    NOT NULL,
    storage_type    TEXT    NOT NULL,
    entity_id       TEXT    NOT NULL,
    read_iops_min REAL, read_iops_max REAL, read_iops_avg REAL,
    write_iops_min REAL, write_iops_max REAL, write_iops_avg REAL,
    read_bps_min REAL, read_bps_max REAL, read_bps_avg REAL,
    write_bps_min REAL, write_bps_max REAL, write_bps_avg REAL,
    latency_ms_min REAL, latency_ms_max REAL, latency_ms_avg REAL, latency_ms_p95 REAL,
    read_ops_total   INTEGER, write_ops_total   INTEGER,
    read_bytes_total INTEGER, write_bytes_total INTEGER,
    queue_depth_min REAL, queue_depth_max REAL, queue_depth_avg REAL,
    utilization_pct_min REAL, utilization_pct_max REAL, utilization_pct_avg REAL,
    error_events_count INTEGER NOT NULL DEFAULT 0,
    sample_count INTEGER NOT NULL,
    gap_count    INTEGER NOT NULL DEFAULT 0,
    reset_count  INTEGER NOT NULL DEFAULT 0
);
CREATE UNIQUE INDEX IF NOT EXISTS storage_1min_rollup_uq
    ON storage_samples_1min_rollup(rollup_time_utc, storage_target, entity_id);
CREATE INDEX IF NOT EXISTS storage_1min_rollup_entity_idx
    ON storage_samples_1min_rollup(entity_id, rollup_time_utc);

CREATE TABLE IF NOT EXISTS storage_samples_15min_rollup (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    rollup_time_utc TEXT    NOT NULL,
    storage_target  TEXT    NOT NULL,
    storage_type    TEXT    NOT NULL,
    entity_id       TEXT    NOT NULL,
    read_iops_min REAL, read_iops_max REAL, read_iops_avg REAL,
    write_iops_min REAL, write_iops_max REAL, write_iops_avg REAL,
    read_bps_min REAL, read_bps_max REAL, read_bps_avg REAL,
    write_bps_min REAL, write_bps_max REAL, write_bps_avg REAL,
    latency_ms_min REAL, latency_ms_max REAL, latency_ms_avg REAL, latency_ms_p95 REAL,
    read_ops_total   INTEGER, write_ops_total   INTEGER,
    read_bytes_total INTEGER, write_bytes_total INTEGER,
    queue_depth_min REAL, queue_depth_max REAL, queue_depth_avg REAL,
    utilization_pct_min REAL, utilization_pct_max REAL, utilization_pct_avg REAL,
    error_events_count INTEGER NOT NULL DEFAULT 0,
    sample_count INTEGER NOT NULL,
    gap_count    INTEGER NOT NULL DEFAULT 0,
    reset_count  INTEGER NOT NULL DEFAULT 0
);
CREATE UNIQUE INDEX IF NOT EXISTS storage_15min_rollup_uq
    ON storage_samples_15min_rollup(rollup_time_utc, storage_target, entity_id);
CREATE INDEX IF NOT EXISTS storage_15min_rollup_entity_idx
    ON storage_samples_15min_rollup(entity_id, rollup_time_utc);

1.5 DDL — migrations/0008_task_steps_alarms_issues.sql (NEW, verbatim)

-- 0008_task_steps_alarms_issues — nested task steps, rule-based alarms,
-- deduplicated issue occurrences (redesign §4 taxonomy, §8 alarm engine).
--
-- task_steps:        per-task workflow stages (a `tasks` row gets N steps).
-- alarm_rules:       operator-editable threshold/compound/state_change rules.
-- alarm_instances:   per-entity activation of a rule.
-- issue_occurrences: deduplicated issue lifecycle (open→ack→silenced→resolved→reopened).
--
-- FKs are DEFERRABLE INITIALLY DEFERRED to support batch transactions.
-- Idempotency: gated on schema_version by run_migrations; runs once.

CREATE TABLE IF NOT EXISTS task_steps (
    id          TEXT PRIMARY KEY,               -- UUID
    task_id     TEXT NOT NULL REFERENCES tasks(id) ON DELETE CASCADE,
    step_index  INTEGER NOT NULL,
    kind        TEXT NOT NULL,
    description TEXT NOT NULL,
    state       TEXT NOT NULL CHECK (state IN
                    ('pending','running','success','error','skipped')),
    started_at  TEXT,
    ended_at    TEXT,
    error_code  TEXT,
    error_msg   TEXT,
    duration_ms INTEGER
);
CREATE INDEX IF NOT EXISTS task_steps_task_idx ON task_steps(task_id, step_index);

CREATE TABLE IF NOT EXISTS alarm_rules (
    id              TEXT PRIMARY KEY,
    kind            TEXT NOT NULL CHECK (kind IN ('threshold','compound','state_change')),
    is_enabled      INTEGER NOT NULL DEFAULT 1,
    metric          TEXT,
    operator        TEXT,                        -- gt/gte/lt/lte/eq/ne
    threshold_value TEXT,
    threshold_unit  TEXT,
    hysteresis_value TEXT,
    hysteresis_unit TEXT,
    trigger_severity TEXT NOT NULL CHECK (trigger_severity IN ('info','warning','critical')),
    issue_code      TEXT NOT NULL
                        REFERENCES issues_catalog(code) DEFERRABLE INITIALLY DEFERRED,
    applies_to_type TEXT NOT NULL CHECK (applies_to_type IN
                        ('host','vm','pool','vdev','virtswitch','interface','disk')),
    antiflooding_group TEXT,
    auto_resolve_secs  INTEGER NOT NULL DEFAULT 172800,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL,
    created_by_user TEXT
);
CREATE INDEX IF NOT EXISTS alarm_rules_applies_idx ON alarm_rules(applies_to_type);
CREATE INDEX IF NOT EXISTS alarm_rules_issue_idx   ON alarm_rules(issue_code);

CREATE TABLE IF NOT EXISTS alarm_instances (
    id               TEXT PRIMARY KEY,
    rule_id          TEXT NOT NULL
                        REFERENCES alarm_rules(id) ON DELETE CASCADE
                        DEFERRABLE INITIALLY DEFERRED,
    target_type      TEXT NOT NULL,
    target_id        TEXT NOT NULL,
    target_name      TEXT NOT NULL,
    is_active        INTEGER NOT NULL DEFAULT 0,
    triggered_count  INTEGER NOT NULL DEFAULT 0,
    last_triggered_at TEXT,
    last_value       TEXT
);
CREATE INDEX IF NOT EXISTS alarm_instances_rule_idx   ON alarm_instances(rule_id);
CREATE INDEX IF NOT EXISTS alarm_instances_target_idx ON alarm_instances(target_type, target_id);
CREATE UNIQUE INDEX IF NOT EXISTS alarm_instances_rule_target_uq
    ON alarm_instances(rule_id, target_type, target_id);

CREATE TABLE IF NOT EXISTS issue_occurrences (
    id               TEXT PRIMARY KEY,
    issue_code       TEXT NOT NULL
                        REFERENCES issues_catalog(code) DEFERRABLE INITIALLY DEFERRED,
    target_type      TEXT NOT NULL,
    target_id        TEXT NOT NULL,
    target_name      TEXT NOT NULL,
    state            TEXT NOT NULL CHECK (state IN
                        ('open','acknowledged','silenced','resolved','reopened')),
    first_detected_at TEXT NOT NULL,
    last_seen_at     TEXT NOT NULL,
    resolved_at      TEXT,
    occurrence_count INTEGER NOT NULL DEFAULT 1,
    ack_user         TEXT,
    ack_at           TEXT,
    ack_note         TEXT,
    resolved_user    TEXT,
    resolved_note    TEXT,
    is_silenced      INTEGER NOT NULL DEFAULT 0,
    silence_until    TEXT,
    silence_reason   TEXT
);
CREATE INDEX IF NOT EXISTS issue_occurrences_target_idx
    ON issue_occurrences(target_type, target_id);
CREATE INDEX IF NOT EXISTS issue_occurrences_state_idx ON issue_occurrences(state);
-- One live row per (code, target) — dedup uses state='reopened' + occurrence_count,
-- it does NOT insert a second active row (see risks).
CREATE UNIQUE INDEX IF NOT EXISTS issue_occurrences_active_unique
    ON issue_occurrences(issue_code, target_type, target_id)
    WHERE state IN ('open','acknowledged','reopened');

1.6 DDL — migrations/0009_collector_health_entity_relationships.sql (NEW, verbatim)

-- 0009_collector_health_entity_relationships — sampler self-observation +
-- the entity correlation graph (redesign §12.3 root-cause).
--
-- collector_health:    one UPSERT row per collector_name (per host). The daemon
--                      writes it after each sampler cycle. UI alerts if
--                      health_check_at is older than 2x the sample interval.
-- entity_relationships: VM↔zvol↔pool↔device, vNIC↔port↔vSwitch↔uplink edges
--                      for impact ranking.
--
-- Idempotency: gated on schema_version by run_migrations; runs once.

CREATE TABLE IF NOT EXISTS collector_health (
    id                TEXT PRIMARY KEY,          -- collector_name (per host)
    last_success_at   TEXT,
    last_failure_at   TEXT,
    last_failure_code TEXT,
    last_failure_msg  TEXT,
    cycle_duration_ms REAL,
    min_cycle_ms      REAL,
    max_cycle_ms      REAL,
    avg_cycle_ms      REAL,
    samples_collected INTEGER NOT NULL DEFAULT 0,
    samples_dropped   INTEGER NOT NULL DEFAULT 0,
    samples_backpressure INTEGER NOT NULL DEFAULT 0,
    queue_depth       INTEGER NOT NULL DEFAULT 0,
    queue_max_depth   INTEGER NOT NULL DEFAULT 0,
    write_batch_size  INTEGER,
    write_time_ms     REAL,
    write_time_max_ms REAL,
    clock_skew_ms     REAL,
    source_unavailable INTEGER NOT NULL DEFAULT 0,
    source_unavailable_reason TEXT,
    is_healthy        INTEGER NOT NULL DEFAULT 1,
    health_check_at   TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS entity_relationships (
    id                 INTEGER PRIMARY KEY AUTOINCREMENT,
    source_entity_type TEXT NOT NULL,
    source_entity_id   TEXT NOT NULL,
    source_entity_name TEXT NOT NULL,
    target_entity_type TEXT NOT NULL,
    target_entity_id   TEXT NOT NULL,
    target_entity_name TEXT NOT NULL,
    relation_kind      TEXT NOT NULL CHECK (relation_kind IN
                          ('contains','uses','provides','connects_through')),
    established_at     TEXT NOT NULL,
    active             INTEGER NOT NULL DEFAULT 1,
    metadata_json      TEXT
);
CREATE INDEX IF NOT EXISTS entity_relationships_source_idx
    ON entity_relationships(source_entity_type, source_entity_id);
CREATE INDEX IF NOT EXISTS entity_relationships_target_idx
    ON entity_relationships(target_entity_type, target_entity_id);
CREATE INDEX IF NOT EXISTS entity_relationships_kind_idx
    ON entity_relationships(relation_kind, active);

1.7 Batch writer

New repository modules mirror capacity.rs exactly (one txn, prepare once, loop, commit once, all rows share one now_iso8601()). Add to crates/synvirt-observability/src/db/mod.rs:7-12 the pub mod lines and to src/lib.rs:33-40 nothing (db is already public):

New file	Public surface
src/db/network.rs	NetworkSampleInsert struct; insert_network_samples(&[…]) -> Result<usize>, list_network_samples(entity_id, since, limit), purge_network_samples_older_than(secs)
src/db/storage_samples.rs	StorageSampleInsert struct; insert_storage_samples(&[…]) -> Result<usize>, list_storage_samples(…), purge_storage_samples_older_than(secs)
src/db/rollups.rs	RollupAggregator<'db>; rollup_network_1min(window), rollup_network_15min(window), rollup_storage_1min(window), rollup_storage_15min(window) — each -> Result<usize>, idempotent re-aggregation via INSERT OR REPLACE keyed on the UNIQUE rollup index
src/db/collector_health.rs	CollectorHealthUpsert struct; upsert_collector_health(&CollectorHealthUpsert), list_collector_health(), purge_stale_collectors(grace_secs)
src/db/entity_graph.rs	EntityEdgeInsert; upsert_entity_edge(&…), list_edges_from(type,id), list_edges_to(type,id)

Insert template (network, mirrors capacity.rs:72-101):

pub fn insert_network_samples(&self, samples: &[NetworkSampleInsert]) -> Result<usize> {
    if samples.is_empty() { return Ok(0); }
    let now = super::now_iso8601();
    self.with_conn_mut(|c| {
        let tx = c.transaction()?;
        {
            let mut stmt = tx.prepare(
                "INSERT INTO network_samples
                   (sampled_at, interface_name, interface_type, entity_id,
                    rx_bytes_abs, rx_packets_abs, rx_errors_abs, rx_dropped_abs,
                    tx_bytes_abs, tx_packets_abs, tx_errors_abs, tx_dropped_abs,
                    rx_bytes_delta, rx_packets_delta, tx_bytes_delta, tx_packets_delta,
                    rx_bytes_rate_bps, tx_bytes_rate_bps, rx_pps, tx_pps,
                    interval_secs, has_reset, has_wrap, has_gap,
                    collector_name, sample_sequence)
                 VALUES (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14,?15,?16,
                         ?17,?18,?19,?20,?21,?22,?23,?24,?25,?26)",
            )?;
            for s in samples { stmt.execute(/* params, now first */)?; }
        }
        tx.commit()?;
        Ok(samples.len())
    })
}

The “BulkWriter” abstraction is a thin owner of a Vec<SampleVariant> plus a flush that dispatches to the per-table insert_*_samples — not a new threading model. Batching is per-cycle (the sampler accumulates one tick’s rows, then flushes once), never one txn per metric. Keep it inside synvirt-observability so the daemon stays the sole linker.

1.8 Rollup loop

A single tokio task in the daemon (crates/daemon/src/main.rs, spawned right after the capacity/vm_disk samplers at main.rs:~1467-1495). It is single-threaded by construction (one tokio::spawn, sequential awaits) so 1-min and 15-min passes never contend on the same raw table:

fn spawn_rollup_scheduler(db: Arc<Database>, cfg: RetentionConfig) -> tokio::task::JoinHandle<()> {
    tokio::spawn(async move {
        let interval = Duration::from_secs(60);
        let window   = Duration::from_secs(300); // re-aggregate last 5 min (idempotent)
        loop {
            tokio::time::sleep(interval).await;
            let agg = db::rollups::RollupAggregator::new(&db);
            for (label, r) in [
                ("net.1min",  agg.rollup_network_1min(window)),
                ("net.15min", agg.rollup_network_15min(Duration::from_secs(900))),
                ("sto.1min",  agg.rollup_storage_1min(window)),
                ("sto.15min", agg.rollup_storage_15min(Duration::from_secs(900))),
            ] {
                match r { Ok(n) => tracing::debug!(rows = n, "{label} rollup"),
                          Err(e) => tracing::warn!(error = %e, "{label} rollup failed") }
            }
            // Age-based purge per tier (see retention).
            let _ = db.purge_network_samples_older_than(cfg.raw_samples_secs);
            let _ = db.purge_storage_samples_older_than(cfg.raw_samples_secs);
            // (rollup-table purges run on their own slower cadence)
        }
    })
}

P95 is computed per bucket only when sample_count >= 20 (else NULL) to avoid noise on short windows. Rollup re-aggregation is idempotent because the UNIQUE bucket index turns a re-run of the same window into INSERT OR REPLACE, so an overlapping window never double-counts.

1.9 Retention tiers — crates/synvirt-observability/src/config.rs

Extend RetentionConfig (currently config.rs:162-201) with the three new tiers + the collector grace period. Defaults mirror redesign §5:

// add to RetentionConfig
#[serde(default = "default_raw_samples_secs")]      pub raw_samples_secs: i64,        // 172_800  (48h)
#[serde(default = "default_rollup_1min_secs")]      pub rollup_1min_secs: i64,        // 2_592_000 (30d)
#[serde(default = "default_rollup_15min_secs")]     pub rollup_15min_secs: i64,       // 33_696_000 (13mo)
#[serde(default = "default_collector_grace_secs")]  pub collector_grace_secs: i64,    // 86_400 (24h stale)

Existing tiers stay: events_days=30, tasks_days=90, resolved_alarms_days=180, tick_secs=3600. The purge of the longer tiers (1-min @ 30d, 15-min @ 13mo) runs on the existing retention purger cadence, not the 60s rollup loop.

1.10 Files to touch — Phase 3

File	Change
crates/synvirt-observability/src/db/mod.rs	SCRIPTS += versions 6-9; pub mod network/storage_samples/rollups/collector_health/entity_graph
crates/synvirt-observability/src/db/migrations/0006_network_storage_samples.sql	NEW (§1.3)
crates/synvirt-observability/src/db/migrations/0007_rollup_tables.sql	NEW (§1.4)
crates/synvirt-observability/src/db/migrations/0008_task_steps_alarms_issues.sql	NEW (§1.5)
crates/synvirt-observability/src/db/migrations/0009_collector_health_entity_relationships.sql	NEW (§1.6)
crates/synvirt-observability/src/db/network.rs	NEW batch writer (§1.7)
crates/synvirt-observability/src/db/storage_samples.rs	NEW batch writer
crates/synvirt-observability/src/db/rollups.rs	NEW RollupAggregator
crates/synvirt-observability/src/db/collector_health.rs	NEW upsert/list/purge
crates/synvirt-observability/src/db/entity_graph.rs	NEW edge upsert/list
crates/synvirt-observability/src/config.rs	RetentionConfig += 4 tier fields
crates/daemon/src/main.rs	spawn spawn_rollup_scheduler after the existing samplers (main.rs:~1495)

Phase 3 adds 15 new tables (network_samples, storage_samples, 4 rollup tables, task_steps, alarm_rules, alarm_instances, issue_occurrences, collector_health, entity_relationships). No REST endpoints — Phase 3 is persistence + writers only; read models are Phase 5, so the OpenAPI snapshot is untouched.

---

2. §7-1 — Wire the 24 stub detectors

The registry already ships 40 catalog rows; 7 are real (read /proc + subprocess, no callback) and the remaining 33 are stubs that today close over an empty-vec/false supplier (crates/daemon/src/main.rs:2048-2071). The 33 stubs collapse to 24 distinct detector IDs once heartbeat reuse and the 10 VM-resource detectors that share one VmTelemetryFn are counted by supplier — but to be unambiguous the table below lists every stub detector code, grouped by the single supplier closure that unblocks it. Wiring = replace the stub closure in main.rs:2048-2071 with a real one that calls the named daemon subsystem; the detector constructors and register() calls below them are unchanged.

Callback type signatures live in crates/synvirt-observability/src/issues/callbacks.rs (already defined: PoolHealthFn, BridgeStatusFn, CertExpiryFn, BmcSensorsFn, VmTelemetryFn, OrphanDiskFn, KernelPanicFn) plus the module-local ListFn / PoolListFn aliases.

2.1 Supplier → detector map

Supplier closure (type)	Real source / how the daemon fills it	Stub detector codes it unblocks	Detector module
PoolListFn	storage_client::fetch_pool_entries over the synvirt-storaged UDS (/v1/storage/pools); map PoolEntry.used_pct	E_POOL_USAGE_WARN, E_POOL_USAGE_CRIT	host_pool_usage.rs
PoolHealthFn	synvirt-storaged pool detail — needs status + resilver_active + last_scrub_at (add /v1/storage/pools/{id}/health if absent)	E_POOL_DEGRADED, E_POOL_SCRUB_OVERDUE, E_POOL_RESILVER_ACTIVE	host_storage.rs
BridgeStatusFn	in-process AppState::network_controller.list_bridges() (wrap in spawn_blocking if sync); map NetworkBridge → BridgeStatus incl. OVS reconcile state, bond members, MTU, VLANs	E_OVS_BRIDGE_MISSING, E_NETWORK_UPLINK_DEGRADED, E_NETWORK_UPLINK_LOST, E_VLAN_TRUNK_MISMATCH, E_MTU_MISMATCH	host_network.rs
CertExpiryFn	synvirt-certs cert loader — parse /etc/synvirt/tls/cert.pem (daemon) + console-ticket + ACME-managed certs; days_until_expiry	E_CERT_EXPIRY_WARN, E_CERT_EXPIRY_CRIT	host_cert.rs
BmcSensorsFn	synvirt-bmc via ipmitool sdr list (thresholds) + sel info (capacity); arc-cache result for the detector interval	E_HOST_HW_FAN, E_HOST_HW_TEMP, E_HOST_HW_PSU, E_HOST_HW_BATTERY, E_HOST_IPMI_SEL_FULL	host_hardware.rs
KernelPanicFn	journalctl -S '1 day ago' --priority err scanned for panic/BUG signature; cache + rescan on interval	E_KERNEL_PANIC_SEEN	host_kernel.rs
vm_heartbeat::ListFn	guest-tools supervisor / telemetry persistor last_heartbeat (or vsock CID→UUID active-connection timestamp)	E_VM_HEARTBEAT_STALE	vm_heartbeat.rs
VmTelemetryFn (heartbeat reuse)	same telemetry snapshot, last_heartbeat threshold	E_VM_GUEST_AGENT_DOWN	vm_guest_agent.rs
vm_consolidation::ListFn	snapshot-chain depth per VM via synvirt-storaged snapshot API (preferred) or zfs list -rH (needs storaged endpoint)	E_VM_CONSOLIDATION_NEEDED	vm_consolidation.rs
vm_snapshot_old::ListFn	oldest-snapshot timestamp per VM (same snapshot listing)	E_VM_SNAPSHOT_OLD	vm_snapshot_old.rs
VmTelemetryFn (main)	guest-tools telemetry persistor per-VM SQLite (5-min avg CPU/mem, top-procs latency, inventory tools/version) + heartbeat + boot/migration failure events	E_VM_CPU_HIGH, E_VM_CPU_READY_HIGH, E_VM_MEM_HIGH, E_VM_DISK_LATENCY, E_VM_DISK_USAGE_HIGH, E_VM_TOOLS_NOT_RUNNING, E_VM_DRIVERS_OUTDATED, E_VM_BOOT_FAILED, E_VM_MIGRATION_ABORTED, E_VM_SNAPSHOT_LARGE	vm_resources.rs
OrphanDiskFn	scan each pool’s <pool>/vms dataset + legacy /var/lib/libvirt/images, cross-check against the libvirt VM registry	E_VM_ORPHAN_DISK	vm_orphan.rs

Total: 33 stub detector codes across 12 supplier closures → wiring 12 closures lights up all of them. (The “24 stubs” count = the distinct non-VM-resource detectors plus the single VM-telemetry supplier; either way the deliverable is the 12 supplier closures above.)

2.2 The 7 real detectors (no wiring needed)

E_HOST_CPU_HIGH (host_cpu.rs, /proc/stat 250ms delta), E_HOST_MEM_HIGH (host_mem.rs), E_HOST_LOAD_HIGH (host_load.rs), E_HOST_SWAP_USED (host_swap.rs), E_NTP_DRIFT (host_ntp_drift.rs, chronyc), E_DNS_UNREACHABLE (host_dns.rs). The 7th, E_DAEMON_LOG_ERROR_RATE (host_log_rate.rs), ships a placeholder Arc::new(|_| Box::pin(async { Ok(0u64) })) at main.rs:2040 — it needs a tracing-subscriber eventcount hook to become real, tracked as an open question (it is NOT one of the 24 stubs, it is a degraded real detector).

2.3 Files to touch — §7-1

File	Change
crates/daemon/src/main.rs	Replace the 12 stub closures at main.rs:2048-2071 one supplier at a time; each new closure calls into storage_client / network_controller / guest-tools cache / ipmitool / journalctl and returns the populated snapshot
crates/daemon/src/api/storage_client.rs	reuse fetch_pool_entries (storage_client.rs:30); add a pool-health fetch if /v1/storage/pools lacks resilver_active/last_scrub_at
crates/daemon/src/api/network.rs	expose network_controller.list_bridges() snapshot for BridgeStatusFn
crates/daemon/src/api/observability/state.rs	hold any new supplier handles needed at registry-spawn time (ObservabilityState::open, state.rs:43)
crates/synvirt-guest-tools/src/telemetry/persistor.rs	expose a per-VM telemetry snapshot read for VmTelemetryFn / heartbeat
(read-only) crates/synvirt-observability/src/issues/callbacks.rs	callback signatures already defined; no change unless a field is missing

Recommended staging: deploy PoolHealth first (storaged is already a clean UDS), then BridgeStatus, CertExpiry, BmcSensors, KernelPanic, then the VM-telemetry suppliers last (they touch the most surface). Each supplier is independently testable: replace the stub, run one tick_once(), confirm alarms populate.

---

3. §7-2 — TaskRegistry + Audit emission

TaskRegistry + AuditStore are deployed but unwired across ~40 long-running call sites. The pattern is fixed (crates/daemon/src/api/task_helper.rs):

let task = crate::api::task_helper::start_with_actor(
    &state, Some(&actor.0), "<kind>", "<target_type>", "<target_id>", "<target_name>");
// ... async work producing Result<Response, ApiError> ...
crate::api::task_helper::settle(task, &result);

start_with_actor already no-ops when observability is unavailable, and settle maps Ok→succeed / Err→fail(code,msg) — both failure and success paths are covered. Each handler also needs Extension(actor): Extension<AuthUser> added to its signature to attribute the task. Task kinds follow the existing dot taxonomy <target_type>.<category>.<verb>.

3.1 TaskRegistry call-site inventory

File	Line (approx)	Handler	Add — task_helper::start_with_actor(..., kind, target_type, target_id, target_name)
crates/daemon/src/api/network.rs	308	set_vmnic_alias	"network.vmnic.set_alias", "network", &nic, &nic
crates/daemon/src/api/network.rs	339	forget_vmnic	"network.vmnic.forget", "network", &nic, &nic
crates/daemon/src/api/network.rs	402	create_vswitch	"network.vswitch.create", "network", "", &name
crates/daemon/src/api/network.rs	432	delete_vswitch	"network.vswitch.delete", "network", &name, &name
crates/daemon/src/api/network.rs	468	set_vswitch_uplinks	"network.vswitch.set_uplinks", "network", &name, &name
crates/daemon/src/api/network.rs	507	set_vswitch_mtu	"network.vswitch.set_mtu", "network", &name, &name
crates/daemon/src/api/network.rs	576	create_network	"network.network.create", "network", "", &name
crates/daemon/src/api/network.rs	609	delete_network	"network.network.delete", "network", &name, &name
crates/daemon/src/api/network.rs	653	update_network	"network.network.update", "network", &name, &name
crates/daemon/src/api/vm_nic_edit.rs	541	put_vm_nic_edit	"vm.nic.edit", "vm", &name, &name
crates/daemon/src/api/vm_hardware.rs	98	edit_hardware_vm_hardware	"vm.hardware.edit", "vm", &name, &name (or delegate to per-field PATCHes below)
crates/daemon/src/api/vm_edit/patch.rs	32	patch_cpu_vm_edit	"vm.cpu.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	243	patch_memory_vm_edit	"vm.memory.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	456	patch_firmware_vm_edit	"vm.firmware.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	608	patch_chipset_vm_edit	"vm.chipset.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	697	patch_boot_vm_edit	"vm.boot.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	781	patch_display_vm_edit	"vm.display.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/patch.rs	875	patch_tpm_vm_edit	"vm.tpm.edit", "vm", &name, &name
crates/daemon/src/api/vm_edit/controllers.rs	76	add_controller_vm_edit	"vm.controller.add", "vm", &name, &name
crates/daemon/src/api/vm_edit/controllers.rs	143	remove_controller_vm_edit	"vm.controller.remove", "vm", &name, &name
crates/daemon/src/api/vm_edit/hostpci.rs	95	assign_host_pci_vm_edit	"vm.hostpci.assign", "vm", &name, &name
crates/daemon/src/api/vm_edit/hostpci.rs	145	unassign_host_pci_vm_edit	"vm.hostpci.unassign", "vm", &name, &name
crates/daemon/src/api/storage_proxy.rs	276	pool rename (PATCH)	"storage.pool.rename", "pool", &id, &name
crates/daemon/src/api/storage_proxy.rs	300	pool destroy (DELETE)	"storage.pool.destroy", "pool", &id, &name
crates/daemon/src/api/update_v2.rs	182	apply	"system.update.apply", "host", "", "host"
crates/daemon/src/api/update_v2.rs	235	rollback	"system.update.rollback", "host", "", "host"

Already wired (verify naming only): vm_edit/nics.rs:70/171 (vm.nic.attach/detach), vm_edit/disks.rs:231/670/784 (vm.disk.attach/detach/resize), storage_proxy.rs:177 (storage.pool.create). Note vm_edit/disks.rs:515 (disk edit PUT) emits no task — add vm.disk.edit if it mutates config.

Optional (recommend emit-on-start): update_v2.rs preflight (~107), plan (~133), create_checkpoint for full update-stage visibility.

That is 26 new TaskRegistry call sites (+3 verify-only, +1 optional disk-edit).

3.2 source_ip = None fix list (audit emission)

Settings service crates hardcode source_ip: None. The daemon middleware extracts the client IP (ConnectInfo<SocketAddr> or X-Forwarded-For for the Caddy-proxied path) and must thread it to AuditStore::record(). Confirmed sites (grep source_ip: None):

File:line	Crate / section
crates/synvirt-resources/src/lib.rs:230	Resource reservation audit
crates/daemon/src/api/events.rs:761	Events audit record (login/hardware)
crates/daemon/src/settings/ssh_keys.rs:244	SSH keys audit
crates/synvirt-services/src/lib.rs:457	Services audit
crates/synvirt-certs/src/audit.rs:38	Certificate audit
crates/synvirt-bmc/src/lib.rs:412	BMC audit
crates/synvirt-firewall/src/lib.rs:512	Firewall audit
crates/synvirt-swap/src/lib.rs:461	Swap audit
crates/synvirt-time/src/lib.rs:259	Time audit
crates/synvirt-packages/src/lib.rs:435	Packages audit

10 sites. AuditRecord.source_ip is Option<String> with #[serde(default)] (synvirt-time/src/audit.rs:32) so legacy records stay deserializable — purely additive. Extract the IP once in the settings route layer and pass it into each service constructor/handler; validate proxy trust (whitelist Caddy’s loopback origin) before honoring X-Forwarded-For.

3.3 Files to touch — §7-2

26 task sites across network.rs, vm_nic_edit.rs, vm_hardware.rs, vm_edit/{patch,controllers,hostpci}.rs, storage_proxy.rs, update_v2.rs; 10 source_ip sites above. Adding Extension<AuthUser> to a handler does not change the wire contract, but per the frozen-contract rule re-dump openapi.snapshot.json (synvirt-daemon --dump-openapi) and regenerate the frontend client if any handler signature visibly changes.

---

4. Build / verify order (stop-gate discipline)

One slice per cycle. After each slice: cargo build + cargo test + cargo fmt --check + cargo clippy -- -D warnings on the touched crates, then the frontend npm run typecheck && npm run build in crates/web-ux-v2/, then hand off and wait. Do not chain slices.

Slice A — Phase 3 persistence (this doc §1).

• cargo build -p synvirt-observability && cargo test -p synvirt-observability — migrations 6-9 compile + apply on a fresh in-memory DB; round-trip insert/list/purge for network + storage; rollup 1-min-from-raw and 15-min-from-1-min aggregate correctly; collector_health upsert + stale purge.
• cargo build -p synvirt-daemon (rollup scheduler spawn compiles).
• cargo fmt --check, cargo clippy -- -D warnings.
• Frontend: unaffected (no endpoints) — still run npm run typecheck && npm run build to prove no drift. Hand off.

Slice B — §7-1 detector wiring (this doc §2).

• Wire suppliers in main.rs, one closure at a time. cargo build -p synvirt-daemon, fmt, clippy.
• Lab-verify on synvirt-01 (10.10.26.65, nested dev appliance): deploy, run a detector tick, confirm pool/cert/bmc/vm alarms populate the dashboard Issues view. Hand off.

Slice C — §7-2 task/audit emission (this doc §3).

• Add 26 task sites + 10 source_ip fixes. cargo build, fmt, clippy.
• Re-dump openapi.snapshot.json; npm run generate:api:from-file && npm run typecheck && npm run build.
• Functional check: a network op emits a tasks row + a correlated events row; a Time settings change writes an AuditRecord with source_ip populated (not None). Hand off.

4.1 Phase 4 collectors — BUILT + lab-only, NOT fleet-deployed

The actual network/storage delta collectors (the samplers in synvirt-network / synvirt-storaged that feed network_samples / storage_samples, and the daemon-side spawn_network_sampler / spawn_storage_delta_sampler) are Phase 4. They may be built and lab-verified on .65 only. They must NOT be deployed to the fleet (.11/.12/.13) without a separate explicit “go”.

Reasons, per project memory:

• SEV-0 Fix70 — a daemon restart drains/bounces guests on pre-Fix70 builds; any change that forces a daemon roll on a production node is SEV-0 on .11/.12. New sampler wiring changes the daemon’s startup task graph, so it inherits this gate.
• Autonomous-execution rule — pause + ask before mass runtime changes / high-regression arch / hard-to-revert chains. Adding fleet-wide collectors writing to a 15-table schema at 5s cadence is exactly that class of change.
• Lockstep deploy of all three nodes is mandatory when a fleet deploy does happen; never partial.

The Phase 3 schema + writers (Slice A) are inert until a collector feeds them, so landing Slice A on the fleet is safe (no behavior change, additive migrations only). The collectors are the behavior change and stay gated.

---

5. Open questions / risks (per slice)

Slice A — persistence

Open questions. (1) Retention configurable vs hardcoded — add a retention_policy table in a 0010 migration, or keep §5 as the canonical hardcoded policy? (2) Issue dedup strategy — when an issue re-fires on the same entity, set state='reopened' + bump occurrence_count, OR insert a new row? The schema’s issue_occurrences_active_unique partial index enforces one live row, so the answer must be reopened-in-place; document it and make the alarm engine honor it. (3) collector_health.id = collector_name implies one row per collector per host — for federation/cross-host rollup, does it need a (hostname, collector_name) composite? Current scope is per-host observability, so single-key is correct until federation lands. (4) sample_sequence wrap at 2^32 vs unbounded — track (collector_name, sample_sequence) with wrap detection; reset on collector restart and rely on has_gap rather than absolute sequence.

Risks. (a) Delta/reset false positives on sampler restart — wrap/reset detection needs the previous sample in a volatile (non-persisted) cache; a daemon restart must NOT mark has_reset. Only flag when the counter genuinely goes backwards. (b) Rollup backpressure — if raw arrives faster than rollups aggregate, the raw table grows before the 48h purge; expose via collector_health.queue_depth/samples_backpressure and set source_unavailable to shed load. (c) SQLite growth — many interfaces × vdevs × 5s can reach GBs in months; auto_vacuum is on (mod.rs:54), schedule a nightly VACUUM timer. (d) Out-of-order inserts — no UNIQUE on (sampled_at, interface) (legitimate same-instant duplicates across vNICs); UI must sort by sampled_at client-side. (e) Rollup timezone — bucket flooring is UTC-only; all timestamps are UTC by rule — operators must not change system TZ.

Most important Slice-A risk: delta wrap/reset false positives after a daemon restart polluting the series — mitigated by a volatile previous-sample cache and a “counter went strictly backwards” gate, never a missing-previous gate.

Slice B — detector wiring

Open questions. Pool-health endpoint may lack resilver_active/last_scrub_at on the fleet’s deployed synvirt-storaged (needs an API extension + version probe); kernel-panic scan policy (err-priority kernel lines vs strict “Kernel panic” match); snapshot depth via storaged API (preferred, decoupled) vs daemon-side zfs; orphan-disk scan via full dataset walk (comprehensive, slow) vs libvirt registry (fast, may miss datasets); whether the guest-tools persistor already broadcasts a snapshot the daemon can read or needs per-VM SQLite polling.

Risks. BridgeStatusFn couples the daemon tightly to the in-process NetworkController (brittle across hot restarts); ipmitool/journalctl subprocesses must be arc-cached per interval (no spawn-per-tick) and fail-soft on no-BMC/large-journal hosts; the 7 real /proc detectors already sleep 250ms — keep the detector interval ≥ 5s so they stay off the hot path.

Most important Slice-B risk: the synvirt-storaged pool-health endpoint may not exist on the deployed fleet build — probe the API version before wiring PoolHealthFn, feature-gate the three pool-health detectors otherwise.

Slice C — task/audit emission

Open questions. VM hardware: one vm.hardware.edit task per HTTP request vs per-field tasks (per-field = granularity but task-table flooding; single = simpler, less visibility) — recommend single task when the PATCH batches fields; per-field when they arrive separately. X-Forwarded-For vs ConnectInfo precedence; audit-table retention (add a weekly >90d purge). Whether backupd/migratord proxies should emit daemon-side tasks (recommend yes for user-initiated ops).

Risks. Proxy task sites (storage_proxy, update_v2) can leave a task stuck running if the downstream daemon hangs — align task timeouts with the storaged/updaterd SLA; a daemon restart mid-update can orphan the task unless updaterd persists its own job mapping. X-Forwarded-For is spoofable unless proxy trust is whitelisted at the middleware (validate Caddy’s loopback origin, not at the audit layer). 40+ new task kinds need the dot-taxonomy convention enforced to avoid collisions.

Most important Slice-C risk: trusting an unvalidated X-Forwarded-For writes attacker- controlled IPs into the forensic audit trail — extract source IP only behind a whitelisted trusted-proxy check at the daemon middleware.