Refactor R1 — Network service migration runbook

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/operations/refactor-r1-network.md. Edit at the source, not here.

Refactor R1 — Network service migration runbook

This document covers the 6-step migration of an existing SynVirt host (e.g. synvirt-01) onto the new synvirt-network.service introduced in Refactor R1.

Superseded (Commit 6): the D-Bus model this runbook describes (the network D-Bus interface, the legacy-network-direct feature, the dbus-monitor step) no longer applies — Commit 6 collapsed the network surface back in-process (the daemon calls synvirt-network’s NetworkController directly). The network.yaml source of truth, the synvirt-network.service boot path, and the OVS reconcile rules below are still accurate; the D-Bus-specific steps are historical.

For greenfield installs (1.0+) the live ISO writes /etc/synvirt/network.yaml and enables both services at install time — this runbook does not apply.

Why this exists

R1 splits OVS, vSwitches, the firewall toggle, and the management interface out of the monolithic synvirt-daemon into their own systemd-managed daemon (synvirt-network). The win is structural: restarting the API daemon (synvirt-daemon) no longer touches OVS. The May 5 2026 incident (3-hour host outage on synvirt-01 after a systemctl restart synvirt-daemon) cannot recur because the data plane no longer lives in the API daemon’s process.

Locked rules during migration

• No auto-delete. If the YAML is missing entries that live OVS has, the new service refuses to delete them. They surface as “drift” on the D-Bus bus and the operator decides.
• No auto-start on package install. The unit ships disabled. The runbook below is what brings it up.
• YAML is the source of truth. Sled is a fast read cache, rebuilt from YAML on every start. Edit /etc/synvirt/network.yaml with vi from a console session when nothing else works.

The 6 steps

1. Lay down the unit + binary + D-Bus policy

The 0.16.0+ package install drops:

/opt/synvirt/bin/synvirt-network
/etc/systemd/system/synvirt-network.service

then runs systemctl daemon-reload. It does not start or enable the service automatically.

2. Verify nothing about the running host changed yet

systemctl status synvirt-daemon          # still active, unchanged
systemctl status synvirt-network         # inactive (dead)
ovs-vsctl list-br                        # current bridges as expected
ls -l /etc/synvirt/network.yaml          # should NOT exist yet

3. Run the migration generator

The first start of synvirt-network runs the migration. We start the unit once with the Type=oneshot override so it exits cleanly after writing /etc/synvirt/network.yaml:

sudo systemctl start synvirt-network
journalctl -u synvirt-network -b -t synvirt.network.migrate

Expect one of three outcomes in the journal:

Outcome	What it means
outcome=AlreadyMigrated	network.yaml already present — operator dropped one in by hand. Skip to step 4.
outcome=FromSled { virtswitches: N }	YAML built from the legacy daemon’s sled at /var/lib/synvirt/policies.db. This is the common path.
outcome=FromOvs { bridges: N }	Sled was empty; YAML built from live OVS introspection. The marker imported_from_ovs: true is set. Audit and remove the marker once verified.

If the unit fails with E_NETWORK_RECONCILE_REFUSE, the YAML is empty AND OVS holds bridges. The service refuses to start as a safety measure — never auto-delete operator state. Hand-edit /etc/synvirt/network.yaml to declare the bridges (or pass the service the legacy sled path explicitly via SYNVIRT_NETWORK_LEGACY_SLED) and retry.

4. Verify /etc/synvirt/network.yaml matches reality

sudo cat /etc/synvirt/network.yaml
ovs-vsctl list-br

Cross-check that every bridge in ovs-vsctl list-br appears as a virtswitches[*].name entry in the YAML. If a bridge is in OVS but not YAML, the service flagged it as drift — decide whether to add it to YAML or delete it from OVS by hand.

5. Flip synvirt-daemon into D-Bus client mode

R1 ships with the daemon’s Cargo feature legacy-network-direct = ["legacy-network-direct"] ON by default, so synvirt-daemon keeps touching OVS directly until you swap it. The hand-off is a recompile:

cd /home/synvirt/dev
cargo build -p synvirt-daemon --no-default-features --release
sudo install -m 0755 target/release/synvirt-daemon /opt/synvirt/bin/synvirt-daemon
sudo systemctl restart synvirt-daemon

In the journal you should now see the daemon’s network handlers log synvirt.daemon.network_client instead of the legacy api::virtswitch target.

(If you skip this step, the API daemon still works but bypasses the D-Bus path entirely — defeats the point of R1 but keeps the host serving.)

6. Validate end-to-end

Run both smoke scripts from a peer host (e.g. synvirtdev):

./scripts/r1-network-restart-test.sh         <target>     # restart synvirt-daemon
./scripts/r1-network-restart-test-network.sh <target>     # restart synvirt-network

Each prints PASS on success and a non-zero exit code on the specific assertion that failed. Re-run them whenever you upgrade either daemon or change the network YAML.

Rollback

If something goes wrong, the rollback is symmetric:

1. sudo systemctl stop synvirt-network && sudo systemctl disable synvirt-network
2. Rebuild synvirt-daemon with legacy-network-direct ON (the default) and reinstall the binary.
3. sudo systemctl restart synvirt-daemon

Operator is back to the pre-R1 monolith. The network.yaml is left in place for the next attempt — no data loss.

Drift detection in steady state

On every reconcile pass (startup, SIGHUP, on-demand reconcile) the service logged drift whenever YAML and live OVS disagreed.

Superseded (Commit 6): the DriftDetected D-Bus signal and the dbus-monitor subscription this section described are gone — the network surface is in-process now; drift surfaces via the daemon’s reconcile-status reporting instead.