Migrator — warm (live) migration via block-level disk read (VDDK/NBD)

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/specs/2026-06-23-migrator-warm-vddk-design.md. Edit at the source, not here.

Migrator — warm (live) migration via block-level disk read (VDDK/NBD)

Date: 2026-06-23 Status: approved (autonomous execution authorized by Tony — “dale”) Scope: make warm migration of a running vim_api source VM work end to end — minimal-downtime import driven by snapshot + change-block tracking (CBT) + a block-level disk reader — and fix vNIC mapping so the migrated NIC lands on the right vSwitch and network tag. First target: synnet.siem (running on the esx15 source) → synvirt-02 (.12).

Goal (operator words)

“elimina esa maquina en synvirt02 e intenta migrarla de nuevo con CBT y que la tarjeta de red se migre correctamente” … “usa la VADP si eso funciona, dale”.

Restated: today only cold (powered-off) import works. We want the industry-standard live path — snapshot the running VM, copy the base, then converge with CBT deltas, and cut over with a short final downtime — plus a correct vNIC.

Current state — why warm does not work today

The orchestrator already models warm end to end, but the disk reader is powered-off-only, so a warm job over a running VM fails mid-copy.

• crates/migrator/src/orchestrator.rs — resolve_mode (lines ~407-434) enters Warm when adapter.supports_warm() && manifest.cbt_available, else auto-downgrades to Cold (records E_* CbtUnsupported, never hard fails). The warm body (lines ~282-334) keeps the VM running through copy_full, then runs warm_delta_loop → CuttingOver (shutdown + final snapshot + copy_delta).
• crates/migrator/src/adapters/vim_api/mod.rs —
  • supports_warm() = cfg!(vim_shim) (line ~835): true on the deployed binaries because the govmomi shim is linked (build.rs:80 cargo:rustc-cfg=vim_shim).
  • cbt_available is set from gov_cbt_status (lines ~1094-1107).
  • create_migration_snapshot (line ~1123) already returns a warm SnapshotRef carrying the snapshot MOR + per-disk change-ids.
  • read_disk_full (line ~1139) is the problem: it streams disk bytes through the govmomi shim’s NFC export.
• crates/migrator/src/adapters/vim_api/go-shim/shim.go — gov_nbd_open_full (line ~987) opens the disk with vm.Export() (NFC export lease, line ~1068). The file header documents it literally: “open an NFC export lease for a powered-off disk”. NFC export refuses a running VM — that is the gov_nbd_open_full failed symptom.

So supports_warm + cbt_available are both true on the running source, the orchestrator enters Warm, and copy_full dies because vm.Export() cannot read a powered-on VM. The snapshot + CBT plumbing (gov_snapshot, gov_delta_extents / QueryChangedDiskAreas, lines ~772-928) is correct and reused unchanged.

Conclusion: the only missing piece is a block-level reader that works on a running VM’s snapshot. The supported mechanism is VDDK (VixDiskLib) over the NBDSSL transport. There is no govmomi-only path: NFC export is powered-off-only, and the datastore /folder HTTP service returns HTTP 500 for raw -flat.vmdk extents (proven 2026-06-23). SAN and HotAdd transports do not apply to our topology (no shared LUN visibility; the destination is a KVM host, not a hypervisor peer that can hot-add the snapshot disk). NBDSSL reads from the source host over the management network and is the only cross-platform transport available to us.

Design

Data engine — nbdkit-vddk subprocess + a Rust NBD reader

Replace the NFC reader (for the warm path only) with a block reader backed by nbdkit running the vddk plugin. The migratord launches one short-lived nbdkit per disk and reads it over a private Unix-domain NBD socket:

nbdkit --exit-with-parent --unix <sock> --readonly \
  vddk libdir=/opt/synvirt/vddk \
       server=<source-host> user=<user> password=+<pwfile> \
       thumbprint=<cert-SHA1> \
       vm=moref=<vm-moref> snapshot=<snapshot-moref> \
       file="<[datastore] path/disk.vmdk>" \
       transports=nbdssl

read_disk_full / read_disk_delta connect to <sock> and stream DiskChunks through the existing BlockStream → sink path, so zero promotion (thinning), progress deltas, and the warm delta loop are reused verbatim.

Why subprocess, not cgo-in-shim: the proprietary VixDiskLib stays isolated in its own process — a VDDK crash kills the nbdkit child, not the migratord. It is the same isolation the in-place preparer (virt-v2v) relies on. A cgo binding inside the Go shim would put VDDK’s thread/connection state machine inside the daemon address space; rejected for fragility with no upside.

The Go shim is unchanged for VDDK — it stays the owner of connect / snapshot / CBT. VDDK lives only inside nbdkit.

New components (all under crates/migrator/src/adapters/vim_api/)

Unit	Responsibility	Depends on
nbdkit.rs	Spawn + supervise one nbdkit vddk child per disk: build args, own the Unix socket path, --exit-with-parent, readiness wait, teardown on drop. Writes the password to a mode-0600 tmp file (never on argv).	tokio::process, source creds, snapshot MOR, vmdk path, cert thumbprint
nbd_reader.rs	Minimal async NBD client over the Unix socket (newstyle fixed handshake + NBD_CMD_READ). read_full() streams the whole device sequentially; read_extents(ranges) reads only CBT-changed ranges. Yields BlockStream. No C FFI.	nbdkit.rs socket
mod.rs (edit)	When the SnapshotRef is warm (WARM_PREFIX) and VDDK is present, read_disk_full / new read_disk_delta route through nbdkit.rs + nbd_reader.rs. Otherwise keep today’s NFC path (cold).	both above

Supporting bits:

• Cert thumbprint: VDDK wants the source cert’s SHA-1 thumbprint; probe_source_health records SHA-256. Derive SHA-1 from the pinned cert (we already hold it) in a small helper; do not re-fetch.
• VMDK path + snapshot MOR: already available — vmdk backing path is cached (mod.rs ~1075-1086, detail.disks[].backing.vmdk_file); the snapshot MOR + change-ids come from the warm SnapshotRef.
• Capability gate: capabilities gains warm_available = vim_shim && nbdkit-vddk present && VDDK libdir present. When false, resolve_mode keeps its existing warm→cold auto-downgrade (no behavior change for hosts without VDDK).

Warm flow (orchestration — already wired, now it actually reads)

enable CBT (gov_enable_cbt)
  → initial snapshot (gov_snapshot)
  → CopyingFull: full copy via NBD reader (source VM stays running)
  → DeltaSyncing loop:
        gov_delta_extents (QueryChangedDiskAreas, sinceChangeId)
        → read only changed ranges via NBD reader → apply to sink
        → repeat until converged (delta_convergence: max_iters / max_bytes / max_wall)
  → CuttingOver: shutdown source → final snapshot → final delta via NBD
  → preparer (Linux guests: effectively no-op; Windows: virtio inject where preparer exists)
  → ProfileDomainStarter define + start on destination

delta_convergence already exists on the plan; cutover already shuts the source down and takes the final snapshot. The only swapped organ is the reader.

vNIC fix (second deliverable)

The earlier synnet.siem job mapped the NIC with target_bridge:"" / target_vlan:0 while the source NIC is VLAN 15 / “LAN15_SOC” / vmxnet3, so it landed on vSwitch0 with no tag → wrong network.

Fix in crates/daemon/src/migrator_domain.rs:

• resolve_bridge already falls back an empty target_bridge to the host default vSwitch (correct, keep).
• New: when nic_mapping.target_vlan == 0, inherit the source NIC’s VLAN from the manifest (source_vlan) instead of emitting an untagged port. Preserve the source MAC (already supported via source_mac).
• Applies identically to warm and cold jobs. Pure-function change with unit coverage (vlan inherited when 0; explicit non-zero override respected).

Packaging + provisioning

• Host tooling baked into the ISO (apt in the build chroot, which has network): nbdkit, nbdkit-vddk-plugin, libnbd0 (+ libnbd-bin for diagnostics). Add to the iso-builder package list.
• VDDK redistributable = external artifact, pinned by SHA-256 under iso-builder/vendor/ following the guest-payloads pattern (iso-builder/vendor/guest-payloads/MANIFEST.toml + prepare-*.sh + verify-on-build). Staged to /opt/synvirt/vddk/. Not committed to git; a missing/hash-mismatched artifact is a hard build failure for the warm feature, never a silent stub.
• Runtime detection: warm is enabled only when the VDDK libdir is present; otherwise the existing warm→cold auto-downgrade keeps standalone hosts working byte-identically.

Error handling

• nbdkit fails to start / VDDK rejects the connect → typed E_MIG_* error, surface “live read engine unavailable; retry as cold or provision the live disk engine”, and let resolve_mode fall back to cold when the job allows it.
• NBD socket read error mid-copy → fail the job with the disk + offset in the record (today’s gov_nbd_read symptom gets a real message); no partial “completed”.
• Source snapshot create/remove failures are best-effort logged; the job always attempts to delete the migration snapshot on success and on failure (no orphaned snapshots left on the source — matches the cleanup discipline already used for cold).
• Cutover guard: the destination domain is defined/started only after the final delta is applied and flushed; boot-gate (running + stabilization + boot reads) from the existing E2E work still gates Completed.

Testing

• Unit (no VDDK, no source): nbd_reader.rs against a local nbdkit memory (or nbdkit data) server — validates handshake, read_full, and read_extents sparse-range reads. nbdkit.rs arg-builder is a pure function with a snapshot test (password never on argv; thumbprint/socket wired). vNIC vlan-inheritance helper gets pure unit tests.
• E2E (gated SYNVIRT_E2E=1): extend crates/synvirt-migratord/tests/e2e_external_source_uefi.rs with a warm case — mode: warm against a running source, assert the job converges through DeltaSyncing → CuttingOver → Completed and the destination VM is running with the NIC on the expected vSwitch/VLAN. Runs only once the VDDK artifact is provisioned on the destination host.

Scope (YAGNI)

In: warm full + CBT delta read via nbdkit-vddk over NBDSSL; the warm delta-convergence loop (already present) reading through the new path; cutover; vNIC VLAN inheritance + MAC preservation; ISO packaging + pinned VDDK artifact + runtime gate; unit + gated E2E.

Out (later): SAN / HotAdd transports; application-aware quiescing (VSS); warm multi-disk parallelism; per-chunk read retry; warm for non-vim_api adapters.

Open dependency (tracked, not blocking the code)

The VDDK redistributable is proprietary (Broadcom developer portal login). Per Tony’s “dale”, we build and unit-test the entire warm path now behind the runtime VDDK gate; the binary artifact is provisioned to .12 (and pinned in iso-builder/vendor/) before the real E2E. Until then the host keeps auto-downgrading warm → cold, so nothing regresses.

Touch list (files)

• crates/migrator/src/adapters/vim_api/nbdkit.rs (new)
• crates/migrator/src/adapters/vim_api/nbd_reader.rs (new)
• crates/migrator/src/adapters/vim_api/mod.rs (warm read routing, thumbprint helper, capability gate)
• crates/migrator/src/orchestrator.rs (wire read_disk_delta into the delta loop if not already symmetric)
• crates/daemon/src/migrator_domain.rs (vNIC VLAN inheritance)
• crates/synvirt-migratord/tests/e2e_external_source_uefi.rs (warm case)
• iso-builder/ package list + iso-builder/vendor/ VDDK MANIFEST + prepare script

Cross-refs

See 2026-06-22-migrator-context-aware-destination-design.md (destination routing — unchanged here; warm runs on the chosen host’s migratord) and the migrator module deep-dive docs/modules/migrator/.