Migrator Warm (Live) Migration via VDDK/NBD — Implementation Plan

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/plans/2026-06-23-migrator-warm-vddk.md. Edit at the source, not here.

Migrator Warm (Live) Migration via VDDK/NBD — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make warm (minimal-downtime) migration of a running vim_api source VM work end to end by replacing the powered-off-only NFC disk reader with a block-level reader over nbdkit-vddk (NBDSSL), and fix vNIC VLAN mapping.

Architecture: The orchestrator already drives warm (snapshot → CBT → full → delta loop → cutover). We swap only the disk reader: a new minimal async NBD client (nbd_reader.rs) reads a per-disk nbdkit vddk subprocess (nbdkit.rs) over a private Unix socket; vim_api/mod.rs routes read_disk_full/read_disk_delta through it when the snapshot is warm and VDDK is present, else keeps the NFC path. The Go shim is unchanged (still owns connect/snapshot/CBT). A runtime gate keeps the existing warm→cold auto-downgrade when VDDK is absent, so standalone hosts are byte-identical.

Tech Stack: Rust (tokio, futures BoxStream, bytes::Bytes, async-trait), the NBD protocol (newstyle), nbdkit + nbdkit-vddk-plugin + VixDiskLib (VDDK), the existing Go govmomi shim (cgo), ZFS sink.

Global Constraints

Rust: one Error enum per crate via thiserror with E_* discriminant codes; no unwrap()/expect() in library code; anyhow only in main/tests.
Public async fns annotated #[tracing::instrument(skip_all)]; use tracing macros, never println!.
Naming verbs: get_* fails-if-absent, find_* returns Option, list_* returns Vec, ensure_* idempotent, read_* pure read.
Subprocess calls: tokio::process::Command with a timeout, stdout/stderr captured separately, never build shell strings.
cargo fmt, cargo clippy -- -D warnings must pass; rustdoc on every pub item (first line a full sentence with a period).
No third-party product names in code/comments/commits. Use vim_api, VDDK, NBD, CBT, “source control plane”. Never name the source hypervisor vendor.
Versioning: version.workspace = true; never hardcode a version literal.
Conventional Commits with crate scope, e.g. feat(synvirt-migrator): .... Local-only git, no push. One commit per task (atomic).
All code, comments, and commit messages in English.
The warm path is gated: when VDDK is unavailable the behavior must be byte-identical to today (warm→cold auto-downgrade in orchestrator.rs::resolve_mode).

File Structure

File	Responsibility	New/Mod
`crates/daemon/src/migrator_domain.rs`	vNIC VLAN inheritance (`source_vlan` helper + use when `target_vlan==0`)	Modify
`crates/migrator/src/adapters/vim_api/nbd_reader.rs`	Minimal async NBD client over a Unix socket: handshake + `NBD_CMD_READ`; `read_full` + `read_ranges` → `BlockStream`	Create
`crates/migrator/src/adapters/vim_api/nbdkit.rs`	`nbdkit vddk` subprocess: pure arg-builder + spawn/readiness/teardown	Create
`crates/migrator/src/adapters/vim_api/thumbprint.rs`	Derive a cert SHA-1 thumbprint (VDDK input) from the pinned cert	Create
`crates/migrator/src/adapters/vim_api/mod.rs`	Route warm reads through NBD; `vddk_available()` gate; declare submodules	Modify
`crates/migrator/src/http.rs`	`CapabilitiesResponse.warm_available`	Modify
`crates/migrator/src/jobs.rs`	carry `warm_available` flag (mirror `preparer_available`)	Modify
`iso-builder/` (package list)	bake `nbdkit`, `nbdkit-vddk-plugin`, `libnbd0`, `libnbd-bin`	Modify
`iso-builder/vendor/vddk/MANIFEST.toml` + `prepare-vddk.sh`	pinned VDDK redistributable, staged to `/opt/synvirt/vddk/`	Create
`crates/synvirt-migratord/tests/e2e_external_source_uefi.rs`	gated warm E2E case	Modify

Task 1: vNIC VLAN inheritance (second deliverable, independent)

Fixes the “NIC didn’t migrate correctly” bug: when a plan omits the VLAN (target_vlan == 0) inherit the source NIC’s VLAN from the manifest instead of emitting an untagged port. Independent of the warm work; ship it first.

Files:

Modify: crates/daemon/src/migrator_domain.rs (NIC mapping block ~222-243; add source_vlan helper next to source_mac ~509-518)
Test: same file (#[cfg(test)] module already present — it holds the resolve_bridge/render tests)

Interfaces:

Consumes: VmManifest.nics: Vec<NicSpec> where NicSpec { label: String, source_vlan: Option<u16>, mac, source_network, model } (crates/migrator/src/types.rs:377-391); NicMapping { source_nic_label, target_bridge, target_vlan: u16, preserve_mac } (:531-539).
Produces: fn source_vlan(manifest: &VmManifest, label: &str) -> Option<u16>.
Step 1: Write the failing test

Add to the #[cfg(test)] module in crates/daemon/src/migrator_domain.rs:

#[test]
fn source_vlan_is_inherited_when_label_matches() {
    let manifest = test_manifest_with_nic("Network adapter 1", Some(15));
    assert_eq!(source_vlan(&manifest, "Network adapter 1"), Some(15));
    assert_eq!(source_vlan(&manifest, "missing"), None);
}

#[test]
fn nic_attachment_inherits_source_vlan_when_target_is_zero() {
    // target_vlan == 0 in the mapping must NOT mean "untagged" when the
    // source NIC carries a VLAN; it means "inherit the source VLAN".
    let manifest = test_manifest_with_nic("Network adapter 1", Some(15));
    let mapping = NicMapping {
        source_nic_label: "Network adapter 1".into(),
        target_bridge: String::new(),
        target_vlan: 0,
        preserve_mac: true,
    };
    let vlan = effective_vlan(&mapping, &manifest);
    assert_eq!(vlan, Some(15));
}

#[test]
fn explicit_target_vlan_overrides_source() {
    let manifest = test_manifest_with_nic("Network adapter 1", Some(15));
    let mapping = NicMapping {
        source_nic_label: "Network adapter 1".into(),
        target_bridge: String::new(),
        target_vlan: 42,
        preserve_mac: true,
    };
    assert_eq!(effective_vlan(&mapping, &manifest), Some(42));
}

Add a test_manifest_with_nic helper in the test module (mirror existing manifest fixtures already in this file; set one NicSpec with the given label + vlan, empty disks ok).

Step 2: Run test to verify it fails

Run: cargo test -p synvirt-daemon --lib migrator_domain::tests::source_vlan -- --nocapture Expected: FAIL — source_vlan / effective_vlan not found.

Step 3: Write minimal implementation

Add the two helpers near source_mac (crates/daemon/src/migrator_domain.rs ~518):

/// The source VLAN for a NIC label, or `None` when the NIC is untagged
/// or absent. Used to inherit the source VLAN when the plan leaves
/// `target_vlan` at 0.
fn source_vlan(manifest: &VmManifest, label: &str) -> Option<u16> {
    manifest
        .nics
        .iter()
        .find(|n| n.label == label)
        .and_then(|n| n.source_vlan)
}

/// The VLAN to apply to a migrated NIC: an explicit non-zero
/// `target_vlan` wins; otherwise inherit the source NIC's VLAN.
fn effective_vlan(mapping: &NicMapping, manifest: &VmManifest) -> Option<u16> {
    if mapping.target_vlan > 0 {
        Some(mapping.target_vlan)
    } else {
        source_vlan(manifest, &mapping.source_nic_label)
    }
}

Then change the vlan: field in the net_atts builder (~235-239) from:

                    vlan: if m.target_vlan > 0 {
                        Some(m.target_vlan)
                    } else {
                        None
                    },

to:

                    vlan: effective_vlan(m, manifest),

Step 4: Run tests to verify they pass

Run: cargo test -p synvirt-daemon --lib migrator_domain -- --nocapture Expected: PASS (new vlan tests + existing resolve_bridge/render tests still green).

Step 5: fmt + clippy + commit

cargo fmt -p synvirt-daemon
cargo clippy -p synvirt-daemon -- -D warnings
git add crates/daemon/src/migrator_domain.rs
git commit -m "fix(synvirt-daemon): inherit source VLAN for migrated vNIC when plan omits it"

Task 2: Minimal async NBD client (`nbd_reader.rs`)

The foundational reader. Testable WITHOUT VDDK against nbdkit memory. Speaks NBD newstyle: handshake via NBD_OPT_EXPORT_NAME (default export), then NBD_CMD_READ simple replies.

Pre-req: nbdkit must be on PATH on the build/test box. Install once: apt-get install -y nbdkit (the unit test skips with a clear message if nbdkit is absent).

Files:

Create: crates/migrator/src/adapters/vim_api/nbd_reader.rs
Modify: crates/migrator/src/adapters/vim_api/mod.rs (add mod nbd_reader;)

Interfaces:

Consumes: crate::types::DiskChunk { offset: u64, length: u64, data: Bytes, is_zero: bool }; crate::adapter::BlockStream<'a>; crate::Result / crate::Error.
Produces:
- pub struct NbdClient with pub async fn connect(socket: &std::path::Path) -> Result<NbdClient>, pub fn size(&self) -> u64, pub async fn read_at(&mut self, offset: u64, len: u32) -> Result<Bytes>.
- pub fn stream_full(client: NbdClient, chunk: u32) -> BlockStream<'static> — sequential whole-device chunks.
- pub fn stream_ranges(client: NbdClient, ranges: Vec<(u64, u64)>, chunk: u32) -> BlockStream<'static> — only the given byte ranges (CBT deltas).
Step 1: Write the failing test

Create crates/migrator/src/adapters/vim_api/nbd_reader.rs with the test first (implementation stubs added in Step 3):

#[cfg(test)]
mod tests {
    use super::*;
    use tokio::process::Command;

    /// Spawn `nbdkit memory <size>` on a unix socket; skip if nbdkit absent.
    async fn spawn_memory_nbdkit(sock: &std::path::Path, size: u64) -> Option<tokio::process::Child> {
        if which_nbdkit().is_none() {
            eprintln!("SKIP: nbdkit not on PATH");
            return None;
        }
        let child = Command::new("nbdkit")
            .arg("--exit-with-parent")
            .arg("-U").arg(sock)
            .arg("memory").arg(size.to_string())
            .kill_on_drop(true)
            .spawn()
            .ok()?;
        // wait for the socket to appear
        for _ in 0..50 {
            if sock.exists() { break; }
            tokio::time::sleep(std::time::Duration::from_millis(40)).await;
        }
        Some(child)
    }

    #[tokio::test]
    async fn reads_back_what_it_handshakes() {
        let dir = tempfile::tempdir().unwrap();
        let sock = dir.path().join("nbd.sock");
        let Some(_child) = spawn_memory_nbdkit(&sock, 1 << 20).await else { return };

        let mut client = NbdClient::connect(&sock).await.expect("connect");
        assert_eq!(client.size(), 1 << 20);
        // memory plugin starts zero-filled; reading any range yields zeros.
        let buf = client.read_at(0, 4096).await.expect("read");
        assert_eq!(buf.len(), 4096);
        assert!(buf.iter().all(|&b| b == 0));
    }

    #[tokio::test]
    async fn stream_full_covers_whole_device() {
        let dir = tempfile::tempdir().unwrap();
        let sock = dir.path().join("nbd.sock");
        let Some(_child) = spawn_memory_nbdkit(&sock, 1 << 20).await else { return };

        let client = NbdClient::connect(&sock).await.expect("connect");
        let mut s = stream_full(client, 64 * 1024);
        let mut total = 0u64;
        use futures::StreamExt;
        while let Some(chunk) = s.next().await {
            total += chunk.expect("chunk").length;
        }
        assert_eq!(total, 1 << 20);
    }
}

Add tempfile to [dev-dependencies] of crates/migrator/Cargo.toml if absent.

Step 2: Run test to verify it fails

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::nbd_reader -- --nocapture Expected: FAIL — NbdClient / stream_full not defined (compile error).

Step 3: Write minimal implementation

Prepend the module body (above the #[cfg(test)]):

//! Minimal asynchronous NBD client for reading a disk exported by a
//! local `nbdkit` instance over a Unix-domain socket. Speaks the NBD
//! "newstyle" handshake via `NBD_OPT_EXPORT_NAME` (default export) and
//! issues `NBD_CMD_READ` simple-reply requests. No external C
//! dependency; the proprietary disk engine lives entirely in `nbdkit`.

use bytes::{Bytes, BytesMut};
use futures::stream::{self, BoxStream, StreamExt};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::UnixStream;

use crate::adapter::{zero_chunk, BlockStream};
use crate::types::DiskChunk;
use crate::{Error, Result};

const NBD_OPT_MAGIC: u64 = 0x4942_4D41_4749_4300 | 0x5054; // "IHAVEOPT"
const NBDMAGIC: u64 = 0x4E42_444D_4147_4943; // "NBDMAGIC"
const IHAVEOPT: u64 = 0x4948_4156_454F_5054; // "IHAVEOPT"
const NBD_FLAG_C_FIXED_NEWSTYLE: u32 = 1;
const NBD_OPT_EXPORT_NAME: u32 = 1;
const NBD_REQUEST_MAGIC: u32 = 0x2560_9513;
const NBD_SIMPLE_REPLY_MAGIC: u32 = 0x6744_6698;
const NBD_CMD_READ: u16 = 0;

/// Locate the `nbdkit` binary on PATH (used by tests + readiness checks).
pub fn which_nbdkit() -> Option<std::path::PathBuf> {
    std::env::var_os("PATH").and_then(|paths| {
        std::env::split_paths(&paths)
            .map(|p| p.join("nbdkit"))
            .find(|p| p.exists())
    })
}

/// A connected NBD session against one export.
pub struct NbdClient {
    stream: UnixStream,
    size: u64,
}

impl NbdClient {
    /// Connect to an `nbdkit` Unix socket and negotiate the default export.
    pub async fn connect(socket: &std::path::Path) -> Result<NbdClient> {
        let mut stream = UnixStream::connect(socket)
            .await
            .map_err(|e| Error::Internal(format!("nbd connect {}: {e}", socket.display())))?;

        let magic = stream.read_u64().await.map_err(nbd_io)?;
        if magic != NBDMAGIC {
            return Err(Error::Internal("nbd: bad server magic".into()));
        }
        let opt_magic = stream.read_u64().await.map_err(nbd_io)?;
        if opt_magic != IHAVEOPT {
            return Err(Error::Internal("nbd: server not newstyle".into()));
        }
        let _handshake_flags = stream.read_u16().await.map_err(nbd_io)?;

        // Client flags.
        stream.write_u32(NBD_FLAG_C_FIXED_NEWSTYLE).await.map_err(nbd_io)?;

        // NBD_OPT_EXPORT_NAME with an empty name selects nbdkit's default export.
        stream.write_u64(IHAVEOPT).await.map_err(nbd_io)?;
        stream.write_u32(NBD_OPT_EXPORT_NAME).await.map_err(nbd_io)?;
        stream.write_u32(0).await.map_err(nbd_io)?; // name length 0
        // Server replies: 64-bit export size, 16-bit transmission flags,
        // then 124 zero bytes (we did not send NBD_FLAG_C_NO_ZEROES).
        let size = stream.read_u64().await.map_err(nbd_io)?;
        let _tflags = stream.read_u16().await.map_err(nbd_io)?;
        let mut pad = [0u8; 124];
        stream.read_exact(&mut pad).await.map_err(nbd_io)?;

        Ok(NbdClient { stream, size })
    }

    /// Total export size in bytes.
    pub fn size(&self) -> u64 {
        self.size
    }

    /// Read `len` bytes at `offset` via a single `NBD_CMD_READ`.
    pub async fn read_at(&mut self, offset: u64, len: u32) -> Result<Bytes> {
        // Request: magic, cmd flags(0), type, handle, offset, length.
        self.stream.write_u32(NBD_REQUEST_MAGIC).await.map_err(nbd_io)?;
        self.stream.write_u16(0).await.map_err(nbd_io)?;
        self.stream.write_u16(NBD_CMD_READ).await.map_err(nbd_io)?;
        self.stream.write_u64(offset).await.map_err(nbd_io)?; // handle == offset
        self.stream.write_u64(offset).await.map_err(nbd_io)?;
        self.stream.write_u32(len).await.map_err(nbd_io)?;

        let magic = self.stream.read_u32().await.map_err(nbd_io)?;
        if magic != NBD_SIMPLE_REPLY_MAGIC {
            return Err(Error::Internal("nbd: bad reply magic".into()));
        }
        let err = self.stream.read_u32().await.map_err(nbd_io)?;
        let _handle = self.stream.read_u64().await.map_err(nbd_io)?;
        if err != 0 {
            return Err(Error::Internal(format!("nbd read error {err} at {offset}")));
        }
        let mut buf = BytesMut::zeroed(len as usize);
        self.stream.read_exact(&mut buf).await.map_err(nbd_io)?;
        Ok(buf.freeze())
    }
}

fn nbd_io(e: std::io::Error) -> Error {
    Error::Internal(format!("nbd io: {e}"))
}

/// Stream the whole export sequentially as `DiskChunk`s of up to `chunk`
/// bytes. All-zero chunks are emitted as zero-runs so the sink thins them.
pub fn stream_full(client: NbdClient, chunk: u32) -> BlockStream<'static> {
    let size = client.size();
    let ranges = vec![(0u64, size)];
    stream_ranges(client, ranges, chunk)
}

/// Stream only the given byte ranges (CBT deltas) as `DiskChunk`s.
pub fn stream_ranges(client: NbdClient, ranges: Vec<(u64, u64)>, chunk: u32) -> BlockStream<'static> {
    let st = stream::unfold(
        (client, ranges, 0usize, 0u64),
        move |(mut client, ranges, mut idx, mut pos)| async move {
            loop {
                if idx >= ranges.len() {
                    return None;
                }
                let (start, len) = ranges[idx];
                if pos >= len {
                    idx += 1;
                    pos = 0;
                    continue;
                }
                let want = std::cmp::min(chunk as u64, len - pos) as u32;
                let offset = start + pos;
                match client.read_at(offset, want).await {
                    Ok(data) => {
                        pos += want as u64;
                        let chunk = if data.iter().all(|&b| b == 0) {
                            zero_chunk(offset, want as u64)
                        } else {
                            DiskChunk { offset, length: want as u64, data, is_zero: false }
                        };
                        return Some((Ok(chunk), (client, ranges, idx, pos)));
                    }
                    Err(e) => return Some((Err(e), (client, ranges, idx + ranges.len() + 1, 0))),
                }
            }
        },
    );
    st.boxed()
}

Note: if cargo clippy flags the NBD_OPT_MAGIC const as unused, delete it — only IHAVEOPT/NBDMAGIC are needed. Keep the const list minimal.

Step 4: Run tests to verify they pass

apt-get install -y nbdkit   # once, if missing
cargo test -p synvirt-migrator --lib adapters::vim_api::nbd_reader -- --nocapture

Expected: PASS (or a clear SKIP: nbdkit not on PATH if it could not be installed — then install it and re-run; do not commit on SKIP).

Step 5: fmt + clippy + commit

cargo fmt -p synvirt-migrator
cargo clippy -p synvirt-migrator -- -D warnings
git add crates/migrator/src/adapters/vim_api/nbd_reader.rs crates/migrator/src/adapters/vim_api/mod.rs crates/migrator/Cargo.toml
git commit -m "feat(synvirt-migrator): minimal async NBD client for nbdkit-backed disk reads"

Task 3: cert SHA-1 thumbprint helper (`thumbprint.rs`)

VDDK requires the source host certificate’s SHA-1 thumbprint (colon-separated upper hex). We already pin the cert during probe_source_health; derive the thumbprint from its DER bytes — do not re-fetch.

Files:

Create: crates/migrator/src/adapters/vim_api/thumbprint.rs
Modify: crates/migrator/src/adapters/vim_api/mod.rs (mod thumbprint;)

Interfaces:

Produces: pub fn sha1_thumbprint(cert_der: &[u8]) -> String → e.g. "AA:BB:..:FF".
Step 1: Write the failing test

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn known_vector() {
        // SHA-1 of the 3 bytes {0x01,0x02,0x03} = 7037807198c22a7d2b0807371d763779a84fdfcf
        let tp = sha1_thumbprint(&[0x01, 0x02, 0x03]);
        assert_eq!(tp, "70:37:80:71:98:C2:2A:7D:2B:08:07:37:1D:76:37:79:A8:4F:DF:CF");
    }
}

Step 2: Run test to verify it fails

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::thumbprint -- --nocapture Expected: FAIL — sha1_thumbprint not defined.

Step 3: Write minimal implementation

//! Derive the SHA-1 certificate thumbprint that the VDDK transport
//! requires, from the already-pinned source certificate's DER bytes.

use sha1::{Digest, Sha1};

/// Colon-separated, upper-hex SHA-1 thumbprint of a DER certificate.
pub fn sha1_thumbprint(cert_der: &[u8]) -> String {
    let digest = Sha1::digest(cert_der);
    digest
        .iter()
        .map(|b| format!("{b:02X}"))
        .collect::<Vec<_>>()
        .join(":")
}

Add sha1 = "0.10" to crates/migrator/Cargo.toml [dependencies] if not present (check first — the workspace may already pull it transitively; prefer a workspace dep if one exists).

Step 4: Run test to verify it passes

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::thumbprint -- --nocapture Expected: PASS.

Step 5: fmt + clippy + commit

cargo fmt -p synvirt-migrator && cargo clippy -p synvirt-migrator -- -D warnings
git add crates/migrator/src/adapters/vim_api/thumbprint.rs crates/migrator/src/adapters/vim_api/mod.rs crates/migrator/Cargo.toml
git commit -m "feat(synvirt-migrator): SHA-1 cert thumbprint helper for the VDDK transport"

Task 4: `nbdkit vddk` subprocess supervisor (`nbdkit.rs`)

Pure arg-builder (fully unit-tested) + spawn/readiness/teardown. The password goes in a mode-0600 file referenced as password=+<file>, never on argv.

Files:

Create: crates/migrator/src/adapters/vim_api/nbdkit.rs
Modify: crates/migrator/src/adapters/vim_api/mod.rs (mod nbdkit;)

Interfaces:

Consumes: nbd_reader::which_nbdkit.
Produces:
- pub struct VddkParams { pub libdir: String, pub server: String, pub user: String, pub vm_moref: String, pub snapshot_moref: String, pub vmdk_file: String, pub thumbprint: String }
- pub fn build_nbdkit_args(p: &VddkParams, socket: &str, password_file: &str) -> Vec<String> (pure)
- pub struct NbdkitProcess { socket: std::path::PathBuf, _child: tokio::process::Child, _pw: tempfile::NamedTempFile } with pub async fn start(p: &VddkParams, password: &str) -> Result<NbdkitProcess> and pub fn socket(&self) -> &std::path::Path. Drop kills the child (use .kill_on_drop(true)).
Step 1: Write the failing test (pure arg-builder)

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn args_wire_everything_and_keep_password_off_argv() {
        let p = VddkParams {
            libdir: "/opt/synvirt/vddk".into(),
            server: "10.0.0.5".into(),
            user: "root".into(),
            vm_moref: "vm-73".into(),
            snapshot_moref: "snapshot-12".into(),
            vmdk_file: "[ds1] synnet.siem/synnet.siem.vmdk".into(),
            thumbprint: "AA:BB".into(),
        };
        let args = build_nbdkit_args(&p, "/run/x.sock", "/run/pw");
        let joined = args.join(" ");
        assert!(joined.contains("vddk"));
        assert!(joined.contains("libdir=/opt/synvirt/vddk"));
        assert!(joined.contains("server=10.0.0.5"));
        assert!(joined.contains("vm=moref=vm-73"));
        assert!(joined.contains("snapshot=snapshot-12"));
        assert!(joined.contains("transports=nbdssl"));
        assert!(joined.contains("thumbprint=AA:BB"));
        assert!(joined.contains("password=+/run/pw"));
        assert!(joined.contains("-U") && joined.contains("/run/x.sock"));
        // the literal password must never appear
        assert!(!joined.contains("hunter2"));
        // readonly export
        assert!(joined.contains("--readonly"));
    }
}

Step 2: Run test to verify it fails

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::nbdkit -- --nocapture Expected: FAIL — build_nbdkit_args not defined.

Step 3: Write minimal implementation

//! Supervise one `nbdkit` instance running the `vddk` plugin to export a
//! source VM's snapshot disk over a private Unix socket. The proprietary
//! VixDiskLib stays isolated inside this child process.

use std::io::Write;
use std::path::{Path, PathBuf};

use tokio::process::Command;

use super::nbd_reader::which_nbdkit;
use crate::{Error, Result};

/// Inputs needed to export one snapshot disk over VDDK/NBDSSL.
pub struct VddkParams {
    pub libdir: String,
    pub server: String,
    pub user: String,
    pub vm_moref: String,
    pub snapshot_moref: String,
    pub vmdk_file: String,
    pub thumbprint: String,
}

/// Build the full `nbdkit` argv (pure; password is referenced by file).
pub fn build_nbdkit_args(p: &VddkParams, socket: &str, password_file: &str) -> Vec<String> {
    vec![
        "--exit-with-parent".into(),
        "--readonly".into(),
        "-U".into(),
        socket.into(),
        "vddk".into(),
        format!("libdir={}", p.libdir),
        format!("server={}", p.server),
        format!("user={}", p.user),
        format!("password=+{password_file}"),
        format!("thumbprint={}", p.thumbprint),
        format!("vm=moref={}", p.vm_moref),
        format!("snapshot={}", p.snapshot_moref),
        format!("file={}", p.vmdk_file),
        "transports=nbdssl".into(),
    ]
}

/// A running `nbdkit vddk` child plus the temp password file. Dropping it
/// kills the child (`kill_on_drop`) and unlinks the password file.
pub struct NbdkitProcess {
    socket: PathBuf,
    _child: tokio::process::Child,
    _pw: tempfile::NamedTempFile,
}

impl NbdkitProcess {
    /// Spawn `nbdkit vddk` and wait until its Unix socket is accepting.
    #[tracing::instrument(skip_all, fields(vm = %p.vm_moref))]
    pub async fn start(p: &VddkParams, password: &str) -> Result<NbdkitProcess> {
        if which_nbdkit().is_none() {
            return Err(Error::Internal("nbdkit not installed on host".into()));
        }
        let mut pw = tempfile::Builder::new()
            .prefix("synvirt-vddk-")
            .tempfile()
            .map_err(|e| Error::Internal(format!("vddk pw tmpfile: {e}")))?;
        pw.write_all(password.as_bytes())
            .map_err(|e| Error::Internal(format!("vddk pw write: {e}")))?;
        let pw_path = pw.path().to_string_lossy().to_string();

        let socket = std::env::temp_dir().join(format!("synvirt-nbd-{}.sock", p.vm_moref));
        let _ = std::fs::remove_file(&socket);
        let args = build_nbdkit_args(p, &socket.to_string_lossy(), &pw_path);

        let child = Command::new("nbdkit")
            .args(&args)
            .kill_on_drop(true)
            .stdout(std::process::Stdio::piped())
            .stderr(std::process::Stdio::piped())
            .spawn()
            .map_err(|e| Error::Internal(format!("spawn nbdkit: {e}")))?;

        // Readiness: socket appears once nbdkit has connected to the source.
        for _ in 0..150 {
            if socket.exists() {
                return Ok(NbdkitProcess { socket, _child: child, _pw: pw });
            }
            tokio::time::sleep(std::time::Duration::from_millis(100)).await;
        }
        Err(Error::Internal("nbdkit vddk did not become ready in time".into()))
    }

    /// Path to the NBD Unix socket this process serves.
    pub fn socket(&self) -> &Path {
        &self.socket
    }
}

Ensure tempfile is a normal [dependencies] entry now (it was dev-only in Task 2; promote it).

Step 4: Run test to verify it passes

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::nbdkit -- --nocapture Expected: PASS (arg-builder test; start is exercised by the E2E in Task 7).

Step 5: fmt + clippy + commit

cargo fmt -p synvirt-migrator && cargo clippy -p synvirt-migrator -- -D warnings
git add crates/migrator/src/adapters/vim_api/nbdkit.rs crates/migrator/src/adapters/vim_api/mod.rs crates/migrator/Cargo.toml
git commit -m "feat(synvirt-migrator): supervise an nbdkit-vddk subprocess per snapshot disk"

Task 5: Route warm reads through NBD + VDDK gate (`mod.rs`)

Branch read_disk_full/read_disk_delta: when the snapshot is warm (WARM_PREFIX) and VDDK is available, read via NbdkitProcess + nbd_reader; otherwise keep the existing NFC path unchanged. Add vddk_available() and expose warm_available in capabilities.

Files:

Modify: crates/migrator/src/adapters/vim_api/mod.rs
Modify: crates/migrator/src/http.rs (CapabilitiesResponse.warm_available), crates/migrator/src/jobs.rs (carry the flag)

Interfaces:

Consumes: WARM_PREFIX (existing const in mod.rs); create_migration_snapshot warm SnapshotRef JSON (snapshot MOR + per-disk change-ids); ffi::gov_delta_extents(vm, disk_key, since_change_id, current_snap_ref) -> Result<String> (JSON extents); the vmdk path cache (mod.rs ~1075-1086); nbdkit::{VddkParams, NbdkitProcess}; nbd_reader::{NbdClient, stream_full, stream_ranges}; thumbprint::sha1_thumbprint.
Produces: fn vddk_available() -> bool; warm-aware read_disk_full/read_disk_delta.
Step 1: Write the failing test (gate logic, pure)

In mod.rs #[cfg(test)]:

#[test]
fn vddk_gate_is_false_without_libdir() {
    // With no VDDK libdir env/path set, the gate is off and the
    // orchestrator keeps auto-downgrading warm -> cold.
    std::env::remove_var("SYNVIRT_VDDK_LIBDIR");
    assert!(!vddk_available());
}

Step 2: Run test to verify it fails

Run: cargo test -p synvirt-migrator --lib adapters::vim_api::tests::vddk_gate -- --nocapture Expected: FAIL — vddk_available not defined.

Step 3: Write minimal implementation

Add the gate (VDDK libdir defaults to /opt/synvirt/vddk, overridable for tests/dev):

/// Directory holding the VixDiskLib redistributable, overridable via env.
fn vddk_libdir() -> std::path::PathBuf {
    std::env::var_os("SYNVIRT_VDDK_LIBDIR")
        .map(std::path::PathBuf::from)
        .unwrap_or_else(|| std::path::PathBuf::from("/opt/synvirt/vddk"))
}

/// Whether the warm (live) disk reader can run: the govmomi shim is
/// linked, `nbdkit` is installed, and the VDDK libdir is present.
pub fn vddk_available() -> bool {
    cfg!(vim_shim)
        && nbd_reader::which_nbdkit().is_some()
        && vddk_libdir().join("lib64").exists()
}

Then wire the warm branch into read_disk_full (top of the existing fn, before the NFC tokio::spawn):

// Warm path: a WARM_PREFIX snapshot + an available VDDK engine reads the
// running VM's snapshot disk over nbdkit/NBD instead of NFC export.
if snapshot.0.starts_with(WARM_PREFIX) && vddk_available() {
    return self.read_via_nbd_full(vm_id, disk_id, snapshot);
}

Add read_via_nbd_full / read_via_nbd_delta methods on VimApiAdapter that:

parse the warm SnapshotRef JSON → snapshot_moref + the disk’s change_id;
resolve vmdk_file from the cache and the source creds (server, user, password) from self;
compute thumbprint via sha1_thumbprint(self.pinned_cert_der());
NbdkitProcess::start(&params, &password) then NbdClient::connect(proc.socket());
full: stream_full(client, CHUNK); delta: gov_delta_extents(...) → parse (offset,length) ranges → stream_ranges(client, ranges, CHUNK).

Keep the NbdkitProcess alive for the stream’s lifetime by moving it into the stream’s state (return a stream that owns (proc, client); extend stream_full/stream_ranges signatures to take an optional owner, or wrap with stream::unfold holding proc). Use CHUNK = 4 * 1024 * 1024.

read_disk_delta gets the same branch at its top; on the cold/no-VDDK path it keeps today’s behavior.

Implementation detail to verify against the live shim: confirm the warm SnapshotRef JSON field names emitted by gov_snapshot (snapshot MOR + per-disk changeId) at crates/migrator/src/adapters/vim_api/go-shim/shim.go ~765-820 and parse exactly those. Mirror the JSON shape read_disk_delta already parses today for gov_delta_extents.

Step 4: capabilities flag

In crates/migrator/src/http.rs add to CapabilitiesResponse (~882): pub warm_available: bool, and set it from app state (mirror preparer_available, ~979). In crates/migrator/src/jobs.rs carry a warm_available field next to preparer_available (~66/92/113/293) defaulting to vim_api::vddk_available() at construction. Add a builder with_warm_available.

Step 5: Run tests + build

cargo test -p synvirt-migrator --lib adapters::vim_api -- --nocapture
cargo build -p synvirt-migrator -p synvirt-migratord -p synvirt-daemon

Expected: gate test PASS; workspace builds.

Step 6: fmt + clippy + commit

cargo fmt -p synvirt-migrator && cargo clippy -p synvirt-migrator -- -D warnings
git add crates/migrator/src/adapters/vim_api/mod.rs crates/migrator/src/http.rs crates/migrator/src/jobs.rs
git commit -m "feat(synvirt-migrator): read running-VM snapshot disks over nbdkit-vddk on the warm path"

Task 6: ISO packaging + pinned VDDK artifact

Bake the NBD host tooling into the ISO and stage the VDDK redistributable as a pinned external artifact (the binary itself is never committed).

Files:

Modify: iso-builder package list (the live-build package-list file under iso-builder/live-build/config/package-lists/ — grep for where qemu-img/nbdkit siblings are declared; add nbdkit, nbdkit-vddk-plugin, libnbd0, libnbd-bin).
Create: iso-builder/vendor/vddk/MANIFEST.toml (sha256 + filename + staged path /opt/synvirt/vddk/), modeled on iso-builder/vendor/guest-payloads/MANIFEST.toml.
Create: iso-builder/scripts/prepare-vddk.sh (verify sha256, extract to staging), modeled on iso-builder/scripts/prepare-guest-payloads.sh.
Step 1: Inspect the existing pattern

Run: sed -n '1,60p' iso-builder/vendor/guest-payloads/MANIFEST.toml && ls iso-builder/scripts/prepare-guest-payloads.sh && ls iso-builder/live-build/config/package-lists/ Expected: see the manifest schema + prepare-script structure + where package lists live.

Step 2: Add host packages

Append nbdkit, nbdkit-vddk-plugin, libnbd0, libnbd-bin to the package list that already carries qemu-utils/migrator tooling. (Verify the exact list file by grep: grep -rl qemu-utils iso-builder/live-build/config/package-lists/.)

Step 3: Author the VDDK manifest + prepare script

iso-builder/vendor/vddk/MANIFEST.toml:

# VixDiskLib (VDDK) redistributable — external, NOT committed.
# Obtain the tarball out-of-band; pin its sha256 here. A missing or
# mismatched artifact is a hard build failure for the warm migrator.
[vddk]
filename = "VMware-vix-disklib.tar.gz"   # operator-provided
sha256   = "PROVIDE_ON_PROVISION"        # filled when the tarball is pinned
staged_to = "/opt/synvirt/vddk"          # nbdkit-vddk libdir

iso-builder/scripts/prepare-vddk.sh (mirror prepare-guest-payloads.sh): verify the sha256 against MANIFEST.toml, extract the tarball, and stage lib64/ under the squashfs /opt/synvirt/vddk/. If the artifact is absent, skip with a loud warning (warm stays auto-downgraded) rather than failing the whole ISO — gate this behind a WARM_VDDK=1 build flag so default ISO builds don’t require the proprietary blob.

Step 4: Commit (no binary)

git add iso-builder/vendor/vddk/MANIFEST.toml iso-builder/scripts/prepare-vddk.sh iso-builder/live-build/config/package-lists/
git commit -m "build(iso): bake nbdkit/libnbd and stage a pinned VDDK redistributable for warm migration"

Do NOT run the ISO build here (per project rule: never run iso-builder autonomously). The build runs later on Tony’s explicit “haz el iso”.

Task 7: Gated warm E2E case

Extend the existing env-gated E2E with a warm run against a running source. Runs only with SYNVIRT_E2E=1 and a VDDK-provisioned destination; otherwise skips+passes.

Files:

Modify: crates/synvirt-migratord/tests/e2e_external_source_uefi.rs
Step 1: Add the warm test

Mirror the existing cold flow but mode: warm, assert the job transitions CopyingFull → DeltaSyncing → CuttingOver → Completed, then assert the destination VM is running with the NIC on the expected vSwitch/VLAN. Guard the whole body behind if std::env::var("SYNVIRT_E2E").is_err() { return; } and a capability probe asserting warm_available == true (skip with a logged reason otherwise).

#[tokio::test]
async fn warm_external_source_running_vm() {
    if std::env::var("SYNVIRT_E2E").is_err() {
        eprintln!("SKIP: set SYNVIRT_E2E=1 to run");
        return;
    }
    // ... probe-cert -> source -> discover -> inspect (assert power_state running)
    // ... assert GET /migrator/capabilities warm_available == true (else SKIP)
    // ... create plan { mode: warm, nic_mappings: [{ source_nic_label, target_bridge:"", target_vlan:0, preserve_mac:true }] }
    // ... dry-run (no blockers) -> start -> poll until Completed
    // ... assert GET /vms/{name} state == running
    // ... assert the NIC bridge == default vSwitch and vlan == source vlan (inherited)
}

Step 2: Compile-check the test (no infra)

Run: cargo test -p synvirt-migratord --test e2e_external_source_uefi -- --list Expected: lists both *_uefi (cold) and warm_external_source_running_vm without compiling errors; with no SYNVIRT_E2E they skip.

Step 3: Commit

git add crates/synvirt-migratord/tests/e2e_external_source_uefi.rs
git commit -m "test(synvirt-migratord): gated warm E2E for a running source VM over VDDK"

Self-Review

Spec coverage:

Data engine (nbdkit-vddk subprocess + Rust NBD reader) → Tasks 2, 4, 5. ✓
Warm read routing + capability gate → Task 5. ✓
Snapshot/CBT reuse (orchestrator unchanged) → consumed in Task 5 via gov_delta_extents; no orchestrator rewrite needed (warm loop already calls read_disk_delta). ✓
Cert SHA-1 thumbprint → Task 3. ✓
vNIC VLAN fix → Task 1. ✓
Packaging + pinned VDDK → Task 6. ✓
Unit (nbdkit memory) + gated E2E → Tasks 2, 4, 7. ✓
Runtime gate / no regression → Task 5 vddk_available() + existing resolve_mode downgrade. ✓

Placeholder scan: The only deferred literal is the VDDK sha256 = "PROVIDE_ON_PROVISION" in Task 6 — intentional and documented (the proprietary tarball is provisioned out-of-band; the build skips warm without it). No code step is left as TODO.

Type consistency: DiskChunk { offset, length, data, is_zero }, BlockStream<'a>, SnapshotRef(pub String), NicMapping { source_nic_label, target_bridge, target_vlan: u16, preserve_mac }, NicSpec { label, source_vlan: Option<u16>, mac, source_network, model }, gov_delta_extents(vm, disk_key, since_change_id, current_snap_ref) -> Result<String> all match the live signatures (adapter.rs:19/76/85, types.rs:176/377-391/531-539, ffi.rs:319). which_nbdkit/VddkParams/build_nbdkit_args/NbdkitProcess/NbdClient/stream_full/stream_ranges/sha1_thumbprint/vddk_available/source_vlan/effective_vlan are each defined before use.

Known follow-up during execution: Task 5 must verify the warm SnapshotRef JSON field names against shim.go (snapshot MOR + per-disk changeId) and reuse the exact parse read_disk_delta already does for gov_delta_extents. Flagged in the task, not a placeholder.

Migrator Warm (Live) Migration via VDDK/NBD — Implementation Plan

Migrator Warm (Live) Migration via VDDK/NBD — Implementation Plan

Global Constraints

File Structure

Task 1: vNIC VLAN inheritance (second deliverable, independent)

Task 2: Minimal async NBD client (nbd_reader.rs)

Task 3: cert SHA-1 thumbprint helper (thumbprint.rs)

Task 4: nbdkit vddk subprocess supervisor (nbdkit.rs)

Task 5: Route warm reads through NBD + VDDK gate (mod.rs)

Task 6: ISO packaging + pinned VDDK artifact

Task 7: Gated warm E2E case

Self-Review

Task 2: Minimal async NBD client (`nbd_reader.rs`)

Task 3: cert SHA-1 thumbprint helper (`thumbprint.rs`)

Task 4: `nbdkit vddk` subprocess supervisor (`nbdkit.rs`)

Task 5: Route warm reads through NBD + VDDK gate (`mod.rs`)