PCIe passthrough + Sunshine streaming

This document is the design north star for SynVirt’s “Horizon-class” remote workstation path: a discrete GPU (or vGPU mediated device) exclusively assigned to one guest, with the guest streaming its display via Sunshine to remote Moonlight clients.

Why

SPICE’s compression pipeline is QXL-bound and the vendored spice-html5 client is frozen at 2021. For workloads that need modern paravirt / native graphics performance — CAD, video editing, gaming, 3D-heavy admin tools, multi-monitor remote desktops — the realistic 2026 path is to give the VM a real GPU and let it encode its own framebuffer into a low-latency H.264/HEVC/AV1 stream that a modern client decodes.

This is the architecture that Horizon Cloud, Citrix VAD, AWS Workspaces with Hardware GPU, and Stadia-style cloud gaming all converge on — modulo who licenses the streaming protocol.

Stack

┌──────────────────────── host (SynVirt) ────────────────────────┐
│                                                                 │
│  pcie passthrough         libvirt domain XML                    │
│      ▲                       ▲                                  │
│      │ vfio-pci              │ <hostdev mode='subsystem'>       │
│      │ binding via           │   <source><address.../></source> │
│      │ driverctl             │ </hostdev>                       │
│  ┌───┴───────┐                                                  │
│  │ /sys/bus/ │                                                  │
│  │ pci/...   │                                                  │
│  └───────────┘                                                  │
│      ▲                                                          │
│      │ enumerate                                                │
│      │ + iommu_groups/                                          │
│  GET /api/v1/host/pci-devices                                   │
│  POST /api/v1/host/pci-devices/:bdf/bind-vfio                   │
│  POST /api/v1/host/pci-devices/:bdf/release                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │  PCIe passthrough
                              ▼
┌──────────────────────── guest VM ──────────────────────────────┐
│                                                                 │
│  Operator sees an NVIDIA / AMD / Intel GPU as a real PCIe       │
│  device, installs vendor driver, runs Sunshine as a service.    │
│                                                                 │
│  Sunshine                                                       │
│   ├── captures framebuffer (DXGI / NvFBC / VAAPI)               │
│   ├── encodes via NVENC / AMF / QSV (H.264, H.265, AV1)         │
│   └── serves Moonlight protocol over UDP/RTP on :47984/:47989   │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │  Moonlight protocol
                              ▼
                       Moonlight client
                       (Win/Mac/Linux/iOS/Android/web)

Sunshine + Moonlight is the open-source reimplementation of NVIDIA GameStream. It runs on any guest OS that has a GPU and a hardware encoder. It’s the de-facto open standard for low-latency desktop streaming since GeForce Experience deprecated GameStream in 2023.

Why Sunshine and not WebRTC pixel streaming

Both are valid for 2026. Sunshine wins for this layer because:

Hardware encoder + protocol are co-designed. Latency target is <10 ms encode + transmit on LAN. WebRTC adds a SFU jitter buffer.
Codec parity with the GPU. AV1 if hardware supports it (Arc / RTX 40 / RX 7000), HEVC otherwise, H.264 fallback. WebRTC is catching up but H.264 is still the safe path.
Multi-touch + gamepad + USB redirection work out of the box.
Operator runs nothing in the host. All encoding is in-guest. The host’s job ends at “GPU is bound to the VM”.

Counter-arg: WebRTC pixel streaming gives a browser-native client without an install. SynVirt should support both (Sunshine for the high-perf workstation, WebRTC for the dashboard quick-look).

What SynVirt needs to ship

Five stages, ordered by independence:

Stage A — Architecture + ADR (this doc)

Done.

Stage B — Host PCI inventory (read-only)

Endpoint that enumerates everything in /sys/bus/pci/devices/ and groups them by /sys/kernel/iommu_groups/. Answers the operator question “what GPUs does this host have, and can I passthrough one without dragging unrelated devices along?”

GET /api/v1/host/pci-devices → list of PciDeviceView with:
- BDF (0000:01:00.0)
- Vendor + device IDs + names (read from /usr/share/misc/pci.ids if available, else hex)
- Class (display, audio, network, …)
- Current driver binding (nvidia, amdgpu, vfio-pci, …)
- IOMMU group ID
- Boolean: passthrough_eligible (in a non-empty group, not in use by host, not the boot GPU on a non-headless server)
GET /api/v1/host/iommu → IOMMU status (enabled / cmdline / mode intel_iommu=on vs amd_iommu=on vs vIOMMU nested)

Testable in this dev appliance because nested vIOMMU exposes groups.

Stage C — Renderer `<hostdev>` element

Add <hostdev mode='subsystem' type='pci' managed='yes'> rendering to libvirt-renderer. managed='yes' lets libvirt handle the detach-from-host-driver / re-attach lifecycle automatically — no manual virsh nodedev-detach dance from operators.

New HostDevice field on VmContext (Vec, ordered).
New <hostdev> element renderer next to <disk> / <interface> in elements/devices/.
Insta snapshot for a “windows-11 + GPU” sample profile.

Pure rendering; no runtime side effects. Fully testable on the dev box.

Stage D — Dashboard UX

Add a “Passthrough hardware” tab to the VM detail (or a step in the create wizard).
Read /api/v1/host/pci-devices, group by IOMMU group, surface whole-group selection (you can’t passthrough one function of a multi-function device without taking the rest).
Show eligibility + “currently bound to driver X” + “in use by VM Y” with helpful hints.
Empty state: “This host has no IOMMU enabled” with a one-button “Enable IOMMU and reboot” affordance (stage E feature).

Stage E — Runtime binding (`vfio-pci`)

The hard part. Touches host kernel state.

POST /api/v1/host/pci-devices/{bdf}/bind-vfio — driverctl unbind from current driver, bind to vfio-pci. Persisted in /etc/modprobe.d/synvirt-vfio.conf so it survives reboot.
Boot config: ensure intel_iommu=on iommu=pt (or AMD equiv) is in GRUB_CMDLINE_LINUX_DEFAULT. Ditto vfio-pci early-load via initramfs-tools. Idempotent helper.
Reset method discovery: prefer FLR, fall back to bus reset. Surface the picked method on the device row so operators know what to expect on stop/restart.
Hot-attach/detach via virsh attach-device <vm> <hostdev.xml> for already-running VMs (only works for some devices; primary GPU changes need a power cycle).
ACS override module loader for hosts where the BIOS doesn’t publish ACS capabilities cleanly — gated behind a “force unsafe groups” toggle with a giant warning, since splitting an unsafe IOMMU group is a security regression.

Testable only on a real host with a real GPU. Not on this nested appliance.

Stage F — Sunshine guest documentation

A short markdown page covering:

“Install Sunshine in your Windows / Linux guest”
Recommended firewall rules (Sunshine’s Moonlight ports)
Pairing flow (4-digit PIN against the Sunshine web UI)
Codec / bitrate guidance per use case (admin desktop / CAD / gaming)

Sunshine is not bundled in guest-tools.iso — it’s a 50 MB binary the operator installs once. Bundling it would balloon the ISO and duplicate Sunshine’s update cadence.

Hardware support matrix (informational)

GPU class	vfio-pci	Reset	Sunshine encoder	Notes
NVIDIA RTX consumer (3060+/4060+)	✅	FLR (since 30xx)	NVENC AV1 (40+), HEVC (20+), H.264	Pre-30xx need `kvm.ignore_msrs=1` and Code 43 workaround
NVIDIA Tesla / Quadro / RTX A-series	✅ + vGPU mediated	FLR	NVENC enterprise grade	Optional NVIDIA vGPU Manager for multi-tenant slicing (license required)
AMD RX 6000 / 7000 / 9000	✅	FLR	AMF HEVC, AV1 (7000+)	“AMD reset bug” historical; mostly resolved in Linux 6.x kernels
AMD Radeon Pro / Instinct	✅ + SR-IOV (Instinct MI series)	FLR	AMF	SR-IOV needs `amdgpu` Linux 5.15+
Intel Arc A-series / Arc Pro	✅ + SR-IOV (Arc Pro)	FLR	QSV AV1, HEVC, H.264	Arc SR-IOV is the most operator-friendly multi-tenant path in 2026
Apple Silicon GPU	❌	n/a	n/a	No PCIe in modern Macs; out of scope

Open questions

Audio. GPU audio function (e.g. 0000:01:00.1) is in the same IOMMU group. Pass it through too, surface guest audio via Sunshine. Document that the host’s Bluetooth/USB audio controllers are separate.
Multi-GPU hosts. Should we support multiple passthrough’d GPUs to multiple VMs simultaneously? Yes by default — IOMMU groups make this clean. The dashboard just filters currently_bound_vm == null.
Headless host. If the only GPU on the host is passed through, the host loses console output (text mode tty falls back to VGA framebuffer if available). Block this in the bind flow until the operator confirms with a typed PIN or the host has a second GPU/iGPU.
Live migration. vfio-pci passthrough breaks libvirt migration. Document as a known limitation; offer no-migration badge on passthrough’d VMs.

PCIe passthrough + Sunshine streaming

PCIe passthrough + Sunshine streaming

Why

Stack

Why Sunshine and not WebRTC pixel streaming

What SynVirt needs to ship

Stage A — Architecture + ADR (this doc)

Stage B — Host PCI inventory (read-only)

Stage C — Renderer <hostdev> element

Stage D — Dashboard UX

Stage E — Runtime binding (vfio-pci)

Stage F — Sunshine guest documentation

Hardware support matrix (informational)

Open questions

See also

Stage C — Renderer `<hostdev>` element

Stage E — Runtime binding (`vfio-pci`)