WebRTC video pipeline — fixed-cadence rewrite

Synced read-only from /home/synnet/mirrors/synvirt-product/docs/superpowers/specs/2026-05-12-webrtc-fixed-cadence-pipeline-design.md. Edit at the source, not here.

WebRTC video pipeline — fixed-cadence rewrite

Author: Tony Castrejón + Claude Date: 2026-05-12 Scope: crates/synvirt-webrtc Target host for validation: 10.10.26.52 (synvirt-01 baremetal)

---

1. Problem

Browser WebRTC stats on a live console session show:

Metric	Observed	Expected
Interframe LIVE	534 ms	~16 ms (60 fps)
Processing LIVE	6.8 ms	low, as observed
Jitter buf cum.	201 ms	<50 ms
Interframe cum.	343 ms	~16 ms

Daemon and browser are on the same LAN (no RTT issue). The pipeline degrades progressively over the lifetime of a session: latency starts low and accumulates.

2. Root cause

Two independent flaws compound:

2.1 Cadence mismatch + keyframe death spiral

• Encoder thread ticks at frame_period = 1_000_000 / fps µs (vm_source.rs:610) — 16.6 ms at 60 fps.
• Mux drain ticks at hardcoded 20 ms (session.rs:528, :548, :610).
• Drift: every ~5 mux ticks, the broadcast holds 2 frames → write_live_frames sees intermediate_dropped > 0 → calls source.request_keyframe() (session.rs:537-541). The same call also fires on every TryRecvError::Lagged (session.rs:519-521).
• VP8 keyframes at 1080p are 5-10× the size of P-frames. Bigger frames → more UDP fragments → slower mux iteration → larger drain backlog → another keyframe request → positive feedback loop. Steady-state degrades into a near-all-keyframe stream where useful bitrate collapses and browser-perceived latency grows over time.

2.2 Skip-when-unchanged + 500 ms heartbeat

When QEMU’s -display dbus is quiet (Windows DWM not pushing resource_flushs), the encoder skip-gate at vm_source.rs:908-919 drops frames until last_real_encode_at.elapsed() >= heartbeat_interval where heartbeat_interval = 500 ms (:679). Effective frame rate collapses to 2 fps with 534 ms interframe — exactly matching the observed metric when the guest is between flushes.

2.3 Secondary accumulators

• mpsc::unbounded_channel<InputCommand> (vm_source.rs:381) — no cap. If the encoder thread stalls, input commands pile up indefinitely.
• UDP socket bound with kernel-default SO_SNDBUF (~208 KB on Linux). A single 1080p keyframe (~500 KB after RTP) saturates it and forces send_to.await to block, which freezes the entire mux task.

3. Goals

• Deterministic 60 fps end-to-end when a session is open and the guest is active.
• No accumulators. Every queue/buffer in the pipeline is either bounded with a drop policy, or self-draining at the same rate it fills.
• No server-initiated keyframes except when the browser sends PLI/FIR via Event::KeyframeRequest.
• CPU-idle when nobody is watching. Preserve the tx.receiver_count() == 0 → sleep 1s gate.

4. Non-goals

• Changing the codec (still VP8 via libvpx).
• Hardware-accelerated encoding (vaapi, NVENC).
• Changing the transport (still str0m + UDP).
• Replacing dbus-cpu with another capture backend.
• Per-session encoders (single encoder per VM stays).

5. Architecture

5.1 Before

[zbus listener task]                  [encoder OS thread]                [mux tokio task]
  scanout/update           ─push─▶   loop @ frame_period:                loop:
  → fb.lock().await                    block_on(next_frame()) ◀──────      for c in clients:
  → bgra in-place                      ↳ awaits dirty_rx.changed()         c.maybe_write_frame()
  → dirty_tx.send(gen)                 SKIP gate (raw_seq, 500ms heartbeat)  ↳ try_recv loop on broadcast(8)
                                       bgra_to_i420 (rayon)                  ↳ "Lagged" or drained>=2:
                                       libvpx encode                            request_keyframe() ← spiral
                                       broadcast.send(Arc<VmFrame>)        ↳ rtc.writer.write(newest)
                                                                            ↳ tick = now + 20ms (hardcoded)
                                                                          drain transmit_queue → send_to.await
                                                                          select! (cmd / recv / sleep)

5.2 After

[zbus listener task]                  [encoder OS thread]                [mux tokio task]
  scanout/update           ─push─▶   loop @ frame_period:                loop:
  → fb.lock().await                    sleep until next tick               for c in clients:
  → bgra in-place                      grab latest fb (lock + clone)       c.maybe_write_frame()
  → dirty_tx.send(gen) [optional]      bgra_to_i420 (rayon)                  ↳ try_recv loop on broadcast(8)
                                       libvpx encode (P-frame default)        ↳ "Lagged" / drained>=2:
                                       broadcast.send(Arc<VmFrame>)             METRIC ONLY — no keyframe req
                                                                              ↳ rtc.writer.write(newest)
                                                                              ↳ tick = now + frame_period
                                                                          drain transmit_queue → send_to.await
                                                                          select! (cmd / recv / sleep)

Behavioural delta vs 5.1:

• No dirty_rx.changed().await gate in the encoder critical path.
• No skip-when-unchanged. No heartbeat path.
• No request_keyframe() from the mux on drain backlog.
• Mux per-client tick aligns with encoder tick.

6. Detailed changes

All changes confined to crates/synvirt-webrtc/.

6.1 src/vm_source.rs

Where	What
:381	mpsc::unbounded_channel::<InputCommand>() → mpsc::channel::<InputCommand>(256). Callers (axum handlers, dispatcher) switch from unbounded_send to try_send. On TrySendError::Full: drop the command and emit a tracing::warn! once per second per VM (rate-limited). 256 slots is enough headroom for 2 s of 120 Hz mouse-move bursts even if the encoder is briefly stalled; if it ever fills, the encoder is wedged and dropping is the only sane outcome.
:667 (resolve_keepalive_config_with_default)	Remove function. Encoder no longer uses keepalive.
:668-680	Remove last_real_fb, keepalive_enabled, keepalive_period, heartbeat_interval, last_real_encode_at, related counters.
:823-870 (next_frame timeout block)	Replace with a non-blocking current_frame() call on the backend. Encoder ticks on its own clock; if the backend has no fb yet (very first frame after start), continue and wait one tick.
:908-919 (skip-when-unchanged gate)	Delete. Always encode.
:786-789 (idle gate receiver_count() == 0)	Keep unchanged.

6.2 src/capture/mod.rs (trait CaptureBackend)

Add:

/// Non-blocking snapshot of the most recent framebuffer. Returns
/// `Ok(None)` if no scanout has landed yet. Never awaits I/O.
async fn current_frame(&mut self) -> Result<Option<Framebuffer>, CaptureError>;

async fn is fine because the dbus backend takes a tokio::sync::Mutex, but the contract is: the lock acquisition is the only await, and it never waits for a remote producer.

6.3 src/capture/dbus.rs

Implement current_frame:

async fn current_frame(&mut self) -> Result<Option<Framebuffer>, CaptureError> {
    let g = self.state.fb.lock().await;
    let Some(fb) = g.as_ref() else { return Ok(None) };
    Ok(Some(Framebuffer {
        width: fb.width,
        height: fb.height,
        stride: fb.width * 4,
        format: PixelFormat::Bgra8,
        data: Arc::new(fb.bgra.clone()),
        captured_at: fb.captured_at,
        raw_seq: fb.generation,
    }))
}

The existing next_frame stays for compatibility but the encoder no longer calls it on the dbus backend.

6.4 src/capture/rfb.rs

Implement current_frame returning the last decoded framebuffer the RFB poll task wrote into the shared slot. No new RFB request is issued.

6.5 src/session.rs

Where	What
:519-521 (Lagged arm)	Keep the counter increment + debug log. Delete the s.request_keyframe() call.
:537-541 (intermediate-dropped block)	Keep the counter. Delete the s.request_keyframe() call.
:528, :548, :610 (return now + 20ms)	Replace Duration::from_millis(20) with Duration::from_micros(1_000_000 / source.fps().max(1) as u64). write_live_frames already takes source: Option<&Arc<VmSource>> (:499), so no schema change is needed; VmSource::fps() exists (lib.rs:273). For synthetic source the path keeps FRAME_DURATION from test_pattern.

6.6 src/lib.rs — UDP socket buffer

Replace tokio::net::UdpSocket::bind(...).await (:127-136) with:

use socket2::{Socket, Domain, Type};
let sock2 = Socket::new(Domain::IPV4, Type::DGRAM, None)?;
sock2.set_send_buffer_size(4 * 1024 * 1024)?;
sock2.set_recv_buffer_size(4 * 1024 * 1024)?;
sock2.set_nonblocking(true)?;
sock2.bind(&bind_addr.into())?;
let std_sock: std::net::UdpSocket = sock2.into();
let socket = tokio::net::UdpSocket::from_std(std_sock)?;

Add socket2 to Cargo.toml. Log a warning if the kernel clamps the buffer below 1 MB (requires sysctl net.core.{rmem,wmem}_max).

6.7 Telemetry preserved

All VIDEO_RX_LAGGED_FRAMES, STALE_VIDEO_FRAMES_DROPPED, VIDEO_FRAMES_SKIPPED_BEFORE_ENCODE counters stay (they now stop climbing in steady state — that’s the proof). The vm-source profile log (:1005-1018) stays. The new design’s success looks like:

• avg_encode_us ~3000-8000 (1080p) / ~15000-25000 (4K).
• video_rx_lagged_frames flat after warm-up.
• stale_video_frames_dropped near zero.
• newest_frame_seq climbs at ~60/s.
• video_frames_skipped_before_encode flat at zero (gate removed).

7. Behaviour contracts

• First frame after session open. Mux loop starts ticking immediately; encoder force-keyframes on the very first frame (vm_source.rs:380 initial force_keyframe = true stays). Browser PLI not needed.
• Browser PLI/FIR. udp_mux.rs:261-271 keeps routing Event::KeyframeRequest to source.request_keyframe(). This is the only path that requests a server-side keyframe.
• Transient lag (1-3 frames delayed in broadcast). Latest-frame-wins still drops intermediates; the missing P-frame reference causes ~16-50 ms smear, no more. Browser sees no glitch unless lag > broadcast depth (8 frames = 133 ms); at that point the browser will likely PLI on its own.
• Guest idle / no Update from QEMU. Encoder still ticks at 60 Hz and encodes the same pixel content as P-frames (libvpx outputs tiny ~100-300 byte P-frames for unchanged input). Bandwidth ~50-200 kbps idle, ~3-4 Mbps active. No more 500 ms blackouts.
• No session open. receiver_count() == 0 gate at :786-789 puts the encoder loop to sleep for 1 s per tick. CPU idle.

8. Validation plan

Manual on 10.10.26.52:

1. Open browser console session to a Windows VM.
2. Capture 60 s of chrome://webrtc-internals graphs before deploy.
3. Deploy patched binary via scripts/deploy-to-host.sh hot (HOST=10.10.26.52).
4. Re-open session, capture 60 s of graphs after.
5. Confirm:
   • Interframe LIVE ≤ 20 ms steady-state.
   • No keyframe storm visible in framesEncoded vs keyFramesEncoded.
   • journalctl shows avg_encode_us consistent with 1080p baremetal budget.
   • VM idle for 30 s with session open → bandwidth stays at floor, interframe still ~16 ms (no 500 ms blackout).

No unit tests added; the existing offer_carries_sctp_and_two_distinct_input_channels and label_constants_match_frontend_contract cover SDP wire contract. The behaviour we’re changing is loop dynamics, which a unit test can’t catch reliably — the validation lives in the runtime metrics above.

9. Risks

• current_frame blocking under fb lock contention. The dbus listener holds fb.lock() for the duration of one row-by-row memcpy. At 4K that’s a few ms. Encoder waiting on the lock = ticked late. Mitigation: none needed at 1080p; if 4K becomes routine, switch to arc_swap::ArcSwap for the framebuffer slot (separate change).
• CPU floor with session open + guest idle: ~50% of one core per VM at 1080p, ~150% at 4K. Acceptable per goals. If multiple VMs run sessions simultaneously on a small host, operator should size accordingly. Document in CHANGELOG only.
• Browser BWE behaviour change. Without artificial keyframe storms, TWCC will see a smoother bitrate profile. Browsers may adjust jitter buffer down (better). No expected regression.

10. Rollback

Single git revert of the implementation commit. State on the host before deploy is captured by the deploy-to-host.sh hot workflow keeping prior binaries at /opt/synvirt/bin/synvirt-daemon.prev.

11. Out of scope (deliberately deferred)

• Per-session encoders (Sunshine-style fanout).
• Hardware VP8/AV1 encoding.
• Replacing broadcast with watch::channel<Option<Arc<VmFrame>>> per session. Worth measuring after this change ships.
• Tuning MAX_LOOP_TIMEOUT from 50 ms downward.
• str0m pacer / BWE configuration.
• 4K-class capture path optimisations (ArcSwap, dmabuf zero-copy).