Outline the active raster layer's canvas bounds (document space) so its size is
visible, especially when it differs from the document after a resize. Two strokes
sharing one dash pattern, the yellow offset by a dash to fill the black's gaps -
interlocking marching-ants. Sizes divided by zoom so the dashes stay ~constant on
screen.
ImageExportSettings gains `fit: ExportFitMode` (default Letterbox) + a "Fit"
dropdown in advanced image settings; the image render path uses it instead of the
hardcoded Letterbox. Image export supports a resolution override, so the mode
matters when the override aspect differs from the document.
A) Imported video had a different aspect ratio depending on how it was placed:
direct import used a uniform scale + center (preserve aspect), while the
asset-library timeline drag used an independent scale_x/scale_y (stretch to fill).
Add Transform::fit_centered (uniform scale, centered, aspect-preserving) and route
both paths through it, so a clip looks identical however it's added.
B) Exported video was stretched when the export resolution's aspect differed from
the document's (base_transform was always scale_non_uniform). Add
ExportFitMode {Stretch, Letterbox (default), Crop} on VideoExportSettings + a "Fit"
dropdown in advanced export settings, and a shared export_base_transform() helper:
Letterbox = uniform fit centered (black bars), Crop = uniform fill centered (trim),
Stretch = the old distort-to-fill. Threaded through the software, HDR, and
zero-copy export render paths (image export is doc-sized → identity).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two bugs made 4K (and any) video playback wrong:
1. Sped-up playback. The VideoManager frame cache was keyed by milliseconds, but
GPU (hardware-decoded) frames bypass the decoder's internal cache. A UI that
refreshes finer than the video frame rate (a 60Hz canvas on a 30fps clip)
therefore missed the cache on every sub-frame request and decoded the NEXT
frame each time — advancing the video ~2x faster than the clock, and racing the
decoder toward EOF. Key the cache on the video frame index (round(ts·fps))
instead, so all requests within one frame share an entry and each frame decodes
exactly once → correct speed.
2. Periodic seek/jerk. The seek decision compared the rounded request frame_ts to
the decoded frame's exact PTS, which sit on slightly different grids — so
frame_ts landed ~1 frame "behind" the just-decoded PTS every ~10 frames and
falsely read as a backward seek (40ms seek + GOP re-decode). Track the previous
request (last_requested_ts), which is strictly monotonic during forward
playback, and detect backward from that instead. Real scrubs still seek.
Per-frame HW decode is ~3-5ms; both bugs were spurious decode/seek work on top.
Keeps a gated LB_VIDEO_DEBUG [Video Seek?] diagnostic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The asset-library thumbnail called VideoManager::get_frame, which honors the
render-pass hardware flag — left ON by the preview, so the thumbnail got a GPU
frame with empty rgba_data → an all-black thumbnail. The thumbnail is also the
only consumer that requests a fixed low timestamp (1.0s) on the shared per-clip
decoder, so when it re-decodes during playback it yanks the decoder back to ~1s;
the next playback frame (6.x) is then ">2s forward" and re-seeks to the keyframe
+ re-decodes the whole GOP (the jerk: per-frame decode is ~5ms, but these seeks
cost 40ms + N-frame catch-up).
Add VideoManager::get_frame_cpu (forces want_gpu=false regardless of the render
flag); the thumbnail uses it. Now it produces a real RGBA thumbnail that the
editor texture-caches once, instead of re-decoding black frames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes the common Linux H.264 export fully GPU-resident: decode (HW NV12) →
composite → VAAPI encode all on one device, no CPU round-trips.
- gpu-video-encoder: ZeroCopyEncoder now holds wgpu (device,queue,adapter) handles
instead of owning a DrmDevice. New `new_on_device(device,queue,adapter,...)` runs
the RGBA→NV12 render + DMA-BUF import on a passed device; `new` keeps building its
own. The encoder only *imports* VAAPI surfaces (not export), so the shared
import-capable device works directly.
- editor: stash the shared device handles on EditorApp (set in the creation closure
from the eframe render_state when the shared device is active) and thread them
through start_video_export / start_video_with_audio_export → try_build_zero_copy,
which uses new_on_device when available. ZeroCopyVideo carries on_shared_device →
the export composite uses hardware_ok=true (consuming HW-decoded GPU frames from
pt 1) only then.
Falls back to the own-device encoder (decode downloads to CPU) when no shared
device. Tradeoff: the export now shares the GPU with the UI thread (was a separate
device); acceptable for the GPU-resident-decode win. Compiles; crate tests build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The software and HDR export paths already composite on the shared device (the
eframe device, via render_next_video_frame), so they can consume the same
hardware-decoded NV12 GPU frames the preview uses — no CPU download/upload:
- ExportGpuResources gains the Nv12BlitPipeline; the export compositor's Video arm
branches on inst.gpu (NV12 blit, mirroring stage.rs) vs the RGBA upload path.
- render_frame_to_gpu_rgba takes a hardware_ok flag: true for the software video +
image export (shared device), false for the zero-copy path (own device, must
download). render_frame_to_yuv10_hdr sets it true.
Drops the 4K software/HDR-export decode wall (HW decode + GPU composite, no
per-frame RGBA upload). The common Linux H.264 zero-copy path is unchanged
(separate device) — that's pt 2. Falls back to software when no shared device /
importer (flag is harmless then: get_frame returns CPU anyway). Compiles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire HDR frame production end-to-end (encoder side was pt 1):
- linear_to_pq.wgsl: linear scene HDR (BT.709, white=1.0) → BT.2020 primaries →
PQ (203-nit) or HLG OETF → gamma-encoded R'G'B'. Inverse of the nv12 decode, so
a decode→encode round-trip is the identity.
- hdr_frame.rs (isolated module): runs that pass into an Rgba16Float target, does a
synchronous readback (f16-decoded on CPU), then BT.2020 R'G'B'→Y'CbCr limited
4:2:0 10-bit pack → YUV420P10LE planes. Rgba16Float avoids the 16BIT_NORM device
feature dependency.
- video_exporter::render_frame_to_yuv10_hdr: composite + HDR encode, returning
10-bit planes; lazily builds the pipeline.
- orchestrator: HDR uses a synchronous 1-frame-per-call path (the async RGBA
pipeline is 8-bit only); zero-copy is skipped for HDR.
- dialog: "Dynamic range" dropdown (SDR / HDR10 PQ / HLG).
Software HEVC Main10 path; favors correctness over throughput. Compiles; runtime
verification needs a GPU + HDR-capable player.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings + encoder side of HDR video export (frame data still SDR until pt 2):
- core: HdrExportMode {Sdr (default), Pq, Hlg} on VideoExportSettings (serde
default), with transfer-name helpers.
- setup_video_encoder takes the mode: HDR → YUV420P10LE, BT.2020 NCL matrix,
limited range, color_primaries=bt2020, color_trc=smpte2084/arib-std-b67,
profile=main10. SDR path unchanged (8-bit full-range BT.709).
- run_video_encoder forces HEVC when HDR is selected (the only 10-bit codec wired
up). encode_frame is parameterized by pixel format and now copies planes
row-by-row honoring stride (10-bit / non-aligned widths can have row padding).
Dormant: no UI exposes the mode yet and the frame data is still 8-bit SDR, so
selecting HDR is not yet wired. The render→PQ/HLG 10-bit frame path is pt 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A failed DMA-BUF import returned None silently → core falls back to software,
whose RGBA path does no BT.2020→709 gamut conversion, so HDR clips render washed
out. The detection log fires before import_raw, so it didn't reveal this. Log the
actual import error (with ten_bit) instead of swallowing it, to confirm whether
10-bit P010 import is failing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Symptom: properly-converted BT.2020 PQ/HLG clips rendered progressively
desaturated vs the SDR reference — the signature of the transfer/primaries tags
not reaching the NV12→RGB shader, so HDR code values were decoded as plain
sRGB/BT.709 (no gamut expansion, wrong EOTF).
VAAPI hardware-decoded frames frequently leave color_trc/color_primaries/
colorspace/color_range unspecified — the authoritative values live on the codec
context (parsed from the bitstream), which the importer never sees. Before
importing, copy any unspecified colour field from the codec context onto the
frame, so the importer detects PQ/HLG + BT.2020 correctly.
Also add a one-time importer log (LB_VIDEO_DEBUG) of the detected
ten_bit/full_range/colorspace/primaries/trc to confirm what's picked up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
R16Unorm/Rg16Unorm aren't renderable, but the import gave every plane texture
RENDER_ATTACHMENT/COLOR_ATTACHMENT usage (needed only for the encoder's RGBA→NV12
write path) — so create_view panicked on 10-bit decode ("R16Unorm is not
renderable"). Gate the render-target usage (vk COLOR_ATTACHMENT, hal COLOR_TARGET,
wgpu RENDER_ATTACHMENT) on 8-bit; 16-bit P010 planes are sample-only
(TEXTURE_BINDING/RESOURCE + COPY_SRC), which is all the decode path needs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
10/12-bit HDR decodes to P010-style VAAPI surfaces (16-bit planes), which the
8-bit NV12 import couldn't handle — real HDR clips fell back to software (losing
the HDR). Import them as R16/Rg16 plane textures:
- dmabuf::Nv12DmaBuf gains `ten_bit`; import_raw picks R16_UNORM/R16G16_UNORM (vk)
+ R16Unorm/Rg16Unorm (wgpu) when set, else the existing R8/Rg8.
- hw_video.rs detects 10-bit from the hw frames context sw_format (P010/P012/P016)
and sets it. The NV12→RGB shader is unchanged: it samples normalized floats, and
the limited/full de-quant lands at the same values regardless of bit depth.
- vk_device requests TEXTURE_FORMAT_16BIT_NORM when the adapter supports it (R16/
Rg16 need it); absent → 10-bit falls back to software, 8-bit unaffected.
Pairs with the PQ/HLG/BT.2020 colour math so HDR10/HLG clips now reach the GPU
path end-to-end. NV12 callers (encoder, decode primitive) pass ten_bit=false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a per-document HDR→SDR mapping applied at the final linear→sRGB encode, so
super-white (HDR) video highlights can be recovered instead of hard-clipped:
- core: HdrOutputMode {Clip (default), HighlightRolloff} on Document (serde
default), with a SetDocumentPropertiesAction variant for undoable edits.
- shaders: a fs_main_rolloff entry point (preview linear_to_srgb.wgsl + the export
inline shader) applying a C1 highlight knee — identity below 0.8, smooth rolloff
[0.8,∞)→[0.8,1). SDR below the knee is untouched; Clip stays the historical path.
- preview (stage.rs) and both export encodes (video_exporter.rs) pick the pipeline
variant from document.hdr_output_mode — one value per frame, so no per-pixel
uniform; mirrors the existing fs_main_straight pattern.
- UI: an "HDR output" dropdown in the Document section of the info panel.
Default (Clip) is bit-identical to previous behaviour. Completes Stage A:
HDR-correct input (pt 1) + SDR-safe output mapping. HDR export (10-bit P010/PQ)
and HDR display remain Stages B/C.
Decode HDR video into the linear compositor correctly instead of approximating
everything as sRGB/BT.709:
- Read the frame's color_trc and color_primaries in the importer → VideoTransfer
{Gamma,Pq,Hlg} + VideoPrimaries {Bt709,Bt2020} on GpuVideoFrame.
- nv12_blit.wgsl: branch the EOTF — sRGB gamma (SDR), SMPTE2084 PQ (normalized so
203-nit graphics white = 1.0; highlights exceed 1.0), or HLG inverse-OETF
(reference white ≈ 1.0). Then BT.2020→BT.709 primaries in linear light when
wide-gamut, clamping out-of-709 colours.
Establishes the white=1.0 scene-linear convention: SDR content is unchanged
(stays in [0,1]); HDR video carries super-white highlights through compositing.
SDR-output mapping (clip default vs highlight rolloff) is Part 2. HLG's display
OOTF is omitted (scene-referred) — approximate but reasonable for SDR-out.
The NV12→RGB pass hardcoded BT.709, so SD (BT.601) clips had slightly wrong hues.
Read each frame's AVColorSpace in the importer and derive the Y'CbCr→R'G'B'
matrix (BT.709/601/240M/2020; Unspecified guessed by height like swscale/players),
carry the four coefficients on GpuVideoFrame, and apply them in the shader.
- core: GpuVideoFrame.coeffs + ycbcr_coeffs(kr, kb) helper.
- hw_video.rs: map AVColorSpace → (kr, kb) → coeffs.
- nv12_blit{.rs,.wgsl}: uniform grows to 80 bytes (adds a coeffs vec4); the matrix
multiply uses params.coeffs instead of literals.
BT.2020's transfer is still approximated as sRGB. The DRM-modifier-without-SAMPLED case stays a graceful software fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The WGSL struct trailed `full_range: u32` with `_pad: vec3<u32>`, but vec3 has
16-byte alignment so the struct rounded up to 80 bytes while the Rust-side
Nv12Params (BlitTransform + u32 + [u32;3]) is 64 — wgpu rejected the draw
("Buffer is bound with size 64 where the shader expects 80"). Pack the flag as a
single `flags: vec4<u32>` (.x = full_range) so both sides are 64 bytes.
Make the dormant core HW-decode engine live for the preview path:
- hw_video.rs: editor's HwVideoImporter — maps a decoded VAAPI surface to a
DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane textures on the *shared*
device (the only one with the import extensions). install() creates the VAAPI
device and injects it + the importer into the VideoManager.
- main.rs: track whether the shared device is actually in use; only then (Linux,
not LB_NO_SHARED_DEVICE) install hardware decode, using the CreationContext's
shared device + adapter.
- nv12_blit.rs + nv12_blit.wgsl: NV12 plane textures → BT.709 → sRGB-encoded →
linear, written straight into the Rgba16Float HDR layer (no CPU upload). Colour
math mirrors the software path so HW/SW video match; honours full_range.
- stage.rs: the preview Video arm branches on inst.gpu (NV12 blit) vs rgba_data
(existing upload+blit_straight); sets render_hardware_ok = !cpu_renderer so the
CPU fallback still gets software frames.
- video_exporter.rs: sets render_hardware_ok(false) before both compositing
passes — export composites on the encoder's separate device, so a hardware
decoder downloads to CPU instead (export stays software, correct).
- dmabuf.rs: imported plane textures now also carry SAMPLED/TEXTURE_BINDING so
they can be sampled by the NV12 blit (they were render-target-only); into_planes
hands the textures to the longer-lived GpuVideoFrame.
- video.rs: cache-key the GPU/CPU representation on want_gpu (HW-configured AND
render_hardware_ok) so software-only decode keeps a single cache entry.
Preview only this pass; export GPU-residency is the 3c-export follow-up. Untested
at runtime here (no GPU/display in container) — both crates compile.
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):
- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
downloads the surface via av_hwframe_transfer_data then swscales — so export
stays software/correct and only the preview goes GPU-resident. HW init or
import failure falls back to software per clip.
Dormant until the editor injects an importer (next): no importer => software,
unchanged.
import_raw now takes (&wgpu::Device, &wgpu::Adapter) and extracts the raw Vulkan
handles via as_hal, instead of taking the crate's own DrmDevice. This lets a
VAAPI surface be imported onto ANY DMA-BUF-import-capable wgpu device — the
encoder/decoder's own device today, and the editor's shared device (Stage 3a)
next, so hardware-decoded frames are usable by the preview compositor.
Encoder/decoder pass their DrmDevice's device+adapter; round-trip decode test
still passes.
Foundation for hardware video decode in BOTH preview and export: wgpu textures
can't cross devices, and a hardware-decoded frame is a DMA-BUF-imported texture
that needs the import extensions (only addable via wgpu-hal device_from_raw). So
eframe + the compositor + decode + encode must share ONE custom device.
- vk_device::create_windowed(): the existing import-capable DrmDevice plus
VK_KHR_swapchain (the WSI surface instance extensions are already enabled by
Instance::init), for use as the editor's main device.
- main.rs: on Linux, build the shared device and inject it into eframe via
WgpuSetup::Existing (the egui fork supports it); fall back to wgpu's normal
device + software decode on any failure, on other platforms, or via
LB_NO_SHARED_DEVICE. The DrmDevice's wgpu handles are cloned into eframe
(Arc-backed), so the VkDevice persists with them.
Runtime-verified: the editor launches and renders identically (canvas/vello,
video, panels) on the shared device, and the env-var fallback works.
Headless decode primitive, the mirror of the encoder: VaapiDecoder opens a file,
hardware-decodes H.264 into VAAPI NV12 surfaces (hw_device_ctx + a get_format
callback selecting AV_PIX_FMT_VAAPI), maps each surface to a DRM-PRIME DMA-BUF,
and imports it as two wgpu plane textures via the existing dmabuf::import_raw —
the same path the encoder uses, in the read direction. Frames stay GPU-resident
(no CPU copy).
Validated by a round-trip test: encode solid gray with ZeroCopyEncoder, decode
it back, read the Y plane (mean ≈ 128). All 9 crate tests pass on the container's
Intel GPU.
Next (Stage 3): wire this into VideoManager/get_frame so the compositor consumes
a GPU texture directly (no write_texture upload), with software decode fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The zero-copy VAAPI encoder emitted full-range BT.709 NV12 but wrote no color
tags, so players assumed limited range and stretched it — the H.264 output
looked dark and oversaturated (preview and VP9/software were fine).
- Rgba2Nv12 takes `full_range` and applies the matching Y/chroma scale+offset
(limited 16-235 / full 0-255) via a uniform; the encoder sets color_range +
bt709 colorspace/primaries/transfer tags to match. ffprobe-confirmed.
- New ColorRange { Limited, Full } on VideoExportSettings (default Limited, the
universally-correct choice; serde(default) so saved settings still load),
surfaced as a "Color range" dropdown in advanced export settings for H.264.
The swscale software fallback still emits Limited regardless of the toggle
(Full only affects the VAAPI zero-copy path).
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.
Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
The export render bucket was dominated by re-rendering the static document
background through Vello every frame and by nearest-sampling the video on
upscale.
- Background cache: render the (static) background through Vello once, snapshot
the composited HDR accumulator, and restore it with a single texture copy on
every later frame instead of a Vello render + sRGB-convert + composite (+2
submits). Invalidated on resize. background-render 3.6ms -> 0.56ms (1080p),
7.5ms -> ~0.5ms (4K).
- blit_straight now uses the bilinear sampler — video frames are scaled to the
output size, and nearest made that blocky. Fixes export and live preview.
- LB_RENDER_PROFILE: gated per-frame timing split (build/decode vs composite/
upload vs srgb, and background vs layers) used to find all of the above. Kept
as a debug aid for the remaining decode stages.
Net: export render ~12.8ms -> ~7.4ms/frame on a 1080p video clip.
get_frame rebuilt the RGBA swscale context on every decoded frame and printed
[Video Timing] lines unconditionally. A stream's frames share one input
format/size, so build the scaler once (keyed on format+dims, rebuilt only if
they change) and reuse it; gate the per-frame traces behind LB_VIDEO_DEBUG.
Cuts export wall time ~10% on a 1080p video clip (the scaler rebuild was the
bulk of the per-frame "scale" cost; it's now ~0ms). SwsContext is !Send, so the
cached scaler is wrapped in a SendScaler — sound because a VideoDecoder is only
ever touched under the VideoManager mutex (same invariant as its decoder/input).
The libva guard fired: the release ffmpeg had no h264_vaapi. ffmpeg-sys-next
configures with --disable-autodetect and only passes --enable-vaapi when its
build-vaapi feature is set; ffmpeg-next 8.0 doesn't forward it, so installing
libva-dev alone did nothing.
Inject a direct ffmpeg-sys-next dependency with the build-vaapi feature
(Linux only) in both the GitHub Actions and container build paths. With it,
FFmpeg gets --enable-vaapi (and fails loudly if libva-dev is missing rather
than silently shipping software-only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gpu_video_encoder's encoder/vaapi/vk_device/dmabuf modules are #[cfg(linux)]
(VAAPI is Linux-only), but the editor referenced them unconditionally, so
macOS/Windows failed to compile (cannot find `encoder` in `gpu_video_encoder`).
Gate the zero-copy path (ZeroCopyVideo, try_build_zero_copy,
run_zerocopy_video_export, and the branches in start_video_export /
start_video_with_audio_export) behind cfg(target_os = "linux"). Factor the
software encoder-thread spawn into spawn_software_video so both the None arm
and the non-Linux path share it without duplication. Non-Linux always uses
the software path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Video-only (no-audio) H.264 export now uses the same background-thread zero-copy
VAAPI path as video+audio, instead of the software encoder. It writes the output
.mp4 directly (no mux step) and reports through the single-export progress
channel; completion clears that channel so the dialog closes cleanly.
Factored the encoder/renderer/resources setup shared by both entry points into
ExportOrchestrator::try_build_zero_copy. start_video_export gains the document/
video_manager/raster_store/container_path inputs to seed the off-thread render
from a Document snapshot. Non-H264 / non-VAAPI still falls back to software.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Linux release ffmpeg is built statically from source, and ffmpeg only
autodetects --enable-vaapi when libva headers are present at configure time.
Neither build path installed them, so every release shipped a static ffmpeg
with no h264_vaapi encoder -- the zero-copy export silently fell back to
software 100% of the time.
- Containerfile + build.yml: install libva-dev + libdrm-dev so the from-source
ffmpeg gets VAAPI.
- build.yml: assert the release binary links libva, so a missing dep can't
silently regress to a software-only build again.
- deb/rpm: libva (libva2/libva-drm2/libdrm2, libva/libdrm) is a hard runtime
dep -- the vaapi-enabled ffmpeg DT_NEEDEDs it, so the app won't launch
without it. The VA driver is a soft recommends (absent it, export falls back
to software): va-driver-all (deb), intel-media-driver/mesa-va-drivers (rpm).
build.rs needs no change: releases link ffmpeg statically (its bundling path
is skipped) and the AppImage is thin, taking libva from the host like libvulkan.
The zero-copy VAAPI export previously rendered one frame per egui repaint on
the UI thread, which pinned throughput to the 60Hz vsync of the present loop
(measured exactly 16ms/frame) -- so the near-free hardware encode bought
nothing. Because the export runs on its own VAAPI device (independent of
eframe), it can run entirely off the UI thread.
- run_zerocopy_video_export: a background thread owning the encoder + its own
vello renderer/device, a Document snapshot (Document is Clone+Send; the UI
keeps the live one), its own ImageCache, and a RasterStore clone. Renders +
hardware-encodes every frame and reports through the same video_progress
channel the software encoder thread uses.
- start_video_with_audio_export takes the document/video_manager/raster_store/
container_path to seed the thread; video_state is None for this path.
- Throttle export-time UI repaints (~6Hz) and the thread's progress sends so
the render thread keeps the cores; the breakdown print stays.
- cancel() tears down parallel_export (detaches threads, removes temp files)
so the progress dialog dismisses; the call site closes the dialog.
- Gate the progress poll loop on has_pending_progress() so it stops once the
export ends instead of polling/logging every repaint forever; the single
export path clears its channel on the terminal event.
Vsync overhead is gone (0.1ms/frame); export is now render-bound (~11ms/frame
Vello scene-build). ~1:50 -> ~56s (~2x) on the validation clip.
Three fixes found while running the zero-copy export on real Intel hardware:
- vaapi::create_device() retries LIBVA_DRIVER_NAME in order iHD -> auto ->
i965 -> radeonsi. libva was auto-selecting the legacy i965 driver, which
fails on newer Intel GPUs; the modern iHD (intel-media-driver) is needed.
encoder.rs now builds its hwdevice through this helper.
- vk_device: request the adapter's full limits instead of downlevel_defaults.
Vello's compute pipelines need max_storage_buffers_per_shader_stage >= 5
(downlevel caps at 4), which panicked Vello's shader init on the export
device. This device only ever runs on a real VAAPI GPU.
- ZeroCopyEncoder: unsafe impl Send. It owns its FFmpeg/Vulkan handles
exclusively and is only moved (onto the export thread), never shared.
When exporting H.264 with audio, try the gpu-video-encoder ZeroCopyEncoder:
render each frame to RGBA and hardware-encode it into a VAAPI surface
inline, on the encoder's own VAAPI-capable wgpu device — no GPU->CPU
readback, no swscale, no software-encoder thread. Falls back to the
existing software path verbatim when VAAPI/the device is unavailable
(non-Linux, non-H264, or init failure), so it's additive.
- VideoExportState gains zero_copy: Option<ZeroCopyVideo> (encoder + its own
vello renderer + ExportGpuResources + a reused RGBA target, all on the
encoder's device).
- start_video_with_audio_export builds it for H.264 and skips spawning the
software encoder thread when present.
- render_next_video_frame routes to a zero-copy arm that reuses
render_frame_to_gpu_rgba on the encoder's device, then encode_rgba; on the
last frame finish() writes the temp .mp4 and sets video_progress=Complete
so the existing mux runs. video_thread=None makes the mux join a no-op.
Separate export device (vs modifying the eframe device) keeps this contained
to export. Video-only export stays on the software path for now. Runtime
verification (an actual H.264 export) is pending — cannot run the editor in
the dev container.
ZeroCopyEncoder::new now takes an output path and writes a real container
(format inferred from the extension, e.g. .mp4): create an output format
context, add the h264 stream from the encoder, write header; encode_rgba
rescales each packet's ts and av_interleaved_write_frame's it; finish
flushes + writes the trailer + closes. Sets AV_CODEC_FLAG_GLOBAL_HEADER for
mp4/mov so SPS/PPS land in extradata. This lets the editor's existing
mux_video_and_audio consume the temp video file unchanged.
The zerocopy_encode test now writes a .mp4 and ffprobe-verifies the codec,
dimensions, and frame count. Also let wgpu own the imported plane-image
destruction via texture_from_raw drop callbacks (clears two warnings).
Build the full end-to-end zero-copy encoder, validated on Intel/VAAPI:
- render_nv12: fragment-shader RGBA->NV12 that renders luma/chroma into
the imported R8/RG8 plane render targets (compute storage can't write
the DMA-BUF-backed planes; render attachments can).
- dmabuf: import_raw imports an NV12 DMA-BUF by explicit layout; the two
plane images + shared memory are now destroyed by wgpu via texture_from_raw
drop callbacks (Arc MemoryGuard frees the memory once both images are
gone, in wgpu's wait-idle'd deferred pass) -- fixes the teardown segfault.
- encoder::ZeroCopyEncoder: renders an RGBA texture straight into a pooled
VAAPI surface (imports cached by VASurface id) and encodes with h264_vaapi.
encode_rgba + finish; the caller renders on device().
Tests: real-frame render into the surface matches the CPU NV12 reference,
and a 30-frame encode produces valid H.264 (ffprobe-verified) with clean
teardown. Not yet wired into the editor.
New workspace crate isolating the unsafe GPU<->encoder interop for
zero-copy hardware video encoding. Every link is validated by a test on
real Intel/Mesa/iHD hardware:
- nv12: GPU RGBA->NV12 compute (BT.709 full-range), byte-exact vs a CPU
reference.
- vaapi: VAAPI hwcontext + h264_vaapi encode (CPU-fed NV12 -> valid H.264),
and DRM-PRIME surface layout probing.
- vk_device: a custom wgpu Vulkan device that adds
VK_EXT_image_drm_format_modifier (+ external-memory fd/dma-buf) via the
wgpu-hal device-from-raw path, so a tiled VAAPI surface can be imported.
- dmabuf: import a VAAPI NV12 surface's tiled DMA-BUF as two aliasing wgpu
textures (Y=R8, UV=RG8) at the plane offsets.
- zerocopy test: render values via Vulkan straight into the VAAPI surface
and read them back 100% correct -- proving the GPU writes into the
encoder surface with no CPU copy.
Not yet wired into the editor; real-frame render + encode-from-surface +
fallback wiring follow. Linux-only (libva); other platforms fall back.
The suite had accumulated breakage from prior refactors:
- selection unification: rewrite the integration tests for the single
unified clip_instances collection (shapes+clips are one set now).
- tempo-map: thread a TempoMap (constant 60 BPM = identity) into the
clip remap_time tests so the second-based expectations hold.
- drop two dead rgba_to_yuv420p tests that asserted tight plane sizes
incompatible with the function's 16-macroblock alignment.
- ignore the WIP theme var() cascade test (theme system not wired up).
Also a real bug the tests caught: auto_key_ranges produced overlapping
sample key ranges (the midpoint key mapped to both adjacent samples).
Start each range one past the previous midpoint.
The export read back 8MB RGBA per frame and ran swscale RGBA->YUV420p on
the UI thread (~6ms/frame). Add a tight GPU compute converter (gpu_yuv,
BT.709 full-range matching the encoder tags) and wire it into the
triple-buffered ReadbackPipeline: render to RGBA, convert on the GPU, read
back ~3MB of planar YUV, and skip the CPU pass. Gated on a runtime check
that the encoder's YUV420P plane strides are tight (no linesize padding),
with the swscale path as fallback for other dimensions; LB_DISABLE_GPU_YUV
forces the CPU path. Includes a CPU reference + unit tests for the packing.
Also guard render_next_video_frame against re-initializing/re-emitting
"Complete" every frame after the render finishes while the encoder/mux
drains (the completion nulled gpu_resources but left video_state set).
Offline export blocks each streaming source until decoded frames are
available. For a video whose audio track ends before the video (or a
container like WebM/Opus with no exact stream duration), the tail chunks
wait for frames that never arrive — the 10s safety valve fires per chunk,
so the export appears to hang indefinitely.
Add a `finished` flag to ReadAheadBuffer, set when the disk reader hits
EOF (cleared on seek). The export-mode wait now breaks immediately once
the source is finished and the requested frames aren't present, rendering
silence for the missing tail instead of timing out chunk-by-chunk. Only
the export path reads the flag, so playback is unaffected.
Imported video is a Group[Video, Audio] that rendered as a Vello-baked
Vector layer, re-uploading the full frame to Vello's image atlas every
frame (~17ms/frame at 1080p, hitting playback and export alike). Extract
video frames out of the Group/clip scene recursion into
VideoRenderInstances so they composite via the GPU Video path; mixed
video+vector containers fall back to Vello (correct, unaccelerated).
Also route video through hardware sRGB decode: upload raw sRGB bytes to an
Rgba8UnormSrgb texture and blit with a non-unpremultiplying shader variant
(blit_straight), removing the per-frame per-pixel CPU sRGB->linear pass.
Add an F3 GPU-timestamp timer and a per-frame video texture cache.
Drops the live composite of a 1080p video from ~17ms to ~2-3ms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Video was the only media type always kept external (VideoClip.file_path),
so a project with video wasn't self-contained. Now video packs into the
SQLite container under the same large-media policy as audio (pack < 2 GB
unless the user chose Reference), and both the frames and the embedded audio
track decode by streaming directly from the blob — no temp files.
- New crate ffmpeg-blob-io: an AVIOContext-over-Read+Seek shim (BlobInput)
that lets ffmpeg demux from an arbitrary byte source. Isolates all the
unsafe FFI + ffmpeg ABI coupling (version-pinned =8.0.0/=8.0.1). Manual
Drop teardown order; AVSEEK_SIZE restores the read position (FFmpeg assumes
a size query doesn't move it — required for MP4 moov-at-end).
- Schema/save/load: VideoClip.media_id; save_beam packs/references video as
MediaKind::Video (keyed by clip id); load resolves packed vs referenced and
reports missing sources. A packed clip points its linked video-audio pool
entry's media_id at the video row so the audio streams from the same blob.
- Frames: video.rs VideoSource{Path,Packed} threaded through new/seek/scan/
probe/thumbnails (a fresh BlobReader per open); editor builds the source
from current_file_path (now set before register_loaded_videos).
- Audio: VideoAudioReader::open_source via BlobInput; the disk_reader
StreamSource block on packed video-audio is removed; the engine's existing
factory activation routes it unchanged.
Tests: ffmpeg-blob-io AVIO unit tests (WAV via Cursor, seek, open/drop loop);
core packed_video_stream (blob->AVIO->Input) and beam_archive video round-trip;
daw-backend open_source test (compiles; links/runs only off-container).
Runtime-verified: a packed video plays frames + audio after the source file
is removed.
Address the code smells flagged in the .beam format spec:
- Write the project's actual sample rate on save instead of a hardcoded 48000
(add AudioProject::sample_rate()).
- Remove the vestigial RasterKeyframe.media_path field (it was only used by the
legacy ZIP loader, which now derives "media/raster/<id>.png" from the keyframe
id) and the dead buffer_path_at_time accessor. Backward-compatible: older files
carrying media_path deserialize fine (the field is ignored).
- Drop the unused SaveSettings fields auto_embed_threshold_bytes / force_embed_all
/ force_link_all; only large_media_mode was ever consulted. Un-prefix the now-used
`settings` parameter.
Update BEAM_FILE_FORMAT.md to match. The remaining notes (reserved MediaKind::Video,
exact-match version check) are design choices, left as-is.
The .beam container moved from ZIP to SQLite, but the docs/tooling still
described the old ZIP format.
- Rewrite BEAM_FILE_FORMAT.md as a normative spec of the current SQLite
container: file-magic identification, the four-table schema, the two version
numbers, the media model (kinds, packed/referenced storage, 4 MiB chunking),
derived media-id formulas, project.json top-level + key entities, save/load
semantics, the legacy-ZIP format + migration, large-media policy, conformance
and security sections, and known quirks.
- Port beam_inspector.py to read SQLite (auto-detecting container by magic,
with the legacy ZIP path preserved): a --media store view, packed/referenced
storage detection, and chunk-reassembling media extraction.
- Update beam_inspector_README.md to match.
Push to the 'all' remote (GitHub + Gitea pushurls) instead of origin so
releases are mirrored to Gitea too. CI still triggers via the GitHub pushurl.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shipping product is the pure-Rust app in lightningbeam-ui/; the old
JavaScript UI in src/ is browser-only (window.__TAURI__ is shimmed by
src/tauri_polyfill.js), so the src-tauri Tauri backend is dead. It also no
longer compiled (mid-refactor: missing video_server module, handler list
referencing deleted commands), and CI only ever built lightningbeam-ui.
- Delete src-tauri/ entirely (commands, native renderer/render-window,
websocket frame streamers, Tauri config, generated schemas, icons).
- Relocate icon.icns into lightningbeam-ui/lightningbeam-editor/assets/icons/
(the only src-tauri asset still needed) and repoint the editor's window-icon
include_bytes! at assets/icons/256x256.png (was src-tauri/icons/icon.png).
- CI/packaging: drop the redundant icon-copy steps (PNGs are committed in the
editor assets) and point macOS bundling at the relocated icns.
- package.json: drop @tauri-apps/* deps and the tauri script.
- Docs/.gitignore: drop src-tauri references.
daw-backend/ (shared audio engine) and the top-level WASM lightningbeam-core/
are untouched. Editor builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rust-ui is being merged into master and retired as the primary branch.
Update README, ARCHITECTURE, and CONTRIBUTING to reference master.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cmake-rs 0.1.54 panics with "unsupported or unknown VisualStudio version: 18.0"
because the windows-latest runner now ships Visual Studio 18 (2026), which the
crate's VS-generator detection doesn't recognize.
Install Ninja and set CMAKE_GENERATOR=Ninja for the Windows build so the cmake
crate skips VS-version detection and drives cl.exe (from the active MSVC env)
directly. build.rs already searches the single-config (non-Release) output dirs,
so no linking changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Git for Windows puts its usr\bin on PATH ahead of the MSVC tools, so under
Git Bash rustc picked up coreutils link.exe instead of MSVC link.exe and
every link (even host build scripts like proc-macro2/serde) failed with
"/usr/bin/link: extra operand".
Split the build step: non-Windows keeps the bash script (needs --target
handling); Windows builds under pwsh, where the MSVC environment from
msvc-dev-cmd stays at the front of PATH.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skyler Lehmkuhl