Decode HDR video into the linear compositor correctly instead of approximating
everything as sRGB/BT.709:
- Read the frame's color_trc and color_primaries in the importer → VideoTransfer
{Gamma,Pq,Hlg} + VideoPrimaries {Bt709,Bt2020} on GpuVideoFrame.
- nv12_blit.wgsl: branch the EOTF — sRGB gamma (SDR), SMPTE2084 PQ (normalized so
203-nit graphics white = 1.0; highlights exceed 1.0), or HLG inverse-OETF
(reference white ≈ 1.0). Then BT.2020→BT.709 primaries in linear light when
wide-gamut, clamping out-of-709 colours.
Establishes the white=1.0 scene-linear convention: SDR content is unchanged
(stays in [0,1]); HDR video carries super-white highlights through compositing.
SDR-output mapping (clip default vs highlight rolloff) is Part 2. HLG's display
OOTF is omitted (scene-referred) — approximate but reasonable for SDR-out.
The NV12→RGB pass hardcoded BT.709, so SD (BT.601) clips had slightly wrong hues.
Read each frame's AVColorSpace in the importer and derive the Y'CbCr→R'G'B'
matrix (BT.709/601/240M/2020; Unspecified guessed by height like swscale/players),
carry the four coefficients on GpuVideoFrame, and apply them in the shader.
- core: GpuVideoFrame.coeffs + ycbcr_coeffs(kr, kb) helper.
- hw_video.rs: map AVColorSpace → (kr, kb) → coeffs.
- nv12_blit{.rs,.wgsl}: uniform grows to 80 bytes (adds a coeffs vec4); the matrix
multiply uses params.coeffs instead of literals.
BT.2020's transfer is still approximated as sRGB. The DRM-modifier-without-SAMPLED case stays a graceful software fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the dormant core HW-decode engine live for the preview path:
- hw_video.rs: editor's HwVideoImporter — maps a decoded VAAPI surface to a
DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane textures on the *shared*
device (the only one with the import extensions). install() creates the VAAPI
device and injects it + the importer into the VideoManager.
- main.rs: track whether the shared device is actually in use; only then (Linux,
not LB_NO_SHARED_DEVICE) install hardware decode, using the CreationContext's
shared device + adapter.
- nv12_blit.rs + nv12_blit.wgsl: NV12 plane textures → BT.709 → sRGB-encoded →
linear, written straight into the Rgba16Float HDR layer (no CPU upload). Colour
math mirrors the software path so HW/SW video match; honours full_range.
- stage.rs: the preview Video arm branches on inst.gpu (NV12 blit) vs rgba_data
(existing upload+blit_straight); sets render_hardware_ok = !cpu_renderer so the
CPU fallback still gets software frames.
- video_exporter.rs: sets render_hardware_ok(false) before both compositing
passes — export composites on the encoder's separate device, so a hardware
decoder downloads to CPU instead (export stays software, correct).
- dmabuf.rs: imported plane textures now also carry SAMPLED/TEXTURE_BINDING so
they can be sampled by the NV12 blit (they were render-target-only); into_planes
hands the textures to the longer-lived GpuVideoFrame.
- video.rs: cache-key the GPU/CPU representation on want_gpu (HW-configured AND
render_hardware_ok) so software-only decode keeps a single cache entry.
Preview only this pass; export GPU-residency is the 3c-export follow-up. Untested
at runtime here (no GPU/display in container) — both crates compile.
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):
- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
downloads the surface via av_hwframe_transfer_data then swscales — so export
stays software/correct and only the preview goes GPU-resident. HW init or
import failure falls back to software per clip.
Dormant until the editor injects an importer (next): no importer => software,
unchanged.
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.
Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
get_frame rebuilt the RGBA swscale context on every decoded frame and printed
[Video Timing] lines unconditionally. A stream's frames share one input
format/size, so build the scaler once (keyed on format+dims, rebuilt only if
they change) and reuse it; gate the per-frame traces behind LB_VIDEO_DEBUG.
Cuts export wall time ~10% on a 1080p video clip (the scaler rebuild was the
bulk of the per-frame "scale" cost; it's now ~0ms). SwsContext is !Send, so the
cached scaler is wrapped in a SendScaler — sound because a VideoDecoder is only
ever touched under the VideoManager mutex (same invariant as its decoder/input).
Video was the only media type always kept external (VideoClip.file_path),
so a project with video wasn't self-contained. Now video packs into the
SQLite container under the same large-media policy as audio (pack < 2 GB
unless the user chose Reference), and both the frames and the embedded audio
track decode by streaming directly from the blob — no temp files.
- New crate ffmpeg-blob-io: an AVIOContext-over-Read+Seek shim (BlobInput)
that lets ffmpeg demux from an arbitrary byte source. Isolates all the
unsafe FFI + ffmpeg ABI coupling (version-pinned =8.0.0/=8.0.1). Manual
Drop teardown order; AVSEEK_SIZE restores the read position (FFmpeg assumes
a size query doesn't move it — required for MP4 moov-at-end).
- Schema/save/load: VideoClip.media_id; save_beam packs/references video as
MediaKind::Video (keyed by clip id); load resolves packed vs referenced and
reports missing sources. A packed clip points its linked video-audio pool
entry's media_id at the video row so the audio streams from the same blob.
- Frames: video.rs VideoSource{Path,Packed} threaded through new/seek/scan/
probe/thumbnails (a fresh BlobReader per open); editor builds the source
from current_file_path (now set before register_loaded_videos).
- Audio: VideoAudioReader::open_source via BlobInput; the disk_reader
StreamSource block on packed video-audio is removed; the engine's existing
factory activation routes it unchanged.
Tests: ffmpeg-blob-io AVIO unit tests (WAV via Cursor, seek, open/drop loop);
core packed_video_stream (blob->AVIO->Input) and beam_archive video round-trip;
daw-backend open_source test (compiles; links/runs only off-container).
Runtime-verified: a packed video plays frames + audio after the source file
is removed.
Thumbnail rendering fixes:
- Strip now tiles from each clip's true (unclamped) origin and draws only the
tiles intersecting the visible rect, so it scrolls correctly and shows the
right frames when a clip is scrolled partly off the left. Both render sites
(collapsed group + expanded track) share one draw_video_thumbnail_strip helper.
- On-clip strip no longer freezes on the first thumbnail: get_thumbnail_at now
returns the actual thumbnail timestamp and the GPU texture cache keys on it, so
tiles refresh as closer thumbnails finish generating.
- Hover preview derives content time from the clip's true origin too (matches the
strip when scrolled off-screen).
- insert_thumbnail keeps the cache sorted + deduped (fixes a latent unsorted
binary_search bug, and makes concurrent restore + resume race-safe).
Thumbnail persistence (mirrors waveform persistence):
- MediaKind::Thumbnail rows, keyed by thumbnail_media_id(clip_id) (clip id XOR a
sentinel). Each clip's thumbnails PNG-encoded into one opaque LBTN blob (editor
owns the format), snapshotted cheaply (Arc clones) and encoded off the UI thread.
- Save writes the packs (kept in place on re-save); load reads them into
LoadedProject.thumbnail_blobs; the editor decodes + inserts them on a background
thread, so reload shows thumbnails instantly with no re-decode (even if the
source video file is missing).
- Partial sets are persisted with a complete flag and RESUMED on load:
generate_keyframe_thumbnails takes a should_skip predicate so a save made
mid-generation continues from where it left off instead of redoing the work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- VideoManager.frame_cache: unbounded HashMap (grew per distinct frame during
playback) -> LruCache evicted by a 256MB byte budget. Byte-budget rather than
frame count is robust across resolutions (a 4K frame is ~33MB vs ~2MB at
800x600). unload_video pops per-clip keys (LruCache has no retain).
- mux_video_and_audio: stream-merge the two inputs by PTS with one pending
packet per stream (O(1) memory) instead of collecting every packet into Vecs
first (O(duration)). Output is byte-identical.
- export AAC: sanitize the planar-f32 path (non-finite -> 0, finite clamped to
[-1,1]) like the integer paths, with a one-time warning. A stray NaN/Inf
render sample no longer fails the whole export.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the .beam container to SQLite and stream media from it instead of
decoding whole files into RAM on import/load.
Container & large files:
- SQLite .beam container (beam_archive) with in-place transactional saves and an
incremental BlobReader; supports both packed (chunked blobs) and referenced
(external path) media, with a user preference + first-import prompt for files
over the large-media threshold.
Audio streaming:
- Stream packed compressed audio on load via an inversion-of-control blob factory
(AudioBlobSourceFactory): daw-backend defines the trait, core implements it
over BlobReader, so the audio engine stays container-agnostic.
- Bulk-activate disk streaming for all loaded clips after SetProject.
- Sample-accurate compressed seek (SeekMode::Accurate; Coarse mislands on VBR).
Video:
- Video frames decoded/streamed on demand; thumbnails generated asynchronously
on a dedicated decoder so import/load never blocks the UI.
- The video's audio track is streamed on demand via an ffmpeg VideoAudioReader
as a separate editable AudioClip (no /tmp WAV extraction).
Waveform overview:
- Streaming min/max LOD pyramid (waveform_pyramid), bounded memory, configurable
floor B; serialized into the container and restored on load (or generated in
the background from the packed blob when absent), so no re-decode on reload.
- GPU min/max upload path; integer-LOD textureLoad fixes zoom-dependent wobble.
Skyler Lehmkuhl