Symptom: properly-converted BT.2020 PQ/HLG clips rendered progressively
desaturated vs the SDR reference — the signature of the transfer/primaries tags
not reaching the NV12→RGB shader, so HDR code values were decoded as plain
sRGB/BT.709 (no gamut expansion, wrong EOTF).
VAAPI hardware-decoded frames frequently leave color_trc/color_primaries/
colorspace/color_range unspecified — the authoritative values live on the codec
context (parsed from the bitstream), which the importer never sees. Before
importing, copy any unspecified colour field from the codec context onto the
frame, so the importer detects PQ/HLG + BT.2020 correctly.
Also add a one-time importer log (LB_VIDEO_DEBUG) of the detected
ten_bit/full_range/colorspace/primaries/trc to confirm what's picked up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a per-document HDR→SDR mapping applied at the final linear→sRGB encode, so
super-white (HDR) video highlights can be recovered instead of hard-clipped:
- core: HdrOutputMode {Clip (default), HighlightRolloff} on Document (serde
default), with a SetDocumentPropertiesAction variant for undoable edits.
- shaders: a fs_main_rolloff entry point (preview linear_to_srgb.wgsl + the export
inline shader) applying a C1 highlight knee — identity below 0.8, smooth rolloff
[0.8,∞)→[0.8,1). SDR below the knee is untouched; Clip stays the historical path.
- preview (stage.rs) and both export encodes (video_exporter.rs) pick the pipeline
variant from document.hdr_output_mode — one value per frame, so no per-pixel
uniform; mirrors the existing fs_main_straight pattern.
- UI: an "HDR output" dropdown in the Document section of the info panel.
Default (Clip) is bit-identical to previous behaviour. Completes Stage A:
HDR-correct input (pt 1) + SDR-safe output mapping. HDR export (10-bit P010/PQ)
and HDR display remain Stages B/C.
Decode HDR video into the linear compositor correctly instead of approximating
everything as sRGB/BT.709:
- Read the frame's color_trc and color_primaries in the importer → VideoTransfer
{Gamma,Pq,Hlg} + VideoPrimaries {Bt709,Bt2020} on GpuVideoFrame.
- nv12_blit.wgsl: branch the EOTF — sRGB gamma (SDR), SMPTE2084 PQ (normalized so
203-nit graphics white = 1.0; highlights exceed 1.0), or HLG inverse-OETF
(reference white ≈ 1.0). Then BT.2020→BT.709 primaries in linear light when
wide-gamut, clamping out-of-709 colours.
Establishes the white=1.0 scene-linear convention: SDR content is unchanged
(stays in [0,1]); HDR video carries super-white highlights through compositing.
SDR-output mapping (clip default vs highlight rolloff) is Part 2. HLG's display
OOTF is omitted (scene-referred) — approximate but reasonable for SDR-out.
The NV12→RGB pass hardcoded BT.709, so SD (BT.601) clips had slightly wrong hues.
Read each frame's AVColorSpace in the importer and derive the Y'CbCr→R'G'B'
matrix (BT.709/601/240M/2020; Unspecified guessed by height like swscale/players),
carry the four coefficients on GpuVideoFrame, and apply them in the shader.
- core: GpuVideoFrame.coeffs + ycbcr_coeffs(kr, kb) helper.
- hw_video.rs: map AVColorSpace → (kr, kb) → coeffs.
- nv12_blit{.rs,.wgsl}: uniform grows to 80 bytes (adds a coeffs vec4); the matrix
multiply uses params.coeffs instead of literals.
BT.2020's transfer is still approximated as sRGB. The DRM-modifier-without-SAMPLED case stays a graceful software fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the dormant core HW-decode engine live for the preview path:
- hw_video.rs: editor's HwVideoImporter — maps a decoded VAAPI surface to a
DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane textures on the *shared*
device (the only one with the import extensions). install() creates the VAAPI
device and injects it + the importer into the VideoManager.
- main.rs: track whether the shared device is actually in use; only then (Linux,
not LB_NO_SHARED_DEVICE) install hardware decode, using the CreationContext's
shared device + adapter.
- nv12_blit.rs + nv12_blit.wgsl: NV12 plane textures → BT.709 → sRGB-encoded →
linear, written straight into the Rgba16Float HDR layer (no CPU upload). Colour
math mirrors the software path so HW/SW video match; honours full_range.
- stage.rs: the preview Video arm branches on inst.gpu (NV12 blit) vs rgba_data
(existing upload+blit_straight); sets render_hardware_ok = !cpu_renderer so the
CPU fallback still gets software frames.
- video_exporter.rs: sets render_hardware_ok(false) before both compositing
passes — export composites on the encoder's separate device, so a hardware
decoder downloads to CPU instead (export stays software, correct).
- dmabuf.rs: imported plane textures now also carry SAMPLED/TEXTURE_BINDING so
they can be sampled by the NV12 blit (they were render-target-only); into_planes
hands the textures to the longer-lived GpuVideoFrame.
- video.rs: cache-key the GPU/CPU representation on want_gpu (HW-configured AND
render_hardware_ok) so software-only decode keeps a single cache entry.
Preview only this pass; export GPU-residency is the 3c-export follow-up. Untested
at runtime here (no GPU/display in container) — both crates compile.
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):
- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
downloads the surface via av_hwframe_transfer_data then swscales — so export
stays software/correct and only the preview goes GPU-resident. HW init or
import failure falls back to software per clip.
Dormant until the editor injects an importer (next): no importer => software,
unchanged.
The zero-copy VAAPI encoder emitted full-range BT.709 NV12 but wrote no color
tags, so players assumed limited range and stretched it — the H.264 output
looked dark and oversaturated (preview and VP9/software were fine).
- Rgba2Nv12 takes `full_range` and applies the matching Y/chroma scale+offset
(limited 16-235 / full 0-255) via a uniform; the encoder sets color_range +
bt709 colorspace/primaries/transfer tags to match. ffprobe-confirmed.
- New ColorRange { Limited, Full } on VideoExportSettings (default Limited, the
universally-correct choice; serde(default) so saved settings still load),
surfaced as a "Color range" dropdown in advanced export settings for H.264.
The swscale software fallback still emits Limited regardless of the toggle
(Full only affects the VAAPI zero-copy path).
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.
Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
get_frame rebuilt the RGBA swscale context on every decoded frame and printed
[Video Timing] lines unconditionally. A stream's frames share one input
format/size, so build the scaler once (keyed on format+dims, rebuilt only if
they change) and reuse it; gate the per-frame traces behind LB_VIDEO_DEBUG.
Cuts export wall time ~10% on a 1080p video clip (the scaler rebuild was the
bulk of the per-frame "scale" cost; it's now ~0ms). SwsContext is !Send, so the
cached scaler is wrapped in a SendScaler — sound because a VideoDecoder is only
ever touched under the VideoManager mutex (same invariant as its decoder/input).
The suite had accumulated breakage from prior refactors:
- selection unification: rewrite the integration tests for the single
unified clip_instances collection (shapes+clips are one set now).
- tempo-map: thread a TempoMap (constant 60 BPM = identity) into the
clip remap_time tests so the second-based expectations hold.
- drop two dead rgba_to_yuv420p tests that asserted tight plane sizes
incompatible with the function's 16-macroblock alignment.
- ignore the WIP theme var() cascade test (theme system not wired up).
Also a real bug the tests caught: auto_key_ranges produced overlapping
sample key ranges (the midpoint key mapped to both adjacent samples).
Start each range one past the previous midpoint.
Imported video is a Group[Video, Audio] that rendered as a Vello-baked
Vector layer, re-uploading the full frame to Vello's image atlas every
frame (~17ms/frame at 1080p, hitting playback and export alike). Extract
video frames out of the Group/clip scene recursion into
VideoRenderInstances so they composite via the GPU Video path; mixed
video+vector containers fall back to Vello (correct, unaccelerated).
Also route video through hardware sRGB decode: upload raw sRGB bytes to an
Rgba8UnormSrgb texture and blit with a non-unpremultiplying shader variant
(blit_straight), removing the per-frame per-pixel CPU sRGB->linear pass.
Add an F3 GPU-timestamp timer and a per-frame video texture cache.
Drops the live composite of a 1080p video from ~17ms to ~2-3ms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Video was the only media type always kept external (VideoClip.file_path),
so a project with video wasn't self-contained. Now video packs into the
SQLite container under the same large-media policy as audio (pack < 2 GB
unless the user chose Reference), and both the frames and the embedded audio
track decode by streaming directly from the blob — no temp files.
- New crate ffmpeg-blob-io: an AVIOContext-over-Read+Seek shim (BlobInput)
that lets ffmpeg demux from an arbitrary byte source. Isolates all the
unsafe FFI + ffmpeg ABI coupling (version-pinned =8.0.0/=8.0.1). Manual
Drop teardown order; AVSEEK_SIZE restores the read position (FFmpeg assumes
a size query doesn't move it — required for MP4 moov-at-end).
- Schema/save/load: VideoClip.media_id; save_beam packs/references video as
MediaKind::Video (keyed by clip id); load resolves packed vs referenced and
reports missing sources. A packed clip points its linked video-audio pool
entry's media_id at the video row so the audio streams from the same blob.
- Frames: video.rs VideoSource{Path,Packed} threaded through new/seek/scan/
probe/thumbnails (a fresh BlobReader per open); editor builds the source
from current_file_path (now set before register_loaded_videos).
- Audio: VideoAudioReader::open_source via BlobInput; the disk_reader
StreamSource block on packed video-audio is removed; the engine's existing
factory activation routes it unchanged.
Tests: ffmpeg-blob-io AVIO unit tests (WAV via Cursor, seek, open/drop loop);
core packed_video_stream (blob->AVIO->Input) and beam_archive video round-trip;
daw-backend open_source test (compiles; links/runs only off-container).
Runtime-verified: a packed video plays frames + audio after the source file
is removed.
Address the code smells flagged in the .beam format spec:
- Write the project's actual sample rate on save instead of a hardcoded 48000
(add AudioProject::sample_rate()).
- Remove the vestigial RasterKeyframe.media_path field (it was only used by the
legacy ZIP loader, which now derives "media/raster/<id>.png" from the keyframe
id) and the dead buffer_path_at_time accessor. Backward-compatible: older files
carrying media_path deserialize fine (the field is ignored).
- Drop the unused SaveSettings fields auto_embed_threshold_bytes / force_embed_all
/ force_link_all; only large_media_mode was ever consulted. Un-prefix the now-used
`settings` parameter.
Update BEAM_FILE_FORMAT.md to match. The remaining notes (reserved MediaKind::Video,
exact-match version check) are design choices, left as-is.
Resolve all compiler warnings across daw-backend, lightningbeam-core, and
lightningbeam-editor:
- Delete dead code: the superseded CPU raster tools in raster_tool.rs
(EffectBrush/Smudge/Gradient/Transform/Warp/Liquify/Selection — replaced by
the GPU path), plus orphaned helpers and never-read struct fields.
- Mechanical fixes: drop unused imports/variables/mut, underscore unused params,
`drop(&x)` -> `let _ = x`, deprecated egui::Rounding -> CornerRadius, snake_case
rename, elided-lifetime Cow<'_, [u8]>.
- Keep the WIP CSS theming system (theme.rs/theme_render.rs) under
#[allow(dead_code)] rather than deleting it.
Editor checks warning-free; 293 core tests pass.
Block geometry/clip edits on frames that fall strictly inside a tween span,
where an edit would silently mutate the bracketing keyframe instead:
- Shape tweens: gate vector editing (vertices, curves, DCEL hits) and all
geometry tools in stage.rs behind VectorLayer::is_tween_inbetween.
- Motion tweens: block selecting/dragging/transforming clip instances whose
transform is mid-tween, via AnimationData::is_object_tweened_at.
Also: inserting a keyframe mid-tween now captures the interpolated geometry
shown at that frame (not the left keyframe's) and inherits the shape tween,
so the new keyframe continues morphing toward the right keyframe.
`tween_after == Shape` was stored on keyframes but never read. Now the
render path morphs geometry across a shape-tween span:
- VectorGraph::interpolated(other, t): same-topology lerp of vertex
positions, edge curves, stroke widths and stroke/fill colours. Returns
None when topology differs (counts, deleted flags, edge endpoints, fill
boundaries), so the caller holds the source keyframe.
- VectorLayer::tweened_graph_at(time): returns an owned morphed graph for
a shape-tween span whose two keyframes share topology, else borrows the
held keyframe. Editing still uses graph_at_time (the held keyframe).
- Renderer (Vello + CPU paths) renders via tweened_graph_at.
- SetTweenAction + wired the previously-stubbed "Add Shape Tween" menu.
The typical workflow — keyframe, duplicate it (same topology), move
vertices, Add Shape Tween — now morphs between the two. Non-matching
topology falls back to a hold.
Within a vector layer, groups and movie clips (clip instances) were drawn
before the layer's own VectorGraph, so they appeared underneath the loose
shapes. Draw the loose shapes first and the clip instances on top, in both
the Vello and CPU render paths.
Creating the first transform keyframe for a clip instance at frame N left
the curve with a single keyframe, which Hold-extrapolates backward — so
moving it at frame N also moved it on every earlier frame (frame 1).
When SetKeyframeAction creates a brand-new curve for a clip instance and
the clip already existed before `time`, also anchor a keyframe at the
clip's start (its group visibility start, or timeline_start for movie
clips) with the original value. Earlier frames now hold the original
position and the move produces a proper tween from start to N.
Also capture the clip instance's actual on-stage value when keying (its
base transform), instead of a generic 0/identity default, so a new
keyframe doesn't snap the clip to the origin.
Adds VectorLayer::group_visibility_start and tests covering the anchor and
the no-double-anchor case (keying at the clip's own start).
A dense self-intersecting freehand lasso leaves clusters of near-coincident
duplicate sub-pixel edges (split products the coincident-edge dedupe can't
reach). The planar face trace bounces back and forth across them, producing
a degenerate "spike" boundary (an edge used twice). Add
`collapse_boundary_spikes`, run on each traced face before it becomes a
fill: it removes consecutive out-and-back entries (where the boundary
returns to where it started) until the loop is simple.
Embed the captured region-select dumps as committed fixtures under
tests/region_dumps/ (replayed by `dumped_region_selects_are_valid`) so the
regression survives /tmp being cleared. dump3 is the boundary-spike repro;
it fails this test without the collapse and passes with it. Loosen the
boundary-connectivity test tolerance to 1e-2 (above sub-micropixel float
drift, far below any real gap).
Collapse the two parallel selection systems into one. The RegionSelect
tool (rect + lasso) now cuts the geometry along the region outline and
selects the resulting sub-pieces into the standard `Selection` ID-sets,
exactly like every other tool. The vestigial floating `RegionSelection`
(drag never wired; commit/delete/copy were stubbed) and all its plumbing
are removed, so Group, Convert-to-Movie-Clip, Delete, and Properties all
operate uniformly from lasso, rect, marquee, and click selections.
Region cutting is reworked onto a robust planar arrangement:
- Replace fragile incremental "split a fill by one cut edge" logic with
planar face re-tracing (`retrace_fills_after_cut` + `trace_faces`),
which correctly handles arbitrary holed/concave fills.
- `extract_subgraph` no longer frees vertices still referenced by kept
boundary edges (fixed Group leaving freed-but-referenced vertices that
a later alloc reused and corrupted).
- `split_fill_by_*` direction fix (was producing disconnected boundaries
rendered as stray diagonals).
- `fill_interior_point` (area-centroid + inward-step fallback) for
reliable inside/outside classification of non-convex pieces.
- Coincident-edge dedupe + degenerate-fill removal (edge-adjacent shapes
no longer make zero-area sliver fills).
- Dangling-edge pruning, near-coincident endpoint welding, induced-
subgraph expansion, and tracking of `split_edge` sub-edges, so
self-intersecting freehand lassos cut correctly.
Region-select capture is available behind LIGHTNINGBEAM_DUMP_REGION=1 for
turning a misbehaving cut into a deterministic test. Extensive regression
tests added in vector_graph/tests/region_cut_select.rs.
Both actions were DCEL-stubs (no-ops). They now extract the selected geometry from a
vector layer's active keyframe into a new VectorClip (group vs movie clip) and place a
ClipInstance in its place (identity transform → renders where the geometry was), which
the existing transform-animation system can motion-tween.
- Shared `clip_from_geometry::extract_geometry_to_clip` (+ undo) does the work; the
actions are thin wrappers. Undo snapshots the source graph + removes the clip/instance.
- `extract_subgraph` now DERIVES shared-fill boundary edges internally (an inside edge
still used by a non-extracted fill must be duplicated, not moved) and unions them with
the caller's `explicit_boundary` — so a plain geometry selection needs no boundary
analysis (the selection already includes fill boundary edges via `select_fill`). The
region-select caller keeps passing its lasso boundary.
- Handlers: Group / Convert to Movie Clip on a geometry selection now build these actions
from `selected_fills`/`selected_edges`.
Next: shape tweens (same-topology lerp).
`assets_needed_at(document, time)` (core) enumerates the image asset ids referenced by
the visible vector layers' active keyframes at a time (top-level + group children).
During playback the stage decodes the images needed ~0.5s ahead into the bounded
ImageCache, so a keyframe that swaps image fills doesn't hitch when the playhead
reaches it. Gated on is_playing; nested clip-instance recursion + background decode
are refinements.
Completes Phase 4 (image asset paging: Tier 2 decoded-cache LRU, Tier 1 lazy bytes,
playback prefetch).
Project load no longer eager-reads all image bytes — `ImageAsset.data` stays empty
and the renderer's ImageCache pages compressed bytes from the `.beam` on a decode
miss (read_packed_media_readonly by asset id), decoding into the byte-bounded Tier-2
cache. Result: instant load, and compressed bytes don't accumulate on the heap.
- ImageCache: `container_path` + `resolve_bytes` (asset.data if resident — fresh
import or old base64 project — else page from the container); decoders take `&[u8]`
and use the decoded dimensions.
- Container path threaded App.current_file_path → SharedPaneState → VelloRenderContext,
set on the cache each prepare.
- load_beam_sqlite drops the 3.5b eager read.
(Refinement: a persistent read connection instead of open-per-miss.)
The decoded-image cache (peniko ImageBrush + tiny-skia Pixmap, ~w·h·4 each) was
unbounded — the main asset-memory cost. Now capped at 256 MB with usage-LRU eviction:
every get_or_decode bumps the asset's recency, and inserts past the budget evict the
least-recently-used (a miss re-decodes from the resident asset.data). Images actually
rendered each frame stay resident; unused ones age out under pressure. invalidate/clear
keep the lru + byte accounting in sync.
Next (4b): lazy-load asset.data from the container instead of eager on project open.
Image asset bytes are now stored as MediaKind::ImageAsset rows in the SQLite
container (chunked, kept-in-place on re-save) instead of base64-embedded in the
project JSON — the pageable storage Phase 4 needs.
- ImageAsset.data is `#[serde(default, skip_serializing)]`: never written to JSON,
but still deserialized for old projects (base64) which then migrate to the
container on the next save.
- save_beam writes each asset's bytes (keyed by asset id; ext from the source path),
keeping an existing row when bytes aren't resident; live_media covers them so orphan
cleanup doesn't drop them.
- load_beam_sqlite eager-reads the bytes back into `data` (Phase 4 makes this lazy +
LRU). Old base64 projects keep their JSON-deserialized data (no container row).
- SetImageFillAction (core): set/clear `image_fill` on the selected VectorGraph fills,
with per-fill undo (mirrors SetFillPaintAction). Image takes render priority; clearing
reveals the colour/gradient underneath.
- Info Panel Shape section: an "Image:" combo listing the document's image assets (+ None)
for the selected fill(s), showing the current assignment. Assign/clear pushes the action.
This lets an existing shape be given (or cleared of) an image fill, complementing the
import/drop placement. Next: 3.5b — persist image assets in the .beam container.
The renderer painted the image brush at its native origin (0,0) with no brush
transform, so an image-filled rect drawn anywhere but the world origin only showed
the overlapping corner of the image. Both render paths now map the image's native
pixel space onto the fill's bounding box (Vello brush_transform; tiny-skia Pattern
transform) — 1:1 for an image-sized rectangle, stretch-to-bbox for arbitrary shapes.
Replaces the DCEL "not yet supported" stubs so importing/dropping an image actually
puts it in the vector scene.
- AddShapeAction gains an `image_fill` + `AddShapeAction::image_rect(...)` constructor:
a borderless rectangle (invisible edges) whose enclosed region is paint-bucketed and
tagged with an image asset id. The renderer already prioritises `image_fill`.
- Direct import (auto_place_asset): an imported image is placed centered on the canvas
at native size on a vector layer.
- Drag from the asset library onto the stage: image-filled rect at the drop point
(centered), native size, using the asset's dimensions.
Next: SetImageFillAction + an Info-Panel image-fill picker for existing shapes; then
3.5b container persistence.
- Core compositor gains an optional screen-blend tint per CompositorLayer
([0,0,0,0] default = no-op, so existing compositing is unaffected);
CompositorLayer::with_tint sets it — giving the Vello/vector path a tint hook.
- For the active VECTOR layer with onion on, build ghost scenes at each neighbouring
keyframe's time via render_layer_isolated (reusing the prepare's image cache), then
render each scene → sRGB → linear → composite with warm/cool tint + opacity falloff,
behind the current frame. Off during playback; active layer only.
Completes onion skinning for all layer types (raster + vector).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make raster layers behave like vector on the timeline.
- Timeline: draw a diamond per `RasterKeyframe` (mirrors the vector keyframe block).
- New Keyframe (K / menu): on a raster layer, insert a BLANK cel at the playhead via
a new undoable `AddRasterKeyframeAction` (+ `RasterLayer::insert_blank_keyframe_at`
/ `remove_keyframe`). Vector path unchanged.
- Stop lazy creation: paint tools now edit the ACTIVE keyframe (at-or-before the
playhead) instead of creating one. The brush captures the active keyframe's exact
time; `RasterStroke`/`RasterFillAction` resolve via `keyframe_at_mut` (error if
none); the tool-site `ensure_keyframe_at` blocks (brush/fill/bucket/wand/quick-
select/floating-lift) are removed — each read already bails when no keyframe exists.
New layers still seed a keyframe at the playhead, so there's normally one to paint
into; painting before the first keyframe is now a no-op (as intended).
Next: onion skinning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The brush commits via a GPU-canvas readback and relies on the action's first
execute() to SET raw_pixels from that readback — at that point raw_pixels is empty
(new keyframe) or the pre-stroke state, never something a dirty-rect diff can stamp
onto. My initial diff-only execute skipped (to avoid corruption), so the stroke
disappeared.
Fix: the action keeps the full post-edit buffer ONLY for the first execute (the
commit), assigning raw_pixels outright exactly like the old code; it's taken/dropped
immediately, so the action sitting in the undo stack still retains just the small
diff. Redo replays via the diff onto the now-resident base.
Also harden the diff itself for the blank-base case: `before_blank` lets apply_after
build from a transparent buffer (redo of a first stroke after undo-to-blank) and
apply_before restore to empty; the resident-base skip is kept only for non-blank
bases (faulted in before undo/redo). Tests cover commit/redo from empty.
`RasterStrokeAction`/`RasterFillAction` stored the whole before+after RGBA frame
(~16 MB/action at 1080p → up to ~1.6 GB at the 100-action cap). They now store a
`RasterDiff` — only the changed bounding box's pixels before and after — computed
once in `new()` from the full buffers, which are then dropped. A brush dab shrinks
from ~16 MB to tens of KB; a full-canvas fill is unchanged (its bbox is the frame).
Paging interaction: a diff overwrites just the bbox, so the keyframe's pixels must
be resident when undo/redo applies. A clean evicted frame's container bytes equal
its current logical state, so the editor faults the target frame in (synchronously)
before undo/redo via a new `Action::raster_resident_hint` + `peek_undo/redo_raster_hint`.
Dirty frames are never evicted, so they're already resident. If a base is somehow
not resident the apply is skipped (logged), never resized-and-corrupted.
Unit tests cover exact before/after round-trip, blank-first-stroke, no-op, and the
non-resident-base skip.
Scrubbing onto a paged-out raster keyframe flashed blank for the 1-2 frames its
full pixels took to page in. Now a low-res proxy is shown in that gap.
- core: `MediaKind::RasterProxy` (id derived from the keyframe id via
`raster_proxy_media_id`); `brush_engine::encode_raster_proxy_png` downscales a full
RGBA buffer to a ≤192px-long-edge PNG. Save writes a proxy beside each resident
frame's full PNG (paged-out frames keep their existing proxy row, like the full).
Load eagerly decodes proxies (small) into `RasterKeyframe::proxy`.
- editor: a separate `proxy_layer_cache` in the GPU brush (own recency LRU, budget 64
since each is ~1/100th a full frame) + `ensure_proxy_texture`/`get_proxy_texture`.
The raster render, when the full texture isn't resident, blits the proxy mapped to
the keyframe's FULL logical dims so it upscales via the sampler. F3 VRAM figure now
includes proxy textures.
When the full pixels land (async fault-in), the full path takes over automatically.
Proxies only exist after a save+reload; freshly-painted unsaved frames stay resident
so they need none.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`AudioPool::load_from_serialized` sizes the slot Vec by pool_index and fills gaps
with empty `AudioFile::new(PathBuf::new(), …)` placeholders. Two bugs let a
placeholder reach the next save and abort it with "Is a directory":
- Off-by-one: `entries.max().unwrap_or(0) + 1` made an *empty* pool length 1, so a
project with no audio still got one placeholder. Size by `max(pool_index + 1)`
→ empty entries yield length 0.
- `serialize()` emitted placeholder slots: an empty path round-trips to
`relative_path = Some("")`, which `save_beam` resolves to the project directory
(`join("")`) and tries to read as media. Skip empty-path / no-packed-media slots.
- Defense in `save_beam`: gate referenced-media packing on `full.is_file()` (not
`exists()`), so any blank/dir path falls through to embedded data instead of
reading a directory.
Pre-existing; surfaced by a save → reload → save cycle on a raster-only project.
Scrubbing a large paint project no longer accumulates every visited frame in RAM.
A fault-in-recency LRU keeps the most-recently-paged-in RASTER_RESIDENT_MAX (12)
keyframes resident and drops the pixels of older *clean* ones (re-arming their
fault-in so they re-page on revisit). The shown frame is always the most-recent
fault-in, so it's never evicted.
Data-loss safety: a new `dirty` flag marks any keyframe whose `raw_pixels` were
mutated by editing (stroke/fill/paint-bucket/floating-lift + their undo/redo) and
is NOT yet in the container. Dirty keyframes are NEVER evicted — they're only
unpinned from the LRU. The flag is cleared on a successful save, which also re-arms
the LRU for the now-clean resident frames so the bound still applies to frames
edited this session.
Also: the save loop now walks all layers (incl. nested) to match the load path's
recursive fault-in arming — evicted frames keep their existing container row
(media_exists), and nested raster keyframes are persisted + covered by live_media.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Raster keyframes are no longer eagerly decoded at load — `raw_pixels` stays empty
and is paged in on demand from the project container, so a big paint project opens
instantly and only touched frames hit RAM.
- core: `read_packed_media_readonly` (fresh read-only connection, can't conflict
with an in-place save) + `RasterStore` (holds the container path; `load_pixels`
reads+decodes a keyframe's PNG by id). `load_beam_sqlite` stops eager-decoding and
instead marks every raster keyframe `needs_fault_in` (recursively, incl. nested);
a freshly-created keyframe stays false (blank-resident, nothing to page). Added
`Document::all_layers_mut`.
- editor: the canvas records a fault-in request when it needs a paged-out keyframe
(empty pixels && needs_fault_in); the App drains the sink at the top of update(),
pages the pixels in via the store, clears the flag, and repaints. Store path is set
on load and after save. Export faults in synchronously per frame.
Cold-scrub still shows a 1-frame gap and the page-in is synchronous; the image proxy
(3a-2) and async load (3a-3) remove those next.
The lib unit tests had gone stale (time values became newtypes) and no longer
compiled. Updated the test code to the current API and fixed the few real issues
the now-running tests surfaced.
Test-only:
- Wrap raw f64 time literals in Beats(...) where the API now takes Beats
(automation.rs); pass &TempoMap / Beats where signatures changed (clip.rs,
effect_layer.rs).
- shape.rs: assert the documented no-fill default (fill_color None) instead of Some.
- add_clip_instance / trim_clip_instances tests: register a vector clip with the
test's clip_id so the action's get_clip_duration lookup succeeds.
Production fix (delete_folder.rs):
- DeleteFolderAction(MoveToParent) reparented child subfolders to the deleted
folder's parent but never restored them on undo, orphaning them. Track the moved
subfolder ids and restore their parent on rollback.
Result: daw-backend lib 17 passed; lightningbeam-core lib 264 passed.
Surround → stereo downmix:
- render_from_file folds multichannel sources (5.1/7.1/…) down to stereo with
proper coefficients (full level for the matching front channel, 1/√2 for centre
+ each surround, LFE dropped), normalized per row to avoid clipping (matching
ffmpeg's default). Applied uniformly to both the direct-copy and sinc-resample
paths and to every storage type (PCM, compressed, video audio), only when
dst==2 && src>2; unknown layouts fall back to front L/R. Previously it just took
FL/FR, dropping centre dialog + surrounds.
Proper video-audio reload:
- A video's audio track is now stored as a path reference to the video (never
packed/embedded as audio media) and re-probed via FFmpeg on load into a
streaming VideoAudio entry, so multichannel audio survives reload (the old
Symphonia reconstitution collapsed it, breaking the downmix). Driven by a new
AudioPoolEntry.is_video_audio flag across serialize / save_beam / load. Also
removes the decode-whole-video-to-RAM + temp-file path on load.
Fix video scaling:
- Any video with dimensions larger than the stage was being scaled down into the corner incorrectly; we now bake the frame-clip scale into the instance transform.
Thumbnail rendering fixes:
- Strip now tiles from each clip's true (unclamped) origin and draws only the
tiles intersecting the visible rect, so it scrolls correctly and shows the
right frames when a clip is scrolled partly off the left. Both render sites
(collapsed group + expanded track) share one draw_video_thumbnail_strip helper.
- On-clip strip no longer freezes on the first thumbnail: get_thumbnail_at now
returns the actual thumbnail timestamp and the GPU texture cache keys on it, so
tiles refresh as closer thumbnails finish generating.
- Hover preview derives content time from the clip's true origin too (matches the
strip when scrolled off-screen).
- insert_thumbnail keeps the cache sorted + deduped (fixes a latent unsorted
binary_search bug, and makes concurrent restore + resume race-safe).
Thumbnail persistence (mirrors waveform persistence):
- MediaKind::Thumbnail rows, keyed by thumbnail_media_id(clip_id) (clip id XOR a
sentinel). Each clip's thumbnails PNG-encoded into one opaque LBTN blob (editor
owns the format), snapshotted cheaply (Arc clones) and encoded off the UI thread.
- Save writes the packs (kept in place on re-save); load reads them into
LoadedProject.thumbnail_blobs; the editor decodes + inserts them on a background
thread, so reload shows thumbnails instantly with no re-decode (even if the
source video file is missing).
- Partial sets are persisted with a complete flag and RESUMED on load:
generate_keyframe_thumbnails takes a should_skip predicate so a save made
mid-generation continues from where it left off instead of redoing the work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- VideoManager.frame_cache: unbounded HashMap (grew per distinct frame during
playback) -> LruCache evicted by a 256MB byte budget. Byte-budget rather than
frame count is robust across resolutions (a 4K frame is ~33MB vs ~2MB at
800x600). unload_video pops per-clip keys (LruCache has no retain).
- mux_video_and_audio: stream-merge the two inputs by PTS with one pending
packet per stream (O(1) memory) instead of collecting every packet into Vecs
first (O(duration)). Output is byte-identical.
- export AAC: sanitize the planar-f32 path (non-finite -> 0, finite clamped to
[-1,1]) like the integer paths, with a one-time warning. A stray NaN/Inf
render sample no longer fails the whole export.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the .beam container to SQLite and stream media from it instead of
decoding whole files into RAM on import/load.
Container & large files:
- SQLite .beam container (beam_archive) with in-place transactional saves and an
incremental BlobReader; supports both packed (chunked blobs) and referenced
(external path) media, with a user preference + first-import prompt for files
over the large-media threshold.
Audio streaming:
- Stream packed compressed audio on load via an inversion-of-control blob factory
(AudioBlobSourceFactory): daw-backend defines the trait, core implements it
over BlobReader, so the audio engine stays container-agnostic.
- Bulk-activate disk streaming for all loaded clips after SetProject.
- Sample-accurate compressed seek (SeekMode::Accurate; Coarse mislands on VBR).
Video:
- Video frames decoded/streamed on demand; thumbnails generated asynchronously
on a dedicated decoder so import/load never blocks the UI.
- The video's audio track is streamed on demand via an ffmpeg VideoAudioReader
as a separate editable AudioClip (no /tmp WAV extraction).
Waveform overview:
- Streaming min/max LOD pyramid (waveform_pyramid), bounded memory, configurable
floor B; serialized into the container and restored on load (or generated in
the background from the packed blob when absent), so no re-decode on reload.
- GPU min/max upload path; integer-LOD textureLoad fixes zoom-dependent wobble.
Color correctness:
- Unpremultiply before the sRGB OETF on the display and export blits;
encoding premultiplied color corrupted antialiased/transparent edges.
- Tag exported video as full-range BT.709 (matrix/primaries/transfer).
- Run perception effects (invert, brightness/contrast, hue/saturation)
in gamma space to match standard editors.
- Interpolate gradients in gamma space across the raster and vector paths.
- Render effect thumbnails in the same linear space as the live pipeline.
Brush performance:
- Store the raster canvas as Rgba16Float (no shadow banding from 8-bit
linear), with an incremental per-tile ping-pong sync replacing the
per-frame full-canvas copy.
- Do the linear->sRGB readback conversion on the GPU and reuse a cached
scratch texture, dropping a ~110ms-per-stroke CPU decode.
Cleanup:
- Single COLOR_WGSL prelude and shared CPU sRGB scalars instead of ~8
duplicated copies of the transfer functions.
- Shared compute-pipeline builder; smudge folded onto the tile-sync path.
Skyler Lehmkuhl