CpuYuvConverter::convert rebuilt the swscale context + both ffmpeg frames on every
call (per output frame), despite the converter persisting for the whole export. Now
the scaler + reusable source/dest frames are built once in new() and reused each
convert(). (convert is &mut self; the caller already held it mutably.)
The export pump built a fresh `vello::Renderer` (full wgpu pipeline init, ~tens of ms)
AND an empty `ImageCache` on every egui repaint — i.e. per output frame. That was the
dominant per-frame cost (and the code comment already flagged it). Now the renderer +
image cache are built once on the first export pump, reused for every frame, and
dropped when the export finishes.
Also fixes a latent bug: the throwaway export ImageCache had no container path, so with
Phase 4 Tier 1 (lazy image bytes) images stored only in the container wouldn't render
in exports. The persistent cache now gets the container path.
Audit found the original "per-frame seek" theory was wrong — export decodes the source
sequentially; readback is already async/triple-buffered. Remaining wins: cache swscale
contexts (rebuilt per frame), GPU YUV conversion, decouple from the repaint cadence.
`assets_needed_at(document, time)` (core) enumerates the image asset ids referenced by
the visible vector layers' active keyframes at a time (top-level + group children).
During playback the stage decodes the images needed ~0.5s ahead into the bounded
ImageCache, so a keyframe that swaps image fills doesn't hitch when the playhead
reaches it. Gated on is_playing; nested clip-instance recursion + background decode
are refinements.
Completes Phase 4 (image asset paging: Tier 2 decoded-cache LRU, Tier 1 lazy bytes,
playback prefetch).
Project load no longer eager-reads all image bytes — `ImageAsset.data` stays empty
and the renderer's ImageCache pages compressed bytes from the `.beam` on a decode
miss (read_packed_media_readonly by asset id), decoding into the byte-bounded Tier-2
cache. Result: instant load, and compressed bytes don't accumulate on the heap.
- ImageCache: `container_path` + `resolve_bytes` (asset.data if resident — fresh
import or old base64 project — else page from the container); decoders take `&[u8]`
and use the decoded dimensions.
- Container path threaded App.current_file_path → SharedPaneState → VelloRenderContext,
set on the cache each prepare.
- load_beam_sqlite drops the 3.5b eager read.
(Refinement: a persistent read connection instead of open-per-miss.)
Image fill is now a tab in the Fill type row rather than a separate dropdown. When
Image is active, an asset-picker combo selects which image; switching to None/Solid/
Gradient clears the image fill (it otherwise overrides them). The Image tab only
appears when there are imported image assets.
- SetImageFillAction (core): set/clear `image_fill` on the selected VectorGraph fills,
with per-fill undo (mirrors SetFillPaintAction). Image takes render priority; clearing
reveals the colour/gradient underneath.
- Info Panel Shape section: an "Image:" combo listing the document's image assets (+ None)
for the selected fill(s), showing the current assignment. Assign/clear pushes the action.
This lets an existing shape be given (or cleared of) an image fill, complementing the
import/drop placement. Next: 3.5b — persist image assets in the .beam container.
Replaces the DCEL "not yet supported" stubs so importing/dropping an image actually
puts it in the vector scene.
- AddShapeAction gains an `image_fill` + `AddShapeAction::image_rect(...)` constructor:
a borderless rectangle (invisible edges) whose enclosed region is paint-bucketed and
tagged with an image asset id. The renderer already prioritises `image_fill`.
- Direct import (auto_place_asset): an imported image is placed centered on the canvas
at native size on a vector layer.
- Drag from the asset library onto the stage: image-filled rect at the drop point
(centered), native size, using the asset's dimensions.
Next: SetImageFillAction + an Info-Panel image-fill picker for existing shapes; then
3.5b container persistence.
The standalone egui window didn't fit the UI. Replaced it with a collapsible
"Onion Skin" section at the bottom of the Info Panel (Enabled checkbox + frames
before/after + opacity), available regardless of selection. SharedPaneState carries
a mutable `onion_skin` ref for the controls (distinct from the gated `onion` copy
used by rendering).
- Core compositor gains an optional screen-blend tint per CompositorLayer
([0,0,0,0] default = no-op, so existing compositing is unaffected);
CompositorLayer::with_tint sets it — giving the Vello/vector path a tint hook.
- For the active VECTOR layer with onion on, build ghost scenes at each neighbouring
keyframe's time via render_layer_isolated (reusing the prepare's image cache), then
render each scene → sRGB → linear → composite with warm/cool tint + opacity falloff,
behind the current frame. Off during playback; active layer only.
Completes onion skinning for all layer types (raster + vector).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A small 'Onion Skin' egui window (shown while onion skinning is enabled) with
DragValues for frames_before/after (0..=5) and an opacity slider, rendered next to
the F3 debug overlay. Lets the user tune the configurable settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Removed the early `continue` that skipped a layer with no resident content — it
also skipped the active raster layer's onion ghosts when the current cel was blank
(the exact case you trace a new cel from neighbours). Each render arm already
guards its own empty case, so the skip was redundant.
- Ghost texture resolution now falls through cache → resident raw_pixels (upload) →
in-memory proxy → request fault-in, so neighbours that were paged out (e.g. after a
seek, or saved-this-session with no decoded proxy) page in and ghost in, instead of
silently rendering nothing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Multiplicative tint left blacks black, so outlines (the main thing to ghost in line
art) never picked up the warm/cool color. Switched the canvas-blit tint to a screen
blend (out = base + tint - base*tint): black → tint color, white unchanged, and a
clean no-op at tint=(0,0,0) for all normal blits. Reverted the default .w slots to 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- OnionSkinSettings (editor view state): enabled, frames_before/after (default 2/2),
opacity (0.35), warm past / cool future tints, linear falloff. Threaded App →
SharedPaneState (gated off during playback) → VelloRenderContext.
- Toggle: View ▸ Onion Skinning + the `O` shortcut (AppAction::ToggleOnionSkin).
- Tint: packed an RGB multiply into the canvas-blit matrix uniform's unused .w slots
(1,1,1 = no tint for all normal blits); BlitTransform::with_tint. Shader multiplies.
- Render: for the ACTIVE raster layer only, blit the N neighbouring keyframes (full
texture if resident, else the low-res proxy — uploaded on demand) tinted + faint,
composited behind the current frame. Off during playback.
Next: a settings panel (frame count/opacity) and vector-layer ghosts.
- Pointing-hand cursor when hovering a clickable keyframe diamond.
- Prefetch (Phase 3e, playback only): each update during playback, page in the next
few upcoming keyframes (PREFETCH_AHEAD=4) per raster layer that aren't resident, via
the existing async worker. Their full pixels land before the playhead reaches them,
so playback shows full frames instead of the low-res proxy on every frame (the
proxy→full pop was the "flicker"). Reactive faults still cover scrubbing.
render_layers now records each drawn keyframe diamond's screen rect + exact time in
`keyframe_diamond_hits`; handle_input hit-tests a click against them and sets the
playhead (and seeks the audio controller) to the keyframe's exact time. Uses the
previous frame's rects — diamonds don't move between frames, so the click lands
right — which sidesteps the input-before-render ordering and the drag/scroll Y math.
Works for both raster and vector keyframes.
Make raster layers behave like vector on the timeline.
- Timeline: draw a diamond per `RasterKeyframe` (mirrors the vector keyframe block).
- New Keyframe (K / menu): on a raster layer, insert a BLANK cel at the playhead via
a new undoable `AddRasterKeyframeAction` (+ `RasterLayer::insert_blank_keyframe_at`
/ `remove_keyframe`). Vector path unchanged.
- Stop lazy creation: paint tools now edit the ACTIVE keyframe (at-or-before the
playhead) instead of creating one. The brush captures the active keyframe's exact
time; `RasterStroke`/`RasterFillAction` resolve via `keyframe_at_mut` (error if
none); the tool-site `ensure_keyframe_at` blocks (brush/fill/bucket/wand/quick-
select/floating-lift) are removed — each read already bails when no keyframe exists.
New layers still seed a keyframe at the playhead, so there's normally one to paint
into; painting before the first keyframe is now a no-op (as intended).
Next: onion skinning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`RasterStrokeAction`/`RasterFillAction` stored the whole before+after RGBA frame
(~16 MB/action at 1080p → up to ~1.6 GB at the 100-action cap). They now store a
`RasterDiff` — only the changed bounding box's pixels before and after — computed
once in `new()` from the full buffers, which are then dropped. A brush dab shrinks
from ~16 MB to tens of KB; a full-canvas fill is unchanged (its bbox is the frame).
Paging interaction: a diff overwrites just the bbox, so the keyframe's pixels must
be resident when undo/redo applies. A clean evicted frame's container bytes equal
its current logical state, so the editor faults the target frame in (synchronously)
before undo/redo via a new `Action::raster_resident_hint` + `peek_undo/redo_raster_hint`.
Dirty frames are never evicted, so they're already resident. If a base is somehow
not resident the apply is skipped (logged), never resized-and-corrupted.
Unit tests cover exact before/after round-trip, blank-first-stroke, no-op, and the
non-resident-base skip.
The canvas blit used a nearest sampler, so the upscaled low-res proxy looked
blocky. Added a Linear sampler + `CanvasBlitPipeline::blit_smooth`; the raster
render uses it only for the proxy path (full-res canvas stays nearest/crisp). The
bind-group layout already declares the canvas texture filterable, so no layout
change was needed.
Scrubbing onto a paged-out raster keyframe flashed blank for the 1-2 frames its
full pixels took to page in. Now a low-res proxy is shown in that gap.
- core: `MediaKind::RasterProxy` (id derived from the keyframe id via
`raster_proxy_media_id`); `brush_engine::encode_raster_proxy_png` downscales a full
RGBA buffer to a ≤192px-long-edge PNG. Save writes a proxy beside each resident
frame's full PNG (paged-out frames keep their existing proxy row, like the full).
Load eagerly decodes proxies (small) into `RasterKeyframe::proxy`.
- editor: a separate `proxy_layer_cache` in the GPU brush (own recency LRU, budget 64
since each is ~1/100th a full frame) + `ensure_proxy_texture`/`get_proxy_texture`.
The raster render, when the full texture isn't resident, blits the proxy mapped to
the keyframe's FULL logical dims so it upscales via the sampler. F3 VRAM figure now
includes proxy textures.
When the full pixels land (async fault-in), the full path takes over automatically.
Proxies only exist after a save+reload; freshly-painted unsaved frames stay resident
so they need none.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`raster_layer_cache` (one ~w·h·16-byte Rgba16Float CanvasPair per keyframe) had no
size cap — scrubbing a long timeline grew VRAM without bound (~33 MB/frame at 1080p),
the largest unbounded consumer. Added a recency LRU (RASTER_LAYER_CACHE_MAX = 12):
`ensure_layer_texture` bumps the frame to most-recent and evicts the oldest past the
budget; the shown frame (and any rendered this pass) is always most-recent so it's
never the victim. Evicted textures re-upload cheaply from the resident/faulted-in
pixels on revisit. `remove_layer_texture` keeps the LRU in sync.
F3 debug overlay now reports the tracked VRAM (raster cache MB + frame count), pushed
from the GPU brush whenever the cache changes (wgpu exposes no allocator query).
Scrubbing a large paint project no longer accumulates every visited frame in RAM.
A fault-in-recency LRU keeps the most-recently-paged-in RASTER_RESIDENT_MAX (12)
keyframes resident and drops the pixels of older *clean* ones (re-arming their
fault-in so they re-page on revisit). The shown frame is always the most-recent
fault-in, so it's never evicted.
Data-loss safety: a new `dirty` flag marks any keyframe whose `raw_pixels` were
mutated by editing (stroke/fill/paint-bucket/floating-lift + their undo/redo) and
is NOT yet in the container. Dirty keyframes are NEVER evicted — they're only
unpinned from the LRU. The flag is cleared on a successful save, which also re-arms
the LRU for the now-clean resident frames so the bound still applies to frames
edited this session.
Also: the save loop now walks all layers (incl. nested) to match the load path's
recursive fault-in arming — evicted frames keep their existing container row
(media_exists), and nested raster keyframes are persisted + covered by live_media.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fault-in drain no longer decodes on the UI thread. It now:
- dispatches each newly-requested keyframe's page-in to a background thread
(deduped via an in-flight set, store cloned per request so path changes are
picked up), and
- applies completed results from a channel at the top of update(), keeping the UI
ticking while loads are outstanding.
Cold scrubs no longer freeze. The brief blank gap before a frame lands is removed
by the image proxy (3a-3); eviction to bound RAM while scrubbing is 3b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Raster keyframes are no longer eagerly decoded at load — `raw_pixels` stays empty
and is paged in on demand from the project container, so a big paint project opens
instantly and only touched frames hit RAM.
- core: `read_packed_media_readonly` (fresh read-only connection, can't conflict
with an in-place save) + `RasterStore` (holds the container path; `load_pixels`
reads+decodes a keyframe's PNG by id). `load_beam_sqlite` stops eager-decoding and
instead marks every raster keyframe `needs_fault_in` (recursively, incl. nested);
a freshly-created keyframe stays false (blank-resident, nothing to page). Added
`Document::all_layers_mut`.
- editor: the canvas records a fault-in request when it needs a paged-out keyframe
(empty pixels && needs_fault_in); the App drains the sink at the top of update(),
pages the pixels in via the store, clears the flag, and repaints. Store path is set
on load and after save. Export faults in synchronously per frame.
Cold-scrub still shows a 1-frame gap and the page-in is synchronous; the image proxy
(3a-2) and async load (3a-3) remove those next.
Thumbnail rendering fixes:
- Strip now tiles from each clip's true (unclamped) origin and draws only the
tiles intersecting the visible rect, so it scrolls correctly and shows the
right frames when a clip is scrolled partly off the left. Both render sites
(collapsed group + expanded track) share one draw_video_thumbnail_strip helper.
- On-clip strip no longer freezes on the first thumbnail: get_thumbnail_at now
returns the actual thumbnail timestamp and the GPU texture cache keys on it, so
tiles refresh as closer thumbnails finish generating.
- Hover preview derives content time from the clip's true origin too (matches the
strip when scrolled off-screen).
- insert_thumbnail keeps the cache sorted + deduped (fixes a latent unsorted
binary_search bug, and makes concurrent restore + resume race-safe).
Thumbnail persistence (mirrors waveform persistence):
- MediaKind::Thumbnail rows, keyed by thumbnail_media_id(clip_id) (clip id XOR a
sentinel). Each clip's thumbnails PNG-encoded into one opaque LBTN blob (editor
owns the format), snapshotted cheaply (Arc clones) and encoded off the UI thread.
- Save writes the packs (kept in place on re-save); load reads them into
LoadedProject.thumbnail_blobs; the editor decodes + inserts them on a background
thread, so reload shows thumbnails instantly with no re-decode (even if the
source video file is missing).
- Partial sets are persisted with a complete flag and RESUMED on load:
generate_keyframe_thumbnails takes a should_skip predicate so a save made
mid-generation continues from where it left off instead of redoing the work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- VideoManager.frame_cache: unbounded HashMap (grew per distinct frame during
playback) -> LruCache evicted by a 256MB byte budget. Byte-budget rather than
frame count is robust across resolutions (a 4K frame is ~33MB vs ~2MB at
800x600). unload_video pops per-clip keys (LruCache has no retain).
- mux_video_and_audio: stream-merge the two inputs by PTS with one pending
packet per stream (O(1) memory) instead of collecting every packet into Vecs
first (O(duration)). Output is byte-identical.
- export AAC: sanitize the planar-f32 path (non-finite -> 0, finite clamped to
[-1,1]) like the integer paths, with a one-time warning. A stray NaN/Inf
render sample no longer fails the whole export.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the .beam container to SQLite and stream media from it instead of
decoding whole files into RAM on import/load.
Container & large files:
- SQLite .beam container (beam_archive) with in-place transactional saves and an
incremental BlobReader; supports both packed (chunked blobs) and referenced
(external path) media, with a user preference + first-import prompt for files
over the large-media threshold.
Audio streaming:
- Stream packed compressed audio on load via an inversion-of-control blob factory
(AudioBlobSourceFactory): daw-backend defines the trait, core implements it
over BlobReader, so the audio engine stays container-agnostic.
- Bulk-activate disk streaming for all loaded clips after SetProject.
- Sample-accurate compressed seek (SeekMode::Accurate; Coarse mislands on VBR).
Video:
- Video frames decoded/streamed on demand; thumbnails generated asynchronously
on a dedicated decoder so import/load never blocks the UI.
- The video's audio track is streamed on demand via an ffmpeg VideoAudioReader
as a separate editable AudioClip (no /tmp WAV extraction).
Waveform overview:
- Streaming min/max LOD pyramid (waveform_pyramid), bounded memory, configurable
floor B; serialized into the container and restored on load (or generated in
the background from the packed blob when absent), so no re-decode on reload.
- GPU min/max upload path; integer-LOD textureLoad fixes zoom-dependent wobble.
- Scale the document to the selected output resolution. It was rendered
at document size regardless of the export dimensions, so picking a
different resolution didn't scale the stage.
- Run the audio+video mux on a background thread instead of the UI
thread, keeping the app responsive (showing "Finalizing") during the
re-mux pass.
- Send desktop notifications fire-and-forget. notify_rust's show() is a
synchronous D-Bus call that blocked the UI for the full service
activation timeout (~25s) when no notification daemon is running.
Color correctness:
- Unpremultiply before the sRGB OETF on the display and export blits;
encoding premultiplied color corrupted antialiased/transparent edges.
- Tag exported video as full-range BT.709 (matrix/primaries/transfer).
- Run perception effects (invert, brightness/contrast, hue/saturation)
in gamma space to match standard editors.
- Interpolate gradients in gamma space across the raster and vector paths.
- Render effect thumbnails in the same linear space as the live pipeline.
Brush performance:
- Store the raster canvas as Rgba16Float (no shadow banding from 8-bit
linear), with an incremental per-tile ping-pong sync replacing the
per-frame full-canvas copy.
- Do the linear->sRGB readback conversion on the GPU and reuse a cached
scratch texture, dropping a ~110ms-per-stroke CPU decode.
Cleanup:
- Single COLOR_WGSL prelude and shared CPU sRGB scalars instead of ~8
duplicated copies of the transfer functions.
- Shared compute-pipeline builder; smudge folded onto the tile-sync path.
Skyler Lehmkuhl