save_beam re-encoded every resident raster keyframe to PNG (+ proxy) on every
save, even untouched frames — the dominant per-save cost for painting/animation
projects (the code noted this as deferred "Phase 3").
The infrastructure to do it incrementally already exists: kf.dirty means "current
pixels not yet in the container" (set on any edit, cleared on a successful save,
per main.rs), and it's preserved in the document clone the save worker receives.
Gate the encode on it: a clean keyframe already stored keeps its full + proxy rows
untouched; only dirty (or not-yet-stored) frames are re-encoded. Media blobs were
already incremental; this closes the raster gap.
No new data-loss risk: the mid-save-edit race (edit between the document clone and
save completion) is pre-existing and identical to the old full-write path.
Introduce editable text layers: a resizable text box with editable text,
font size, color, font family, and alignment.
Core:
- New TextLayer/TextContent (text_layer.rs), wired into AnyLayer/LayerType
and all the exhaustive match sites; structured so content can be keyframed
later via content_at().
- fonts.rs: thread-local parley FontContext with three bundled fonts
(Liberation Sans/Serif/Mono, SIL OFL), system-font enumeration consolidated
to base families, document-embedded fonts, glyph/caret/selection geometry,
and a background preloader for the picker fonts.
- Rendering via parley layout + Scene::draw_glyphs (renderer.rs); text
composites through the vector path.
- Actions: CreateTextClipAction (vector-layer branch, undoable),
SetTextContentAction, ResizeTextBoxAction.
- .beam font embedding: MediaKind::Font rows (content-hash dedupe) written on
save and registered on load, with bundled-default fallback.
- VectorClip content bounds include text boxes so text-only clips are
selectable/draggable.
Editor:
- Text tool: click empty/raster/video to create a top-level text layer, or a
vector layer to create+enter a clip containing the text; click an existing
box to edit it.
- Hybrid in-place editing: a hidden egui TextEdit drives input/IME/caret while
the text and caret/selection render in Vello; empty just-created layers are
removed on commit.
- Selection outline + 8 resize handles (re-wrap text) with hover cursors;
factored the corner/edge resize-cursor mapping shared with the Transform tool.
- Info panel: edit text, size, color, alignment, box size, and a font-family
picker that previews each entry in its own font (fonts preloaded in the
background to avoid hitches).
Deps: add parley (git, pinned to match vello's peniko); bundle Liberation
fonts under lightningbeam-core/assets/fonts. Gitignore the local
.cargo/config.toml used to select a machine's ffmpeg.
Video was the only media type always kept external (VideoClip.file_path),
so a project with video wasn't self-contained. Now video packs into the
SQLite container under the same large-media policy as audio (pack < 2 GB
unless the user chose Reference), and both the frames and the embedded audio
track decode by streaming directly from the blob — no temp files.
- New crate ffmpeg-blob-io: an AVIOContext-over-Read+Seek shim (BlobInput)
that lets ffmpeg demux from an arbitrary byte source. Isolates all the
unsafe FFI + ffmpeg ABI coupling (version-pinned =8.0.0/=8.0.1). Manual
Drop teardown order; AVSEEK_SIZE restores the read position (FFmpeg assumes
a size query doesn't move it — required for MP4 moov-at-end).
- Schema/save/load: VideoClip.media_id; save_beam packs/references video as
MediaKind::Video (keyed by clip id); load resolves packed vs referenced and
reports missing sources. A packed clip points its linked video-audio pool
entry's media_id at the video row so the audio streams from the same blob.
- Frames: video.rs VideoSource{Path,Packed} threaded through new/seek/scan/
probe/thumbnails (a fresh BlobReader per open); editor builds the source
from current_file_path (now set before register_loaded_videos).
- Audio: VideoAudioReader::open_source via BlobInput; the disk_reader
StreamSource block on packed video-audio is removed; the engine's existing
factory activation routes it unchanged.
Tests: ffmpeg-blob-io AVIO unit tests (WAV via Cursor, seek, open/drop loop);
core packed_video_stream (blob->AVIO->Input) and beam_archive video round-trip;
daw-backend open_source test (compiles; links/runs only off-container).
Runtime-verified: a packed video plays frames + audio after the source file
is removed.
Address the code smells flagged in the .beam format spec:
- Write the project's actual sample rate on save instead of a hardcoded 48000
(add AudioProject::sample_rate()).
- Remove the vestigial RasterKeyframe.media_path field (it was only used by the
legacy ZIP loader, which now derives "media/raster/<id>.png" from the keyframe
id) and the dead buffer_path_at_time accessor. Backward-compatible: older files
carrying media_path deserialize fine (the field is ignored).
- Drop the unused SaveSettings fields auto_embed_threshold_bytes / force_embed_all
/ force_link_all; only large_media_mode was ever consulted. Un-prefix the now-used
`settings` parameter.
Update BEAM_FILE_FORMAT.md to match. The remaining notes (reserved MediaKind::Video,
exact-match version check) are design choices, left as-is.
Project load no longer eager-reads all image bytes — `ImageAsset.data` stays empty
and the renderer's ImageCache pages compressed bytes from the `.beam` on a decode
miss (read_packed_media_readonly by asset id), decoding into the byte-bounded Tier-2
cache. Result: instant load, and compressed bytes don't accumulate on the heap.
- ImageCache: `container_path` + `resolve_bytes` (asset.data if resident — fresh
import or old base64 project — else page from the container); decoders take `&[u8]`
and use the decoded dimensions.
- Container path threaded App.current_file_path → SharedPaneState → VelloRenderContext,
set on the cache each prepare.
- load_beam_sqlite drops the 3.5b eager read.
(Refinement: a persistent read connection instead of open-per-miss.)
Image asset bytes are now stored as MediaKind::ImageAsset rows in the SQLite
container (chunked, kept-in-place on re-save) instead of base64-embedded in the
project JSON — the pageable storage Phase 4 needs.
- ImageAsset.data is `#[serde(default, skip_serializing)]`: never written to JSON,
but still deserialized for old projects (base64) which then migrate to the
container on the next save.
- save_beam writes each asset's bytes (keyed by asset id; ext from the source path),
keeping an existing row when bytes aren't resident; live_media covers them so orphan
cleanup doesn't drop them.
- load_beam_sqlite eager-reads the bytes back into `data` (Phase 4 makes this lazy +
LRU). Old base64 projects keep their JSON-deserialized data (no container row).
Scrubbing onto a paged-out raster keyframe flashed blank for the 1-2 frames its
full pixels took to page in. Now a low-res proxy is shown in that gap.
- core: `MediaKind::RasterProxy` (id derived from the keyframe id via
`raster_proxy_media_id`); `brush_engine::encode_raster_proxy_png` downscales a full
RGBA buffer to a ≤192px-long-edge PNG. Save writes a proxy beside each resident
frame's full PNG (paged-out frames keep their existing proxy row, like the full).
Load eagerly decodes proxies (small) into `RasterKeyframe::proxy`.
- editor: a separate `proxy_layer_cache` in the GPU brush (own recency LRU, budget 64
since each is ~1/100th a full frame) + `ensure_proxy_texture`/`get_proxy_texture`.
The raster render, when the full texture isn't resident, blits the proxy mapped to
the keyframe's FULL logical dims so it upscales via the sampler. F3 VRAM figure now
includes proxy textures.
When the full pixels land (async fault-in), the full path takes over automatically.
Proxies only exist after a save+reload; freshly-painted unsaved frames stay resident
so they need none.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`AudioPool::load_from_serialized` sizes the slot Vec by pool_index and fills gaps
with empty `AudioFile::new(PathBuf::new(), …)` placeholders. Two bugs let a
placeholder reach the next save and abort it with "Is a directory":
- Off-by-one: `entries.max().unwrap_or(0) + 1` made an *empty* pool length 1, so a
project with no audio still got one placeholder. Size by `max(pool_index + 1)`
→ empty entries yield length 0.
- `serialize()` emitted placeholder slots: an empty path round-trips to
`relative_path = Some("")`, which `save_beam` resolves to the project directory
(`join("")`) and tries to read as media. Skip empty-path / no-packed-media slots.
- Defense in `save_beam`: gate referenced-media packing on `full.is_file()` (not
`exists()`), so any blank/dir path falls through to embedded data instead of
reading a directory.
Pre-existing; surfaced by a save → reload → save cycle on a raster-only project.
Scrubbing a large paint project no longer accumulates every visited frame in RAM.
A fault-in-recency LRU keeps the most-recently-paged-in RASTER_RESIDENT_MAX (12)
keyframes resident and drops the pixels of older *clean* ones (re-arming their
fault-in so they re-page on revisit). The shown frame is always the most-recent
fault-in, so it's never evicted.
Data-loss safety: a new `dirty` flag marks any keyframe whose `raw_pixels` were
mutated by editing (stroke/fill/paint-bucket/floating-lift + their undo/redo) and
is NOT yet in the container. Dirty keyframes are NEVER evicted — they're only
unpinned from the LRU. The flag is cleared on a successful save, which also re-arms
the LRU for the now-clean resident frames so the bound still applies to frames
edited this session.
Also: the save loop now walks all layers (incl. nested) to match the load path's
recursive fault-in arming — evicted frames keep their existing container row
(media_exists), and nested raster keyframes are persisted + covered by live_media.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Raster keyframes are no longer eagerly decoded at load — `raw_pixels` stays empty
and is paged in on demand from the project container, so a big paint project opens
instantly and only touched frames hit RAM.
- core: `read_packed_media_readonly` (fresh read-only connection, can't conflict
with an in-place save) + `RasterStore` (holds the container path; `load_pixels`
reads+decodes a keyframe's PNG by id). `load_beam_sqlite` stops eager-decoding and
instead marks every raster keyframe `needs_fault_in` (recursively, incl. nested);
a freshly-created keyframe stays false (blank-resident, nothing to page). Added
`Document::all_layers_mut`.
- editor: the canvas records a fault-in request when it needs a paged-out keyframe
(empty pixels && needs_fault_in); the App drains the sink at the top of update(),
pages the pixels in via the store, clears the flag, and repaints. Store path is set
on load and after save. Export faults in synchronously per frame.
Cold-scrub still shows a 1-frame gap and the page-in is synchronous; the image proxy
(3a-2) and async load (3a-3) remove those next.
Surround → stereo downmix:
- render_from_file folds multichannel sources (5.1/7.1/…) down to stereo with
proper coefficients (full level for the matching front channel, 1/√2 for centre
+ each surround, LFE dropped), normalized per row to avoid clipping (matching
ffmpeg's default). Applied uniformly to both the direct-copy and sinc-resample
paths and to every storage type (PCM, compressed, video audio), only when
dst==2 && src>2; unknown layouts fall back to front L/R. Previously it just took
FL/FR, dropping centre dialog + surrounds.
Proper video-audio reload:
- A video's audio track is now stored as a path reference to the video (never
packed/embedded as audio media) and re-probed via FFmpeg on load into a
streaming VideoAudio entry, so multichannel audio survives reload (the old
Symphonia reconstitution collapsed it, breaking the downmix). Driven by a new
AudioPoolEntry.is_video_audio flag across serialize / save_beam / load. Also
removes the decode-whole-video-to-RAM + temp-file path on load.
Fix video scaling:
- Any video with dimensions larger than the stage was being scaled down into the corner incorrectly; we now bake the frame-clip scale into the instance transform.
Thumbnail rendering fixes:
- Strip now tiles from each clip's true (unclamped) origin and draws only the
tiles intersecting the visible rect, so it scrolls correctly and shows the
right frames when a clip is scrolled partly off the left. Both render sites
(collapsed group + expanded track) share one draw_video_thumbnail_strip helper.
- On-clip strip no longer freezes on the first thumbnail: get_thumbnail_at now
returns the actual thumbnail timestamp and the GPU texture cache keys on it, so
tiles refresh as closer thumbnails finish generating.
- Hover preview derives content time from the clip's true origin too (matches the
strip when scrolled off-screen).
- insert_thumbnail keeps the cache sorted + deduped (fixes a latent unsorted
binary_search bug, and makes concurrent restore + resume race-safe).
Thumbnail persistence (mirrors waveform persistence):
- MediaKind::Thumbnail rows, keyed by thumbnail_media_id(clip_id) (clip id XOR a
sentinel). Each clip's thumbnails PNG-encoded into one opaque LBTN blob (editor
owns the format), snapshotted cheaply (Arc clones) and encoded off the UI thread.
- Save writes the packs (kept in place on re-save); load reads them into
LoadedProject.thumbnail_blobs; the editor decodes + inserts them on a background
thread, so reload shows thumbnails instantly with no re-decode (even if the
source video file is missing).
- Partial sets are persisted with a complete flag and RESUMED on load:
generate_keyframe_thumbnails takes a should_skip predicate so a save made
mid-generation continues from where it left off instead of redoing the work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the .beam container to SQLite and stream media from it instead of
decoding whole files into RAM on import/load.
Container & large files:
- SQLite .beam container (beam_archive) with in-place transactional saves and an
incremental BlobReader; supports both packed (chunked blobs) and referenced
(external path) media, with a user preference + first-import prompt for files
over the large-media threshold.
Audio streaming:
- Stream packed compressed audio on load via an inversion-of-control blob factory
(AudioBlobSourceFactory): daw-backend defines the trait, core implements it
over BlobReader, so the audio engine stays container-agnostic.
- Bulk-activate disk streaming for all loaded clips after SetProject.
- Sample-accurate compressed seek (SeekMode::Accurate; Coarse mislands on VBR).
Video:
- Video frames decoded/streamed on demand; thumbnails generated asynchronously
on a dedicated decoder so import/load never blocks the UI.
- The video's audio track is streamed on demand via an ffmpeg VideoAudioReader
as a separate editable AudioClip (no /tmp WAV extraction).
Waveform overview:
- Streaming min/max LOD pyramid (waveform_pyramid), bounded memory, configurable
floor B; serialized into the container and restored on load (or generated in
the background from the packed blob when absent), so no re-decode on reload.
- GPU min/max upload path; integer-LOD textureLoad fixes zoom-dependent wobble.
Skyler Lehmkuhl