diff --git a/STREAMING_TO_DISK_PLAN.md b/STREAMING_TO_DISK_PLAN.md deleted file mode 100644 index abd9921..0000000 --- a/STREAMING_TO_DISK_PLAN.md +++ /dev/null @@ -1,859 +0,0 @@ -# Streaming Media To/From Disk — Plan - -**Goal:** Lightningbeam must handle audio and video files (and raster animation, and -image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from -and to disk when the data is too large to fit comfortably in memory, rather than loading -the entire file regardless of size. - -**Scope of this document:** audio, video, raster frames, image-asset paging, **and the -`.beam` container format** — these turned out to be one problem, not two. Streaming on load -is impossible while the container forces a full decode, so the container decision (below) -is now part of this plan. - -## Deferred bugs (do at the end) -- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the - clip, so when a clip was scrolled partly off the left it showed the clip's start content at the - viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing - only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both - render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)* -- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested* - content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as - closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail - timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail - finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)* - -## Raster-keyframe-UI bugs — **[DONE]** (built the raster keyframe timeline UI, 2026-06-20) -Both resolved by the raster-keyframe-timeline-UI work: timeline now draws a diamond per -`RasterKeyframe` (mirrors vector), `K`/New Keyframe inserts a blank cel via `AddRasterKeyframeAction` -(canvas refreshes), paint tools edit the active keyframe instead of lazily creating, diamonds are -click-to-seek (pointing-hand cursor), playback prefetches frames, and onion skinning (raster+vector, -tinted, Info-Panel settings) is in. (a) canvas-refresh-on-new-keyframe and (b) keyframes-on-timeline -are both fixed. - -## Noted enhancements (later, after the phases) -- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it - covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all - flow through this mixer with the source kept multichannel in the read-ahead buffer). New - `stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional - interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching - front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to - prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample - paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean. - *(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)* - Native multichannel support remains a separate, larger project. -- **Export speed (audited 2026-06-21):** a 1:14 1080p MP4 took ~9:06 (~7.4x realtime, ~135 ms/frame). - Audit **refuted** the per-frame-seek theory — export decodes the source *sequentially* - (`video.rs` `need_seek` is false once advancing forward), and readback is already async + - triple-buffered. Real hotspots: - - **[DONE] #1 — per-frame renderer rebuild.** The export pump built a fresh `vello::Renderer` - (full wgpu pipeline init) + empty `ImageCache` *every egui repaint* (`main.rs` ~6218). Now built - once per export and reused; also fixed lazy-image export (the throwaway cache had no container - path). **Expected the dominant win.** - - **[DONE] #2a — encode swscale rebuilt per frame.** `CpuYuvConverter::convert` now caches the - RGBA→YUV420p `scaling::Context` + frames in `new()` instead of per call. - - **[TODO] #2b — decode swscale + stride-repack** per frame in `video.rs:294-320` (shared with - scrubbing; cache the YUV→RGBA scaler on the decoder). Small win, modest risk. - - **Result of #1+#2a (measured):** ~7.4x → **~1.74x realtime** (130.7 s for 4488 frames @ 60 fps; - 34 fps). Per-stage avg: Render(CPU build) 15 ms, **Readback(GPU latency) 42 ms**, Extract 1.3 ms, - Convert 5.7 ms. - - **Now GPU-bound.** Per ~87 ms poll cycle the CPU does ~66 ms (3× build 45 + convert 17 + extract 4) - but the GPU does ~87 ms (3 × ~29 ms composite) → GPU saturated at ~29 ms/frame; "Readback 42 ms" is - queue latency, not transfer (8 MB is sub-ms). - - **[SKIP] #3 GPU YUV / #5 pacing** — both only trim the CPU side, which is already *under* the GPU. - Won't move a GPU-bound throughput. - - **[TODO, big] Reduce the GPU composite (~29 ms/frame).** The per-layer HDR pipeline (Vello render → - linear → composite, ×layers) is the wall, shared with live rendering. Options: batch composite - passes; a fast-path skipping HDR compositing for simple single-layer/no-blend docs; cache unchanged - layers' scenes (CPU-side, only helps if it later becomes CPU-bound). Render-architecture project. - - Non-issues: per-frame seek, blocking readback, audio. (`video.rs:237` container-reopen-on-seek is - a latent cost but doesn't fire on forward export.) -- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples - (NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray - non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/ - decode) still worth chasing if it recurs. -- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are - PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob` - in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR - a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the - `FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place - on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor - decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos` - skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing. - **Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete` - flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete - packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed — - `generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes - the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent - unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a - 2 h video keeps that work and continues from there on reload. - Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for - complete clips; a mid-generation save resumes from where it left off on reload.)* - **Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 → - 128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`, - at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for - on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap - (≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) + - their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror - the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if - absent). The 5 s interval already bounds count; no extra budget needed. -- **Progressive waveform on first import:** generation streams the whole file before the - waveform appears (several seconds for large files). Since `build_waveform_pyramid` already - streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via - the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across - the clip left-to-right instead of appearing all at once. Persistence saves only the final - complete pyramid. - -## Guiding principle -Three subsystems already have the right streaming primitive; most of the work is wiring, -bounding caches, and adding a residency window. The recurring pattern: - -> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a -> stable ID, and evict everything outside a window around the playhead. - ---- - -## Audit summary (where we stand today) - -### Correctly streaming / bounded -- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` — - keyframe-index seek + decode-until-target, one frame resident). -- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`). -- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`). -- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3 - frames in flight). -- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs` - (`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a). - -### Fully-loaded, unbounded by file length (the problems) -| Site | Issue | -|---|---| -| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec`; de-facto playback source. | -| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. | -| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec`. | -| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. | -| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). | -| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. | -| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. | -| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). | -| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. | - ---- - -## Container format decision: `.beam` → SQLite *(DECIDED)* - -The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same -`.beam` extension). This is the foundation the rest of the plan builds on. - -### Why -ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place -mutation** — every save and every raster frame write-back rewrites the whole archive — and -embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each -ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file -→ full Symphonia decode → resident `Vec` (`file_io.rs:513-604`, `pool.rs:1071`). - -SQLite dissolves the single-file-vs-performance tension: -- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder - confusion; we have no bundle magic on Linux/Windows). -- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable, - chunked reads through the pager (mmap mode for the DB). For chunked streaming the - pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter. -- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`; - save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss / - crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll - atomicity). -- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly. - -**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob -reads (or external paths); raster frames live in blob rows and write back transactionally. - -### Large-media policy: packed OR referenced -Two storage modes per media item, both supported: -- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to - make reads naturally chunked), large media is split into **multiple blob-chunk rows** - (e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`. -- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media - on a network drive, or media too large/volatile to pack). - -**Default-mode preference for files over the per-blob limit (~2GB):** -- A user preference `large_media_default: Pack | Reference` controls what happens to - imports above the threshold. -- The **first time** the user imports a media file over the limit, **prompt** them - (Pack vs Reference), apply it, and **persist the choice** as the preference for future - large imports (changeable later in settings). -- Files under the limit are packed by default (chunked only if needed). - -### Schema sketch -``` -media( - id BLOB PRIMARY KEY, -- stable Uuid - kind INTEGER, -- audio | video | raster | image-asset - codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving) - storage INTEGER, -- 0 = packed, 1 = referenced - ext_path TEXT, -- set when storage = referenced - total_len INTEGER, -- bytes (packed) for chunk math - channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta -) -media_chunk( - media_id BLOB, chunk_index INTEGER, bytes BLOB, - PRIMARY KEY (media_id, chunk_index) -) -project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim -meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified -``` -`project.json` stays the same serialized `BeamProject` for now — only its container and the -media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on -first open/save. - -### Streaming reads from packed media -A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing -streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a -`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a -plain `File` exactly as `do_import_audio` already does for originals today. - ---- - -## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)* - -### 1a. Turn on the compressed-audio disk reader -The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but -never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never -called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`). -- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and - assign `AudioClip::read_ahead`. -- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled - waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes - from the ring buffer, not RAM. -- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty. - -**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s -ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs` -already distinguishes. - -### 1b. Stream on project load *(depends on the SQLite container)* -Three coupled changes (none works alone): -1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as - `do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed - (incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File` - (referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted. -2. **Bulk read-ahead activation:** loaded clips are deserialized directly - (`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires - for them. After the engine installs the project, walk all audio clips and - `create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a - `Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a - `BlobReader` instead of a path for packed media.) -3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media` - table instead of base64 `embedded_data`. - -### 1c. Video's embedded audio track — stream from the video via ffmpeg - -**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to -a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole -uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive -save/reload. **Superseded by the design below.** - -**Proper design — stream the video's audio track on demand, never materialized.** - -*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the -ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No -cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed -interface. - -1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`: - `open(path)`, `decode_next(&mut Vec) -> frames` (resample → interleaved f32 at native - rate; reuse the old extraction resampler), `seek(target_frame) -> actual`, - `sample_rate`/`channels`/`total_frames`. -2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }` - (or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch / - export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`. -3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames, - total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool - entry `path` = the video file. -4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe - the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index. -5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`, - make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`. - One ffmpeg context + 3 s buffer per active clip instance. -6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` → - `AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV - handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM. -7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media - bytes — the video is already referenced by its `VideoClip`); reconstruct on load by - re-probing. Fixes the stopgap's reload fragility (nothing to persist). -8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded - memory) into the existing waveform path — shared with the Phase 1a `decode_progressive` - cleanup. - -**Sample accuracy (required — video audio must stay frame-synced with other clips):** -Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must: -- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on - `target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole - frames before target; for the frame straddling target, drop its leading samples). After - `seek`, `decode_next` yields samples starting at exactly `target_frame`. -- This makes frame N of the video-audio pool entry correspond to the exact timeline position, - so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact. -- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its - current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in - while here, or at least flag. - -*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on -an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it. - -*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant → -`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate -seek (both readers) → waveform-peaks pass. - ---- - -## Phase 2 — Video: bound the caches *(small, isolated)* - -### 2a. Bound `VideoManager.frame_cache` -`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc>` to an LRU -mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget. - -### 2b. Stream the export mux -`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS, -write the earlier stream) instead of collecting all then writing. O(duration) → O(1). - ---- - -## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]** - -Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's -`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @ -1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame -undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it), -`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and -the container is opened only at load/save (no live handle). - -**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit -via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async -fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands. -Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident** -(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image** -(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win). - -### Drive-by (Arc pixels): DROPPED -Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need -an owned `Vec` (undo buffers, export, GPU readback) so `Arc>` would force `(*p).clone()` -and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the -**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU -upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame -clone). Not worth 64 edits. Start at 3a. - -### 3a. Lazy async fault-in + image proxy -- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in` - armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`. -- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`); - results applied at top of `update()`. No UI block on cold scrub. -- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written - beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`. - Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to - the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident. - *(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge - paint projects.)* - -- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)` - reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as. -- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe - (`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`). -- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere); - leave full `raw_pixels` empty. -- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster - layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**; - on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the - upscaled proxy. Reused by the exporter (already frame-by-frame). - -### 3b. Residency window + eviction **[DONE]** -- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on - stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU). -- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window: - evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is - always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames' - rows via `media_exists` (no data loss) and walks all layers to match load. - *(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)* - -### 3c. Bound the GPU cache **[DONE for raster_layer_cache]** -`raster_layer_cache` (`gpu_brush.rs`, `HashMap`, Rgba16Float ping-pong -≈ `w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in -`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay -now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom -if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello -`ImageCache` is image *assets* → Phase 4.)* - -### 3d. Undo memory **[DONE]** -`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames. -Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in -`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor -faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`), -correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒ -skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose -bbox is the whole frame.)* - -### 3e. Prefetch frames **[DONE for playback]** -Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4` -upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so -full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during -playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a -prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction -prefetch still TODO.)* - -Original note: *(future, after 3d — pure latency win, no correctness need)* -Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the -proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**: -on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of -playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker + -`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap -concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU -texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res -frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up. - -### Build order & tests -1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted, -PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing, -dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly. - ---- - -## Phase 3.5 — Image textures in vector scenes **[DONE 2026-06-21]** *(prereq for Phase 4; fixed DCEL-broken image import)* - -**Done:** 3.5a — import/drop places an image as a borderless image-filled rectangle -(`AddShapeAction::image_rect`), centered (direct import) or at the drop point (library drag); -renderer now maps the image brush onto the fill's bounding box (was anchored at world origin → -only a corner showed); `SetImageFillAction` + an **Image** fill-type tab (None|Solid|Gradient|Image) -with an asset picker in the Info Panel. 3.5b — image bytes persist as `MediaKind::ImageAsset` rows in -the `.beam` (kept-in-place; `ImageAsset.data` is `skip_serializing` + container-backed; old base64 -projects migrate on re-save); eager-read on load. *(ImageCache still unbounded — Phase 4 adds the -usage-based LRU/lazy paging.)* - -### (original plan below) -## Phase 3.5 — Image textures in vector scenes *(prereq for testing Phase 4; fixes DCEL-broken image import)* - -**Why:** Phase 4 pages *image assets*, but there's currently no way to get an image asset into a -vector scene — so nothing to page. This also repairs image import, half-broken since the DCEL switch. - -**Current state (audited 2026-06-21):** -- *Works:* `import_image` (`main.rs`) decodes dims + creates an `ImageAsset` (raw bytes embedded in - `Document::image_assets`, serialized as **base64 in project JSON**). The renderer's image-fill paths - are **complete** — GPU/Vello (`renderer.rs:~1160`, `ImageBrush` via `ImageCache.get_or_decode`) and - CPU/tiny-skia (`renderer.rs:~1486`). `Fill::image_fill` (`vector_graph/mod.rs:110`) and - `Face::image_fill` (`dcel2/mod.rs:117`) fields exist and render when set. -- *Broken/missing (the workflow):* - 1. **Drop image → canvas is stubbed:** `stage.rs:~11782` and `main.rs:~4924` both just print - "Image drag to stage not yet supported with DCEL backend". Nothing is added to the scene. - 2. **No way to assign an image fill:** no `SetImageFillAction` (only `SetFillPaintAction` for - color/gradient); no Info-Panel picker. `Fill`/`Face.image_fill` are never populated. - 3. **DCEL faces never get `image_fill`** (`dcel2/import.rs:275` always `None`; topology copies from - parent which is also `None`). - 4. **Not in the container:** `MediaKind::ImageAsset` exists but is **dead** — image bytes live only - as base64 in project JSON. Not chunked, not pageable (so Phase 4 can't page them). - -**Tasks:** -- **3.5a — Place + assign.** Replace the two drop stubs: dropping an image onto a vector layer creates - a rectangle face sized to the image at the drop point with `image_fill = asset_id`. Add - `SetImageFillAction` (set/clear an image fill on the selected face/shape; mirrors `SetFillPaintAction`) - + an Info-Panel image-asset picker for the selected shape's fill. Populate `Face.image_fill` in DCEL - (and keep it through topology ops — already copied from parent). -- **3.5b — Persist in the container.** Write image assets as `MediaKind::ImageAsset` rows in the `.beam` - SQLite (like raster/audio: write on save kept-in-place on re-save; read on load), keyed by asset id; - drop the base64-in-JSON embedding (or keep a tiny ref). This is the storage Phase 4 pages from. -- **3.5c — Lazy decode hook.** Image bytes load from the container into `ImageCache` on first render - (decode → `ImageBrush`/`Pixmap`). Leave `ImageCache` **unbounded for now**; Phase 4 adds the - usage-based LRU/eviction (this phase just makes there *be* real, container-backed image assets to page). -- **Tests:** import→drop→render round-trip; save/reload preserves the image fill + reads bytes from the - container (not JSON); CPU and GPU render paths both show the image. - ---- - -## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)* - -Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave -it resident. The heavy, evictable thing is the **image assets** referenced by fills. - -**Data model.** -- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option>` (whole compressed - file bytes) + dims. Imported fully into `data` at `main.rs:3936`. -- All assets resident in `Document.image_assets: HashMap` (`document.rs:206`). -- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap>` + CPU - `Pixmap` map, keyed by asset id, **unbounded**. -- A `Fill` references an asset by `image_fill: Option` (`vector_graph/mod.rs:110`). - Same UUID may appear in many fills/keyframes/layers and recursively through clip instances. - **No asset→frame or frame→asset index exists today.** - -**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload -from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures — -the heavy one). - -**Progress (2026-06-21):** -- **[DONE] Tier 2 — bound the decoded `ImageCache`.** 256 MB **usage-LRU**: every - `get_or_decode`/`_cpu` bumps the asset's recency; inserts past budget evict the least-recently-used - (a miss re-decodes from `asset.data`). Achieves usage-based eviction via render-access recency - (simpler than the frame→asset enumeration below; that enumeration is only needed for *prefetch*). -- **[DONE] Tier 1 — lazy compressed bytes.** `ImageCache` holds the container path (threaded - App.current_file_path → SharedPaneState → VelloRenderContext) and pages bytes on a decode miss via - `read_packed_media_readonly`; `load_beam_sqlite` no longer eager-reads → instant load, compressed - bytes don't accumulate. `asset.data` is still used when resident (fresh import / old base64 project). - *(Refinement: persistent read connection vs open-per-miss.)* -- **[DONE] Prefetch.** `assets_needed_at(document, time)` enumerates image ids in the visible vector - layers' active keyframes; during playback the stage decodes the ~0.5s-ahead set into the cache. - *(Refinements: nested clip-instance recursion; background-thread decode.)* - -**Phase 4 = DONE** (image asset paging by usage + LRU). - -### 4a. Frame→asset enumeration (incl. nested clips — see note below) -A function `assets_needed_at(time) -> HashSet`: walk each visible vector layer's active -`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into -clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning -upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch. - -### 4b. Usage bookkeeping (the multi-frame problem) -Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole -document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation -paths in `vector_graph` and the relevant actions). -- count 0 → dead, fully evictable / GC candidate. -- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to - playhead**, not by count (a high-count asset far from the playhead is still evicted). - -Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte -budget** for referenced-but-distant assets (covers scrubbing back without a reload). -Eviction never touches an asset in needed-now. - -### 4c. Bound the decoded tier -Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU -image-texture cache the same way, keyed to the residency window. - -### Nested-clip prefetch (important) -A clip instance placed on an outer frame has its **own internal timeline of keyframes**, -each of which can reference its own image assets. Prefetch must therefore: -- Recurse through clip instances when computing both needed-now and needed-soon. -- Map outer playhead time → each nested clip's local time, and look ahead along the - **nested** timeline (not just the outer one) so assets used by an upcoming *inner* - keyframe are loaded before the nested clip reaches it. -- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts - once); the usage index handles refcounting. - ---- - -## Cross-cutting: a shared residency abstraction - -A generic **`PagedStore`** with three consumers — always-resident metadata, -disk backing, residency = window/needed-set around playhead + LRU byte budget: - -| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key | -|---|---|---|---|---| -| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer | -| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) | -| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps | - -Audio stays separate (real-time ring buffer, different constraints). The frame→asset -enumeration + usage index is unique to Phase 4. - ---- - -## Sequencing -1. **Phase 1a** — done; independent of the container, works with the current ZIP loader. -2. **Phase 2** — small, isolated, independently shippable; container-independent. -3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy + - legacy-ZIP migration. Prerequisite for 1b/1c/3/4. -4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store). -5. **Phase 1c** — depends on 1b's pool path. -6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows. -7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index. - -Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container). - ---- - -## Status -- [~] Phase 1a — activate compressed-audio disk reader ← **in progress** - - [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed - pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the - existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean. - - [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the - full PCM; emit only the downsampled waveform overview. (Crosses into the UI - waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.) - - [ ] Runtime verification: confirm a compressed clip actually plays from the ring - buffer (was effectively silent before, since `read_ahead` was always `None`). -- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress** - - [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep - (bundled) — `lightningbeam-core/src/beam_archive.rs` - - [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection, - opens a blob handle per read with rowids resolved once) — for `CompressedReader` / - video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json - round-trip, packed full read, streaming reads + seeks across chunk boundaries, - referenced-path, overwrite-replaces-chunks. - - [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection; - `MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`. - - [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only - changed rows written; unchanged large media never rewritten); orphan cleanup via - `retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save - must NOT copy+rename for existing SQLite files. - - [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename - only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows; - raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip - deleted (now packs original bytes with their codec). - - [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP - (`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into - `embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b). - - [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a - ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end. - - [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack, - Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from - disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file). - `Ask` behaves as `Reference` at save time. - - [x] `large_media_default` user preference: persisted in `AppConfig`, editable in - Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt). - - [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into - import_audio/video/image) queues a one-time modal; choice persists to config. - Threshold shown in the modal is derived from the constant. - - [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive - round-trip; confirm an old ZIP `.beam` still opens and migrates on save. - - [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to - skip unchanged frames on in-place save (Phase 3). - -> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have -> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it -> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which -> build the lib in normal mode. Worth fixing separately. -- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load, - waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index - + proper video-audio reload remain as noted follow-ups. - - **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** — - daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio - engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering - violation; referenced-only-first = leaves packed <2GB in RAM.) - - **Current load reality (why this is needed):** *nothing* streams on load today — every entry - is fully decoded to a PCM `Vec`. Packed audio is base64-reconstituted into `embedded_data` - (`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced - audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation - never fires for loaded clips (they bypass `AddAudioClip`). - - [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource: - Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id) - -> Box }`; `SymphoniaByteSource` adapter (impl `MediaSource`, - is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a - refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and - `DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported - `AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test - `tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed - `MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean. - - [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option>` + - `EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before - `SetProject` on the same queue). `AudioFile.packed_media_id: Option` (Some ⇒ open via - factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored - into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk. - - [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory` - implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a - `!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut` - on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed - by the editor between `load_audio_pool` and `set_project`. - - [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec - is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a - Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize` - round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs - (video-container audio tracks) keep the legacy reconstitution path → **no regression**. - - [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` — - walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set - `read_ahead`), the loaded-clip equivalent of the Phase 1a wiring. - - [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came - from the full in-RAM decode, which no longer happens). Added - `disk_reader::build_waveform_pyramid_from_source(Box, ext, B)` (load-time - counterpart of the path-based builder). On load, the editor background-generates a pyramid for any - streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the - floor through the same `waveform_result` channel `update()` drains; the next save persists it. - Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the - overview now fills in shortly after load. - - **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three - crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen), - then instantly on subsequent loads once saved; RAM stays flat on a big project. - - [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking* - (fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3 - byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files - whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels) - landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no - decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`; - the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables); - a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green. - - [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time - header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the - anchor. Matters for multi-hour MP3s; song-length files are fine as-is. - - [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to - the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a - streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize` - (reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized` - (`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload - (the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp - files on load. Old saves (video mis-packed as audio) self-heal on the next save. - - [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace - `load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed - → `from_compressed` placeholder). Higher risk (touches the working referenced path); packed - covers the common <2GB case first. - - [x] **DONE: packed video streaming.** Small videos pack into the `.beam` - (a `MediaKind::Video` blob at the clip id, `VideoClip.media_id` referencing it) and stream **both - frames and audio** from the DB blob via FFmpeg. The `AVIOContext`-over-`Read+Seek` shim lives in - the new `ffmpeg-blob-io` crate (`BlobInput`, version-pinned `=8.0.0`/`=8.0.1`), isolating the - unsafe + ABI coupling. Frames: `video.rs` `VideoSource{Path,Packed}` opens a fresh `BlobReader` - per decoder/seek/scan. Audio: `VideoAudioReader::open_source` over the same blob (the - `disk_reader.rs` `StreamSource` blocker is removed); save points the linked video-audio pool - entry's `media_id` at the video row so it streams from the same blob. Tests: ffmpeg-blob-io AVIO - unit tests (WAV via Cursor + seek + open/drop loop), core `packed_video_stream` (blob→AVIO→Input), - `beam_archive` packed-video round-trip, daw-backend `open_source` (compiles; can't link in the - container — user runtime-verifies actual A/V playback). -- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next** - - [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync` - (mmap). Fixed the ~2.8GB-`Vec` OOM. But writes the whole WAV to `/tmp` (fills - small temp partitions) and the temp path doesn't survive reload. - - [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new - ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path - reference survives save/load. - - [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind` - in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact - frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`): - in-order decode + sample-exact seek at several targets. (Found: mono frames have an - empty channel layout → must `set_channel_layout` before resampling, else swr returns - AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]` - until wired. `VideoAudioReader` made `pub` for the integration test. - - [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames, - total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()` - empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` + - `do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) + - `EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's - waveform overview too. daw-backend compiles clean; probe `total_frames` test passes. - - [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via - `StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`); - `ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip` - activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the - read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is - behaviorally identical (StreamSource::Compressed wraps the same CompressedReader). - daw-backend + editor compile clean; VideoAudioReader tests still pass. - ⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no - regression) and that an activated VideoAudio clip produces sound. - - [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` → - pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the - `AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap - (`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead - `add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync` - / handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept - `import_audio_sync` (still used by normal audio import). daw-backend + editor clean. - **→ Feature is live end-to-end; ready for in-app testing.** - - [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse - symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in - `seek`, applied in `decode_next`, which continues rather than reporting EOF when a - whole packet is discarded). So compressed clips no longer drift vs video audio after - a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek - lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test. - - [~] Step 6: **bounded waveform overview** — replaces today's full-resolution - `raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0, - so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a - `decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder - + (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor. - - [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` — - `WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and - reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B: - `levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors). - Ragged last buckets reduce over available children (no value padding). Bounded - (~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`): - bucket min/max, partial flush, multi-level envelope == global min/max, root-first - ordering, stereo channels, size bound, chunk-agnostic. - - [~] **Slice 1b (data layer DONE; orchestration folded into 1c):** - - [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams - a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded - memory (one chunk + the pyramid). Test: envelope matches the signal through - both backends. - - [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels — - f16 a later size optimization). Round-trip test + rejects truncated/garbage. - - [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id). - - [ ] Orchestration (with 1c). - - [~] **Slice 1c (in-memory floor overview DONE; persistence next):** - - [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio` - threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly). - The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so - only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so - time→texel maps B samples/texel) and `total_frames = floor_texel_count`. - - [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable). - - [x] App: `waveform_minmax_pools: HashMap` (pool → `B`, carries the floor rate - with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel; - drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`. - - [x] Generation: on video-audio import Success, the same bg thread streams - `disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed - `floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.) - - [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` + - main.rs construction) → `render_layers` → **both** render sites (collapsed-group - ~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`, - `eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors). - - [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`** - instead of sampling a fractional mip. Trilinear blends two levels whose row-major - linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x - zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:** - features hold position at every zoom and line up with playback. - See memory `waveform-shader-fractional-mip-offset`. - - [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and - kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform` - row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`, - "LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so - it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row. - Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a - `waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back; - the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs` - + flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's - `sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load. - `register_loaded_videos` only loads frames (not audio), so there is no redundant - regeneration to suppress. Compiles clean across all three crates. - -### Waveform LOD pyramid design (step 6) -A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed -in → per-sample; seamless between. - -- **One streaming decode pass** builds the whole pyramid down to a configurable **floor** - `B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running - per-level min/max accumulator; a filled bucket emits a texel and folds into its parent — - `branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h - stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT - stored. -- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session - temp before first save). `B` is stored with it (preference is just the default for new gen). - Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode. -- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand. - Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is - cleared only after its children") — so rendering can always walk up to a resident ancestor; - detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free. -- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a - tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 — - the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap: - above the floor it's a disk read; below it the span is already small. -- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first - on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a - disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode). -- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of - min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile. - -**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable** -(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4 -viewports of fine tiles; persist pyramid in `.beam`. -- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs - holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second - UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples - at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead - `VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`. -- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded - `HashMap<(Uuid,i64), Arc>` that grew per distinct frame during playback) is now an - `LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame - count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked - on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level - cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)* -- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every - packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending - packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF - behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an - in-app export to confirm A/V sync.)* -- [x] Phase 3a — lazy + async raster fault-in (`RasterStore` + background thread + image proxy) -- [x] Phase 3b — raster residency LRU + eviction (dirty-flag data-loss safety) -- [x] Phase 3c — bound raster GPU texture cache (recency LRU + F3 VRAM readout) -- [x] Phase 3d — raster undo dirty-rect diffs (+ fault-in-before-undo) -- [x] Phase 3.5 — image textures in vector scenes (fixed DCEL-broken image import; image-fill tab + picker; container-persisted) -- [x] Phase 4 — image asset paging: Tier 2 decoded-cache byte-LRU, Tier 1 lazy container bytes, playback prefetch -- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again** - (daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals - in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs, - effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves) - and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real - undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them - on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched. diff --git a/TODO.md b/TODO.md index 4e61b42..a718f1e 100644 --- a/TODO.md +++ b/TODO.md @@ -1,305 +1,25 @@ # Lightningbeam TODO -> ⚠️ **Stale entries:** Lightningbeam was rewritten from JavaScript to Rust. Any entry below -> that cites `src/*.js` / `main.js` / `animation.js` predates that migration — the *issue* may -> or may not still exist in the Rust codebase, but the file/line references are obsolete. -> **Re-verify against the current Rust code before acting** (this covers the "Animation System -> Refactoring" section and the JS-referencing "Known Issues" entries — node editor, default -> interpolation, etc.). Items with no `.js` references are current. +## Known Issues (Rust) -## Animation System Refactoring *(STALE — JS-era migration notes; superseded by the Rust DCEL/keyframe system)* +### Animation: Tweens are broken — LOW PRIORITY +- Shape/vector interpolation between keyframes, and the `tween_after` behavior on + keyframes, don't work correctly in the current app. Needs investigation + fix. + Not urgent — revisit later. -### Completed -- ✅ Implement AnimationData curve-based system (Keyframe, AnimationCurve, AnimationData classes) -- ✅ Add GraphicsObject.currentTime property -- ✅ Migrate shape rendering to use AnimationData curves (exists, zOrder) -- ✅ Binary search optimization for keyframe lookups +## Backlog / Feature ideas -### In Progress -- Migrating from Frame-based to AnimationData curve-based system throughout codebase +### Animation curve enhancements +- [ ] Extrapolation modes, separate for start vs end: hold (default), extend, repeat, decay +- [ ] Position / scale / rotation animation curves for shapes +- [ ] Shape morphing / tweening between keyframes -### Pending Features - -#### Animation Curve Enhancements -- [ ] Implement extrapolation modes (separate for start vs end): - - "hold" (default) - hold value at first/last keyframe - - "extend" - linearly extend the curve beyond keyframes - - "repeat" - repeat the animation - - "decay" - exponential decay to a target value -- [ ] Add position, scale, rotation animation curves for shapes -- [ ] Add shape morphing/tweening between keyframes - -#### Keyframing Behavior -- [ ] Add user preference for keyframing behavior when editing objects: +### Keyframing behavior +- [ ] User preference for keyframing when editing objects: - Auto-keyframe (current default): create/update keyframe at current time - Edit previous (Flash-style): update most recent keyframe before current time - Ephemeral (Blender-style): changes don't persist without manual keyframe - - Optional: Add modifier key (e.g. Shift) to toggle between modes + - Optional modifier key (e.g. Shift) to toggle modes -#### Shape Ordering -- [ ] Add "Bring Forward" menu option (swap zOrder with shape in front) -- [ ] Add "Send Backward" menu option (swap zOrder with shape behind) -- [ ] Add "Bring to Front" menu option (set zOrder to max + 1) -- [ ] Add "Send to Back" menu option (set zOrder to min - 1) - -#### Code Cleanup -- [ ] Remove all remaining references to Frame-based system -- [ ] Remove legacy Frame class once migration is complete -- [ ] Clean up GraphicsObject.shapes[] array (shapes should only live in Layers) - -## Known Issues / Platform Limitations - -### Animation: Tweens are broken (Rust codebase) — LOW PRIORITY -- **Issue**: Animation tweening between keyframes (shape/vector interpolation, and the - `tween_after` behavior on keyframes) does not work correctly in the current Rust app. - Needs investigation + fix. Not urgent — revisit later. -- (Older JS-codebase animation entries below reference `src/*.js` and are stale.) - -### Audio: Oscillator Timbre Drift (Phase Accumulation Error) -- **Issue**: Oscillators exhibit timbre changes over time due to floating-point phase accumulation errors -- **Affected Files**: - - `daw-backend/src/effects/synth.rs:117-120` (SimpleSynth) - - `daw-backend/src/audio/node_graph/nodes/oscillator.rs:167-170` (OscillatorNode) -- **Root Cause**: Current phase wrapping uses conditional subtraction (`if phase >= 1.0 { phase -= 1.0 }`), which accumulates f32 rounding errors over time, especially for long-playing notes -- **Current Code**: - ```rust - self.phase += frequency / sample_rate; - if self.phase >= 1.0 { - self.phase -= 1.0; - } - ``` -- **Recommended Fix**: Replace with `.fract()` for numerically stable wraparound: - ```rust - self.phase += frequency / sample_rate; - self.phase = self.phase.fract(); - ``` -- **Impact**: Medium - affects audio quality for sustained notes, becomes noticeable after several seconds -- **Priority**: Medium - should be addressed before production use - -### UI: Node Connections Render Behind VoiceAllocator Child Nodes -- **Issue**: Connection lines (SVG paths) inside expanded VoiceAllocator nodes render behind child nodes due to z-index stacking -- **Affected File**: `src/styles.css:1128` -- **Root Cause**: Child nodes have `z-index: 10` while connection SVG paths have default/lower z-index -- **Current Code**: - ```css - .drawflow .drawflow-node.child-node { - opacity: 0.9; - border: 1px solid #5a5aaa !important; - box-shadow: 0 2px 8px rgba(90, 90, 170, 0.3); - z-index: 10; - } - ``` -- **Recommended Fix**: Either: - 1. Remove `z-index: 10` from `.child-node` (simplest), or - 2. Add higher z-index to connection SVG paths, or - 3. Use CSS `isolation: isolate` on the VoiceAllocator contents area to create a new stacking context -- **Impact**: Low - visual issue only, connections still function but appear to go "behind" nodes -- **Priority**: Low - cosmetic issue that doesn't affect functionality - -### UI: VoiceAllocator Child Nodes Don't Move with Parent -- **Issue**: When a VoiceAllocator node is moved, its child nodes remain in their original positions instead of moving with the parent -- **Affected File**: `src/main.js:6202-6207` -- **Root Cause**: The `nodeMoved` event handler only handles the case where a child node is moved (resizes parent), but doesn't handle when the VoiceAllocator itself is moved -- **Current Code**: - ```javascript - editor.on("nodeMoved", (nodeId) => { - const node = editor.getNodeFromId(nodeId); - if (node && node.data.parentNodeId) { - resizeVoiceAllocatorToFit(node.data.parentNodeId); - } - }); - ``` -- **Recommended Fix**: Add logic to detect when a VoiceAllocator is moved and update all child node positions: - ```javascript - editor.on("nodeMoved", (nodeId) => { - const node = editor.getNodeFromId(nodeId); - - // Case 1: A child node was moved - resize parent - if (node && node.data.parentNodeId) { - resizeVoiceAllocatorToFit(node.data.parentNodeId); - } - - // Case 2: A VoiceAllocator was moved - move all children - if (node && node.data.nodeType === 'VoiceAllocator') { - // Calculate delta from previous position (need to track) - // Update all child node positions by the delta - // Call editor.updateConnectionNodes() for parent and all children - } - }); - ``` -- **Impact**: High - child nodes become disconnected from parent visually -- **Priority**: High - breaks expected behavior of grouped nodes - -### UI: VoiceAllocator Expansion Doesn't Update Connection Positions -- **Issue**: When expanding/collapsing a VoiceAllocator, connection endpoints don't update to match the new port positions -- **Affected File**: `src/main.js:6496-6555` (handleNodeDoubleClick function) -- **Root Cause**: The expand/collapse logic shows/hides child nodes and resizes the container, but never calls `editor.updateConnectionNodes()` to refresh connection positions -- **Current Code**: In `handleNodeDoubleClick()`, after expanding or collapsing: - ```javascript - // Expand - expandedNodes.add(nodeId); - nodeElement.classList.add('expanded'); - nodeElement.style.width = '600px'; - nodeElement.style.height = '400px'; - // ... shows child nodes ... - // Missing: editor.updateConnectionNodes(`node-${nodeId}`) - ``` -- **Recommended Fix**: Call `editor.updateConnectionNodes()` after resizing: - ```javascript - // After expanding - expandedNodes.add(nodeId); - nodeElement.classList.add('expanded'); - // ... resize and show children ... - - // Update connection positions for VoiceAllocator and all children - editor.updateConnectionNodes(`node-${nodeId}`); - for (const [childId, parentId] of nodeParents.entries()) { - if (parentId === nodeId) { - editor.updateConnectionNodes(`node-${childId}`); - } - } - ``` -- **Impact**: Medium - connections appear in wrong positions until manually moved -- **Priority**: Medium - visual issue that affects usability - -### UI: Node Editor Allows Editing Without MIDI Layer Selected -- **Issue**: The node editor pane allows adding/editing instrument nodes even when no MIDI layer is selected, and always uses hardcoded `trackId: 0` -- **Affected File**: `src/main.js:6045-6920` (nodeEditor function) -- **Root Cause**: The node editor never checks if `context.activeObject.activeLayer` exists or is a MIDI track, and all backend commands use hardcoded `trackId: 0` -- **Current Code**: All graph commands hardcode track 0: - ```javascript - const commandArgs = parentNodeId - ? { - trackId: 0, // HARDCODED! - voiceAllocatorId: editor.getNodeFromId(parentNodeId).data.backendId, - nodeType: nodeType, - x: x, - y: y - } - : { - trackId: 0, // HARDCODED! - nodeType: nodeType, - x: x, - y: y - }; - ``` -- **Recommended Fix**: - 1. Check if activeLayer is a MIDI track before allowing edits: - ```javascript - function getSelectedMidiTrack() { - const activeLayer = context.activeObject?.activeLayer; - if (!activeLayer || activeLayer.type !== 'midi') { - return null; - } - return activeLayer; - } - ``` - 2. Show placeholder when no MIDI track selected: - ```javascript - function nodeEditor() { - const container = document.createElement("div"); - const midiTrack = getSelectedMidiTrack(); - - if (!midiTrack) { - container.innerHTML = '
Select a MIDI layer to edit instruments
'; - return container; - } - // ... rest of node editor code ... - } - ``` - 3. Use actual track ID instead of hardcoded 0: - ```javascript - const trackId = midiTrack.audioTrackId || 0; - const commandArgs = { trackId, nodeType, x, y }; - ``` - 4. Add listener to refresh node editor when layer selection changes -- **Impact**: High - allows editing wrong track's instrument graph, data corruption risk -- **Priority**: High - can cause confusion and data loss - -### Animation: Wrong Default Interpolation for Shape and Object Keyframes -- **Issue**: Shape index and object transform keyframes default to "linear" interpolation but should default to "hold" (step function), and there's no UI to change interpolation after creation -- **Affected Files**: - - `src/models/animation.js:124` (Keyframe constructor defaults to "linear") - - `src/main.js:2161` (shapeIndex keyframes default to "linear") - - `src/main.js:2198` (object position/rotation/scale keyframes default to "linear") - - `src/main.js:5910` (Timeline menu - missing tween options) -- **Root Cause**: - 1. The Keyframe constructor defaults interpolation to "linear" - 2. Shape index keyframes preserve existing interpolation or default to "linear" - 3. Object transform keyframes explicitly use "linear" - 4. No menu options exist to change interpolation mode after keyframe creation -- **Current Code**: - - Keyframe constructor (animation.js:124): - ```javascript - constructor(time, value, interpolation = "linear", uuid = undefined) { - ``` - - Shape index keyframes (main.js:2161): - ```javascript - const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'linear'; - const shapeIndexKeyframe = new Keyframe(currentTime, newShapeIndex, interpolationType); - ``` - - Object keyframes (main.js:2198): - ```javascript - const newKeyframe = new Keyframe( - currentTime, - currentValue, - 'linear' // Default to linear interpolation - ); - ``` -- **Expected Behavior**: - - Shape index keyframes should default to "hold" (shapes shouldn't morph between versions) - - Object transforms should default to "hold" (objects shouldn't move/rotate/scale between keyframes unless explicitly tweened) - - Timeline menu should have options to convert between interpolation modes -- **Recommended Fix**: - 1. Change shapeIndex default to "hold" (main.js:2161): - ```javascript - const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'hold'; - ``` - 2. Change object keyframe default to "hold" (main.js:2198): - ```javascript - const newKeyframe = new Keyframe(currentTime, currentValue, 'hold'); - ``` - 3. Add Timeline menu options (main.js:5910, in timelineSubmenu): - ```javascript - { - text: "Add Shape Tween", - enabled: /* check if shape is selected and has keyframes */, - action: () => { - // Find shapeIndex curve for selected shape - // Change interpolation between keyframes to "linear" - } - }, - { - text: "Add Motion Tween", - enabled: /* check if object is selected and has transform keyframes */, - action: () => { - // Find position/rotation/scale curves for selected object - // Change interpolation between keyframes to "linear" or "bezier" - } - } - ``` -- **Note**: exists and zOrder keyframes already correctly use "hold" (main.js:2139, 2150) -- **Impact**: High - causes unwanted interpolation, shapes morph unexpectedly, objects move when they shouldn't -- **Priority**: High - fundamental animation behavior is incorrect - -### Tauri Pinch-Zoom on Linux -- **Issue**: Two-finger pinch gestures zoom the entire Tauri window instead of individual canvases -- **Status**: Known Tauri limitation on Linux/GTK with no cross-platform solution -- **Tracking**: https://github.com/tauri-apps/tauri/discussions/3843 -- **Workaround attempts**: Tried `zoomHotkeysEnabled: false`, `touch-action: none`, viewport meta tags - none worked -- **Resolution**: Monitor Tauri releases for official fix - -## Notes - -### Architecture -- **GraphicsObject** contains Layers and has `currentTime` (continuous time) -- **Layer** contains `shapes[]` array and `animationData` (AnimationData instance) -- **AnimationData** contains curves dictionary, each curve identified by parameter name - - Shape curves: `shape.{uuid}.exists`, `shape.{uuid}.zOrder` - - Future: `shape.{uuid}.x`, `shape.{uuid}.y`, `shape.{uuid}.rotation`, etc. -- **Shapes render based on curves**: Layer.draw checks exists > 0, sorts by zOrder, draws in order - -### Interpolation Types -- `linear` - Linear interpolation between keyframes -- `bezier` - Cubic Bezier with easing control points -- `step`/`hold` - Step function (jumps to next value) +### Shape ordering +- [ ] Bring Forward / Send Backward / Bring to Front / Send to Back menu options diff --git a/daw-backend/Cargo.lock b/daw-backend/Cargo.lock index 6399b8d..25d2e1e 100644 --- a/daw-backend/Cargo.lock +++ b/daw-backend/Cargo.lock @@ -586,6 +586,7 @@ dependencies = [ "dasp_rms", "dasp_sample", "dasp_signal", + "ffmpeg-blob-io", "ffmpeg-next", "hound", "memmap2", @@ -661,6 +662,15 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365" +[[package]] +name = "ffmpeg-blob-io" +version = "0.1.0" +dependencies = [ + "ffmpeg-next", + "ffmpeg-sys-next", + "libc", +] + [[package]] name = "ffmpeg-next" version = "8.0.0" diff --git a/lightningbeam-ui/Cargo.lock b/lightningbeam-ui/Cargo.lock index efbdbeb..97aed88 100644 --- a/lightningbeam-ui/Cargo.lock +++ b/lightningbeam-ui/Cargo.lock @@ -3628,7 +3628,7 @@ dependencies = [ [[package]] name = "lightningbeam-editor" -version = "1.0.7-alpha" +version = "1.0.8-alpha" dependencies = [ "beamdsp", "bytemuck", diff --git a/lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md b/lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md index 0586549..17e2d19 100644 --- a/lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md +++ b/lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md @@ -67,15 +67,47 @@ at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *an - Result: software exports are full-quality at any export res, and document resizes re-target decode. No hardware needed; this is the correctness fix for the codecs HW can't handle anyway. -### Stage 2 — hardware decode primitive (headless-testable here, like the 8 encode tests) -- In `gpu-video-encoder` (rename → `gpu-video-codec`): `h264_vaapi`-style **decode** → VAAPI surface → - export DMA-BUF → import as a wgpu texture. Hardware test: decode a known file, verify dims/contents. +### Stage 2 — hardware decode primitive (DONE, commit 255e164) +`decoder::VaapiDecoder` in `gpu-video-encoder`: decode → VAAPI surface → DRM-PRIME DMA-BUF → +`dmabuf::import_raw` → wgpu textures. Round-trip test (encode gray → decode → readback Y≈128) passes. -### Stage 3 — wire hardware decode into `get_frame` (blind; user-verifies) -- When the source codec/driver is HW-decodable, `get_frame` returns a **GPU texture** (native res) - instead of `Arc>`; the compositor uses it directly (no `write_texture`), GPU-scaling to the - target. For the zero-copy export the frame never leaves the GPU: **decode → composite → encode** on - one device. Software path is the fallback for everything else. +### The device-affinity problem (drives the whole rest of the design) +wgpu textures can't cross devices, and a decoded frame is a wgpu texture imported from a DMA-BUF — +which **requires a device with the DMA-BUF-import extensions** (`VK_EXT_image_drm_format_modifier` ++ external-memory), built via wgpu-hal `device_from_raw` (the safe `DeviceDescriptor` can't add +them). So a hardware-decoded frame is only usable by a compositor running on **such** a device. +- **Export** composites on the encoder's custom device → already fine. +- **Preview** composites on eframe's *normal* device → can't import DMA-BUFs → can't use HW frames. + +Since **preview must HW-decode 4K** (software 4K decode ≈19 ms/frame), the resolution is a **single +shared custom device** used by eframe + preview compositor + decoder + encoder. eframe 0.33 (local +`egui-fork`) accepts it via `WgpuSetup::Existing { instance, adapter, device, queue }` — confirmed. +The earlier "separate export device" becomes redundant once this lands. + +### Stage 3a — windowed shared `DrmDevice`, injected into eframe (highest-risk; blind) +Today `vk_device::create()` is **headless**. Make a windowed variant (or extend it) that is a +**superset** device: DMA-BUF import ext **+** `VK_KHR_swapchain` (device) and the WSI surface +instance extensions, **+** everything eframe/egui/vello need — `adapter.limits()` (already; Vello +needs `max_storage_buffers_per_shader_stage` ≥ 5), `max_texture_dimension_2d` 8192, and the optional +features main.rs requests (`SHADER_F16`, `TIMESTAMP_QUERY[_INSIDE_ENCODERS]`). Pick the adapter that +is the **VAAPI GPU** (the render node must match libva's, or DMA-BUF sharing fails on multi-GPU). +- main.rs: try to build the shared device; on success pass `WgpuSetup::Existing`, else fall back to + the current `WgpuSetupCreateNew` (software decode only). Gate on Linux + VAAPI + a config/env + override; **must be bulletproof** — this device now renders *every* frame of *every* session for + Linux/VAAPI users, video or not. Milestone: editor runs normally on it with no video involved. + +### Stage 3b — VideoManager hardware decode on the shared device (blind) +- `VideoManager` holds a `VaapiDecoder` per HW-decodable clip (built on the shared device), plus the + software `VideoDecoder` fallback. `get_frame` gains a GPU-returning variant: yields an imported NV12 + texture pair (native res) instead of `Arc>`. Probe HW support per source; non-VAAPI / + unsupported codecs / non-Linux → software path (Stage 1, target-res). +- Cache native GPU textures keyed by (clip, ts); revisit the byte budget (4K NV12 ≈ 12 MB each). + +### Stage 3c — compositor consumes the GPU frame (blind; user-verifies) +- The video-instance composite path takes an NV12 texture (or a small NV12→RGB GPU pass) and blits it + to the target with the existing bilinear blit — **no `write_texture` upload**. GPU scales native→ + target (preview res or export res). Both preview and the zero-copy export become + decode→composite(→encode) with no CPU frame. Software frames still upload as today. ## Critical files - `lightningbeam-core/src/video.rs` — `VideoDecoder` (per-request output size, scaler cache), @@ -86,13 +118,21 @@ at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *an - `gpu-video-encoder/` (→ `gpu-video-codec`) — `dmabuf.rs`/`vk_device.rs` reused for the decode import. ## Risks +- **Shared custom device is the editor's main device (BIGGEST risk)** — Stage 3a makes a hand-built + wgpu-hal Vulkan device render every frame for Linux/VAAPI users. It must satisfy eframe + egui + + vello + winit presentation across varied Intel/AMD/Mesa stacks, or the editor won't start. Mitigate + with a strict try-and-fall-back-to-normal-device path + an env/config kill switch. Test broadly. +- **Multi-GPU** — the shared render device must be the *same* GPU as libva's VAAPI device, or DMA-BUF + import fails. Adapter selection must match the render node to the VAAPI node (laptops with iGPU + + dGPU, PRIME). - **Codec coverage** — only some codecs are HW-decodable per GPU/driver; software must stay correct - and well-tested. Selection must probe support per source, not assume. -- **Cache memory** — native-res GPU textures (esp. 4K) are large; the frame cache budget needs revisiting. -- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; the existing import handles NV12, but - 10-bit/HDR sources (P010) need format handling. -- **Preview vs export sharing** — two live targets (preview res + export res) from the same source; the - cache/scaler design must serve both without thrashing. + and well-tested. Probe support per source, don't assume. +- **Cache memory** — native-res GPU textures (esp. 4K NV12 ≈12 MB) are large; revisit the frame cache + budget, and the two live targets (preview res + export res) shouldn't thrash. +- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; import handles NV12, but 10-bit/HDR + (P010) needs format handling. Decoded NV12 also needs the right BT.601/709 + range on the NV12→RGB + read (mirror the encoder's color tags, [[gpu-video-decode]] color-range work). +- **Non-Linux / no-VAAPI** — must cleanly run on the normal eframe device with software decode. ## Verification - Stage 0/1: visual — export above document res is now full-quality (not upscaled); profile shows diff --git a/lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs b/lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs index 705c9d5..687ed15 100644 --- a/lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs +++ b/lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs @@ -198,7 +198,7 @@ impl Action for AddClipInstanceAction { // Calculate internal start/end from trim parameters let internal_start = self.clip_instance.trim_start; - let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration); + let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native()); let external_start = self.clip_instance.timeline_start; // Calculate external duration (for looping if timeline_duration is set). @@ -240,7 +240,7 @@ impl Action for AddClipInstanceAction { // `trim_*` / `clip.duration` are in SECONDS (audio content time), // while `timeline_*` and the backend's `duration` are in BEATS. let internal_start = self.clip_instance.trim_start; - let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration); + let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native()); let start_time = self.clip_instance.timeline_start; // `effective_duration` is in BEATS. When `timeline_duration` is set // it already is; otherwise the clip occupies its natural content diff --git a/lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs b/lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs index b275b22..4e05db1 100644 --- a/lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs +++ b/lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs @@ -129,7 +129,7 @@ impl LoopClipInstancesAction { }; let content_window = { - let trim_end = instance.trim_end.unwrap_or(clip.duration); + let trim_end = instance.trim_end.unwrap_or(clip.content_duration().native()); (trim_end - instance.trim_start).max(0.0) // seconds }; // Natural content length as a beats span at the clip's start (the diff --git a/lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs b/lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs index 76af639..947e6e0 100644 --- a/lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs +++ b/lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs @@ -170,7 +170,7 @@ impl Action for RemoveClipInstancesAction { use daw_backend::command::{Query, QueryResponse}; let internal_start = instance.trim_start; - let internal_end = instance.trim_end.unwrap_or(clip.duration); + let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native()); let external_start = instance.timeline_start; // MIDI trims are beats-domain, so the fallback span is beats too. let external_duration = instance @@ -198,7 +198,7 @@ impl Action for RemoveClipInstancesAction { } AudioClipType::Sampled { audio_pool_index } => { let internal_start = instance.trim_start; - let internal_end = instance.trim_end.unwrap_or(clip.duration); + let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native()); let start_time = instance.timeline_start; // Fallback span is the content seconds converted to beats at the // clip's start (not the seconds span treated as beats). diff --git a/lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs b/lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs index fb3095e..1e2c6d4 100644 --- a/lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs +++ b/lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs @@ -377,7 +377,7 @@ impl Action for SplitClipInstanceAction { // 1. Trim the original (left) instance let orig_internal_start = original_instance.trim_start; - let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration); + let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native()); // Look up the original backend instance ID if let Some(crate::action::BackendClipInstanceId::Midi(orig_backend_id)) = @@ -388,7 +388,7 @@ impl Action for SplitClipInstanceAction { // 2. Add the new (right) instance let internal_start = new_instance.trim_start; - let internal_end = new_instance.trim_end.unwrap_or(clip.duration); + let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native()); let external_start = new_instance.timeline_start; // MIDI trims are beats-domain, so the fallback span is beats too. let external_duration = new_instance @@ -425,7 +425,7 @@ impl Action for SplitClipInstanceAction { AudioClipType::Sampled { audio_pool_index } => { // 1. Trim the original (left) instance let orig_internal_start = original_instance.trim_start; - let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration); + let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native()); // Look up the original backend instance ID if let Some(crate::action::BackendClipInstanceId::Audio(orig_backend_id)) = @@ -436,7 +436,7 @@ impl Action for SplitClipInstanceAction { // 2. Add the new (right) instance let internal_start = new_instance.trim_start; - let internal_end = new_instance.trim_end.unwrap_or(clip.duration); + let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native()); let start_time = new_instance.timeline_start; // Fallback span is the content seconds converted to beats at the // clip's start (not the seconds span treated as beats). @@ -499,7 +499,7 @@ impl Action for SplitClipInstanceAction { if let Some(instance) = al.clip_instances.iter().find(|ci| ci.id == self.instance_id) { if let Some(clip) = document.get_audio_clip(&instance.clip_id) { let orig_internal_start = instance.trim_start; - let orig_internal_end = self.original_trim_end.unwrap_or(clip.duration); + let orig_internal_end = self.original_trim_end.unwrap_or(clip.content_duration().native()); // Restore based on clip type use crate::clip::AudioClipType; diff --git a/lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs b/lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs index 55fa95d..d4d48bc 100644 --- a/lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs +++ b/lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs @@ -424,7 +424,7 @@ impl Action for TrimClipInstancesAction { // Calculate new internal_start and internal_end for backend // Note: instance already has the new trim values after execute() let internal_start = instance.trim_start; - let internal_end = instance.trim_end.unwrap_or(clip.duration); + let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native()); // Handle trim based on clip type match &clip.clip_type { @@ -517,8 +517,8 @@ impl Action for TrimClipInstancesAction { TrimType::TrimRight => instance.trim_start, // trim_start wasn't changed }; let internal_end = match trim_type { - TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.duration), // trim_end wasn't changed - TrimType::TrimRight => old.trim_value.unwrap_or(clip.duration), + TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.content_duration().native()), // trim_end wasn't changed + TrimType::TrimRight => old.trim_value.unwrap_or(clip.content_duration().native()), }; // Handle trim based on clip type diff --git a/lightningbeam-ui/lightningbeam-core/src/clip.rs b/lightningbeam-ui/lightningbeam-core/src/clip.rs index 2c216df..dd294c4 100644 --- a/lightningbeam-ui/lightningbeam-core/src/clip.rs +++ b/lightningbeam-ui/lightningbeam-core/src/clip.rs @@ -468,6 +468,37 @@ pub enum AudioClipType { Recording, } +/// A clip's content duration, tagged by its native unit. +/// +/// Sampled/recording audio and video measure content in wall-clock **seconds**; MIDI measures +/// it in **beats** (tempo-independent musical length). Carrying the domain in the type means a +/// duration can't be silently read in the wrong unit. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum ClipDuration { + Seconds(Seconds), + Beats(Beats), +} + +impl ClipDuration { + /// Wall-clock seconds. Beats are converted as a length from beat 0 (exact under constant + /// tempo; a reasonable approximation otherwise — durations are position-independent here). + pub fn to_seconds(self, tempo_map: &daw_backend::TempoMap) -> Seconds { + match self { + ClipDuration::Seconds(s) => s, + ClipDuration::Beats(b) => tempo_map.beats_to_seconds(b), + } + } + + /// The raw magnitude in the clip's native unit. Use only in code that already works in that + /// domain (e.g. trim math, whose values share the clip's native domain). + pub fn native(self) -> f64 { + match self { + ClipDuration::Seconds(s) => s.seconds_to_f64(), + ClipDuration::Beats(b) => b.beats_to_f64(), + } + } +} + /// Audio clip /// /// This is compatible with daw-backend's audio system: @@ -481,9 +512,13 @@ pub struct AudioClip { /// Clip name pub name: String, - /// Duration in seconds - /// For sampled audio, this can be set to trim the audio shorter than the source file - pub duration: f64, + /// Raw content duration in the clip's **native domain** — SECONDS for sampled/recording + /// audio, BEATS for MIDI (musical length, tempo-independent). Private on purpose: the domain + /// depends on `clip_type`, so all access goes through [`AudioClip::content_duration`] / + /// [`AudioClip::set_content_duration`], which keep it type-safe. Stored as a bare `f64` + /// because the `.beam` format serializes it as a plain number (serde derives over private + /// fields fine); a domain-tagged newtype would change the on-disk shape. + duration: f64, /// Audio clip type (sampled or MIDI) pub clip_type: AudioClipType, @@ -494,6 +529,31 @@ pub struct AudioClip { } impl AudioClip { + /// The clip's content duration, tagged with its native domain (seconds for sampled/recording, + /// beats for MIDI). This is the only sanctioned way to read the raw `duration` field. + pub fn content_duration(&self) -> ClipDuration { + match self.clip_type { + AudioClipType::Midi { .. } => ClipDuration::Beats(Beats(self.duration)), + AudioClipType::Sampled { .. } | AudioClipType::Recording => { + ClipDuration::Seconds(Seconds(self.duration)) + } + } + } + + /// Set the content duration. Debug-asserts the value's domain matches the clip type so a + /// beats duration can't be stored on a seconds clip (or vice-versa). + pub fn set_content_duration(&mut self, duration: ClipDuration) { + debug_assert!( + matches!( + (&self.clip_type, duration), + (AudioClipType::Midi { .. }, ClipDuration::Beats(_)) + | (AudioClipType::Sampled { .. } | AudioClipType::Recording, ClipDuration::Seconds(_)) + ), + "clip duration domain must match clip type", + ); + self.duration = duration.native(); + } + /// Create a new sampled audio clip /// /// # Arguments diff --git a/lightningbeam-ui/lightningbeam-core/src/document.rs b/lightningbeam-ui/lightningbeam-core/src/document.rs index 9596f74..a26b86b 100644 --- a/lightningbeam-ui/lightningbeam-core/src/document.rs +++ b/lightningbeam-ui/lightningbeam-core/src/document.rs @@ -472,8 +472,10 @@ impl Document { } crate::layer::AnyLayer::Audio(audio_layer) => { for instance in &audio_layer.clip_instances { - if let Some(clip) = self.audio_clips.get(&instance.clip_id) { - let end_time = calculate_instance_end(instance, clip.duration); + // get_clip_duration yields seconds (converting MIDI's beats duration), + // which is what the closure expects. + if let Some(clip_duration) = self.get_clip_duration(&instance.clip_id) { + let end_time = calculate_instance_end(instance, clip_duration.seconds_to_f64()); max_end_time = max_end_time.max(end_time); } } @@ -516,8 +518,8 @@ impl Document { } crate::layer::AnyLayer::Audio(al) => { for inst in &al.clip_instances { - if let Some(clip) = doc.audio_clips.get(&inst.clip_id) { - *max_end = max_end.max(calc_end(inst, clip.duration)); + if let Some(clip_duration) = doc.get_clip_duration(&inst.clip_id) { + *max_end = max_end.max(calc_end(inst, clip_duration.seconds_to_f64())); } } } @@ -908,7 +910,7 @@ impl Document { // Avoid deep recursion — use stored duration for nested vector clips Some(vc.content_duration(self.framerate, tempo_map)) } else if let Some(ac) = self.audio_clips.get(id) { - Some(ac.duration) + Some(ac.content_duration().to_seconds(tempo_map).seconds_to_f64()) } else if let Some(vc) = self.video_clips.get(id) { Some(vc.duration) } else if self.effect_definitions.contains_key(id) { @@ -921,15 +923,9 @@ impl Document { } else if let Some(clip) = self.video_clips.get(clip_id) { Some(Seconds(clip.duration)) } else if let Some(clip) = self.audio_clips.get(clip_id) { - // MIDI clips store `duration` in BEATS (they share the AudioClip struct with - // sampled clips, whose duration is seconds). Convert to wall-clock seconds so - // the content-window sizing works uniformly. - match clip.clip_type { - crate::clip::AudioClipType::Midi { .. } => { - Some(self.tempo_map().beats_to_seconds(Beats(clip.duration))) - } - _ => Some(Seconds(clip.duration)), - } + // Interpret the clip's native-domain duration as wall-clock seconds (MIDI stores + // beats, sampled stores seconds — content_duration keeps that straight). + Some(clip.content_duration().to_seconds(self.tempo_map())) } else if self.effect_definitions.contains_key(clip_id) { // Effects have infinite internal duration - their timeline length // is controlled by ClipInstance.trim_end diff --git a/lightningbeam-ui/lightningbeam-editor/src/main.rs b/lightningbeam-ui/lightningbeam-editor/src/main.rs index 7f98849..6ad2a5b 100644 --- a/lightningbeam-ui/lightningbeam-editor/src/main.rs +++ b/lightningbeam-ui/lightningbeam-editor/src/main.rs @@ -2,6 +2,7 @@ use eframe::egui; use daw_backend::{Beats, Seconds}; +use lightningbeam_core::clip::ClipDuration; use lightningbeam_core::layer::{AnyLayer, AudioLayer}; use lightningbeam_core::layout::{LayoutDefinition, LayoutNode}; use lightningbeam_core::pane::PaneType; @@ -5879,7 +5880,7 @@ impl EditorApp { // Get audio clip duration for logging let duration = self.action_executor.document().audio_clips .get(&audio_clip_id) - .map(|c| c.duration) + .map(|c| c.content_duration().native()) .unwrap_or(0.0); println!("✅ Extracted audio from '{}' ({:.1}s, {}ch, {}Hz) - AudioClip ID: {}", @@ -6477,7 +6478,7 @@ impl eframe::App for EditorApp { if let Some(doc_clip_id) = doc_clip_id { if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { if clip.is_recording() { - clip.duration = duration.seconds_to_f64(); + clip.set_content_duration(ClipDuration::Seconds(duration)); } } } @@ -6650,7 +6651,7 @@ impl eframe::App for EditorApp { } // Update the clip's duration so the timeline bar grows if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { - clip.duration = duration.beats_to_f64(); + clip.set_content_duration(ClipDuration::Beats(duration)); } } } @@ -6686,7 +6687,7 @@ impl eframe::App for EditorApp { .map(|(id, _)| id); if let Some(doc_clip_id) = doc_clip_id { if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { - clip.duration = midi_clip_data.duration; + clip.set_content_duration(ClipDuration::Beats(Beats(midi_clip_data.duration))); clip.name = format!("MIDI Recording {}", clip_id); } } diff --git a/lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs b/lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs index cc39985..d3d0079 100644 --- a/lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs +++ b/lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs @@ -932,7 +932,7 @@ impl AssetLibraryPane { name: clip.name.clone(), category: AssetCategory::Audio, drag_clip_type, - duration: clip.duration, + duration: clip.content_duration().native(), dimensions: None, extra_info, is_builtin: false, @@ -1136,7 +1136,7 @@ impl AssetLibraryPane { name: clip.name.clone(), category: AssetCategory::Audio, drag_clip_type, - duration: clip.duration, + duration: clip.content_duration().native(), dimensions: None, extra_info, is_builtin: false, @@ -1802,7 +1802,7 @@ impl AssetLibraryPane { AudioClipType::Midi { midi_clip_id } => { let note_color = egui::Color32::from_rgb(100, 200, 100); if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { - Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) + Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color)) } else { Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) } @@ -2358,7 +2358,7 @@ impl AssetLibraryPane { AudioClipType::Midi { midi_clip_id } => { let note_color = egui::Color32::from_rgb(100, 200, 100); if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { - Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) + Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color)) } else { Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) } @@ -2495,7 +2495,7 @@ impl AssetLibraryPane { AudioClipType::Midi { midi_clip_id } => { let note_color = egui::Color32::from_rgb(100, 200, 100); if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { - Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) + Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color)) } else { Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) } @@ -2858,7 +2858,7 @@ impl AssetLibraryPane { let note_color = egui::Color32::from_rgb(100, 200, 100); if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { - Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) + Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color)) } else { Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) } diff --git a/lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs b/lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs index d29c26b..1522f5b 100644 --- a/lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs +++ b/lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs @@ -1808,7 +1808,7 @@ impl InfopanelPane { }); ui.horizontal(|ui| { ui.label("Duration:"); - ui.label(format!("{:.2}s", clip.duration)); + ui.label(format!("{:.2}s", clip.content_duration().to_seconds(document.tempo_map()).seconds_to_f64())); }); } else { // Could be an image asset or effect — show ID diff --git a/lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs b/lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs index 34b2145..2505ef3 100644 --- a/lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs +++ b/lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs @@ -466,7 +466,7 @@ impl PianoRollPane { for instance in &audio_layer.clip_instances { if let Some(clip) = document.audio_clips.get(&instance.clip_id) { if let AudioClipType::Midi { midi_clip_id } = clip.clip_type { - let duration = instance.effective_duration(Seconds(clip.duration), document.tempo_map()); + let duration = instance.effective_duration(clip.content_duration().to_seconds(document.tempo_map()), document.tempo_map()); clip_data.push((midi_clip_id, instance.timeline_start.beats_to_f64(), instance.trim_start, duration.beats_to_f64(), instance.id)); } } @@ -2459,7 +2459,7 @@ impl PianoRollPane { // length converted to beats at the clip's start. let duration = instance.timeline_duration.unwrap_or_else(|| { let tmap = document.tempo_map(); - tmap.seconds_to_beats(tmap.beats_to_seconds(instance.timeline_start) + Seconds(clip.duration)) - instance.timeline_start + tmap.seconds_to_beats(tmap.beats_to_seconds(instance.timeline_start) + clip.content_duration().to_seconds(tmap)) - instance.timeline_start }); // Get sample rate from raw_audio_cache if let Some((_samples, sr, _ch)) = shared.raw_audio_cache.get(&audio_pool_index) { diff --git a/lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs b/lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs index 2ed636d..c19b724 100644 --- a/lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs +++ b/lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs @@ -209,7 +209,9 @@ fn effective_clip_duration( document.get_clip_duration(&clip_instance.clip_id) } } - AnyLayer::Audio(_) => document.get_audio_clip(&clip_instance.clip_id).map(|c| Seconds(c.duration)), + // Delegate to get_clip_duration so MIDI clips (whose `duration` is stored in beats, + // not seconds) are converted correctly rather than read as raw seconds. + AnyLayer::Audio(_) => document.get_clip_duration(&clip_instance.clip_id), AnyLayer::Video(_) => document.get_video_clip(&clip_instance.clip_id).map(|c| Seconds(c.duration)), AnyLayer::Effect(_) => Some(Seconds(lightningbeam_core::effect::EFFECT_DURATION)), AnyLayer::Group(_) => None, @@ -3082,7 +3084,7 @@ impl TimelinePane { }; let audio_file_duration = total_frames as f64 / eff_sr as f64; - let clip_dur = Seconds(audio_clip.duration); + let clip_dur = audio_clip.content_duration().to_seconds(document.tempo_map()); let mut ci_start = ci.effective_start(); if is_move_drag && selection.contains_clip_instance(&ci.id) { ci_start = self.moved_start(ci_start, document.tempo_map(), &document.time_signature, document.framerate); @@ -4699,7 +4701,7 @@ impl TimelinePane { if selection.contains_clip_instance(&clip_instance.id) { let clip_duration = match layer { lightningbeam_core::layer::AnyLayer::Audio(_) => { - document.get_audio_clip(&clip_instance.clip_id).map(|c| c.duration) + document.get_audio_clip(&clip_instance.clip_id).map(|c| c.content_duration().native()) } _ => continue, }; @@ -4773,7 +4775,7 @@ impl TimelinePane { if selection.contains_clip_instance(&clip_instance.id) { let clip_duration = match layer { lightningbeam_core::layer::AnyLayer::Audio(_) => { - document.get_audio_clip(&clip_instance.clip_id).map(|c| c.duration) + document.get_audio_clip(&clip_instance.clip_id).map(|c| c.content_duration().native()) } _ => continue, };