skyler
/
Lightningbeam
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Type AudioClip duration by clip kind; fix MIDI clips growing too fast 

The MIDI-clip-end-grows-too-fast bug was a units confusion: AudioClip.duration
is documented "seconds" but MIDI clips store their length in BEATS (they share
the AudioClip struct with sampled clips). The timeline's effective_clip_duration
read it as raw seconds, so at 120 BPM a MIDI clip rendered ~2x too long. The old
backend-snapshot display had hidden this by forcing timeline_duration in beats.

Root fix — make the domain explicit and unforgeable:
- `duration` is now a private field. Reading it in the wrong unit is impossible
  because access goes through typed accessors: AudioClip::content_duration() ->
  ClipDuration (a Seconds|Beats enum tagged by clip_type) and
  set_content_duration(). serde still serializes the private field, so the .beam
  format is unchanged (bare number).
- ClipDuration::to_seconds(tempo_map) for display/sizing; ::native() for code
  that already works in the clip's native domain (trim math shares it).
- get_clip_duration + the timeline-endpoint calc go through to_seconds, so MIDI
  is converted correctly; effective_clip_duration delegates to get_clip_duration.
- Recording mirrors (audio/MIDI progress + finalize) write via set_content_duration
  (debug-asserts the value's domain matches the clip type).

Every former raw read/write of the field (core actions, timeline, piano roll,
infopanel, asset library, recording handlers) now goes through the accessors.
Whole workspace compiles; 299 core tests pass.
This commit is contained in:
Skyler Lehmkuhl 2026-07-11 17:08:09 -04:00
parent c747126759
commit 8ed2320dca
17 changed files with 186 additions and 1216 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

859
STREAMING_TO_DISK_PLAN.md
View File

@ -1,859 +0,0 @@
# Streaming Media To/From Disk — Plan
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
and to disk when the data is too large to fit comfortably in memory, rather than loading
the entire file regardless of size.
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
is impossible while the container forces a full decode, so the container decision (below)
is now part of this plan.
## Deferred bugs (do at the end)
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
## Raster-keyframe-UI bugs — **[DONE]** (built the raster keyframe timeline UI, 2026-06-20)
Both resolved by the raster-keyframe-timeline-UI work: timeline now draws a diamond per
`RasterKeyframe` (mirrors vector), `K`/New Keyframe inserts a blank cel via `AddRasterKeyframeAction`
(canvas refreshes), paint tools edit the active keyframe instead of lazily creating, diamonds are
click-to-seek (pointing-hand cursor), playback prefetches frames, and onion skinning (raster+vector,
tinted, Info-Panel settings) is in. (a) canvas-refresh-on-new-keyframe and (b) keyframes-on-timeline
are both fixed.
## Noted enhancements (later, after the phases)
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
Native multichannel support remains a separate, larger project.
- **Export speed (audited 2026-06-21):** a 1:14 1080p MP4 took ~9:06 (~7.4x realtime, ~135 ms/frame).
Audit **refuted** the per-frame-seek theory — export decodes the source *sequentially*
(`video.rs` `need_seek` is false once advancing forward), and readback is already async +
triple-buffered. Real hotspots:
- **[DONE] #1 — per-frame renderer rebuild.** The export pump built a fresh `vello::Renderer`
(full wgpu pipeline init) + empty `ImageCache` *every egui repaint* (`main.rs` ~6218). Now built
once per export and reused; also fixed lazy-image export (the throwaway cache had no container
path). **Expected the dominant win.**
- **[DONE] #2a — encode swscale rebuilt per frame.** `CpuYuvConverter::convert` now caches the
RGBA→YUV420p `scaling::Context` + frames in `new()` instead of per call.
- **[TODO] #2b — decode swscale + stride-repack** per frame in `video.rs:294-320` (shared with
scrubbing; cache the YUV→RGBA scaler on the decoder). Small win, modest risk.
- **Result of #1+#2a (measured):** ~7.4x → **~1.74x realtime** (130.7 s for 4488 frames @ 60 fps;
34 fps). Per-stage avg: Render(CPU build) 15 ms, **Readback(GPU latency) 42 ms**, Extract 1.3 ms,
Convert 5.7 ms.
- **Now GPU-bound.** Per ~87 ms poll cycle the CPU does ~66 ms (3× build 45 + convert 17 + extract 4)
but the GPU does ~87 ms (3 × ~29 ms composite) → GPU saturated at ~29 ms/frame; "Readback 42 ms" is
queue latency, not transfer (8 MB is sub-ms).
- **[SKIP] #3 GPU YUV / #5 pacing** — both only trim the CPU side, which is already *under* the GPU.
Won't move a GPU-bound throughput.
- **[TODO, big] Reduce the GPU composite (~29 ms/frame).** The per-layer HDR pipeline (Vello render →
linear → composite, ×layers) is the wall, shared with live rendering. Options: batch composite
passes; a fast-path skipping HDR compositing for simple single-layer/no-blend docs; cache unchanged
layers' scenes (CPU-side, only helps if it later becomes CPU-bound). Render-architecture project.
- Non-issues: per-frame seek, blocking readback, audio. (`video.rs:237` container-reopen-on-seek is
a latent cost but doesn't fire on forward export.)
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
decode) still worth chasing if it recurs.
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
2 h video keeps that work and continues from there on reload.
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
complete clips; a mid-generation save resumes from where it left off on reload.)*
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
absent). The 5 s interval already bounds count; no extra budget needed.
- **Progressive waveform on first import:** generation streams the whole file before the
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
the clip left-to-right instead of appearing all at once. Persistence saves only the final
complete pyramid.
## Guiding principle
Three subsystems already have the right streaming primitive; most of the work is wiring,
bounding caches, and adding a residency window. The recurring pattern:
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
> stable ID, and evict everything outside a window around the playhead.
---
## Audit summary (where we stand today)
### Correctly streaming / bounded
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` —
keyframe-index seek + decode-until-target, one frame resident).
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
frames in flight).
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
### Fully-loaded, unbounded by file length (the problems)
| Site | Issue |
|---|---|
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
---
## Container format decision: `.beam` → SQLite *(DECIDED)*
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
`.beam` extension). This is the foundation the rest of the plan builds on.
### Why
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
mutation** — every save and every raster frame write-back rewrites the whole archive — and
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
SQLite dissolves the single-file-vs-performance tension:
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
confusion; we have no bundle magic on Linux/Windows).
- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
atomicity).
- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly.
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
reads (or external paths); raster frames live in blob rows and write back transactionally.
### Large-media policy: packed OR referenced
Two storage modes per media item, both supported:
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
on a network drive, or media too large/volatile to pack).
**Default-mode preference for files over the per-blob limit (~2GB):**
- A user preference `large_media_default: Pack | Reference` controls what happens to
imports above the threshold.
- The **first time** the user imports a media file over the limit, **prompt** them
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
large imports (changeable later in settings).
- Files under the limit are packed by default (chunked only if needed).
### Schema sketch
```
media(
id BLOB PRIMARY KEY, -- stable Uuid
kind INTEGER, -- audio | video | raster | image-asset
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
storage INTEGER, -- 0 = packed, 1 = referenced
ext_path TEXT, -- set when storage = referenced
total_len INTEGER, -- bytes (packed) for chunk math
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
)
media_chunk(
media_id BLOB, chunk_index INTEGER, bytes BLOB,
PRIMARY KEY (media_id, chunk_index)
)
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
```
`project.json` stays the same serialized `BeamProject` for now — only its container and the
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
first open/save.
### Streaming reads from packed media
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
plain `File` exactly as `do_import_audio` already does for originals today.
---
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
### 1a. Turn on the compressed-audio disk reader
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
assign `AudioClip::read_ahead`.
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
from the ring buffer, not RAM.
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
already distinguishes.
### 1b. Stream on project load *(depends on the SQLite container)*
Three coupled changes (none works alone):
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
for them. After the engine installs the project, walk all audio clips and
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
`BlobReader` instead of a path for packed media.)
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
table instead of base64 `embedded_data`.
### 1c. Video's embedded audio track — stream from the video via ffmpeg
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
save/reload. **Superseded by the design below.**
**Proper design — stream the video's audio track on demand, never materialized.**
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
interface.
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
`sample_rate`/`channels`/`total_frames`.
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
entry `path` = the video file.
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
One ffmpeg context + 3 s buffer per active clip instance.
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` →
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
cleanup.
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
frames before target; for the frame straddling target, drop its leading samples). After
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
while here, or at least flag.
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
seek (both readers) → waveform-peaks pass.
---
## Phase 2 — Video: bound the caches *(small, isolated)*
### 2a. Bound `VideoManager.frame_cache`
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
### 2b. Stream the export mux
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
---
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
the container is opened only at load/save (no live handle).
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
### Drive-by (Arc pixels): DROPPED
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
clone). Not worth 64 edits. Start at 3a.
### 3a. Lazy async fault-in + image proxy
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
results applied at top of `update()`. No UI block on cold scrub.
- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written
beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`.
Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to
the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident.
*(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge
paint projects.)*
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
leave full `raw_pixels` empty.
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
upscaled proxy. Reused by the exporter (already frame-by-frame).
### 3b. Residency window + eviction **[DONE]**
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
rows via `media_exists` (no data loss) and walks all layers to match load.
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
### 3c. Bound the GPU cache **[DONE for raster_layer_cache]**
`raster_layer_cache` (`gpu_brush.rs`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
≈ `w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in
`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay
now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom
if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello
`ImageCache` is image *assets* → Phase 4.)*
### 3d. Undo memory **[DONE]**
`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames.
Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in
`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor
faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`),
correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒
skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose
bbox is the whole frame.)*
### 3e. Prefetch frames **[DONE for playback]**
Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4`
upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so
full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during
playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a
prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction
prefetch still TODO.)*
Original note: *(future, after 3d — pure latency win, no correctness need)*
Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the
proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**:
on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of
playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker +
`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap
concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU
texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res
frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up.
### Build order & tests
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
---
## Phase 3.5 — Image textures in vector scenes **[DONE 2026-06-21]** *(prereq for Phase 4; fixed DCEL-broken image import)*
**Done:** 3.5a — import/drop places an image as a borderless image-filled rectangle
(`AddShapeAction::image_rect`), centered (direct import) or at the drop point (library drag);
renderer now maps the image brush onto the fill's bounding box (was anchored at world origin →
only a corner showed); `SetImageFillAction` + an **Image** fill-type tab (None|Solid|Gradient|Image)
with an asset picker in the Info Panel. 3.5b — image bytes persist as `MediaKind::ImageAsset` rows in
the `.beam` (kept-in-place; `ImageAsset.data` is `skip_serializing` + container-backed; old base64
projects migrate on re-save); eager-read on load. *(ImageCache still unbounded — Phase 4 adds the
usage-based LRU/lazy paging.)*
### (original plan below)
## Phase 3.5 — Image textures in vector scenes *(prereq for testing Phase 4; fixes DCEL-broken image import)*
**Why:** Phase 4 pages *image assets*, but there's currently no way to get an image asset into a
vector scene — so nothing to page. This also repairs image import, half-broken since the DCEL switch.
**Current state (audited 2026-06-21):**
- *Works:* `import_image` (`main.rs`) decodes dims + creates an `ImageAsset` (raw bytes embedded in
`Document::image_assets`, serialized as **base64 in project JSON**). The renderer's image-fill paths
are **complete** — GPU/Vello (`renderer.rs:~1160`, `ImageBrush` via `ImageCache.get_or_decode`) and
CPU/tiny-skia (`renderer.rs:~1486`). `Fill::image_fill` (`vector_graph/mod.rs:110`) and
`Face::image_fill` (`dcel2/mod.rs:117`) fields exist and render when set.
- *Broken/missing (the workflow):*
1. **Drop image → canvas is stubbed:** `stage.rs:~11782` and `main.rs:~4924` both just print
"Image drag to stage not yet supported with DCEL backend". Nothing is added to the scene.
2. **No way to assign an image fill:** no `SetImageFillAction` (only `SetFillPaintAction` for
color/gradient); no Info-Panel picker. `Fill`/`Face.image_fill` are never populated.
3. **DCEL faces never get `image_fill`** (`dcel2/import.rs:275` always `None`; topology copies from
parent which is also `None`).
4. **Not in the container:** `MediaKind::ImageAsset` exists but is **dead** — image bytes live only
as base64 in project JSON. Not chunked, not pageable (so Phase 4 can't page them).
**Tasks:**
- **3.5a — Place + assign.** Replace the two drop stubs: dropping an image onto a vector layer creates
a rectangle face sized to the image at the drop point with `image_fill = asset_id`. Add
`SetImageFillAction` (set/clear an image fill on the selected face/shape; mirrors `SetFillPaintAction`)
+ an Info-Panel image-asset picker for the selected shape's fill. Populate `Face.image_fill` in DCEL
(and keep it through topology ops — already copied from parent).
- **3.5b — Persist in the container.** Write image assets as `MediaKind::ImageAsset` rows in the `.beam`
SQLite (like raster/audio: write on save kept-in-place on re-save; read on load), keyed by asset id;
drop the base64-in-JSON embedding (or keep a tiny ref). This is the storage Phase 4 pages from.
- **3.5c — Lazy decode hook.** Image bytes load from the container into `ImageCache` on first render
(decode → `ImageBrush`/`Pixmap`). Leave `ImageCache` **unbounded for now**; Phase 4 adds the
usage-based LRU/eviction (this phase just makes there *be* real, container-backed image assets to page).
- **Tests:** import→drop→render round-trip; save/reload preserves the image fill + reads bytes from the
container (not JSON); CPU and GPU render paths both show the image.
---
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
**Data model.**
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
`Pixmap` map, keyed by asset id, **unbounded**.
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
**No asset→frame or frame→asset index exists today.**
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
the heavy one).
**Progress (2026-06-21):**
- **[DONE] Tier 2 — bound the decoded `ImageCache`.** 256 MB **usage-LRU**: every
`get_or_decode`/`_cpu` bumps the asset's recency; inserts past budget evict the least-recently-used
(a miss re-decodes from `asset.data`). Achieves usage-based eviction via render-access recency
(simpler than the frame→asset enumeration below; that enumeration is only needed for *prefetch*).
- **[DONE] Tier 1 — lazy compressed bytes.** `ImageCache` holds the container path (threaded
App.current_file_path → SharedPaneState → VelloRenderContext) and pages bytes on a decode miss via
`read_packed_media_readonly`; `load_beam_sqlite` no longer eager-reads → instant load, compressed
bytes don't accumulate. `asset.data` is still used when resident (fresh import / old base64 project).
*(Refinement: persistent read connection vs open-per-miss.)*
- **[DONE] Prefetch.** `assets_needed_at(document, time)` enumerates image ids in the visible vector
layers' active keyframes; during playback the stage decodes the ~0.5s-ahead set into the cache.
*(Refinements: nested clip-instance recursion; background-thread decode.)*
**Phase 4 = DONE** (image asset paging by usage + LRU).
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
### 4b. Usage bookkeeping (the multi-frame problem)
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
paths in `vector_graph` and the relevant actions).
- count 0 → dead, fully evictable / GC candidate.
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
playhead**, not by count (a high-count asset far from the playhead is still evicted).
Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
Eviction never touches an asset in needed-now.
### 4c. Bound the decoded tier
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
image-texture cache the same way, keyed to the residency window.
### Nested-clip prefetch (important)
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
each of which can reference its own image assets. Prefetch must therefore:
- Recurse through clip instances when computing both needed-now and needed-soon.
- Map outer playhead time → each nested clip's local time, and look ahead along the
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
keyframe are loaded before the nested clip reaches it.
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
once); the usage index handles refcounting.
---
## Cross-cutting: a shared residency abstraction
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
disk backing, residency = window/needed-set around playhead + LRU byte budget:
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|---|---|---|---|---|
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
enumeration + usage index is unique to Phase 4.
---
## Sequencing
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
2. **Phase 2** — small, isolated, independently shippable; container-independent.
3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy +
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
5. **Phase 1c** — depends on 1b's pool path.
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
---
## Status
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
buffer (was effectively silent before, since `read_ahead` was always `None`).
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
(bundled) — `lightningbeam-core/src/beam_archive.rs`
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
referenced-path, overwrite-replaces-chunks.
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
changed rows written; unchanged large media never rewritten); orphan cleanup via
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
must NOT copy+rename for existing SQLite files.
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
deleted (now packs original bytes with their codec).
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
`Ask` behaves as `Reference` at save time.
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
import_audio/video/image) queues a one-time modal; choice persists to config.
Threshold shown in the modal is derived from the constant.
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
skip unchanged frames on in-place save (Phase 3).
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
> build the lib in normal mode. Worth fixing separately.
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
+ proper video-audio reload remain as noted follow-ups.
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** —
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
violation; referenced-only-first = leaves packed <2GB in RAM.)
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
never fires for loaded clips (they bypass `AddAudioClip`).
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory`
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
by the editor between `load_audio_pool` and `set_project`.
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` —
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
from the full in-RAM decode, which no longer happens). Added
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
overview now fills in shortly after load.
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
then instantly on subsequent loads once saved; RAM stays flat on a big project.
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
→ `from_compressed` placeholder). Higher risk (touches the working referenced path); packed
covers the common <2GB case first.
- [x] **DONE: packed video streaming.** Small videos pack into the `.beam`
(a `MediaKind::Video` blob at the clip id, `VideoClip.media_id` referencing it) and stream **both
frames and audio** from the DB blob via FFmpeg. The `AVIOContext`-over-`Read+Seek` shim lives in
the new `ffmpeg-blob-io` crate (`BlobInput`, version-pinned `=8.0.0`/`=8.0.1`), isolating the
unsafe + ABI coupling. Frames: `video.rs` `VideoSource{Path,Packed}` opens a fresh `BlobReader`
per decoder/seek/scan. Audio: `VideoAudioReader::open_source` over the same blob (the
`disk_reader.rs` `StreamSource` blocker is removed); save points the linked video-audio pool
entry's `media_id` at the video row so it streams from the same blob. Tests: ffmpeg-blob-io AVIO
unit tests (WAV via Cursor + seek + open/drop loop), core `packed_video_stream` (blob→AVIO→Input),
`beam_archive` packed-video round-trip, daw-backend `open_source` (compiles; can't link in the
container — user runtime-verifies actual A/V playback).
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
small temp partitions) and the temp path doesn't survive reload.
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
reference survives save/load.
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
empty channel layout → must `set_channel_layout` before resampling, else swr returns
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
until wired. `VideoAudioReader` made `pub` for the integration test.
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
daw-backend + editor compile clean; VideoAudioReader tests still pass.
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
regression) and that an activated VideoAudio clip produces sound.
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` →
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
**→ Feature is live end-to-end; ready for in-app testing.**
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
whole packet is discarded). So compressed clips no longer drift vs video audio after
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` —
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
Ragged last buckets reduce over available children (no value padding). Bounded
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
ordering, stereo channels, size bound, chunk-agnostic.
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
memory (one chunk + the pyramid). Test: envelope matches the signal through
both backends.
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
- [ ] Orchestration (with 1c).
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
- [x] Generation: on video-audio import Success, the same bg thread streams
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
main.rs construction) → `render_layers` → **both** render sites (collapsed-group
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
features hold position at every zoom and line up with playback.
See memory `waveform-shader-fractional-mip-offset`.
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
`register_loaded_videos` only loads frames (not audio), so there is no redundant
regeneration to suppress. Compiles clean across all three crates.
### Waveform LOD pyramid design (step 6)
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
in → per-sample; seamless between.
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
stored.
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
temp before first save). `B` is stored with it (preference is just the default for new gen).
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
cleared only after its children") — so rendering can always walk up to a resident ancestor;
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 —
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
above the floor it's a disk read; below it the span is already small.
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
viewports of fine tiles; persist pyramid in `.beam`.
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
in-app export to confirm A/V sync.)*
- [x] Phase 3a — lazy + async raster fault-in (`RasterStore` + background thread + image proxy)
- [x] Phase 3b — raster residency LRU + eviction (dirty-flag data-loss safety)
- [x] Phase 3c — bound raster GPU texture cache (recency LRU + F3 VRAM readout)
- [x] Phase 3d — raster undo dirty-rect diffs (+ fault-in-before-undo)
- [x] Phase 3.5 — image textures in vector scenes (fixed DCEL-broken image import; image-fill tab + picker; container-persisted)
- [x] Phase 4 — image asset paging: Tier 2 decoded-cache byte-LRU, Tier 1 lazy container bytes, playback prefetch
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.

310
TODO.md
View File

@ -1,305 +1,25 @@
# Lightningbeam TODO # Lightningbeam TODO
> ⚠️ **Stale entries:** Lightningbeam was rewritten from JavaScript to Rust. Any entry below ## Known Issues (Rust)
> that cites `src/*.js` / `main.js` / `animation.js` predates that migration — the *issue* may
> or may not still exist in the Rust codebase, but the file/line references are obsolete.
> **Re-verify against the current Rust code before acting** (this covers the "Animation System
> Refactoring" section and the JS-referencing "Known Issues" entries — node editor, default
> interpolation, etc.). Items with no `.js` references are current.
## Animation System Refactoring *(STALE — JS-era migration notes; superseded by the Rust DCEL/keyframe system)* ### Animation: Tweens are broken — LOW PRIORITY
- Shape/vector interpolation between keyframes, and the `tween_after` behavior on
keyframes, don't work correctly in the current app. Needs investigation + fix.
Not urgent — revisit later.
### Completed ## Backlog / Feature ideas
- ✅ Implement AnimationData curve-based system (Keyframe, AnimationCurve, AnimationData classes)
- ✅ Add GraphicsObject.currentTime property
- ✅ Migrate shape rendering to use AnimationData curves (exists, zOrder)
- ✅ Binary search optimization for keyframe lookups
### In Progress ### Animation curve enhancements
- Migrating from Frame-based to AnimationData curve-based system throughout codebase - [ ] Extrapolation modes, separate for start vs end: hold (default), extend, repeat, decay
- [ ] Position / scale / rotation animation curves for shapes
- [ ] Shape morphing / tweening between keyframes
### Pending Features ### Keyframing behavior
- [ ] User preference for keyframing when editing objects:
#### Animation Curve Enhancements
- [ ] Implement extrapolation modes (separate for start vs end):
- "hold" (default) - hold value at first/last keyframe
- "extend" - linearly extend the curve beyond keyframes
- "repeat" - repeat the animation
- "decay" - exponential decay to a target value
- [ ] Add position, scale, rotation animation curves for shapes
- [ ] Add shape morphing/tweening between keyframes
#### Keyframing Behavior
- [ ] Add user preference for keyframing behavior when editing objects:
- Auto-keyframe (current default): create/update keyframe at current time - Auto-keyframe (current default): create/update keyframe at current time
- Edit previous (Flash-style): update most recent keyframe before current time - Edit previous (Flash-style): update most recent keyframe before current time
- Ephemeral (Blender-style): changes don't persist without manual keyframe - Ephemeral (Blender-style): changes don't persist without manual keyframe
- Optional: Add modifier key (e.g. Shift) to toggle between modes - Optional modifier key (e.g. Shift) to toggle modes
#### Shape Ordering ### Shape ordering
- [ ] Add "Bring Forward" menu option (swap zOrder with shape in front) - [ ] Bring Forward / Send Backward / Bring to Front / Send to Back menu options
- [ ] Add "Send Backward" menu option (swap zOrder with shape behind)
- [ ] Add "Bring to Front" menu option (set zOrder to max + 1)
- [ ] Add "Send to Back" menu option (set zOrder to min - 1)
#### Code Cleanup
- [ ] Remove all remaining references to Frame-based system
- [ ] Remove legacy Frame class once migration is complete
- [ ] Clean up GraphicsObject.shapes[] array (shapes should only live in Layers)
## Known Issues / Platform Limitations
### Animation: Tweens are broken (Rust codebase) — LOW PRIORITY
- **Issue**: Animation tweening between keyframes (shape/vector interpolation, and the
`tween_after` behavior on keyframes) does not work correctly in the current Rust app.
Needs investigation + fix. Not urgent — revisit later.
- (Older JS-codebase animation entries below reference `src/*.js` and are stale.)
### Audio: Oscillator Timbre Drift (Phase Accumulation Error)
- **Issue**: Oscillators exhibit timbre changes over time due to floating-point phase accumulation errors
- **Affected Files**:
- `daw-backend/src/effects/synth.rs:117-120` (SimpleSynth)
- `daw-backend/src/audio/node_graph/nodes/oscillator.rs:167-170` (OscillatorNode)
- **Root Cause**: Current phase wrapping uses conditional subtraction (`if phase >= 1.0 { phase -= 1.0 }`), which accumulates f32 rounding errors over time, especially for long-playing notes
- **Current Code**:
```rust
self.phase += frequency / sample_rate;
if self.phase >= 1.0 {
self.phase -= 1.0;
}
```
- **Recommended Fix**: Replace with `.fract()` for numerically stable wraparound:
```rust
self.phase += frequency / sample_rate;
self.phase = self.phase.fract();
```
- **Impact**: Medium - affects audio quality for sustained notes, becomes noticeable after several seconds
- **Priority**: Medium - should be addressed before production use
### UI: Node Connections Render Behind VoiceAllocator Child Nodes
- **Issue**: Connection lines (SVG paths) inside expanded VoiceAllocator nodes render behind child nodes due to z-index stacking
- **Affected File**: `src/styles.css:1128`
- **Root Cause**: Child nodes have `z-index: 10` while connection SVG paths have default/lower z-index
- **Current Code**:
```css
.drawflow .drawflow-node.child-node {
opacity: 0.9;
border: 1px solid #5a5aaa !important;
box-shadow: 0 2px 8px rgba(90, 90, 170, 0.3);
z-index: 10;
}
```
- **Recommended Fix**: Either:
1. Remove `z-index: 10` from `.child-node` (simplest), or
2. Add higher z-index to connection SVG paths, or
3. Use CSS `isolation: isolate` on the VoiceAllocator contents area to create a new stacking context
- **Impact**: Low - visual issue only, connections still function but appear to go "behind" nodes
- **Priority**: Low - cosmetic issue that doesn't affect functionality
### UI: VoiceAllocator Child Nodes Don't Move with Parent
- **Issue**: When a VoiceAllocator node is moved, its child nodes remain in their original positions instead of moving with the parent
- **Affected File**: `src/main.js:6202-6207`
- **Root Cause**: The `nodeMoved` event handler only handles the case where a child node is moved (resizes parent), but doesn't handle when the VoiceAllocator itself is moved
- **Current Code**:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
});
```
- **Recommended Fix**: Add logic to detect when a VoiceAllocator is moved and update all child node positions:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
// Case 1: A child node was moved - resize parent
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
// Case 2: A VoiceAllocator was moved - move all children
if (node && node.data.nodeType === 'VoiceAllocator') {
// Calculate delta from previous position (need to track)
// Update all child node positions by the delta
// Call editor.updateConnectionNodes() for parent and all children
}
});
```
- **Impact**: High - child nodes become disconnected from parent visually
- **Priority**: High - breaks expected behavior of grouped nodes
### UI: VoiceAllocator Expansion Doesn't Update Connection Positions
- **Issue**: When expanding/collapsing a VoiceAllocator, connection endpoints don't update to match the new port positions
- **Affected File**: `src/main.js:6496-6555` (handleNodeDoubleClick function)
- **Root Cause**: The expand/collapse logic shows/hides child nodes and resizes the container, but never calls `editor.updateConnectionNodes()` to refresh connection positions
- **Current Code**: In `handleNodeDoubleClick()`, after expanding or collapsing:
```javascript
// Expand
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
nodeElement.style.width = '600px';
nodeElement.style.height = '400px';
// ... shows child nodes ...
// Missing: editor.updateConnectionNodes(`node-${nodeId}`)
```
- **Recommended Fix**: Call `editor.updateConnectionNodes()` after resizing:
```javascript
// After expanding
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
// ... resize and show children ...
// Update connection positions for VoiceAllocator and all children
editor.updateConnectionNodes(`node-${nodeId}`);
for (const [childId, parentId] of nodeParents.entries()) {
if (parentId === nodeId) {
editor.updateConnectionNodes(`node-${childId}`);
}
}
```
- **Impact**: Medium - connections appear in wrong positions until manually moved
- **Priority**: Medium - visual issue that affects usability
### UI: Node Editor Allows Editing Without MIDI Layer Selected
- **Issue**: The node editor pane allows adding/editing instrument nodes even when no MIDI layer is selected, and always uses hardcoded `trackId: 0`
- **Affected File**: `src/main.js:6045-6920` (nodeEditor function)
- **Root Cause**: The node editor never checks if `context.activeObject.activeLayer` exists or is a MIDI track, and all backend commands use hardcoded `trackId: 0`
- **Current Code**: All graph commands hardcode track 0:
```javascript
const commandArgs = parentNodeId
? {
trackId: 0, // HARDCODED!
voiceAllocatorId: editor.getNodeFromId(parentNodeId).data.backendId,
nodeType: nodeType,
x: x,
y: y
}
: {
trackId: 0, // HARDCODED!
nodeType: nodeType,
x: x,
y: y
};
```
- **Recommended Fix**:
1. Check if activeLayer is a MIDI track before allowing edits:
```javascript
function getSelectedMidiTrack() {
const activeLayer = context.activeObject?.activeLayer;
if (!activeLayer || activeLayer.type !== 'midi') {
return null;
}
return activeLayer;
}
```
2. Show placeholder when no MIDI track selected:
```javascript
function nodeEditor() {
const container = document.createElement("div");
const midiTrack = getSelectedMidiTrack();
if (!midiTrack) {
container.innerHTML = '<div class="placeholder">Select a MIDI layer to edit instruments</div>';
return container;
}
// ... rest of node editor code ...
}
```
3. Use actual track ID instead of hardcoded 0:
```javascript
const trackId = midiTrack.audioTrackId || 0;
const commandArgs = { trackId, nodeType, x, y };
```
4. Add listener to refresh node editor when layer selection changes
- **Impact**: High - allows editing wrong track's instrument graph, data corruption risk
- **Priority**: High - can cause confusion and data loss
### Animation: Wrong Default Interpolation for Shape and Object Keyframes
- **Issue**: Shape index and object transform keyframes default to "linear" interpolation but should default to "hold" (step function), and there's no UI to change interpolation after creation
- **Affected Files**:
- `src/models/animation.js:124` (Keyframe constructor defaults to "linear")
- `src/main.js:2161` (shapeIndex keyframes default to "linear")
- `src/main.js:2198` (object position/rotation/scale keyframes default to "linear")
- `src/main.js:5910` (Timeline menu - missing tween options)
- **Root Cause**:
1. The Keyframe constructor defaults interpolation to "linear"
2. Shape index keyframes preserve existing interpolation or default to "linear"
3. Object transform keyframes explicitly use "linear"
4. No menu options exist to change interpolation mode after keyframe creation
- **Current Code**:
- Keyframe constructor (animation.js:124):
```javascript
constructor(time, value, interpolation = "linear", uuid = undefined) {
```
- Shape index keyframes (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'linear';
const shapeIndexKeyframe = new Keyframe(currentTime, newShapeIndex, interpolationType);
```
- Object keyframes (main.js:2198):
```javascript
const newKeyframe = new Keyframe(
currentTime,
currentValue,
'linear' // Default to linear interpolation
);
```
- **Expected Behavior**:
- Shape index keyframes should default to "hold" (shapes shouldn't morph between versions)
- Object transforms should default to "hold" (objects shouldn't move/rotate/scale between keyframes unless explicitly tweened)
- Timeline menu should have options to convert between interpolation modes
- **Recommended Fix**:
1. Change shapeIndex default to "hold" (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'hold';
```
2. Change object keyframe default to "hold" (main.js:2198):
```javascript
const newKeyframe = new Keyframe(currentTime, currentValue, 'hold');
```
3. Add Timeline menu options (main.js:5910, in timelineSubmenu):
```javascript
{
text: "Add Shape Tween",
enabled: /* check if shape is selected and has keyframes */,
action: () => {
// Find shapeIndex curve for selected shape
// Change interpolation between keyframes to "linear"
}
},
{
text: "Add Motion Tween",
enabled: /* check if object is selected and has transform keyframes */,
action: () => {
// Find position/rotation/scale curves for selected object
// Change interpolation between keyframes to "linear" or "bezier"
}
}
```
- **Note**: exists and zOrder keyframes already correctly use "hold" (main.js:2139, 2150)
- **Impact**: High - causes unwanted interpolation, shapes morph unexpectedly, objects move when they shouldn't
- **Priority**: High - fundamental animation behavior is incorrect
### Tauri Pinch-Zoom on Linux
- **Issue**: Two-finger pinch gestures zoom the entire Tauri window instead of individual canvases
- **Status**: Known Tauri limitation on Linux/GTK with no cross-platform solution
- **Tracking**: https://github.com/tauri-apps/tauri/discussions/3843
- **Workaround attempts**: Tried `zoomHotkeysEnabled: false`, `touch-action: none`, viewport meta tags - none worked
- **Resolution**: Monitor Tauri releases for official fix
## Notes
### Architecture
- **GraphicsObject** contains Layers and has `currentTime` (continuous time)
- **Layer** contains `shapes[]` array and `animationData` (AnimationData instance)
- **AnimationData** contains curves dictionary, each curve identified by parameter name
- Shape curves: `shape.{uuid}.exists`, `shape.{uuid}.zOrder`
- Future: `shape.{uuid}.x`, `shape.{uuid}.y`, `shape.{uuid}.rotation`, etc.
- **Shapes render based on curves**: Layer.draw checks exists > 0, sorts by zOrder, draws in order
### Interpolation Types
- `linear` - Linear interpolation between keyframes
- `bezier` - Cubic Bezier with easing control points
- `step`/`hold` - Step function (jumps to next value)

10
daw-backend/Cargo.lock generated
View File

@ -586,6 +586,7 @@ dependencies = [
"dasp_rms", "dasp_rms",
"dasp_sample", "dasp_sample",
"dasp_signal", "dasp_signal",
"ffmpeg-blob-io",
"ffmpeg-next", "ffmpeg-next",
"hound", "hound",
"memmap2", "memmap2",
@ -661,6 +662,15 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365" checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
[[package]]
name = "ffmpeg-blob-io"
version = "0.1.0"
dependencies = [
"ffmpeg-next",
"ffmpeg-sys-next",
"libc",
]
[[package]] [[package]]
name = "ffmpeg-next" name = "ffmpeg-next"
version = "8.0.0" version = "8.0.0"

2
lightningbeam-ui/Cargo.lock generated
View File

@ -3628,7 +3628,7 @@ dependencies = [
[[package]] [[package]]
name = "lightningbeam-editor" name = "lightningbeam-editor"
version = "1.0.7-alpha" version = "1.0.8-alpha"
dependencies = [ dependencies = [
"beamdsp", "beamdsp",
"bytemuck", "bytemuck",

68
lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md
View File

@ -67,15 +67,47 @@ at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *an
- Result: software exports are full-quality at any export res, and document resizes re-target decode. - Result: software exports are full-quality at any export res, and document resizes re-target decode.
No hardware needed; this is the correctness fix for the codecs HW can't handle anyway. No hardware needed; this is the correctness fix for the codecs HW can't handle anyway.
### Stage 2 — hardware decode primitive (headless-testable here, like the 8 encode tests) ### Stage 2 — hardware decode primitive (DONE, commit 255e164)
- In `gpu-video-encoder` (rename → `gpu-video-codec`): `h264_vaapi`-style **decode** → VAAPI surface → `decoder::VaapiDecoder` in `gpu-video-encoder`: decode → VAAPI surface → DRM-PRIME DMA-BUF →
export DMA-BUF → import as a wgpu texture. Hardware test: decode a known file, verify dims/contents. `dmabuf::import_raw` → wgpu textures. Round-trip test (encode gray → decode → readback Y≈128) passes.
### Stage 3 — wire hardware decode into `get_frame` (blind; user-verifies) ### The device-affinity problem (drives the whole rest of the design)
- When the source codec/driver is HW-decodable, `get_frame` returns a **GPU texture** (native res) wgpu textures can't cross devices, and a decoded frame is a wgpu texture imported from a DMA-BUF —
instead of `Arc<Vec<u8>>`; the compositor uses it directly (no `write_texture`), GPU-scaling to the which **requires a device with the DMA-BUF-import extensions** (`VK_EXT_image_drm_format_modifier`
target. For the zero-copy export the frame never leaves the GPU: **decode → composite → encode** on + external-memory), built via wgpu-hal `device_from_raw` (the safe `DeviceDescriptor` can't add
one device. Software path is the fallback for everything else. them). So a hardware-decoded frame is only usable by a compositor running on **such** a device.
- **Export** composites on the encoder's custom device → already fine.
- **Preview** composites on eframe's *normal* device → can't import DMA-BUFs → can't use HW frames.
Since **preview must HW-decode 4K** (software 4K decode ≈19 ms/frame), the resolution is a **single
shared custom device** used by eframe + preview compositor + decoder + encoder. eframe 0.33 (local
`egui-fork`) accepts it via `WgpuSetup::Existing { instance, adapter, device, queue }` — confirmed.
The earlier "separate export device" becomes redundant once this lands.
### Stage 3a — windowed shared `DrmDevice`, injected into eframe (highest-risk; blind)
Today `vk_device::create()` is **headless**. Make a windowed variant (or extend it) that is a
**superset** device: DMA-BUF import ext **+** `VK_KHR_swapchain` (device) and the WSI surface
instance extensions, **+** everything eframe/egui/vello need — `adapter.limits()` (already; Vello
needs `max_storage_buffers_per_shader_stage` ≥ 5), `max_texture_dimension_2d` 8192, and the optional
features main.rs requests (`SHADER_F16`, `TIMESTAMP_QUERY[_INSIDE_ENCODERS]`). Pick the adapter that
is the **VAAPI GPU** (the render node must match libva's, or DMA-BUF sharing fails on multi-GPU).
- main.rs: try to build the shared device; on success pass `WgpuSetup::Existing`, else fall back to
the current `WgpuSetupCreateNew` (software decode only). Gate on Linux + VAAPI + a config/env
override; **must be bulletproof** — this device now renders *every* frame of *every* session for
Linux/VAAPI users, video or not. Milestone: editor runs normally on it with no video involved.
### Stage 3b — VideoManager hardware decode on the shared device (blind)
- `VideoManager` holds a `VaapiDecoder` per HW-decodable clip (built on the shared device), plus the
software `VideoDecoder` fallback. `get_frame` gains a GPU-returning variant: yields an imported NV12
texture pair (native res) instead of `Arc<Vec<u8>>`. Probe HW support per source; non-VAAPI /
unsupported codecs / non-Linux → software path (Stage 1, target-res).
- Cache native GPU textures keyed by (clip, ts); revisit the byte budget (4K NV12 ≈ 12 MB each).
### Stage 3c — compositor consumes the GPU frame (blind; user-verifies)
- The video-instance composite path takes an NV12 texture (or a small NV12→RGB GPU pass) and blits it
to the target with the existing bilinear blit — **no `write_texture` upload**. GPU scales native→
target (preview res or export res). Both preview and the zero-copy export become
decode→composite(→encode) with no CPU frame. Software frames still upload as today.
## Critical files ## Critical files
- `lightningbeam-core/src/video.rs` — `VideoDecoder` (per-request output size, scaler cache), - `lightningbeam-core/src/video.rs` — `VideoDecoder` (per-request output size, scaler cache),
@ -86,13 +118,21 @@ at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *an
- `gpu-video-encoder/` (→ `gpu-video-codec`) — `dmabuf.rs`/`vk_device.rs` reused for the decode import. - `gpu-video-encoder/` (→ `gpu-video-codec`) — `dmabuf.rs`/`vk_device.rs` reused for the decode import.
## Risks ## Risks
- **Shared custom device is the editor's main device (BIGGEST risk)** — Stage 3a makes a hand-built
wgpu-hal Vulkan device render every frame for Linux/VAAPI users. It must satisfy eframe + egui +
vello + winit presentation across varied Intel/AMD/Mesa stacks, or the editor won't start. Mitigate
with a strict try-and-fall-back-to-normal-device path + an env/config kill switch. Test broadly.
- **Multi-GPU** — the shared render device must be the *same* GPU as libva's VAAPI device, or DMA-BUF
import fails. Adapter selection must match the render node to the VAAPI node (laptops with iGPU +
dGPU, PRIME).
- **Codec coverage** — only some codecs are HW-decodable per GPU/driver; software must stay correct - **Codec coverage** — only some codecs are HW-decodable per GPU/driver; software must stay correct
and well-tested. Selection must probe support per source, not assume. and well-tested. Probe support per source, don't assume.
- **Cache memory** — native-res GPU textures (esp. 4K) are large; the frame cache budget needs revisiting. - **Cache memory** — native-res GPU textures (esp. 4K NV12 ≈12 MB) are large; revisit the frame cache
- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; the existing import handles NV12, but budget, and the two live targets (preview res + export res) shouldn't thrash.
10-bit/HDR sources (P010) need format handling. - **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; import handles NV12, but 10-bit/HDR
- **Preview vs export sharing** — two live targets (preview res + export res) from the same source; the (P010) needs format handling. Decoded NV12 also needs the right BT.601/709 + range on the NV12→RGB
cache/scaler design must serve both without thrashing. read (mirror the encoder's color tags, [[gpu-video-decode]] color-range work).
- **Non-Linux / no-VAAPI** — must cleanly run on the normal eframe device with software decode.
## Verification ## Verification
- Stage 0/1: visual — export above document res is now full-quality (not upscaled); profile shows - Stage 0/1: visual — export above document res is now full-quality (not upscaled); profile shows

4
lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs
View File

@ -198,7 +198,7 @@ impl Action for AddClipInstanceAction {
// Calculate internal start/end from trim parameters // Calculate internal start/end from trim parameters
let internal_start = self.clip_instance.trim_start; let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration); let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native());
let external_start = self.clip_instance.timeline_start; let external_start = self.clip_instance.timeline_start;
// Calculate external duration (for looping if timeline_duration is set). // Calculate external duration (for looping if timeline_duration is set).
@ -240,7 +240,7 @@ impl Action for AddClipInstanceAction {
// `trim_*` / `clip.duration` are in SECONDS (audio content time), // `trim_*` / `clip.duration` are in SECONDS (audio content time),
// while `timeline_*` and the backend's `duration` are in BEATS. // while `timeline_*` and the backend's `duration` are in BEATS.
let internal_start = self.clip_instance.trim_start; let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration); let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native());
let start_time = self.clip_instance.timeline_start; let start_time = self.clip_instance.timeline_start;
// `effective_duration` is in BEATS. When `timeline_duration` is set // `effective_duration` is in BEATS. When `timeline_duration` is set
// it already is; otherwise the clip occupies its natural content // it already is; otherwise the clip occupies its natural content

2
lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs
View File

@ -129,7 +129,7 @@ impl LoopClipInstancesAction {
}; };
let content_window = { let content_window = {
let trim_end = instance.trim_end.unwrap_or(clip.duration); let trim_end = instance.trim_end.unwrap_or(clip.content_duration().native());
(trim_end - instance.trim_start).max(0.0) // seconds (trim_end - instance.trim_start).max(0.0) // seconds
}; };
// Natural content length as a beats span at the clip's start (the // Natural content length as a beats span at the clip's start (the

4
lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs
View File

@ -170,7 +170,7 @@ impl Action for RemoveClipInstancesAction {
use daw_backend::command::{Query, QueryResponse}; use daw_backend::command::{Query, QueryResponse};
let internal_start = instance.trim_start; let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.duration); let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
let external_start = instance.timeline_start; let external_start = instance.timeline_start;
// MIDI trims are beats-domain, so the fallback span is beats too. // MIDI trims are beats-domain, so the fallback span is beats too.
let external_duration = instance let external_duration = instance
@ -198,7 +198,7 @@ impl Action for RemoveClipInstancesAction {
} }
AudioClipType::Sampled { audio_pool_index } => { AudioClipType::Sampled { audio_pool_index } => {
let internal_start = instance.trim_start; let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.duration); let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
let start_time = instance.timeline_start; let start_time = instance.timeline_start;
// Fallback span is the content seconds converted to beats at the // Fallback span is the content seconds converted to beats at the
// clip's start (not the seconds span treated as beats). // clip's start (not the seconds span treated as beats).

10
lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs
View File

@ -377,7 +377,7 @@ impl Action for SplitClipInstanceAction {
// 1. Trim the original (left) instance // 1. Trim the original (left) instance
let orig_internal_start = original_instance.trim_start; let orig_internal_start = original_instance.trim_start;
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration); let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native());
// Look up the original backend instance ID // Look up the original backend instance ID
if let Some(crate::action::BackendClipInstanceId::Midi(orig_backend_id)) = if let Some(crate::action::BackendClipInstanceId::Midi(orig_backend_id)) =
@ -388,7 +388,7 @@ impl Action for SplitClipInstanceAction {
// 2. Add the new (right) instance // 2. Add the new (right) instance
let internal_start = new_instance.trim_start; let internal_start = new_instance.trim_start;
let internal_end = new_instance.trim_end.unwrap_or(clip.duration); let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native());
let external_start = new_instance.timeline_start; let external_start = new_instance.timeline_start;
// MIDI trims are beats-domain, so the fallback span is beats too. // MIDI trims are beats-domain, so the fallback span is beats too.
let external_duration = new_instance let external_duration = new_instance
@ -425,7 +425,7 @@ impl Action for SplitClipInstanceAction {
AudioClipType::Sampled { audio_pool_index } => { AudioClipType::Sampled { audio_pool_index } => {
// 1. Trim the original (left) instance // 1. Trim the original (left) instance
let orig_internal_start = original_instance.trim_start; let orig_internal_start = original_instance.trim_start;
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration); let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native());
// Look up the original backend instance ID // Look up the original backend instance ID
if let Some(crate::action::BackendClipInstanceId::Audio(orig_backend_id)) = if let Some(crate::action::BackendClipInstanceId::Audio(orig_backend_id)) =
@ -436,7 +436,7 @@ impl Action for SplitClipInstanceAction {
// 2. Add the new (right) instance // 2. Add the new (right) instance
let internal_start = new_instance.trim_start; let internal_start = new_instance.trim_start;
let internal_end = new_instance.trim_end.unwrap_or(clip.duration); let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native());
let start_time = new_instance.timeline_start; let start_time = new_instance.timeline_start;
// Fallback span is the content seconds converted to beats at the // Fallback span is the content seconds converted to beats at the
// clip's start (not the seconds span treated as beats). // clip's start (not the seconds span treated as beats).
@ -499,7 +499,7 @@ impl Action for SplitClipInstanceAction {
if let Some(instance) = al.clip_instances.iter().find(|ci| ci.id == self.instance_id) { if let Some(instance) = al.clip_instances.iter().find(|ci| ci.id == self.instance_id) {
if let Some(clip) = document.get_audio_clip(&instance.clip_id) { if let Some(clip) = document.get_audio_clip(&instance.clip_id) {
let orig_internal_start = instance.trim_start; let orig_internal_start = instance.trim_start;
let orig_internal_end = self.original_trim_end.unwrap_or(clip.duration); let orig_internal_end = self.original_trim_end.unwrap_or(clip.content_duration().native());
// Restore based on clip type // Restore based on clip type
use crate::clip::AudioClipType; use crate::clip::AudioClipType;

6
lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs
View File

@ -424,7 +424,7 @@ impl Action for TrimClipInstancesAction {
// Calculate new internal_start and internal_end for backend // Calculate new internal_start and internal_end for backend
// Note: instance already has the new trim values after execute() // Note: instance already has the new trim values after execute()
let internal_start = instance.trim_start; let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.duration); let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
// Handle trim based on clip type // Handle trim based on clip type
match &clip.clip_type { match &clip.clip_type {
@ -517,8 +517,8 @@ impl Action for TrimClipInstancesAction {
TrimType::TrimRight => instance.trim_start, // trim_start wasn't changed TrimType::TrimRight => instance.trim_start, // trim_start wasn't changed
}; };
let internal_end = match trim_type { let internal_end = match trim_type {
TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.duration), // trim_end wasn't changed TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.content_duration().native()), // trim_end wasn't changed
TrimType::TrimRight => old.trim_value.unwrap_or(clip.duration), TrimType::TrimRight => old.trim_value.unwrap_or(clip.content_duration().native()),
}; };
// Handle trim based on clip type // Handle trim based on clip type

66
lightningbeam-ui/lightningbeam-core/src/clip.rs
View File

@ -468,6 +468,37 @@ pub enum AudioClipType {
Recording, Recording,
} }
/// A clip's content duration, tagged by its native unit.
///
/// Sampled/recording audio and video measure content in wall-clock **seconds**; MIDI measures
/// it in **beats** (tempo-independent musical length). Carrying the domain in the type means a
/// duration can't be silently read in the wrong unit.
#[derive(Clone, Copy, Debug, PartialEq)]
pub enum ClipDuration {
Seconds(Seconds),
Beats(Beats),
}
impl ClipDuration {
/// Wall-clock seconds. Beats are converted as a length from beat 0 (exact under constant
/// tempo; a reasonable approximation otherwise — durations are position-independent here).
pub fn to_seconds(self, tempo_map: &daw_backend::TempoMap) -> Seconds {
match self {
ClipDuration::Seconds(s) => s,
ClipDuration::Beats(b) => tempo_map.beats_to_seconds(b),
}
}
/// The raw magnitude in the clip's native unit. Use only in code that already works in that
/// domain (e.g. trim math, whose values share the clip's native domain).
pub fn native(self) -> f64 {
match self {
ClipDuration::Seconds(s) => s.seconds_to_f64(),
ClipDuration::Beats(b) => b.beats_to_f64(),
}
}
}
/// Audio clip /// Audio clip
/// ///
/// This is compatible with daw-backend's audio system: /// This is compatible with daw-backend's audio system:
@ -481,9 +512,13 @@ pub struct AudioClip {
/// Clip name /// Clip name
pub name: String, pub name: String,
/// Duration in seconds /// Raw content duration in the clip's **native domain** — SECONDS for sampled/recording
/// For sampled audio, this can be set to trim the audio shorter than the source file /// audio, BEATS for MIDI (musical length, tempo-independent). Private on purpose: the domain
pub duration: f64, /// depends on `clip_type`, so all access goes through [`AudioClip::content_duration`] /
/// [`AudioClip::set_content_duration`], which keep it type-safe. Stored as a bare `f64`
/// because the `.beam` format serializes it as a plain number (serde derives over private
/// fields fine); a domain-tagged newtype would change the on-disk shape.
duration: f64,
/// Audio clip type (sampled or MIDI) /// Audio clip type (sampled or MIDI)
pub clip_type: AudioClipType, pub clip_type: AudioClipType,
@ -494,6 +529,31 @@ pub struct AudioClip {
} }
impl AudioClip { impl AudioClip {
/// The clip's content duration, tagged with its native domain (seconds for sampled/recording,
/// beats for MIDI). This is the only sanctioned way to read the raw `duration` field.
pub fn content_duration(&self) -> ClipDuration {
match self.clip_type {
AudioClipType::Midi { .. } => ClipDuration::Beats(Beats(self.duration)),
AudioClipType::Sampled { .. } | AudioClipType::Recording => {
ClipDuration::Seconds(Seconds(self.duration))
}
}
}
/// Set the content duration. Debug-asserts the value's domain matches the clip type so a
/// beats duration can't be stored on a seconds clip (or vice-versa).
pub fn set_content_duration(&mut self, duration: ClipDuration) {
debug_assert!(
matches!(
(&self.clip_type, duration),
(AudioClipType::Midi { .. }, ClipDuration::Beats(_))
| (AudioClipType::Sampled { .. } | AudioClipType::Recording, ClipDuration::Seconds(_))
),
"clip duration domain must match clip type",
);
self.duration = duration.native();
}
/// Create a new sampled audio clip /// Create a new sampled audio clip
/// ///
/// # Arguments /// # Arguments

24
lightningbeam-ui/lightningbeam-core/src/document.rs
View File

@ -472,8 +472,10 @@ impl Document {
} }
crate::layer::AnyLayer::Audio(audio_layer) => { crate::layer::AnyLayer::Audio(audio_layer) => {
for instance in &audio_layer.clip_instances { for instance in &audio_layer.clip_instances {
if let Some(clip) = self.audio_clips.get(&instance.clip_id) { // get_clip_duration yields seconds (converting MIDI's beats duration),
let end_time = calculate_instance_end(instance, clip.duration); // which is what the closure expects.
if let Some(clip_duration) = self.get_clip_duration(&instance.clip_id) {
let end_time = calculate_instance_end(instance, clip_duration.seconds_to_f64());
max_end_time = max_end_time.max(end_time); max_end_time = max_end_time.max(end_time);
} }
} }
@ -516,8 +518,8 @@ impl Document {
} }
crate::layer::AnyLayer::Audio(al) => { crate::layer::AnyLayer::Audio(al) => {
for inst in &al.clip_instances { for inst in &al.clip_instances {
if let Some(clip) = doc.audio_clips.get(&inst.clip_id) { if let Some(clip_duration) = doc.get_clip_duration(&inst.clip_id) {
*max_end = max_end.max(calc_end(inst, clip.duration)); *max_end = max_end.max(calc_end(inst, clip_duration.seconds_to_f64()));
} }
} }
} }
@ -908,7 +910,7 @@ impl Document {
// Avoid deep recursion — use stored duration for nested vector clips // Avoid deep recursion — use stored duration for nested vector clips
Some(vc.content_duration(self.framerate, tempo_map)) Some(vc.content_duration(self.framerate, tempo_map))
} else if let Some(ac) = self.audio_clips.get(id) { } else if let Some(ac) = self.audio_clips.get(id) {
Some(ac.duration) Some(ac.content_duration().to_seconds(tempo_map).seconds_to_f64())
} else if let Some(vc) = self.video_clips.get(id) { } else if let Some(vc) = self.video_clips.get(id) {
Some(vc.duration) Some(vc.duration)
} else if self.effect_definitions.contains_key(id) { } else if self.effect_definitions.contains_key(id) {
@ -921,15 +923,9 @@ impl Document {
} else if let Some(clip) = self.video_clips.get(clip_id) { } else if let Some(clip) = self.video_clips.get(clip_id) {
Some(Seconds(clip.duration)) Some(Seconds(clip.duration))
} else if let Some(clip) = self.audio_clips.get(clip_id) { } else if let Some(clip) = self.audio_clips.get(clip_id) {
// MIDI clips store `duration` in BEATS (they share the AudioClip struct with // Interpret the clip's native-domain duration as wall-clock seconds (MIDI stores
// sampled clips, whose duration is seconds). Convert to wall-clock seconds so // beats, sampled stores seconds — content_duration keeps that straight).
// the content-window sizing works uniformly. Some(clip.content_duration().to_seconds(self.tempo_map()))
match clip.clip_type {
crate::clip::AudioClipType::Midi { .. } => {
Some(self.tempo_map().beats_to_seconds(Beats(clip.duration)))
}
_ => Some(Seconds(clip.duration)),
}
} else if self.effect_definitions.contains_key(clip_id) { } else if self.effect_definitions.contains_key(clip_id) {
// Effects have infinite internal duration - their timeline length // Effects have infinite internal duration - their timeline length
// is controlled by ClipInstance.trim_end // is controlled by ClipInstance.trim_end

9
lightningbeam-ui/lightningbeam-editor/src/main.rs
View File

@ -2,6 +2,7 @@
use eframe::egui; use eframe::egui;
use daw_backend::{Beats, Seconds}; use daw_backend::{Beats, Seconds};
use lightningbeam_core::clip::ClipDuration;
use lightningbeam_core::layer::{AnyLayer, AudioLayer}; use lightningbeam_core::layer::{AnyLayer, AudioLayer};
use lightningbeam_core::layout::{LayoutDefinition, LayoutNode}; use lightningbeam_core::layout::{LayoutDefinition, LayoutNode};
use lightningbeam_core::pane::PaneType; use lightningbeam_core::pane::PaneType;
@ -5879,7 +5880,7 @@ impl EditorApp {
// Get audio clip duration for logging // Get audio clip duration for logging
let duration = self.action_executor.document().audio_clips let duration = self.action_executor.document().audio_clips
.get(&audio_clip_id) .get(&audio_clip_id)
.map(|c| c.duration) .map(|c| c.content_duration().native())
.unwrap_or(0.0); .unwrap_or(0.0);
println!("✅ Extracted audio from '{}' ({:.1}s, {}ch, {}Hz) - AudioClip ID: {}", println!("✅ Extracted audio from '{}' ({:.1}s, {}ch, {}Hz) - AudioClip ID: {}",
@ -6477,7 +6478,7 @@ impl eframe::App for EditorApp {
if let Some(doc_clip_id) = doc_clip_id { if let Some(doc_clip_id) = doc_clip_id {
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
if clip.is_recording() { if clip.is_recording() {
clip.duration = duration.seconds_to_f64(); clip.set_content_duration(ClipDuration::Seconds(duration));
} }
} }
} }
@ -6650,7 +6651,7 @@ impl eframe::App for EditorApp {
} }
// Update the clip's duration so the timeline bar grows // Update the clip's duration so the timeline bar grows
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
clip.duration = duration.beats_to_f64(); clip.set_content_duration(ClipDuration::Beats(duration));
} }
} }
} }
@ -6686,7 +6687,7 @@ impl eframe::App for EditorApp {
.map(|(id, _)| id); .map(|(id, _)| id);
if let Some(doc_clip_id) = doc_clip_id { if let Some(doc_clip_id) = doc_clip_id {
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) { if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
clip.duration = midi_clip_data.duration; clip.set_content_duration(ClipDuration::Beats(Beats(midi_clip_data.duration)));
clip.name = format!("MIDI Recording {}", clip_id); clip.name = format!("MIDI Recording {}", clip_id);
} }
} }

12
lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs
View File

@ -932,7 +932,7 @@ impl AssetLibraryPane {
name: clip.name.clone(), name: clip.name.clone(),
category: AssetCategory::Audio, category: AssetCategory::Audio,
drag_clip_type, drag_clip_type,
duration: clip.duration, duration: clip.content_duration().native(),
dimensions: None, dimensions: None,
extra_info, extra_info,
is_builtin: false, is_builtin: false,
@ -1136,7 +1136,7 @@ impl AssetLibraryPane {
name: clip.name.clone(), name: clip.name.clone(),
category: AssetCategory::Audio, category: AssetCategory::Audio,
drag_clip_type, drag_clip_type,
duration: clip.duration, duration: clip.content_duration().native(),
dimensions: None, dimensions: None,
extra_info, extra_info,
is_builtin: false, is_builtin: false,
@ -1802,7 +1802,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => { AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100); let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
} else { } else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
} }
@ -2358,7 +2358,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => { AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100); let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
} else { } else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
} }
@ -2495,7 +2495,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => { AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100); let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
} else { } else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
} }
@ -2858,7 +2858,7 @@ impl AssetLibraryPane {
let note_color = egui::Color32::from_rgb(100, 200, 100); let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) { if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color)) Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
} else { } else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200)) Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
} }

2
lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs
View File

@ -1808,7 +1808,7 @@ impl InfopanelPane {
}); });
ui.horizontal(|ui| { ui.horizontal(|ui| {
ui.label("Duration:"); ui.label("Duration:");
ui.label(format!("{:.2}s", clip.duration)); ui.label(format!("{:.2}s", clip.content_duration().to_seconds(document.tempo_map()).seconds_to_f64()));
}); });
} else { } else {
// Could be an image asset or effect — show ID // Could be an image asset or effect — show ID

4
lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs
View File

@ -466,7 +466,7 @@ impl PianoRollPane {
for instance in &audio_layer.clip_instances { for instance in &audio_layer.clip_instances {
if let Some(clip) = document.audio_clips.get(&instance.clip_id) { if let Some(clip) = document.audio_clips.get(&instance.clip_id) {
if let AudioClipType::Midi { midi_clip_id } = clip.clip_type { if let AudioClipType::Midi { midi_clip_id } = clip.clip_type {
let duration = instance.effective_duration(Seconds(clip.duration), document.tempo_map()); let duration = instance.effective_duration(clip.content_duration().to_seconds(document.tempo_map()), document.tempo_map());
clip_data.push((midi_clip_id, instance.timeline_start.beats_to_f64(), instance.trim_start, duration.beats_to_f64(), instance.id)); clip_data.push((midi_clip_id, instance.timeline_start.beats_to_f64(), instance.trim_start, duration.beats_to_f64(), instance.id));
} }
} }
@ -2459,7 +2459,7 @@ impl PianoRollPane {
// length converted to beats at the clip's start. // length converted to beats at the clip's start.
let duration = instance.timeline_duration.unwrap_or_else(|| { let duration = instance.timeline_duration.unwrap_or_else(|| {
let tmap = document.tempo_map(); let tmap = document.tempo_map();
tmap.seconds_to_beats(tmap.beats_to_seconds(instance.timeline_start) + Seconds(clip.duration)) - instance.timeline_start tmap.seconds_to_beats(tmap.beats_to_seconds(instance.timeline_start) + clip.content_duration().to_seconds(tmap)) - instance.timeline_start
}); });
// Get sample rate from raw_audio_cache // Get sample rate from raw_audio_cache
if let Some((_samples, sr, _ch)) = shared.raw_audio_cache.get(&audio_pool_index) { if let Some((_samples, sr, _ch)) = shared.raw_audio_cache.get(&audio_pool_index) {

10
lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs
View File

@ -209,7 +209,9 @@ fn effective_clip_duration(
document.get_clip_duration(&clip_instance.clip_id) document.get_clip_duration(&clip_instance.clip_id)
} }
} }
AnyLayer::Audio(_) => document.get_audio_clip(&clip_instance.clip_id).map(|c| Seconds(c.duration)), // Delegate to get_clip_duration so MIDI clips (whose `duration` is stored in beats,
// not seconds) are converted correctly rather than read as raw seconds.
AnyLayer::Audio(_) => document.get_clip_duration(&clip_instance.clip_id),
AnyLayer::Video(_) => document.get_video_clip(&clip_instance.clip_id).map(|c| Seconds(c.duration)), AnyLayer::Video(_) => document.get_video_clip(&clip_instance.clip_id).map(|c| Seconds(c.duration)),
AnyLayer::Effect(_) => Some(Seconds(lightningbeam_core::effect::EFFECT_DURATION)), AnyLayer::Effect(_) => Some(Seconds(lightningbeam_core::effect::EFFECT_DURATION)),
AnyLayer::Group(_) => None, AnyLayer::Group(_) => None,
@ -3082,7 +3084,7 @@ impl TimelinePane {
}; };
let audio_file_duration = total_frames as f64 / eff_sr as f64; let audio_file_duration = total_frames as f64 / eff_sr as f64;
let clip_dur = Seconds(audio_clip.duration); let clip_dur = audio_clip.content_duration().to_seconds(document.tempo_map());
let mut ci_start = ci.effective_start(); let mut ci_start = ci.effective_start();
if is_move_drag && selection.contains_clip_instance(&ci.id) { if is_move_drag && selection.contains_clip_instance(&ci.id) {
ci_start = self.moved_start(ci_start, document.tempo_map(), &document.time_signature, document.framerate); ci_start = self.moved_start(ci_start, document.tempo_map(), &document.time_signature, document.framerate);
@ -4699,7 +4701,7 @@ impl TimelinePane {
if selection.contains_clip_instance(&clip_instance.id) { if selection.contains_clip_instance(&clip_instance.id) {
let clip_duration = match layer { let clip_duration = match layer {
lightningbeam_core::layer::AnyLayer::Audio(_) => { lightningbeam_core::layer::AnyLayer::Audio(_) => {
document.get_audio_clip(&clip_instance.clip_id).map(|c| c.duration) document.get_audio_clip(&clip_instance.clip_id).map(|c| c.content_duration().native())
} }
_ => continue, _ => continue,
}; };
@ -4773,7 +4775,7 @@ impl TimelinePane {
if selection.contains_clip_instance(&clip_instance.id) { if selection.contains_clip_instance(&clip_instance.id) {
let clip_duration = match layer { let clip_duration = match layer {
lightningbeam_core::layer::AnyLayer::Audio(_) => { lightningbeam_core::layer::AnyLayer::Audio(_) => {
document.get_audio_clip(&clip_instance.clip_id).map(|c| c.duration) document.get_audio_clip(&clip_instance.clip_id).map(|c| c.content_duration().native())
} }
_ => continue, _ => continue,
}; };