skyler
/
Lightningbeam
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: skyler:a94f6004e5a96c405324e3e6b4a4b9946201b931
Branches Tags
skyler:main
skyler:release
skyler:feat/mobile-ui
..
compare: skyler:0eff5f7f33b7c889a614c34eef63bd2980b77934
Branches Tags
skyler:release
skyler:main
skyler:feat/mobile-ui

No commits in common. "a94f6004e5a96c405324e3e6b4a4b9946201b931" and "0eff5f7f33b7c889a614c34eef63bd2980b77934" have entirely different histories.
a94f6004e5 ... 0eff5f7f33

35 changed files with 1930 additions and 1071 deletions
Show Stats Expand all files Collapse all files

8
Changelog.md
View File

@ -1,11 +1,3 @@
# 1.0.9-alpha:
Bugfixes:
- Fix audio recording placement: a second recording landed at the wrong spot (and clicking/dragging clips was similarly off) at any tempo other than 60 BPM — recordings now start exactly at the playhead at any tempo
- While recording a second audio clip, the live bar showed as a zero-length clip until you stopped; it now grows as you record
- MIDI clips were drawn with the wrong length (their end grew too fast) at tempos other than 120 BPM
- Recording a MIDI clip didn't mark the project as having unsaved changes, so closing or starting a new file didn't prompt to save
- Audio and MIDI recording can now be undone (and redone)
# 1.0.8-alpha:
Changes:
- Mobile/touch UI (experimental, testing only — not built or packaged for mobile yet; enabled on desktop with the LB_MOBILE_UI environment variable): work-in-progress phone-friendly interface with a vertical sliding-window pane stack you drag to reveal panes, a new-file intent picker, a selection inspector sheet, a keyboard-primary music surface, a Focus/Patch node editor, long-press context menus, a command palette, and landscape/orientation support

859
STREAMING_TO_DISK_PLAN.md Normal file
View File

@ -0,0 +1,859 @@
# Streaming Media To/From Disk — Plan
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
and to disk when the data is too large to fit comfortably in memory, rather than loading
the entire file regardless of size.
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
is impossible while the container forces a full decode, so the container decision (below)
is now part of this plan.
## Deferred bugs (do at the end)
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
## Raster-keyframe-UI bugs — **[DONE]** (built the raster keyframe timeline UI, 2026-06-20)
Both resolved by the raster-keyframe-timeline-UI work: timeline now draws a diamond per
`RasterKeyframe` (mirrors vector), `K`/New Keyframe inserts a blank cel via `AddRasterKeyframeAction`
(canvas refreshes), paint tools edit the active keyframe instead of lazily creating, diamonds are
click-to-seek (pointing-hand cursor), playback prefetches frames, and onion skinning (raster+vector,
tinted, Info-Panel settings) is in. (a) canvas-refresh-on-new-keyframe and (b) keyframes-on-timeline
are both fixed.
## Noted enhancements (later, after the phases)
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
Native multichannel support remains a separate, larger project.
- **Export speed (audited 2026-06-21):** a 1:14 1080p MP4 took ~9:06 (~7.4x realtime, ~135 ms/frame).
Audit **refuted** the per-frame-seek theory — export decodes the source *sequentially*
(`video.rs` `need_seek` is false once advancing forward), and readback is already async +
triple-buffered. Real hotspots:
- **[DONE] #1 — per-frame renderer rebuild.** The export pump built a fresh `vello::Renderer`
(full wgpu pipeline init) + empty `ImageCache` *every egui repaint* (`main.rs` ~6218). Now built
once per export and reused; also fixed lazy-image export (the throwaway cache had no container
path). **Expected the dominant win.**
- **[DONE] #2a — encode swscale rebuilt per frame.** `CpuYuvConverter::convert` now caches the
RGBA→YUV420p `scaling::Context` + frames in `new()` instead of per call.
- **[TODO] #2b — decode swscale + stride-repack** per frame in `video.rs:294-320` (shared with
scrubbing; cache the YUV→RGBA scaler on the decoder). Small win, modest risk.
- **Result of #1+#2a (measured):** ~7.4x → **~1.74x realtime** (130.7 s for 4488 frames @ 60 fps;
34 fps). Per-stage avg: Render(CPU build) 15 ms, **Readback(GPU latency) 42 ms**, Extract 1.3 ms,
Convert 5.7 ms.
- **Now GPU-bound.** Per ~87 ms poll cycle the CPU does ~66 ms (3× build 45 + convert 17 + extract 4)
but the GPU does ~87 ms (3 × ~29 ms composite) → GPU saturated at ~29 ms/frame; "Readback 42 ms" is
queue latency, not transfer (8 MB is sub-ms).
- **[SKIP] #3 GPU YUV / #5 pacing** — both only trim the CPU side, which is already *under* the GPU.
Won't move a GPU-bound throughput.
- **[TODO, big] Reduce the GPU composite (~29 ms/frame).** The per-layer HDR pipeline (Vello render →
linear → composite, ×layers) is the wall, shared with live rendering. Options: batch composite
passes; a fast-path skipping HDR compositing for simple single-layer/no-blend docs; cache unchanged
layers' scenes (CPU-side, only helps if it later becomes CPU-bound). Render-architecture project.
- Non-issues: per-frame seek, blocking readback, audio. (`video.rs:237` container-reopen-on-seek is
a latent cost but doesn't fire on forward export.)
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
decode) still worth chasing if it recurs.
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
2 h video keeps that work and continues from there on reload.
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
complete clips; a mid-generation save resumes from where it left off on reload.)*
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
absent). The 5 s interval already bounds count; no extra budget needed.
- **Progressive waveform on first import:** generation streams the whole file before the
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
the clip left-to-right instead of appearing all at once. Persistence saves only the final
complete pyramid.
## Guiding principle
Three subsystems already have the right streaming primitive; most of the work is wiring,
bounding caches, and adding a residency window. The recurring pattern:
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
> stable ID, and evict everything outside a window around the playhead.
---
## Audit summary (where we stand today)
### Correctly streaming / bounded
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` —
keyframe-index seek + decode-until-target, one frame resident).
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
frames in flight).
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
### Fully-loaded, unbounded by file length (the problems)
| Site | Issue |
|---|---|
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
---
## Container format decision: `.beam` → SQLite *(DECIDED)*
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
`.beam` extension). This is the foundation the rest of the plan builds on.
### Why
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
mutation** — every save and every raster frame write-back rewrites the whole archive — and
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
SQLite dissolves the single-file-vs-performance tension:
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
confusion; we have no bundle magic on Linux/Windows).
- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
atomicity).
- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly.
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
reads (or external paths); raster frames live in blob rows and write back transactionally.
### Large-media policy: packed OR referenced
Two storage modes per media item, both supported:
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
on a network drive, or media too large/volatile to pack).
**Default-mode preference for files over the per-blob limit (~2GB):**
- A user preference `large_media_default: Pack | Reference` controls what happens to
imports above the threshold.
- The **first time** the user imports a media file over the limit, **prompt** them
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
large imports (changeable later in settings).
- Files under the limit are packed by default (chunked only if needed).
### Schema sketch
```
media(
id BLOB PRIMARY KEY, -- stable Uuid
kind INTEGER, -- audio | video | raster | image-asset
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
storage INTEGER, -- 0 = packed, 1 = referenced
ext_path TEXT, -- set when storage = referenced
total_len INTEGER, -- bytes (packed) for chunk math
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
)
media_chunk(
media_id BLOB, chunk_index INTEGER, bytes BLOB,
PRIMARY KEY (media_id, chunk_index)
)
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
```
`project.json` stays the same serialized `BeamProject` for now — only its container and the
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
first open/save.
### Streaming reads from packed media
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
plain `File` exactly as `do_import_audio` already does for originals today.
---
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
### 1a. Turn on the compressed-audio disk reader
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
assign `AudioClip::read_ahead`.
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
from the ring buffer, not RAM.
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
already distinguishes.
### 1b. Stream on project load *(depends on the SQLite container)*
Three coupled changes (none works alone):
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
for them. After the engine installs the project, walk all audio clips and
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
`BlobReader` instead of a path for packed media.)
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
table instead of base64 `embedded_data`.
### 1c. Video's embedded audio track — stream from the video via ffmpeg
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
save/reload. **Superseded by the design below.**
**Proper design — stream the video's audio track on demand, never materialized.**
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
interface.
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
`sample_rate`/`channels`/`total_frames`.
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
entry `path` = the video file.
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
One ffmpeg context + 3 s buffer per active clip instance.
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` →
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
cleanup.
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
frames before target; for the frame straddling target, drop its leading samples). After
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
while here, or at least flag.
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
seek (both readers) → waveform-peaks pass.
---
## Phase 2 — Video: bound the caches *(small, isolated)*
### 2a. Bound `VideoManager.frame_cache`
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
### 2b. Stream the export mux
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
---
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
the container is opened only at load/save (no live handle).
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
### Drive-by (Arc pixels): DROPPED
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
clone). Not worth 64 edits. Start at 3a.
### 3a. Lazy async fault-in + image proxy
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
results applied at top of `update()`. No UI block on cold scrub.
- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written
beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`.
Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to
the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident.
*(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge
paint projects.)*
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
leave full `raw_pixels` empty.
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
upscaled proxy. Reused by the exporter (already frame-by-frame).
### 3b. Residency window + eviction **[DONE]**
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
rows via `media_exists` (no data loss) and walks all layers to match load.
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
### 3c. Bound the GPU cache **[DONE for raster_layer_cache]**
`raster_layer_cache` (`gpu_brush.rs`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
≈ `w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in
`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay
now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom
if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello
`ImageCache` is image *assets* → Phase 4.)*
### 3d. Undo memory **[DONE]**
`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames.
Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in
`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor
faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`),
correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒
skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose
bbox is the whole frame.)*
### 3e. Prefetch frames **[DONE for playback]**
Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4`
upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so
full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during
playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a
prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction
prefetch still TODO.)*
Original note: *(future, after 3d — pure latency win, no correctness need)*
Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the
proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**:
on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of
playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker +
`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap
concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU
texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res
frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up.
### Build order & tests
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
---
## Phase 3.5 — Image textures in vector scenes **[DONE 2026-06-21]** *(prereq for Phase 4; fixed DCEL-broken image import)*
**Done:** 3.5a — import/drop places an image as a borderless image-filled rectangle
(`AddShapeAction::image_rect`), centered (direct import) or at the drop point (library drag);
renderer now maps the image brush onto the fill's bounding box (was anchored at world origin →
only a corner showed); `SetImageFillAction` + an **Image** fill-type tab (None|Solid|Gradient|Image)
with an asset picker in the Info Panel. 3.5b — image bytes persist as `MediaKind::ImageAsset` rows in
the `.beam` (kept-in-place; `ImageAsset.data` is `skip_serializing` + container-backed; old base64
projects migrate on re-save); eager-read on load. *(ImageCache still unbounded — Phase 4 adds the
usage-based LRU/lazy paging.)*
### (original plan below)
## Phase 3.5 — Image textures in vector scenes *(prereq for testing Phase 4; fixes DCEL-broken image import)*
**Why:** Phase 4 pages *image assets*, but there's currently no way to get an image asset into a
vector scene — so nothing to page. This also repairs image import, half-broken since the DCEL switch.
**Current state (audited 2026-06-21):**
- *Works:* `import_image` (`main.rs`) decodes dims + creates an `ImageAsset` (raw bytes embedded in
`Document::image_assets`, serialized as **base64 in project JSON**). The renderer's image-fill paths
are **complete** — GPU/Vello (`renderer.rs:~1160`, `ImageBrush` via `ImageCache.get_or_decode`) and
CPU/tiny-skia (`renderer.rs:~1486`). `Fill::image_fill` (`vector_graph/mod.rs:110`) and
`Face::image_fill` (`dcel2/mod.rs:117`) fields exist and render when set.
- *Broken/missing (the workflow):*
1. **Drop image → canvas is stubbed:** `stage.rs:~11782` and `main.rs:~4924` both just print
"Image drag to stage not yet supported with DCEL backend". Nothing is added to the scene.
2. **No way to assign an image fill:** no `SetImageFillAction` (only `SetFillPaintAction` for
color/gradient); no Info-Panel picker. `Fill`/`Face.image_fill` are never populated.
3. **DCEL faces never get `image_fill`** (`dcel2/import.rs:275` always `None`; topology copies from
parent which is also `None`).
4. **Not in the container:** `MediaKind::ImageAsset` exists but is **dead** — image bytes live only
as base64 in project JSON. Not chunked, not pageable (so Phase 4 can't page them).
**Tasks:**
- **3.5a — Place + assign.** Replace the two drop stubs: dropping an image onto a vector layer creates
a rectangle face sized to the image at the drop point with `image_fill = asset_id`. Add
`SetImageFillAction` (set/clear an image fill on the selected face/shape; mirrors `SetFillPaintAction`)
+ an Info-Panel image-asset picker for the selected shape's fill. Populate `Face.image_fill` in DCEL
(and keep it through topology ops — already copied from parent).
- **3.5b — Persist in the container.** Write image assets as `MediaKind::ImageAsset` rows in the `.beam`
SQLite (like raster/audio: write on save kept-in-place on re-save; read on load), keyed by asset id;
drop the base64-in-JSON embedding (or keep a tiny ref). This is the storage Phase 4 pages from.
- **3.5c — Lazy decode hook.** Image bytes load from the container into `ImageCache` on first render
(decode → `ImageBrush`/`Pixmap`). Leave `ImageCache` **unbounded for now**; Phase 4 adds the
usage-based LRU/eviction (this phase just makes there *be* real, container-backed image assets to page).
- **Tests:** import→drop→render round-trip; save/reload preserves the image fill + reads bytes from the
container (not JSON); CPU and GPU render paths both show the image.
---
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
**Data model.**
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
`Pixmap` map, keyed by asset id, **unbounded**.
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
**No asset→frame or frame→asset index exists today.**
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
the heavy one).
**Progress (2026-06-21):**
- **[DONE] Tier 2 — bound the decoded `ImageCache`.** 256 MB **usage-LRU**: every
`get_or_decode`/`_cpu` bumps the asset's recency; inserts past budget evict the least-recently-used
(a miss re-decodes from `asset.data`). Achieves usage-based eviction via render-access recency
(simpler than the frame→asset enumeration below; that enumeration is only needed for *prefetch*).
- **[DONE] Tier 1 — lazy compressed bytes.** `ImageCache` holds the container path (threaded
App.current_file_path → SharedPaneState → VelloRenderContext) and pages bytes on a decode miss via
`read_packed_media_readonly`; `load_beam_sqlite` no longer eager-reads → instant load, compressed
bytes don't accumulate. `asset.data` is still used when resident (fresh import / old base64 project).
*(Refinement: persistent read connection vs open-per-miss.)*
- **[DONE] Prefetch.** `assets_needed_at(document, time)` enumerates image ids in the visible vector
layers' active keyframes; during playback the stage decodes the ~0.5s-ahead set into the cache.
*(Refinements: nested clip-instance recursion; background-thread decode.)*
**Phase 4 = DONE** (image asset paging by usage + LRU).
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
### 4b. Usage bookkeeping (the multi-frame problem)
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
paths in `vector_graph` and the relevant actions).
- count 0 → dead, fully evictable / GC candidate.
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
playhead**, not by count (a high-count asset far from the playhead is still evicted).
Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
Eviction never touches an asset in needed-now.
### 4c. Bound the decoded tier
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
image-texture cache the same way, keyed to the residency window.
### Nested-clip prefetch (important)
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
each of which can reference its own image assets. Prefetch must therefore:
- Recurse through clip instances when computing both needed-now and needed-soon.
- Map outer playhead time → each nested clip's local time, and look ahead along the
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
keyframe are loaded before the nested clip reaches it.
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
once); the usage index handles refcounting.
---
## Cross-cutting: a shared residency abstraction
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
disk backing, residency = window/needed-set around playhead + LRU byte budget:
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|---|---|---|---|---|
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
enumeration + usage index is unique to Phase 4.
---
## Sequencing
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
2. **Phase 2** — small, isolated, independently shippable; container-independent.
3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy +
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
5. **Phase 1c** — depends on 1b's pool path.
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
---
## Status
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
buffer (was effectively silent before, since `read_ahead` was always `None`).
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
(bundled) — `lightningbeam-core/src/beam_archive.rs`
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
referenced-path, overwrite-replaces-chunks.
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
changed rows written; unchanged large media never rewritten); orphan cleanup via
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
must NOT copy+rename for existing SQLite files.
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
deleted (now packs original bytes with their codec).
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
`Ask` behaves as `Reference` at save time.
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
import_audio/video/image) queues a one-time modal; choice persists to config.
Threshold shown in the modal is derived from the constant.
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
skip unchanged frames on in-place save (Phase 3).
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
> build the lib in normal mode. Worth fixing separately.
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
+ proper video-audio reload remain as noted follow-ups.
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** —
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
violation; referenced-only-first = leaves packed <2GB in RAM.)
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
never fires for loaded clips (they bypass `AddAudioClip`).
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory`
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
by the editor between `load_audio_pool` and `set_project`.
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` —
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
from the full in-RAM decode, which no longer happens). Added
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
overview now fills in shortly after load.
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
then instantly on subsequent loads once saved; RAM stays flat on a big project.
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
→ `from_compressed` placeholder). Higher risk (touches the working referenced path); packed
covers the common <2GB case first.
- [x] **DONE: packed video streaming.** Small videos pack into the `.beam`
(a `MediaKind::Video` blob at the clip id, `VideoClip.media_id` referencing it) and stream **both
frames and audio** from the DB blob via FFmpeg. The `AVIOContext`-over-`Read+Seek` shim lives in
the new `ffmpeg-blob-io` crate (`BlobInput`, version-pinned `=8.0.0`/`=8.0.1`), isolating the
unsafe + ABI coupling. Frames: `video.rs` `VideoSource{Path,Packed}` opens a fresh `BlobReader`
per decoder/seek/scan. Audio: `VideoAudioReader::open_source` over the same blob (the
`disk_reader.rs` `StreamSource` blocker is removed); save points the linked video-audio pool
entry's `media_id` at the video row so it streams from the same blob. Tests: ffmpeg-blob-io AVIO
unit tests (WAV via Cursor + seek + open/drop loop), core `packed_video_stream` (blob→AVIO→Input),
`beam_archive` packed-video round-trip, daw-backend `open_source` (compiles; can't link in the
container — user runtime-verifies actual A/V playback).
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
small temp partitions) and the temp path doesn't survive reload.
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
reference survives save/load.
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
empty channel layout → must `set_channel_layout` before resampling, else swr returns
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
until wired. `VideoAudioReader` made `pub` for the integration test.
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
daw-backend + editor compile clean; VideoAudioReader tests still pass.
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
regression) and that an activated VideoAudio clip produces sound.
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` →
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
**→ Feature is live end-to-end; ready for in-app testing.**
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
whole packet is discarded). So compressed clips no longer drift vs video audio after
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` —
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
Ragged last buckets reduce over available children (no value padding). Bounded
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
ordering, stereo channels, size bound, chunk-agnostic.
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
memory (one chunk + the pyramid). Test: envelope matches the signal through
both backends.
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
- [ ] Orchestration (with 1c).
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
- [x] Generation: on video-audio import Success, the same bg thread streams
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
main.rs construction) → `render_layers` → **both** render sites (collapsed-group
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
features hold position at every zoom and line up with playback.
See memory `waveform-shader-fractional-mip-offset`.
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
`register_loaded_videos` only loads frames (not audio), so there is no redundant
regeneration to suppress. Compiles clean across all three crates.
### Waveform LOD pyramid design (step 6)
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
in → per-sample; seamless between.
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
stored.
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
temp before first save). `B` is stored with it (preference is just the default for new gen).
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
cleared only after its children") — so rendering can always walk up to a resident ancestor;
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 —
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
above the floor it's a disk read; below it the span is already small.
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
viewports of fine tiles; persist pyramid in `.beam`.
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
in-app export to confirm A/V sync.)*
- [x] Phase 3a — lazy + async raster fault-in (`RasterStore` + background thread + image proxy)
- [x] Phase 3b — raster residency LRU + eviction (dirty-flag data-loss safety)
- [x] Phase 3c — bound raster GPU texture cache (recency LRU + F3 VRAM readout)
- [x] Phase 3d — raster undo dirty-rect diffs (+ fault-in-before-undo)
- [x] Phase 3.5 — image textures in vector scenes (fixed DCEL-broken image import; image-fill tab + picker; container-persisted)
- [x] Phase 4 — image asset paging: Tier 2 decoded-cache byte-LRU, Tier 1 lazy container bytes, playback prefetch
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.

310
TODO.md
View File

@ -1,25 +1,305 @@
# Lightningbeam TODO
## Known Issues (Rust)
> ⚠️ **Stale entries:** Lightningbeam was rewritten from JavaScript to Rust. Any entry below
> that cites `src/*.js` / `main.js` / `animation.js` predates that migration — the *issue* may
> or may not still exist in the Rust codebase, but the file/line references are obsolete.
> **Re-verify against the current Rust code before acting** (this covers the "Animation System
> Refactoring" section and the JS-referencing "Known Issues" entries — node editor, default
> interpolation, etc.). Items with no `.js` references are current.
### Animation: Tweens are broken — LOW PRIORITY
- Shape/vector interpolation between keyframes, and the `tween_after` behavior on
keyframes, don't work correctly in the current app. Needs investigation + fix.
Not urgent — revisit later.
## Animation System Refactoring *(STALE — JS-era migration notes; superseded by the Rust DCEL/keyframe system)*
## Backlog / Feature ideas
### Completed
- ✅ Implement AnimationData curve-based system (Keyframe, AnimationCurve, AnimationData classes)
- ✅ Add GraphicsObject.currentTime property
- ✅ Migrate shape rendering to use AnimationData curves (exists, zOrder)
- ✅ Binary search optimization for keyframe lookups
### Animation curve enhancements
- [ ] Extrapolation modes, separate for start vs end: hold (default), extend, repeat, decay
- [ ] Position / scale / rotation animation curves for shapes
- [ ] Shape morphing / tweening between keyframes
### In Progress
- Migrating from Frame-based to AnimationData curve-based system throughout codebase
### Keyframing behavior
- [ ] User preference for keyframing when editing objects:
### Pending Features
#### Animation Curve Enhancements
- [ ] Implement extrapolation modes (separate for start vs end):
- "hold" (default) - hold value at first/last keyframe
- "extend" - linearly extend the curve beyond keyframes
- "repeat" - repeat the animation
- "decay" - exponential decay to a target value
- [ ] Add position, scale, rotation animation curves for shapes
- [ ] Add shape morphing/tweening between keyframes
#### Keyframing Behavior
- [ ] Add user preference for keyframing behavior when editing objects:
- Auto-keyframe (current default): create/update keyframe at current time
- Edit previous (Flash-style): update most recent keyframe before current time
- Ephemeral (Blender-style): changes don't persist without manual keyframe
- Optional modifier key (e.g. Shift) to toggle modes
- Optional: Add modifier key (e.g. Shift) to toggle between modes
### Shape ordering
- [ ] Bring Forward / Send Backward / Bring to Front / Send to Back menu options
#### Shape Ordering
- [ ] Add "Bring Forward" menu option (swap zOrder with shape in front)
- [ ] Add "Send Backward" menu option (swap zOrder with shape behind)
- [ ] Add "Bring to Front" menu option (set zOrder to max + 1)
- [ ] Add "Send to Back" menu option (set zOrder to min - 1)
#### Code Cleanup
- [ ] Remove all remaining references to Frame-based system
- [ ] Remove legacy Frame class once migration is complete
- [ ] Clean up GraphicsObject.shapes[] array (shapes should only live in Layers)
## Known Issues / Platform Limitations
### Animation: Tweens are broken (Rust codebase) — LOW PRIORITY
- **Issue**: Animation tweening between keyframes (shape/vector interpolation, and the
`tween_after` behavior on keyframes) does not work correctly in the current Rust app.
Needs investigation + fix. Not urgent — revisit later.
- (Older JS-codebase animation entries below reference `src/*.js` and are stale.)
### Audio: Oscillator Timbre Drift (Phase Accumulation Error)
- **Issue**: Oscillators exhibit timbre changes over time due to floating-point phase accumulation errors
- **Affected Files**:
- `daw-backend/src/effects/synth.rs:117-120` (SimpleSynth)
- `daw-backend/src/audio/node_graph/nodes/oscillator.rs:167-170` (OscillatorNode)
- **Root Cause**: Current phase wrapping uses conditional subtraction (`if phase >= 1.0 { phase -= 1.0 }`), which accumulates f32 rounding errors over time, especially for long-playing notes
- **Current Code**:
```rust
self.phase += frequency / sample_rate;
if self.phase >= 1.0 {
self.phase -= 1.0;
}
```
- **Recommended Fix**: Replace with `.fract()` for numerically stable wraparound:
```rust
self.phase += frequency / sample_rate;
self.phase = self.phase.fract();
```
- **Impact**: Medium - affects audio quality for sustained notes, becomes noticeable after several seconds
- **Priority**: Medium - should be addressed before production use
### UI: Node Connections Render Behind VoiceAllocator Child Nodes
- **Issue**: Connection lines (SVG paths) inside expanded VoiceAllocator nodes render behind child nodes due to z-index stacking
- **Affected File**: `src/styles.css:1128`
- **Root Cause**: Child nodes have `z-index: 10` while connection SVG paths have default/lower z-index
- **Current Code**:
```css
.drawflow .drawflow-node.child-node {
opacity: 0.9;
border: 1px solid #5a5aaa !important;
box-shadow: 0 2px 8px rgba(90, 90, 170, 0.3);
z-index: 10;
}
```
- **Recommended Fix**: Either:
1. Remove `z-index: 10` from `.child-node` (simplest), or
2. Add higher z-index to connection SVG paths, or
3. Use CSS `isolation: isolate` on the VoiceAllocator contents area to create a new stacking context
- **Impact**: Low - visual issue only, connections still function but appear to go "behind" nodes
- **Priority**: Low - cosmetic issue that doesn't affect functionality
### UI: VoiceAllocator Child Nodes Don't Move with Parent
- **Issue**: When a VoiceAllocator node is moved, its child nodes remain in their original positions instead of moving with the parent
- **Affected File**: `src/main.js:6202-6207`
- **Root Cause**: The `nodeMoved` event handler only handles the case where a child node is moved (resizes parent), but doesn't handle when the VoiceAllocator itself is moved
- **Current Code**:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
});
```
- **Recommended Fix**: Add logic to detect when a VoiceAllocator is moved and update all child node positions:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
// Case 1: A child node was moved - resize parent
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
// Case 2: A VoiceAllocator was moved - move all children
if (node && node.data.nodeType === 'VoiceAllocator') {
// Calculate delta from previous position (need to track)
// Update all child node positions by the delta
// Call editor.updateConnectionNodes() for parent and all children
}
});
```
- **Impact**: High - child nodes become disconnected from parent visually
- **Priority**: High - breaks expected behavior of grouped nodes
### UI: VoiceAllocator Expansion Doesn't Update Connection Positions
- **Issue**: When expanding/collapsing a VoiceAllocator, connection endpoints don't update to match the new port positions
- **Affected File**: `src/main.js:6496-6555` (handleNodeDoubleClick function)
- **Root Cause**: The expand/collapse logic shows/hides child nodes and resizes the container, but never calls `editor.updateConnectionNodes()` to refresh connection positions
- **Current Code**: In `handleNodeDoubleClick()`, after expanding or collapsing:
```javascript
// Expand
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
nodeElement.style.width = '600px';
nodeElement.style.height = '400px';
// ... shows child nodes ...
// Missing: editor.updateConnectionNodes(`node-${nodeId}`)
```
- **Recommended Fix**: Call `editor.updateConnectionNodes()` after resizing:
```javascript
// After expanding
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
// ... resize and show children ...
// Update connection positions for VoiceAllocator and all children
editor.updateConnectionNodes(`node-${nodeId}`);
for (const [childId, parentId] of nodeParents.entries()) {
if (parentId === nodeId) {
editor.updateConnectionNodes(`node-${childId}`);
}
}
```
- **Impact**: Medium - connections appear in wrong positions until manually moved
- **Priority**: Medium - visual issue that affects usability
### UI: Node Editor Allows Editing Without MIDI Layer Selected
- **Issue**: The node editor pane allows adding/editing instrument nodes even when no MIDI layer is selected, and always uses hardcoded `trackId: 0`
- **Affected File**: `src/main.js:6045-6920` (nodeEditor function)
- **Root Cause**: The node editor never checks if `context.activeObject.activeLayer` exists or is a MIDI track, and all backend commands use hardcoded `trackId: 0`
- **Current Code**: All graph commands hardcode track 0:
```javascript
const commandArgs = parentNodeId
? {
trackId: 0, // HARDCODED!
voiceAllocatorId: editor.getNodeFromId(parentNodeId).data.backendId,
nodeType: nodeType,
x: x,
y: y
}
: {
trackId: 0, // HARDCODED!
nodeType: nodeType,
x: x,
y: y
};
```
- **Recommended Fix**:
1. Check if activeLayer is a MIDI track before allowing edits:
```javascript
function getSelectedMidiTrack() {
const activeLayer = context.activeObject?.activeLayer;
if (!activeLayer || activeLayer.type !== 'midi') {
return null;
}
return activeLayer;
}
```
2. Show placeholder when no MIDI track selected:
```javascript
function nodeEditor() {
const container = document.createElement("div");
const midiTrack = getSelectedMidiTrack();
if (!midiTrack) {
container.innerHTML = '<div class="placeholder">Select a MIDI layer to edit instruments</div>';
return container;
}
// ... rest of node editor code ...
}
```
3. Use actual track ID instead of hardcoded 0:
```javascript
const trackId = midiTrack.audioTrackId || 0;
const commandArgs = { trackId, nodeType, x, y };
```
4. Add listener to refresh node editor when layer selection changes
- **Impact**: High - allows editing wrong track's instrument graph, data corruption risk
- **Priority**: High - can cause confusion and data loss
### Animation: Wrong Default Interpolation for Shape and Object Keyframes
- **Issue**: Shape index and object transform keyframes default to "linear" interpolation but should default to "hold" (step function), and there's no UI to change interpolation after creation
- **Affected Files**:
- `src/models/animation.js:124` (Keyframe constructor defaults to "linear")
- `src/main.js:2161` (shapeIndex keyframes default to "linear")
- `src/main.js:2198` (object position/rotation/scale keyframes default to "linear")
- `src/main.js:5910` (Timeline menu - missing tween options)
- **Root Cause**:
1. The Keyframe constructor defaults interpolation to "linear"
2. Shape index keyframes preserve existing interpolation or default to "linear"
3. Object transform keyframes explicitly use "linear"
4. No menu options exist to change interpolation mode after keyframe creation
- **Current Code**:
- Keyframe constructor (animation.js:124):
```javascript
constructor(time, value, interpolation = "linear", uuid = undefined) {
```
- Shape index keyframes (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'linear';
const shapeIndexKeyframe = new Keyframe(currentTime, newShapeIndex, interpolationType);
```
- Object keyframes (main.js:2198):
```javascript
const newKeyframe = new Keyframe(
currentTime,
currentValue,
'linear' // Default to linear interpolation
);
```
- **Expected Behavior**:
- Shape index keyframes should default to "hold" (shapes shouldn't morph between versions)
- Object transforms should default to "hold" (objects shouldn't move/rotate/scale between keyframes unless explicitly tweened)
- Timeline menu should have options to convert between interpolation modes
- **Recommended Fix**:
1. Change shapeIndex default to "hold" (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'hold';
```
2. Change object keyframe default to "hold" (main.js:2198):
```javascript
const newKeyframe = new Keyframe(currentTime, currentValue, 'hold');
```
3. Add Timeline menu options (main.js:5910, in timelineSubmenu):
```javascript
{
text: "Add Shape Tween",
enabled: /* check if shape is selected and has keyframes */,
action: () => {
// Find shapeIndex curve for selected shape
// Change interpolation between keyframes to "linear"
}
},
{
text: "Add Motion Tween",
enabled: /* check if object is selected and has transform keyframes */,
action: () => {
// Find position/rotation/scale curves for selected object
// Change interpolation between keyframes to "linear" or "bezier"
}
}
```
- **Note**: exists and zOrder keyframes already correctly use "hold" (main.js:2139, 2150)
- **Impact**: High - causes unwanted interpolation, shapes morph unexpectedly, objects move when they shouldn't
- **Priority**: High - fundamental animation behavior is incorrect
### Tauri Pinch-Zoom on Linux
- **Issue**: Two-finger pinch gestures zoom the entire Tauri window instead of individual canvases
- **Status**: Known Tauri limitation on Linux/GTK with no cross-platform solution
- **Tracking**: https://github.com/tauri-apps/tauri/discussions/3843
- **Workaround attempts**: Tried `zoomHotkeysEnabled: false`, `touch-action: none`, viewport meta tags - none worked
- **Resolution**: Monitor Tauri releases for official fix
## Notes
### Architecture
- **GraphicsObject** contains Layers and has `currentTime` (continuous time)
- **Layer** contains `shapes[]` array and `animationData` (AnimationData instance)
- **AnimationData** contains curves dictionary, each curve identified by parameter name
- Shape curves: `shape.{uuid}.exists`, `shape.{uuid}.zOrder`
- Future: `shape.{uuid}.x`, `shape.{uuid}.y`, `shape.{uuid}.rotation`, etc.
- **Shapes render based on curves**: Layer.draw checks exists > 0, sorts by zOrder, draws in order
### Interpolation Types
- `linear` - Linear interpolation between keyframes
- `bezier` - Cubic Bezier with easing control points
- `step`/`hold` - Step function (jumps to next value)

10
daw-backend/Cargo.lock generated
View File

@ -586,7 +586,6 @@ dependencies = [
"dasp_rms",
"dasp_sample",
"dasp_signal",
"ffmpeg-blob-io",
"ffmpeg-next",
"hound",
"memmap2",
@ -662,15 +661,6 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
[[package]]
name = "ffmpeg-blob-io"
version = "0.1.0"
dependencies = [
"ffmpeg-next",
"ffmpeg-sys-next",
"libc",
]
[[package]]
name = "ffmpeg-next"
version = "8.0.0"

93
daw-backend/src/audio/engine.rs
View File

@ -1001,20 +1001,20 @@ impl Engine {
let _ = self.event_tx.push(AudioEvent::AudioFileAdded(pool_index, path));
}
Command::AddAudioClip(track_id, clip_id, pool_index, start_time, duration, offset) => {
// Create a new clip instance with the pre-assigned clip_id.
// start_time/duration are beats; offset (internal_start) is seconds.
let start_beats = start_time;
let end_beats = start_time + duration;
// Create a new clip instance with the pre-assigned clip_id
// start_time and duration are in beats; offset (internal_start) is seconds
let start_beats = Beats(start_time);
let end_beats = Beats(start_time + duration);
let start_secs = self.tempo_map.beats_to_seconds(start_beats);
let end_secs = self.tempo_map.beats_to_seconds(end_beats);
let content_dur_secs = (end_secs - start_secs).seconds_to_f64();
let mut clip = AudioClipInstance::new(
clip_id,
pool_index,
offset,
offset + Seconds(content_dur_secs),
Seconds(offset),
Seconds(offset + content_dur_secs),
start_beats,
duration,
Beats(duration),
);
// If the source is streamed (a compressed audio file, or a video's
@ -3348,8 +3348,8 @@ impl EngineController {
}
/// Seek to a specific position in seconds
pub fn seek(&mut self, seconds: Seconds) {
let _ = self.command_tx.push(Command::Seek(seconds.seconds_to_f64()));
pub fn seek(&mut self, seconds: f64) {
let _ = self.command_tx.push(Command::Seek(seconds));
}
/// Set track volume (0.0 = silence, 1.0 = unity gain)
@ -3380,23 +3380,19 @@ impl EngineController {
}
/// Move a clip to a new timeline position (changes external_start)
pub fn move_clip(&mut self, track_id: TrackId, clip_id: ClipId, new_start_time: Beats) {
let _ = self.command_tx.push(Command::MoveClip(track_id, clip_id, new_start_time.beats_to_f64()));
pub fn move_clip(&mut self, track_id: TrackId, clip_id: ClipId, new_start_time: f64) {
let _ = self.command_tx.push(Command::MoveClip(track_id, clip_id, new_start_time));
}
/// Trim a clip's internal boundaries (changes which portion of source content is used)
/// This also resets external_duration to match internal duration (disables looping)
/// Trim a clip's internal content bounds. The units are content-domain and depend on the
/// track type: SECONDS for a sampled-audio clip, BEATS for a MIDI clip (see the TrimClip
/// handler). Left as raw f64 because a single newtype can't express both; callers pass the
/// clip's own `trim_start`/`trim_end`, which already match its content domain.
pub fn trim_clip(&mut self, track_id: TrackId, clip_id: ClipId, new_internal_start: f64, new_internal_end: f64) {
let _ = self.command_tx.push(Command::TrimClip(track_id, clip_id, new_internal_start, new_internal_end));
}
/// Extend or shrink a clip's external duration (enables looping if > internal duration)
pub fn extend_clip(&mut self, track_id: TrackId, clip_id: ClipId, new_external_duration: Beats) {
let _ = self.command_tx.push(Command::ExtendClip(track_id, clip_id, new_external_duration.beats_to_f64()));
pub fn extend_clip(&mut self, track_id: TrackId, clip_id: ClipId, new_external_duration: f64) {
let _ = self.command_tx.push(Command::ExtendClip(track_id, clip_id, new_external_duration));
}
/// Send a generic command to the audio thread
@ -3444,8 +3440,8 @@ impl EngineController {
/// Set metatrack time offset in seconds
/// Positive = shift content later, negative = shift earlier
pub fn set_offset(&mut self, track_id: TrackId, offset: Seconds) {
let _ = self.command_tx.push(Command::SetOffset(track_id, offset.seconds_to_f64()));
pub fn set_offset(&mut self, track_id: TrackId, offset: f64) {
let _ = self.command_tx.push(Command::SetOffset(track_id, offset));
}
/// Set metatrack pitch shift in semitones (for future use)
@ -3454,13 +3450,13 @@ impl EngineController {
}
/// Set metatrack trim start in seconds
pub fn set_trim_start(&mut self, track_id: TrackId, trim_start: Seconds) {
let _ = self.command_tx.push(Command::SetTrimStart(track_id, trim_start.seconds_to_f64()));
pub fn set_trim_start(&mut self, track_id: TrackId, trim_start: f64) {
let _ = self.command_tx.push(Command::SetTrimStart(track_id, trim_start));
}
/// Set metatrack trim end in seconds (None = no end trim)
pub fn set_trim_end(&mut self, track_id: TrackId, trim_end: Option<Seconds>) {
let _ = self.command_tx.push(Command::SetTrimEnd(track_id, trim_end.map(|s| s.seconds_to_f64())));
pub fn set_trim_end(&mut self, track_id: TrackId, trim_end: Option<f64>) {
let _ = self.command_tx.push(Command::SetTrimEnd(track_id, trim_end));
}
/// Create a new audio track
@ -3529,14 +3525,14 @@ impl EngineController {
/// Add a clip to an audio track (async, fire-and-forget)
/// Returns the pre-assigned clip instance ID so callers can track the clip without a sync round-trip
pub fn add_audio_clip(&mut self, track_id: TrackId, pool_index: usize, start_time: Beats, duration: Beats, offset: Seconds) -> AudioClipInstanceId {
pub fn add_audio_clip(&mut self, track_id: TrackId, pool_index: usize, start_time: f64, duration: f64, offset: f64) -> AudioClipInstanceId {
let clip_id = self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed);
let _ = self.command_tx.push(Command::AddAudioClip(track_id, clip_id, pool_index, start_time, duration, offset));
clip_id
}
/// Add a clip to an audio track with a pre-assigned ID (for undo/redo, restoring deleted clips)
pub fn add_audio_clip_with_id(&mut self, track_id: TrackId, clip_id: AudioClipInstanceId, pool_index: usize, start_time: Beats, duration: Beats, offset: Seconds) {
pub fn add_audio_clip_with_id(&mut self, track_id: TrackId, clip_id: AudioClipInstanceId, pool_index: usize, start_time: f64, duration: f64, offset: f64) {
let _ = self.command_tx.push(Command::AddAudioClip(track_id, clip_id, pool_index, start_time, duration, offset));
}
@ -3611,26 +3607,25 @@ impl EngineController {
}
/// Create a new MIDI clip on a track
pub fn create_midi_clip(&mut self, track_id: TrackId, start_time: Beats, duration: Beats) -> MidiClipId {
pub fn create_midi_clip(&mut self, track_id: TrackId, start_time: f64, duration: f64) -> MidiClipId {
// Peek at the next clip ID that will be used
let clip_id = self.next_midi_clip_id.load(Ordering::Relaxed);
let _ = self.command_tx.push(Command::CreateMidiClip(track_id, start_time.beats_to_f64(), duration.beats_to_f64()));
let _ = self.command_tx.push(Command::CreateMidiClip(track_id, start_time, duration));
clip_id
}
/// Add a MIDI note to a clip
pub fn add_midi_note(&mut self, track_id: TrackId, clip_id: MidiClipId, time_offset: Beats, note: u8, velocity: u8, duration: Beats) {
let _ = self.command_tx.push(Command::AddMidiNote(track_id, clip_id, time_offset.beats_to_f64(), note, velocity, duration.beats_to_f64()));
pub fn add_midi_note(&mut self, track_id: TrackId, clip_id: MidiClipId, time_offset: f64, note: u8, velocity: u8, duration: f64) {
let _ = self.command_tx.push(Command::AddMidiNote(track_id, clip_id, time_offset, note, velocity, duration));
}
/// Add a pre-loaded MIDI clip to a track at the given timeline position (beats)
pub fn add_loaded_midi_clip(&mut self, track_id: TrackId, clip: MidiClip, start_time: Beats) {
let _ = self.command_tx.push(Command::AddLoadedMidiClip(track_id, clip, start_time.beats_to_f64()));
/// Add a pre-loaded MIDI clip to a track at the given timeline position
pub fn add_loaded_midi_clip(&mut self, track_id: TrackId, clip: MidiClip, start_time: f64) {
let _ = self.command_tx.push(Command::AddLoadedMidiClip(track_id, clip, start_time));
}
/// Update all notes in a MIDI clip. Note tuples are (start [beats], note, velocity, duration [beats]).
pub fn update_midi_clip_notes(&mut self, track_id: TrackId, clip_id: MidiClipId, notes: Vec<(Beats, u8, u8, Beats)>) {
let notes = notes.into_iter().map(|(t, n, v, d)| (t.beats_to_f64(), n, v, d.beats_to_f64())).collect();
/// Update all notes in a MIDI clip
pub fn update_midi_clip_notes(&mut self, track_id: TrackId, clip_id: MidiClipId, notes: Vec<(f64, u8, u8, f64)>) {
let _ = self.command_tx.push(Command::UpdateMidiClipNotes(track_id, clip_id, notes));
}
@ -3666,25 +3661,25 @@ impl EngineController {
&mut self,
track_id: TrackId,
lane_id: crate::audio::AutomationLaneId,
time: Beats,
time: f64,
value: f32,
curve: crate::audio::CurveType,
) {
let _ = self.command_tx.push(Command::AddAutomationPoint(
track_id, lane_id, time.beats_to_f64(), value, curve,
track_id, lane_id, time, value, curve,
));
}
/// Remove an automation point at a specific time (beats); tolerance is a beats delta
/// Remove an automation point at a specific time
pub fn remove_automation_point(
&mut self,
track_id: TrackId,
lane_id: crate::audio::AutomationLaneId,
time: Beats,
tolerance: Beats,
time: f64,
tolerance: f64,
) {
let _ = self.command_tx.push(Command::RemoveAutomationPoint(
track_id, lane_id, time.beats_to_f64(), tolerance.beats_to_f64(),
track_id, lane_id, time, tolerance,
));
}
@ -3720,16 +3715,16 @@ impl EngineController {
/// Add a keyframe to an AutomationInput node
pub fn automation_add_keyframe(&mut self, track_id: TrackId, node_id: u32,
time: Beats, value: f32, interpolation: String,
time: f64, value: f32, interpolation: String,
ease_out: (f32, f32), ease_in: (f32, f32)) {
let _ = self.command_tx.push(Command::AutomationAddKeyframe(
track_id, node_id, time.beats_to_f64(), value, interpolation, ease_out, ease_in));
track_id, node_id, time, value, interpolation, ease_out, ease_in));
}
/// Remove a keyframe from an AutomationInput node
pub fn automation_remove_keyframe(&mut self, track_id: TrackId, node_id: u32, time: Beats) {
pub fn automation_remove_keyframe(&mut self, track_id: TrackId, node_id: u32, time: f64) {
let _ = self.command_tx.push(Command::AutomationRemoveKeyframe(
track_id, node_id, time.beats_to_f64()));
track_id, node_id, time));
}
/// Set the display name of an AutomationInput node
@ -3744,8 +3739,8 @@ impl EngineController {
}
/// Start recording on a track
pub fn start_recording(&mut self, track_id: TrackId, start_time: Beats) {
let _ = self.command_tx.push(Command::StartRecording(track_id, start_time));
pub fn start_recording(&mut self, track_id: TrackId, start_time: f64) {
let _ = self.command_tx.push(Command::StartRecording(track_id, Beats(start_time)));
}
/// Stop the current recording
@ -3764,8 +3759,8 @@ impl EngineController {
}
/// Start MIDI recording on a track
pub fn start_midi_recording(&mut self, track_id: TrackId, clip_id: MidiClipId, start_time: Beats) {
let _ = self.command_tx.push(Command::StartMidiRecording(track_id, clip_id, start_time));
pub fn start_midi_recording(&mut self, track_id: TrackId, clip_id: MidiClipId, start_time: f64) {
let _ = self.command_tx.push(Command::StartMidiRecording(track_id, clip_id, Beats(start_time)));
}
/// Stop the current MIDI recording

3
daw-backend/src/command/types.rs
View File

@ -71,8 +71,7 @@ pub enum Command {
AddAudioFile(String, Vec<f32>, u32, u32),
/// Add a clip to an audio track (track_id, clip_id, pool_index, start_time, duration, offset)
/// The clip_id is pre-assigned by the caller (via EngineController::next_audio_clip_id())
/// (track, clip_id, pool_index, start_time [beats], duration [beats], offset [seconds])
AddAudioClip(TrackId, AudioClipInstanceId, usize, Beats, Beats, Seconds),
AddAudioClip(TrackId, AudioClipInstanceId, usize, f64, f64, f64),
// MIDI commands
/// Create a new MIDI track with a name and optional parent group

6
daw-backend/src/tui/mod.rs
View File

@ -790,7 +790,7 @@ fn execute_command(
return Err("Usage: seek <seconds>".to_string());
}
let pos: f64 = parts[1].parse().map_err(|_| "Invalid position")?;
controller.seek(crate::Seconds(pos));
controller.seek(pos);
app.set_status(format!("Seeked to {:.2}s", pos));
}
"track" => {
@ -830,7 +830,7 @@ fn execute_command(
app.next_clip_id += 1;
app.add_clip(track_id, clip_id, start_time, duration, format!("Clip {}", clip_id), Vec::new());
controller.create_midi_clip(track_id, crate::Beats(start_time), crate::Beats(duration));
controller.create_midi_clip(track_id, start_time, duration);
app.set_status(format!("Created MIDI clip on track {} at {:.2}s for {:.2}s", track_id, start_time, duration));
}
"loadmidi" => {
@ -882,7 +882,7 @@ fn execute_command(
app.next_clip_id += 1;
// Send to audio engine with the start_time (clip content is separate from timeline position)
controller.add_loaded_midi_clip(track_id, midi_clip, crate::Beats(start_time));
controller.add_loaded_midi_clip(track_id, midi_clip, start_time);
app.set_status(format!("Loaded {} ({} events, {:.2}s) to track {} at {:.2}s",
file_path, event_count, duration, track_id, start_time));

2
lightningbeam-ui/Cargo.lock generated
View File

@ -3628,7 +3628,7 @@ dependencies = [
[[package]]
name = "lightningbeam-editor"
version = "1.0.8-alpha"
version = "1.0.7-alpha"
dependencies = [
"beamdsp",
"bytemuck",

68
lightningbeam-ui/GPU_VIDEO_DECODE_PLAN.md
View File

@ -67,47 +67,15 @@ at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *an
- Result: software exports are full-quality at any export res, and document resizes re-target decode.
No hardware needed; this is the correctness fix for the codecs HW can't handle anyway.
### Stage 2 — hardware decode primitive (DONE, commit 255e164)
`decoder::VaapiDecoder` in `gpu-video-encoder`: decode → VAAPI surface → DRM-PRIME DMA-BUF →
`dmabuf::import_raw` → wgpu textures. Round-trip test (encode gray → decode → readback Y≈128) passes.
### Stage 2 — hardware decode primitive (headless-testable here, like the 8 encode tests)
- In `gpu-video-encoder` (rename → `gpu-video-codec`): `h264_vaapi`-style **decode** → VAAPI surface →
export DMA-BUF → import as a wgpu texture. Hardware test: decode a known file, verify dims/contents.
### The device-affinity problem (drives the whole rest of the design)
wgpu textures can't cross devices, and a decoded frame is a wgpu texture imported from a DMA-BUF —
which **requires a device with the DMA-BUF-import extensions** (`VK_EXT_image_drm_format_modifier`
+ external-memory), built via wgpu-hal `device_from_raw` (the safe `DeviceDescriptor` can't add
them). So a hardware-decoded frame is only usable by a compositor running on **such** a device.
- **Export** composites on the encoder's custom device → already fine.
- **Preview** composites on eframe's *normal* device → can't import DMA-BUFs → can't use HW frames.
Since **preview must HW-decode 4K** (software 4K decode ≈19 ms/frame), the resolution is a **single
shared custom device** used by eframe + preview compositor + decoder + encoder. eframe 0.33 (local
`egui-fork`) accepts it via `WgpuSetup::Existing { instance, adapter, device, queue }` — confirmed.
The earlier "separate export device" becomes redundant once this lands.
### Stage 3a — windowed shared `DrmDevice`, injected into eframe (highest-risk; blind)
Today `vk_device::create()` is **headless**. Make a windowed variant (or extend it) that is a
**superset** device: DMA-BUF import ext **+** `VK_KHR_swapchain` (device) and the WSI surface
instance extensions, **+** everything eframe/egui/vello need — `adapter.limits()` (already; Vello
needs `max_storage_buffers_per_shader_stage` ≥ 5), `max_texture_dimension_2d` 8192, and the optional
features main.rs requests (`SHADER_F16`, `TIMESTAMP_QUERY[_INSIDE_ENCODERS]`). Pick the adapter that
is the **VAAPI GPU** (the render node must match libva's, or DMA-BUF sharing fails on multi-GPU).
- main.rs: try to build the shared device; on success pass `WgpuSetup::Existing`, else fall back to
the current `WgpuSetupCreateNew` (software decode only). Gate on Linux + VAAPI + a config/env
override; **must be bulletproof** — this device now renders *every* frame of *every* session for
Linux/VAAPI users, video or not. Milestone: editor runs normally on it with no video involved.
### Stage 3b — VideoManager hardware decode on the shared device (blind)
- `VideoManager` holds a `VaapiDecoder` per HW-decodable clip (built on the shared device), plus the
software `VideoDecoder` fallback. `get_frame` gains a GPU-returning variant: yields an imported NV12
texture pair (native res) instead of `Arc<Vec<u8>>`. Probe HW support per source; non-VAAPI /
unsupported codecs / non-Linux → software path (Stage 1, target-res).
- Cache native GPU textures keyed by (clip, ts); revisit the byte budget (4K NV12 ≈ 12 MB each).
### Stage 3c — compositor consumes the GPU frame (blind; user-verifies)
- The video-instance composite path takes an NV12 texture (or a small NV12→RGB GPU pass) and blits it
to the target with the existing bilinear blit — **no `write_texture` upload**. GPU scales native→
target (preview res or export res). Both preview and the zero-copy export become
decode→composite(→encode) with no CPU frame. Software frames still upload as today.
### Stage 3 — wire hardware decode into `get_frame` (blind; user-verifies)
- When the source codec/driver is HW-decodable, `get_frame` returns a **GPU texture** (native res)
instead of `Arc<Vec<u8>>`; the compositor uses it directly (no `write_texture`), GPU-scaling to the
target. For the zero-copy export the frame never leaves the GPU: **decode → composite → encode** on
one device. Software path is the fallback for everything else.
## Critical files
- `lightningbeam-core/src/video.rs` — `VideoDecoder` (per-request output size, scaler cache),
@ -118,21 +86,13 @@ is the **VAAPI GPU** (the render node must match libva's, or DMA-BUF sharing fai
- `gpu-video-encoder/` (→ `gpu-video-codec`) — `dmabuf.rs`/`vk_device.rs` reused for the decode import.
## Risks
- **Shared custom device is the editor's main device (BIGGEST risk)** — Stage 3a makes a hand-built
wgpu-hal Vulkan device render every frame for Linux/VAAPI users. It must satisfy eframe + egui +
vello + winit presentation across varied Intel/AMD/Mesa stacks, or the editor won't start. Mitigate
with a strict try-and-fall-back-to-normal-device path + an env/config kill switch. Test broadly.
- **Multi-GPU** — the shared render device must be the *same* GPU as libva's VAAPI device, or DMA-BUF
import fails. Adapter selection must match the render node to the VAAPI node (laptops with iGPU +
dGPU, PRIME).
- **Codec coverage** — only some codecs are HW-decodable per GPU/driver; software must stay correct
and well-tested. Probe support per source, don't assume.
- **Cache memory** — native-res GPU textures (esp. 4K NV12 ≈12 MB) are large; revisit the frame cache
budget, and the two live targets (preview res + export res) shouldn't thrash.
- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; import handles NV12, but 10-bit/HDR
(P010) needs format handling. Decoded NV12 also needs the right BT.601/709 + range on the NV12→RGB
read (mirror the encoder's color tags, [[gpu-video-decode]] color-range work).
- **Non-Linux / no-VAAPI** — must cleanly run on the normal eframe device with software decode.
and well-tested. Selection must probe support per source, not assume.
- **Cache memory** — native-res GPU textures (esp. 4K) are large; the frame cache budget needs revisiting.
- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; the existing import handles NV12, but
10-bit/HDR sources (P010) need format handling.
- **Preview vs export sharing** — two live targets (preview res + export res) from the same source; the
cache/scaler design must serve both without thrashing.
## Verification
- Stage 0/1: visual — export above document res is now full-quality (not upscaled); profile shows

18
lightningbeam-ui/lightningbeam-core/src/action.rs
View File

@ -236,24 +236,6 @@ impl ActionExecutor {
Ok(())
}
/// Register an action whose effect has **already been applied** to the document (and backend)
/// outside the executor — e.g. a recording, which streams its content into the document live
/// over time and can't be applied by a single synchronous `execute()`.
///
/// Unlike `execute`, this does NOT run `execute()`/`execute_backend()` (the effect is already
/// present). It clears the redo stack, bumps the epoch (so the document reads as modified), and
/// pushes the action so it becomes undoable: undo runs `rollback`/`rollback_backend` to remove
/// the content, redo runs `execute`/`execute_backend` to bring it back. The action must be
/// constructed already in its post-execute state (see e.g. `AddClipInstanceAction::already_applied`).
pub fn push_applied(&mut self, action: Box<dyn Action>) {
self.redo_stack.clear();
self.epoch = self.epoch.wrapping_add(1);
self.undo_stack.push(action);
if self.undo_stack.len() > self.max_undo_depth {
self.undo_stack.remove(0);
}
}
/// Undo the last action
///
/// Returns Ok(true) if an action was undone, Ok(false) if undo stack is empty,

55
lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs
View File

@ -47,31 +47,6 @@ impl AddClipInstanceAction {
}
}
/// Construct the action in its **already-applied** state: the clip instance is already in the
/// document and its backend clip already exists (e.g. a finished recording). Pair with
/// `ActionExecutor::push_applied` so the recording becomes undoable without re-adding anything.
/// Undo will `rollback`/`rollback_backend` (removing the clip from doc + backend via the seeded
/// ids); redo re-adds it via the normal `execute`/`execute_backend` path.
pub fn already_applied(
layer_id: Uuid,
clip_instance: ClipInstance,
backend_track_id: daw_backend::TrackId,
backend_id: crate::action::BackendClipInstanceId,
) -> Self {
let (backend_midi_instance_id, backend_audio_instance_id) = match backend_id {
crate::action::BackendClipInstanceId::Midi(id) => (Some(id), None),
crate::action::BackendClipInstanceId::Audio(id) => (None, Some(id)),
};
Self {
layer_id,
clip_instance,
executed: true, // already present in the document
backend_track_id: Some(backend_track_id),
backend_midi_instance_id,
backend_audio_instance_id,
}
}
/// Get the ID of the clip instance that will be/was added
pub fn clip_instance_id(&self) -> Uuid {
self.clip_instance.id
@ -85,14 +60,13 @@ impl AddClipInstanceAction {
impl Action for AddClipInstanceAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
// Calculate the clip's effective duration in BEATS for overlap testing.
// `get_clip_duration` is the content length in seconds; the placement span
// must be beats (the timeline is beats-domain), so convert via the clip's
// typed helper rather than treating the seconds span as beats.
// Calculate the clip's effective duration
let clip_duration = document.get_clip_duration(&self.clip_instance.clip_id)
.ok_or_else(|| format!("Clip {} not found", self.clip_instance.clip_id))?;
let effective_duration = self.clip_instance
.effective_duration_beats(clip_duration, document.tempo_map());
let trim_start = self.clip_instance.trim_start;
let trim_end = self.clip_instance.trim_end.unwrap_or(clip_duration);
let effective_duration = trim_end - trim_start;
// Auto-adjust position for audio/video layers to avoid overlaps
let adjusted_start = document.find_nearest_valid_position(
@ -223,13 +197,12 @@ impl Action for AddClipInstanceAction {
// Calculate internal start/end from trim parameters
let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
let external_start = self.clip_instance.timeline_start;
// Calculate external duration (for looping if timeline_duration is set).
// MIDI trims are beats-domain, so the fallback span is beats too.
// Calculate external duration (for looping if timeline_duration is set)
let external_duration = self.clip_instance.timeline_duration
.unwrap_or(daw_backend::Beats(internal_end - internal_start));
.unwrap_or(internal_end - internal_start);
// Create MidiClipInstance
let instance = daw_backend::MidiClipInstance::new(
@ -237,8 +210,8 @@ impl Action for AddClipInstanceAction {
*midi_clip_id,
daw_backend::Beats(internal_start),
daw_backend::Beats(internal_end),
external_start,
external_duration,
daw_backend::Beats(external_start),
daw_backend::Beats(external_duration),
);
// Send query to add instance and get instance ID
@ -265,7 +238,7 @@ impl Action for AddClipInstanceAction {
// `trim_*` / `clip.duration` are in SECONDS (audio content time),
// while `timeline_*` and the backend's `duration` are in BEATS.
let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
let start_time = self.clip_instance.timeline_start;
// `effective_duration` is in BEATS. When `timeline_duration` is set
// it already is; otherwise the clip occupies its natural content
@ -274,8 +247,8 @@ impl Action for AddClipInstanceAction {
// the seconds-as-beats bug that made clips stop early off 60 BPM).
let effective_duration = self.clip_instance.timeline_duration.unwrap_or_else(|| {
let tempo_map = document.tempo_map();
let content_secs = daw_backend::Seconds(internal_end - internal_start);
tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(start_time) + content_secs)
let content_secs = internal_end - internal_start;
tempo_map.inverse_transform(tempo_map.transform(start_time) + content_secs)
- start_time
});
@ -284,7 +257,7 @@ impl Action for AddClipInstanceAction {
*audio_pool_index,
start_time,
effective_duration,
daw_backend::Seconds(internal_start),
internal_start,
);
self.backend_track_id = Some(*backend_track_id);

10
lightningbeam-ui/lightningbeam-core/src/actions/add_effect.rs
View File

@ -159,7 +159,7 @@ mod tests {
fn test_add_effect() {
let (mut document, layer_id, def) = create_test_setup();
let instance = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance = def.create_instance(0.0, 10.0);
let instance_id = instance.id;
let mut action = AddEffectAction::new(layer_id, instance);
@ -181,7 +181,7 @@ mod tests {
fn test_add_effect_rollback() {
let (mut document, layer_id, def) = create_test_setup();
let instance = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance = def.create_instance(0.0, 10.0);
let mut action = AddEffectAction::new(layer_id, instance);
action.execute(&mut document).unwrap();
@ -201,19 +201,19 @@ mod tests {
let (mut document, layer_id, def) = create_test_setup();
// Add first effect
let instance1 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance1 = def.create_instance(0.0, 10.0);
let id1 = instance1.id;
let mut action1 = AddEffectAction::new(layer_id, instance1);
action1.execute(&mut document).unwrap();
// Add second effect
let instance2 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance2 = def.create_instance(0.0, 10.0);
let id2 = instance2.id;
let mut action2 = AddEffectAction::new(layer_id, instance2);
action2.execute(&mut document).unwrap();
// Insert third effect at index 1 (between first and second)
let instance3 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance3 = def.create_instance(0.0, 10.0);
let id3 = instance3.id;
let mut action3 = AddEffectAction::at_index(layer_id, instance3, 1);
action3.execute(&mut document).unwrap();

20
lightningbeam-ui/lightningbeam-core/src/actions/loop_clip_instances.rs
View File

@ -9,9 +9,8 @@ use crate::layer::AnyLayer;
use std::collections::HashMap;
use uuid::Uuid;
/// Per-instance loop change: (instance_id, old_timeline_duration, new_timeline_duration, old_loop_before, new_loop_before).
/// All durations/offsets are in beats.
pub type LoopEntry = (Uuid, Option<daw_backend::Beats>, Option<daw_backend::Beats>, Option<daw_backend::Beats>, Option<daw_backend::Beats>);
/// Per-instance loop change: (instance_id, old_timeline_duration, new_timeline_duration, old_loop_before, new_loop_before)
pub type LoopEntry = (Uuid, Option<f64>, Option<f64>, Option<f64>, Option<f64>);
/// Action that changes the loop duration of clip instances
pub struct LoopClipInstancesAction {
@ -129,18 +128,11 @@ impl LoopClipInstancesAction {
};
let content_window = {
let trim_end = instance.trim_end.unwrap_or(clip.content_duration().native());
(trim_end - instance.trim_start).max(0.0) // seconds
let trim_end = instance.trim_end.unwrap_or(clip.duration);
(trim_end - instance.trim_start).max(0.0)
};
// Natural content length as a beats span at the clip's start (the
// fallback when no explicit timeline_duration is set).
let tempo_map = document.tempo_map();
let content_window_beats = tempo_map.seconds_to_beats(
tempo_map.beats_to_seconds(instance.timeline_start)
+ daw_backend::Seconds(content_window),
) - instance.timeline_start;
let right_duration = target_duration.unwrap_or(content_window_beats);
let left_duration = target_loop_before.unwrap_or(daw_backend::Beats::ZERO);
let right_duration = target_duration.unwrap_or(content_window);
let left_duration = target_loop_before.unwrap_or(0.0);
let external_duration = left_duration + right_duration;
let external_start = instance.timeline_start - left_duration;

42
lightningbeam-ui/lightningbeam-core/src/actions/move_clip_instances.rs
View File

@ -6,15 +6,13 @@ use crate::action::Action;
use crate::clip::ClipInstance;
use crate::document::Document;
use crate::layer::AnyLayer;
use daw_backend::Beats;
use std::collections::HashMap;
use uuid::Uuid;
/// Action that moves clip instances to new timeline positions
pub struct MoveClipInstancesAction {
/// Map of layer IDs to vectors of (clip_instance_id, old_timeline_start, new_timeline_start).
/// Timeline positions are in beats.
layer_moves: HashMap<Uuid, Vec<(Uuid, Beats, Beats)>>,
/// Map of layer IDs to vectors of (clip_instance_id, old_timeline_start, new_timeline_start)
layer_moves: HashMap<Uuid, Vec<(Uuid, f64, f64)>>,
}
impl MoveClipInstancesAction {
@ -22,8 +20,8 @@ impl MoveClipInstancesAction {
///
/// # Arguments
///
/// * `layer_moves` - Map of layer IDs to vectors of (clip_instance_id, old_timeline_start, new_timeline_start) in beats
pub fn new(layer_moves: HashMap<Uuid, Vec<(Uuid, Beats, Beats)>>) -> Self {
/// * `layer_moves` - Map of layer IDs to vectors of (clip_instance_id, old_timeline_start, new_timeline_start)
pub fn new(layer_moves: HashMap<Uuid, Vec<(Uuid, f64, f64)>>) -> Self {
Self { layer_moves }
}
}
@ -44,7 +42,7 @@ impl Action for MoveClipInstancesAction {
// Check if this instance is in a group
if let Some(group) = document.find_group_for_instance(instance_id) {
let offset = *new_start - *old_start;
let offset = new_start - old_start;
// Add all group members to the move list
for (member_layer_id, member_instance_id) in group.get_members() {
@ -79,7 +77,7 @@ impl Action for MoveClipInstancesAction {
}
// Auto-adjust moves to avoid overlaps
let mut adjusted_moves: HashMap<Uuid, Vec<(Uuid, Beats, Beats)>> = HashMap::new();
let mut adjusted_moves: HashMap<Uuid, Vec<(Uuid, f64, f64)>> = HashMap::new();
for (layer_id, moves) in &expanded_moves {
let layer = document.get_layer(layer_id)
@ -103,10 +101,10 @@ impl Action for MoveClipInstancesAction {
AnyLayer::Text(_) => &[],
};
let group: Vec<(Uuid, Beats, Beats)> = moves.iter().filter_map(|(id, old_start, _)| {
let group: Vec<(Uuid, f64, f64)> = moves.iter().filter_map(|(id, old_start, _)| {
let inst = clip_instances.iter().find(|ci| &ci.id == id)?;
let dur = document.get_clip_duration(&inst.clip_id)?;
let eff = inst.effective_duration_beats(dur, document.tempo_map());
let eff = inst.trim_end.unwrap_or(dur) - inst.trim_start;
Some((*id, *old_start, eff))
}).collect();
@ -114,7 +112,7 @@ impl Action for MoveClipInstancesAction {
let clamped = document.clamp_group_move_offset(layer_id, &group, desired_offset);
for (instance_id, old_start, _) in moves {
adjusted_layer_moves.push((*instance_id, *old_start, (*old_start + clamped).max(Beats::ZERO)));
adjusted_layer_moves.push((*instance_id, *old_start, (*old_start + clamped).max(0.0)));
}
adjusted_moves.insert(*layer_id, adjusted_layer_moves);
@ -210,9 +208,9 @@ impl Action for MoveClipInstancesAction {
if let Some(instance) = vl.clip_instances.iter().find(|ci| ci.id == *instance_id) {
// Check if this clip has a metatrack
if let Some(&metatrack_id) = backend.layer_to_track_map.get(&instance.clip_id) {
controller.set_offset(metatrack_id, document.tempo_map().beats_to_seconds(*new_start));
controller.set_trim_start(metatrack_id, daw_backend::Seconds(instance.trim_start));
controller.set_trim_end(metatrack_id, instance.trim_end.map(daw_backend::Seconds));
controller.set_offset(metatrack_id, *new_start);
controller.set_trim_start(metatrack_id, instance.trim_start);
controller.set_trim_end(metatrack_id, instance.trim_end);
}
}
}
@ -294,9 +292,9 @@ impl Action for MoveClipInstancesAction {
for (instance_id, old_start, _new_start) in moves {
if let Some(instance) = vl.clip_instances.iter().find(|ci| ci.id == *instance_id) {
if let Some(&metatrack_id) = backend.layer_to_track_map.get(&instance.clip_id) {
controller.set_offset(metatrack_id, document.tempo_map().beats_to_seconds(*old_start));
controller.set_trim_start(metatrack_id, daw_backend::Seconds(instance.trim_start));
controller.set_trim_end(metatrack_id, instance.trim_end.map(daw_backend::Seconds));
controller.set_offset(metatrack_id, *old_start);
controller.set_trim_start(metatrack_id, instance.trim_start);
controller.set_trim_end(metatrack_id, instance.trim_end);
}
}
}
@ -375,15 +373,15 @@ mod tests {
let mut vector_layer = VectorLayer::new("Layer 1");
let mut clip_instance = ClipInstance::new(clip_id);
clip_instance.timeline_start = Beats(1.0); // Start at beat 1
clip_instance.timeline_start = 1.0; // Start at 1 second
let instance_id = clip_instance.id;
vector_layer.clip_instances.push(clip_instance);
let layer_id = document.root.add_child(AnyLayer::Vector(vector_layer));
// Create move action: move from beat 1 to beat 5
// Create move action: move from 1.0 to 5.0 seconds
let mut layer_moves = HashMap::new();
layer_moves.insert(layer_id, vec![(instance_id, Beats(1.0), Beats(5.0))]);
layer_moves.insert(layer_id, vec![(instance_id, 1.0, 5.0)]);
let mut action = MoveClipInstancesAction::new(layer_moves);
@ -397,7 +395,7 @@ mod tests {
.iter()
.find(|ci| ci.id == instance_id)
.unwrap();
assert_eq!(instance.timeline_start, Beats(5.0));
assert_eq!(instance.timeline_start, 5.0);
}
// Rollback
@ -410,7 +408,7 @@ mod tests {
.iter()
.find(|ci| ci.id == instance_id)
.unwrap();
assert_eq!(instance.timeline_start, Beats(1.0));
assert_eq!(instance.timeline_start, 1.0);
}
}
}

27
lightningbeam-ui/lightningbeam-core/src/actions/remove_clip_instances.rs
View File

@ -170,20 +170,19 @@ impl Action for RemoveClipInstancesAction {
use daw_backend::command::{Query, QueryResponse};
let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = instance.trim_end.unwrap_or(clip.duration);
let external_start = instance.timeline_start;
// MIDI trims are beats-domain, so the fallback span is beats too.
let external_duration = instance
.timeline_duration
.unwrap_or(daw_backend::Beats(internal_end - internal_start));
.unwrap_or(internal_end - internal_start);
let midi_instance = daw_backend::MidiClipInstance::new(
0,
*midi_clip_id,
daw_backend::Beats(internal_start),
daw_backend::Beats(internal_end),
external_start,
external_duration,
daw_backend::Beats(external_start),
daw_backend::Beats(external_duration),
);
let query = Query::AddMidiClipInstanceSync(track_id, midi_instance);
@ -198,23 +197,17 @@ impl Action for RemoveClipInstancesAction {
}
AudioClipType::Sampled { audio_pool_index } => {
let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = instance.trim_end.unwrap_or(clip.duration);
let effective_duration = instance.timeline_duration
.unwrap_or(internal_end - internal_start);
let start_time = instance.timeline_start;
// Fallback span is the content seconds converted to beats at the
// clip's start (not the seconds span treated as beats).
let effective_duration = instance.timeline_duration.unwrap_or_else(|| {
let tempo_map = document.tempo_map();
let content_secs = daw_backend::Seconds(internal_end - internal_start);
tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(start_time) + content_secs)
- start_time
});
let new_id = controller.add_audio_clip(
track_id,
*audio_pool_index,
start_time,
effective_duration,
daw_backend::Seconds(internal_start),
internal_start,
);
backend.clip_instance_to_backend_map.insert(
instance.id,
@ -245,11 +238,11 @@ mod tests {
let mut vector_layer = VectorLayer::new("Layer 1");
let mut ci1 = ClipInstance::new(clip_id);
ci1.timeline_start = daw_backend::Beats::ZERO;
ci1.timeline_start = 0.0;
let id1 = ci1.id;
let mut ci2 = ClipInstance::new(clip_id);
ci2.timeline_start = daw_backend::Beats(5.0);
ci2.timeline_start = 5.0;
let id2 = ci2.id;
vector_layer.clip_instances.push(ci1);

10
lightningbeam-ui/lightningbeam-core/src/actions/remove_effect.rs
View File

@ -134,7 +134,7 @@ mod tests {
let (mut document, layer_id, def) = create_test_setup();
// Add an effect first
let instance = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance = def.create_instance(0.0, 10.0);
let instance_id = instance.id;
if let Some(AnyLayer::Effect(el)) = document.get_layer_mut(&layer_id) {
@ -161,7 +161,7 @@ mod tests {
let (mut document, layer_id, def) = create_test_setup();
// Add an effect first
let instance = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance = def.create_instance(0.0, 10.0);
let instance_id = instance.id;
if let Some(AnyLayer::Effect(el)) = document.get_layer_mut(&layer_id) {
@ -185,11 +185,11 @@ mod tests {
let (mut document, layer_id, def) = create_test_setup();
// Add three effects
let instance1 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance1 = def.create_instance(0.0, 10.0);
let id1 = instance1.id;
let instance2 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance2 = def.create_instance(0.0, 10.0);
let id2 = instance2.id;
let instance3 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let instance3 = def.create_instance(0.0, 10.0);
let id3 = instance3.id;
if let Some(AnyLayer::Effect(el)) = document.get_layer_mut(&layer_id) {

4
lightningbeam-ui/lightningbeam-core/src/actions/set_keyframe.rs
View File

@ -81,11 +81,9 @@ impl Action for SetKeyframeAction {
if let Some(AnyLayer::Vector(vl)) = document.get_layer(&self.layer_id) {
for clip_id in &self.clip_instance_ids {
if let Some(ci) = vl.clip_instances.iter().find(|c| c.id == *clip_id) {
// `start` is a keyframe time in seconds; group_visibility_start returns seconds,
// so the fallback must convert the clip's beats start to seconds too.
let start = vl
.group_visibility_start(clip_id, self.time)
.unwrap_or_else(|| document.tempo_map().beats_to_seconds(ci.timeline_start).seconds_to_f64());
.unwrap_or(ci.timeline_start);
clip_info.insert(*clip_id, (ci.transform.clone(), ci.opacity, start));
}
}

75
lightningbeam-ui/lightningbeam-core/src/actions/split_clip_instance.rs
View File

@ -17,8 +17,8 @@ pub struct SplitClipInstanceAction {
/// The clip instance to split
instance_id: Uuid,
/// Timeline position where to split (in beats)
split_time: daw_backend::Beats,
/// Timeline time where to split (in seconds)
split_time: f64,
/// Whether the action has been executed (for rollback)
executed: bool,
@ -26,8 +26,8 @@ pub struct SplitClipInstanceAction {
// Stored during execute for rollback
/// Original trim_end value of the left (original) instance
original_trim_end: Option<f64>,
/// Original timeline_duration value of the left (original) instance (beats)
original_timeline_duration: Option<daw_backend::Beats>,
/// Original timeline_duration value of the left (original) instance
original_timeline_duration: Option<f64>,
/// ID of the new (right) instance created by the split
new_instance_id: Option<Uuid>,
@ -47,8 +47,8 @@ impl SplitClipInstanceAction {
///
/// * `layer_id` - The ID of the layer containing the clip instance
/// * `instance_id` - The ID of the clip instance to split
/// * `split_time` - The timeline position (in beats) where to split
pub fn new(layer_id: Uuid, instance_id: Uuid, split_time: daw_backend::Beats) -> Self {
/// * `split_time` - The timeline time (in seconds) where to split
pub fn new(layer_id: Uuid, instance_id: Uuid, split_time: f64) -> Self {
Self {
layer_id,
instance_id,
@ -72,9 +72,9 @@ impl SplitClipInstanceAction {
///
/// * `layer_id` - The ID of the layer containing the clip instance
/// * `instance_id` - The ID of the clip instance to split
/// * `split_time` - The timeline position (in beats) where to split
/// * `split_time` - The timeline time (in seconds) where to split
/// * `new_instance_id` - The UUID to use for the new (right) clip instance
pub fn with_new_instance_id(layer_id: Uuid, instance_id: Uuid, split_time: daw_backend::Beats, new_instance_id: Uuid) -> Self {
pub fn with_new_instance_id(layer_id: Uuid, instance_id: Uuid, split_time: f64, new_instance_id: Uuid) -> Self {
Self {
layer_id,
instance_id,
@ -132,9 +132,9 @@ impl Action for SplitClipInstanceAction {
let timeline_end = instance.timeline_start + effective_duration;
// Validate: split_time must be strictly within the clip's timeline span
let epsilon = daw_backend::Beats(0.001); // ~1ms tolerance
if self.split_time <= instance.timeline_start + epsilon
|| self.split_time >= timeline_end - epsilon
const EPSILON: f64 = 0.001; // 1ms tolerance
if self.split_time <= instance.timeline_start + EPSILON
|| self.split_time >= timeline_end - EPSILON
{
return Err(format!(
"Split time {} must be within clip bounds ({} to {})",
@ -146,27 +146,21 @@ impl Action for SplitClipInstanceAction {
self.original_trim_end = instance.trim_end;
self.original_timeline_duration = instance.timeline_duration;
// Check if this is a looping clip. `content_duration` is a trim-domain
// span (seconds), so `clip_duration` must be unwrapped as seconds.
// Check if this is a looping clip
let is_looping = instance.timeline_duration.is_some();
let content_duration = instance.trim_end.unwrap_or(clip_duration.seconds_to_f64()) - instance.trim_start;
let content_duration = instance.trim_end.unwrap_or(clip_duration) - instance.trim_start;
// Timeline split point (beats).
// Calculate the split point
let time_into_clip = self.split_time - instance.timeline_start;
let left_duration = time_into_clip;
let right_duration = effective_duration - left_duration;
// How far the split lands into the clip's *content* (seconds, trim domain).
let tempo_map = document.tempo_map();
let time_into_clip_secs = (tempo_map.beats_to_seconds(self.split_time)
- tempo_map.beats_to_seconds(instance.timeline_start)).seconds_to_f64();
// Calculate content split time (seconds)
// Calculate content split time
let content_split_time = if is_looping {
// For looping clips, wrap around content
instance.trim_start + (time_into_clip_secs % content_duration)
instance.trim_start + (time_into_clip % content_duration)
} else {
instance.trim_start + time_into_clip_secs
instance.trim_start + time_into_clip
};
// Clone the instance for the right side
@ -377,7 +371,7 @@ impl Action for SplitClipInstanceAction {
// 1. Trim the original (left) instance
let orig_internal_start = original_instance.trim_start;
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native());
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration);
// Look up the original backend instance ID
if let Some(crate::action::BackendClipInstanceId::Midi(orig_backend_id)) =
@ -388,20 +382,19 @@ impl Action for SplitClipInstanceAction {
// 2. Add the new (right) instance
let internal_start = new_instance.trim_start;
let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = new_instance.trim_end.unwrap_or(clip.duration);
let external_start = new_instance.timeline_start;
// MIDI trims are beats-domain, so the fallback span is beats too.
let external_duration = new_instance
.timeline_duration
.unwrap_or(daw_backend::Beats(internal_end - internal_start));
.unwrap_or(internal_end - internal_start);
let instance = daw_backend::MidiClipInstance::new(
0,
*midi_clip_id,
daw_backend::Beats(internal_start),
daw_backend::Beats(internal_end),
external_start,
external_duration,
daw_backend::Beats(external_start),
daw_backend::Beats(external_duration),
);
let query = Query::AddMidiClipInstanceSync(*backend_track_id, instance);
@ -425,7 +418,7 @@ impl Action for SplitClipInstanceAction {
AudioClipType::Sampled { audio_pool_index } => {
// 1. Trim the original (left) instance
let orig_internal_start = original_instance.trim_start;
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.content_duration().native());
let orig_internal_end = original_instance.trim_end.unwrap_or(clip.duration);
// Look up the original backend instance ID
if let Some(crate::action::BackendClipInstanceId::Audio(orig_backend_id)) =
@ -436,23 +429,17 @@ impl Action for SplitClipInstanceAction {
// 2. Add the new (right) instance
let internal_start = new_instance.trim_start;
let internal_end = new_instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = new_instance.trim_end.unwrap_or(clip.duration);
let effective_duration = new_instance.timeline_duration
.unwrap_or(internal_end - internal_start);
let start_time = new_instance.timeline_start;
// Fallback span is the content seconds converted to beats at the
// clip's start (not the seconds span treated as beats).
let effective_duration = new_instance.timeline_duration.unwrap_or_else(|| {
let tempo_map = document.tempo_map();
let content_secs = daw_backend::Seconds(internal_end - internal_start);
tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(start_time) + content_secs)
- start_time
});
let instance_id = controller.add_audio_clip(
*backend_track_id,
*audio_pool_index,
start_time,
effective_duration,
daw_backend::Seconds(internal_start),
internal_start,
);
self.backend_track_id = Some(*backend_track_id);
@ -499,7 +486,7 @@ impl Action for SplitClipInstanceAction {
if let Some(instance) = al.clip_instances.iter().find(|ci| ci.id == self.instance_id) {
if let Some(clip) = document.get_audio_clip(&instance.clip_id) {
let orig_internal_start = instance.trim_start;
let orig_internal_end = self.original_trim_end.unwrap_or(clip.content_duration().native());
let orig_internal_end = self.original_trim_end.unwrap_or(clip.duration);
// Restore based on clip type
use crate::clip::AudioClipType;
@ -553,7 +540,7 @@ mod tests {
// Create a clip instance at timeline 0, with trim 0-10 (10 seconds)
let mut clip_instance = ClipInstance::new(clip_id);
clip_instance.timeline_start = daw_backend::Beats::ZERO;
clip_instance.timeline_start = 0.0;
clip_instance.trim_start = 0.0;
clip_instance.trim_end = Some(10.0);
let instance_id = clip_instance.id;
@ -562,7 +549,7 @@ mod tests {
let layer_id = document.root.add_child(AnyLayer::Vector(vector_layer));
// Split at timeline 5.0
let mut action = SplitClipInstanceAction::new(layer_id, instance_id, daw_backend::Beats(5.0));
let mut action = SplitClipInstanceAction::new(layer_id, instance_id, 5.0);
// Execute - this will fail because we don't have a real clip in the document
// In a real test, we'd need to add a VectorClip first
@ -572,7 +559,7 @@ mod tests {
#[test]
fn test_split_action_description() {
let action = SplitClipInstanceAction::new(Uuid::new_v4(), Uuid::new_v4(), daw_backend::Beats(5.0));
let action = SplitClipInstanceAction::new(Uuid::new_v4(), Uuid::new_v4(), 5.0);
assert_eq!(action.description(), "Split clip instance");
}
}

76
lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs
View File

@ -6,7 +6,6 @@ use crate::action::Action;
use crate::clip::ClipInstance;
use crate::document::Document;
use crate::layer::AnyLayer;
use daw_backend::{Beats, Seconds};
use std::collections::HashMap;
use uuid::Uuid;
@ -33,14 +32,14 @@ pub struct TrimData {
/// For TrimLeft: trim_start value
/// For TrimRight: trim_end value (Option because it can be None)
pub trim_value: Option<f64>,
/// For TrimLeft: timeline_start value (where the clip appears on timeline, beats)
/// For TrimLeft: timeline_start value (where the clip appears on timeline)
/// For TrimRight: unused (None)
pub timeline_start: Option<Beats>,
pub timeline_start: Option<f64>,
}
impl TrimData {
/// Create TrimData for left trim
pub fn left(trim_start: f64, timeline_start: Beats) -> Self {
pub fn left(trim_start: f64, timeline_start: f64) -> Self {
Self {
trim_value: Some(trim_start),
timeline_start: Some(timeline_start),
@ -204,60 +203,51 @@ impl Action for TrimClipInstancesAction {
{
// If extending to the left (new_trim < old_trim)
if should_validate && new_trim < old_trim {
// Max leftward extension as content seconds (the gap's wall-clock span).
let max_extend_secs = document.find_max_trim_extend_left(
// Find the maximum we can extend left
let max_extend = document.find_max_trim_extend_left(
layer_id,
instance_id,
instance.timeline_start,
).seconds_to_f64();
);
// Calculate how much we want to extend (content seconds)
// Calculate how much we want to extend
let desired_extend = old_trim - new_trim;
// Clamp to max allowed
let actual_extend = desired_extend.min(max_extend_secs);
let actual_extend = desired_extend.min(max_extend);
let clamped_trim_start = old_trim - actual_extend;
// Move the timeline left by the same wall-clock seconds.
let tempo_map = document.tempo_map();
let clamped_timeline_start = tempo_map
.seconds_to_beats(tempo_map.beats_to_seconds(old_timeline) - Seconds(actual_extend))
.max(Beats::ZERO);
let clamped_timeline_start = (old_timeline - actual_extend).max(0.0);
clamped_new = TrimData::left(clamped_trim_start, clamped_timeline_start);
}
}
}
TrimType::TrimRight => {
let old_trim_end = old.trim_value.unwrap_or(clip_duration.seconds_to_f64());
let new_trim_end = new.trim_value.unwrap_or(clip_duration.seconds_to_f64());
let old_trim_end = old.trim_value.unwrap_or(clip_duration);
let new_trim_end = new.trim_value.unwrap_or(clip_duration);
// If extending to the right (new_trim_end > old_trim_end)
if should_validate && new_trim_end > old_trim_end {
let tempo_map = document.tempo_map();
// Current effective duration in beats (content seconds
// converted to beats at the clip's start).
let content_secs = Seconds(old_trim_end - instance.trim_start);
let current_effective_duration = tempo_map.seconds_to_beats(
tempo_map.beats_to_seconds(instance.timeline_start) + content_secs,
) - instance.timeline_start;
// Calculate current effective duration
let current_effective_duration = old_trim_end - instance.trim_start;
// Max rightward extension as content seconds (the gap's wall-clock span).
let max_extend_secs = document.find_max_trim_extend_right(
// Find the maximum we can extend right
let max_extend = document.find_max_trim_extend_right(
layer_id,
instance_id,
instance.timeline_start,
current_effective_duration,
).seconds_to_f64();
);
// Calculate how much we want to extend (content seconds)
// Calculate how much we want to extend
let desired_extend = new_trim_end - old_trim_end;
// Clamp to max allowed
let actual_extend = desired_extend.min(max_extend_secs);
let actual_extend = desired_extend.min(max_extend);
let clamped_trim_end = old_trim_end + actual_extend;
// Don't exceed clip duration
let final_trim_end = clamped_trim_end.min(clip_duration.seconds_to_f64());
let final_trim_end = clamped_trim_end.min(clip_duration);
clamped_new = TrimData::right(Some(final_trim_end));
}
@ -386,9 +376,9 @@ impl Action for TrimClipInstancesAction {
if let Some(instance) = vl.clip_instances.iter().find(|ci| ci.id == *instance_id) {
if let Some(&metatrack_id) = backend.layer_to_track_map.get(&instance.clip_id) {
// Instance already has new values after execute()
controller.set_offset(metatrack_id, document.tempo_map().beats_to_seconds(instance.timeline_start));
controller.set_trim_start(metatrack_id, daw_backend::Seconds(instance.trim_start));
controller.set_trim_end(metatrack_id, instance.trim_end.map(daw_backend::Seconds));
controller.set_offset(metatrack_id, instance.timeline_start);
controller.set_trim_start(metatrack_id, instance.trim_start);
controller.set_trim_end(metatrack_id, instance.trim_end);
}
}
}
@ -424,7 +414,7 @@ impl Action for TrimClipInstancesAction {
// Calculate new internal_start and internal_end for backend
// Note: instance already has the new trim values after execute()
let internal_start = instance.trim_start;
let internal_end = instance.trim_end.unwrap_or(clip.content_duration().native());
let internal_end = instance.trim_end.unwrap_or(clip.duration);
// Handle trim based on clip type
match &clip.clip_type {
@ -476,9 +466,9 @@ impl Action for TrimClipInstancesAction {
if let Some(instance) = vl.clip_instances.iter().find(|ci| ci.id == *instance_id) {
if let Some(&metatrack_id) = backend.layer_to_track_map.get(&instance.clip_id) {
// Instance already has old values after rollback()
controller.set_offset(metatrack_id, document.tempo_map().beats_to_seconds(instance.timeline_start));
controller.set_trim_start(metatrack_id, daw_backend::Seconds(instance.trim_start));
controller.set_trim_end(metatrack_id, instance.trim_end.map(daw_backend::Seconds));
controller.set_offset(metatrack_id, instance.timeline_start);
controller.set_trim_start(metatrack_id, instance.trim_start);
controller.set_trim_end(metatrack_id, instance.trim_end);
}
}
}
@ -517,8 +507,8 @@ impl Action for TrimClipInstancesAction {
TrimType::TrimRight => instance.trim_start, // trim_start wasn't changed
};
let internal_end = match trim_type {
TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.content_duration().native()), // trim_end wasn't changed
TrimType::TrimRight => old.trim_value.unwrap_or(clip.content_duration().native()),
TrimType::TrimLeft => instance.trim_end.unwrap_or(clip.duration), // trim_end wasn't changed
TrimType::TrimRight => old.trim_value.unwrap_or(clip.duration),
};
// Handle trim based on clip type
@ -568,7 +558,7 @@ mod tests {
let mut vector_layer = VectorLayer::new("Layer 1");
let mut clip_instance = ClipInstance::new(clip_id);
clip_instance.timeline_start = Beats::ZERO;
clip_instance.timeline_start = 0.0;
clip_instance.trim_start = 0.0;
let instance_id = clip_instance.id;
vector_layer.clip_instances.push(clip_instance);
@ -582,8 +572,8 @@ mod tests {
vec![(
instance_id,
TrimType::TrimLeft,
TrimData::left(0.0, Beats::ZERO),
TrimData::left(2.0, Beats(2.0)),
TrimData::left(0.0, 0.0),
TrimData::left(2.0, 2.0),
)],
);
@ -600,7 +590,7 @@ mod tests {
.find(|ci| ci.id == instance_id)
.unwrap();
assert_eq!(instance.trim_start, 2.0);
assert_eq!(instance.timeline_start, Beats(2.0));
assert_eq!(instance.timeline_start, 2.0);
}
// Rollback
@ -614,7 +604,7 @@ mod tests {
.find(|ci| ci.id == instance_id)
.unwrap();
assert_eq!(instance.trim_start, 0.0);
assert_eq!(instance.timeline_start, Beats::ZERO);
assert_eq!(instance.timeline_start, 0.0);
}
}

11
lightningbeam-ui/lightningbeam-core/src/actions/update_midi_notes.rs
View File

@ -1,13 +1,7 @@
use crate::action::Action;
use crate::document::Document;
use daw_backend::Beats;
use uuid::Uuid;
/// Convert editor-side beats-domain note tuples to the typed backend form.
fn notes_to_beats(notes: &[(f64, u8, u8, f64)]) -> Vec<(Beats, u8, u8, Beats)> {
notes.iter().map(|&(t, n, v, d)| (Beats(t), n, v, Beats(d))).collect()
}
/// Action to update MIDI notes in a clip (supports undo/redo)
///
/// Stores the before and after note states. MIDI note data lives in the backend,
@ -55,8 +49,7 @@ impl Action for UpdateMidiNotesAction {
.get(&self.layer_id)
.ok_or_else(|| format!("Layer {} not mapped to backend track", self.layer_id))?;
// Note times/durations are beats (MIDI content domain); assert that at the typed boundary.
controller.update_midi_clip_notes(*track_id, self.midi_clip_id, notes_to_beats(&self.new_notes));
controller.update_midi_clip_notes(*track_id, self.midi_clip_id, self.new_notes.clone());
Ok(())
}
@ -75,7 +68,7 @@ impl Action for UpdateMidiNotesAction {
.get(&self.layer_id)
.ok_or_else(|| format!("Layer {} not mapped to backend track", self.layer_id))?;
controller.update_midi_clip_notes(*track_id, self.midi_clip_id, notes_to_beats(&self.old_notes));
controller.update_midi_clip_notes(*track_id, self.midi_clip_id, self.old_notes.clone());
Ok(())
}

150
lightningbeam-ui/lightningbeam-core/src/clip.rs
View File

@ -14,7 +14,6 @@
use crate::layer::AnyLayer;
use crate::layer_tree::LayerTree;
use crate::object::Transform;
use daw_backend::{Beats, Seconds};
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use uuid::Uuid;
@ -111,7 +110,7 @@ impl VectorClip {
pub fn content_duration_with(&self, framerate: f64, tempo_map: &crate::tempo_map::TempoMap, clip_duration_fn: impl Fn(&Uuid) -> Option<f64>) -> f64 {
let frame_duration = 1.0 / framerate;
// Work in beats, convert to seconds at the end.
let mut last_beats: Option<Beats> = None;
let mut last_beats: Option<f64> = None;
let mut last_secs: Option<f64> = None;
for layer_node in self.layers.iter() {
@ -127,18 +126,18 @@ impl VectorClip {
};
for ci in clip_instances {
// Compute end position of this clip instance in beats
let end_beats: Beats = if let Some(td_beats) = ci.timeline_duration {
let end_beats = if let Some(td_beats) = ci.timeline_duration {
ci.timeline_start + td_beats
} else if let Some(te) = ci.trim_end {
let secs = (te - ci.trim_start).max(0.0);
tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(ci.timeline_start) + Seconds(secs))
ci.timeline_start + tempo_map.inverse_transform(tempo_map.transform(ci.timeline_start) + secs) - ci.timeline_start
} else if let Some(clip_dur_secs) = clip_duration_fn(&ci.clip_id) {
let secs = (clip_dur_secs - ci.trim_start).max(0.0);
tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(ci.timeline_start) + Seconds(secs))
ci.timeline_start + tempo_map.inverse_transform(tempo_map.transform(ci.timeline_start) + secs) - ci.timeline_start
} else {
continue;
};
last_beats = Some(last_beats.map_or(end_beats, |t: Beats| t.max(end_beats)));
last_beats = Some(last_beats.map_or(end_beats, |t: f64| t.max(end_beats)));
}
// Vector layer keyframes are in seconds
@ -149,7 +148,7 @@ impl VectorClip {
}
}
let from_clips = last_beats.map(|b| tempo_map.beats_to_seconds(b).seconds_to_f64());
let from_clips = last_beats.map(|b| tempo_map.transform(b));
let combined = match (from_clips, last_secs) {
(Some(a), Some(b)) => Some(a.max(b)),
(Some(a), None) => Some(a),
@ -200,7 +199,7 @@ impl VectorClip {
for clip_instance in &vector_layer.clip_instances {
// Convert parent clip time (seconds) to nested clip local time (seconds).
// timeline_start is in beats; convert to seconds using document BPM.
let start_secs = document.tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let start_secs = document.tempo_map().transform(clip_instance.timeline_start);
let nested_clip_time = ((clip_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
// Look up the nested clip definition
@ -468,37 +467,6 @@ pub enum AudioClipType {
Recording,
}
/// A clip's content duration, tagged by its native unit.
///
/// Sampled/recording audio and video measure content in wall-clock **seconds**; MIDI measures
/// it in **beats** (tempo-independent musical length). Carrying the domain in the type means a
/// duration can't be silently read in the wrong unit.
#[derive(Clone, Copy, Debug, PartialEq)]
pub enum ClipDuration {
Seconds(Seconds),
Beats(Beats),
}
impl ClipDuration {
/// Wall-clock seconds. Beats are converted as a length from beat 0 (exact under constant
/// tempo; a reasonable approximation otherwise — durations are position-independent here).
pub fn to_seconds(self, tempo_map: &daw_backend::TempoMap) -> Seconds {
match self {
ClipDuration::Seconds(s) => s,
ClipDuration::Beats(b) => tempo_map.beats_to_seconds(b),
}
}
/// The raw magnitude in the clip's native unit. Use only in code that already works in that
/// domain (e.g. trim math, whose values share the clip's native domain).
pub fn native(self) -> f64 {
match self {
ClipDuration::Seconds(s) => s.seconds_to_f64(),
ClipDuration::Beats(b) => b.beats_to_f64(),
}
}
}
/// Audio clip
///
/// This is compatible with daw-backend's audio system:
@ -512,13 +480,9 @@ pub struct AudioClip {
/// Clip name
pub name: String,
/// Raw content duration in the clip's **native domain** — SECONDS for sampled/recording
/// audio, BEATS for MIDI (musical length, tempo-independent). Private on purpose: the domain
/// depends on `clip_type`, so all access goes through [`AudioClip::content_duration`] /
/// [`AudioClip::set_content_duration`], which keep it type-safe. Stored as a bare `f64`
/// because the `.beam` format serializes it as a plain number (serde derives over private
/// fields fine); a domain-tagged newtype would change the on-disk shape.
duration: f64,
/// Duration in seconds
/// For sampled audio, this can be set to trim the audio shorter than the source file
pub duration: f64,
/// Audio clip type (sampled or MIDI)
pub clip_type: AudioClipType,
@ -529,31 +493,6 @@ pub struct AudioClip {
}
impl AudioClip {
/// The clip's content duration, tagged with its native domain (seconds for sampled/recording,
/// beats for MIDI). This is the only sanctioned way to read the raw `duration` field.
pub fn content_duration(&self) -> ClipDuration {
match self.clip_type {
AudioClipType::Midi { .. } => ClipDuration::Beats(Beats(self.duration)),
AudioClipType::Sampled { .. } | AudioClipType::Recording => {
ClipDuration::Seconds(Seconds(self.duration))
}
}
}
/// Set the content duration. Debug-asserts the value's domain matches the clip type so a
/// beats duration can't be stored on a seconds clip (or vice-versa).
pub fn set_content_duration(&mut self, duration: ClipDuration) {
debug_assert!(
matches!(
(&self.clip_type, duration),
(AudioClipType::Midi { .. }, ClipDuration::Beats(_))
| (AudioClipType::Sampled { .. } | AudioClipType::Recording, ClipDuration::Seconds(_))
),
"clip duration domain must match clip type",
);
self.duration = duration.native();
}
/// Create a new sampled audio clip
///
/// # Arguments
@ -708,12 +647,12 @@ pub struct ClipInstance {
/// When this instance starts on the timeline, in **beats**.
/// Default: 0.0
pub timeline_start: Beats,
pub timeline_start: f64,
/// How long this instance appears on the timeline, in **beats**.
/// If set and longer than the trimmed content, the content will loop.
/// Default: None (use trimmed clip duration, no looping)
pub timeline_duration: Option<Beats>,
pub timeline_duration: Option<f64>,
/// Trim start: offset into the clip's internal content, in **seconds**.
/// - For audio: byte-offset into the audio file
@ -740,7 +679,7 @@ pub struct ClipInstance {
/// When set, loop iterations are drawn/played before the content start.
/// Default: None (no pre-loop)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub loop_before: Option<Beats>,
pub loop_before: Option<f64>,
}
/// High 64-bit sentinel used to identify UUIDs that encode a backend audio clip instance ID.
@ -792,7 +731,7 @@ impl ClipInstance {
transform: Transform::default(),
opacity: 1.0,
name: None,
timeline_start: Beats::ZERO,
timeline_start: 0.0,
timeline_duration: None,
trim_start: 0.0,
trim_end: None,
@ -810,7 +749,7 @@ impl ClipInstance {
transform: Transform::default(),
opacity: 1.0,
name: None,
timeline_start: Beats::ZERO,
timeline_start: 0.0,
timeline_duration: None,
trim_start: 0.0,
trim_end: None,
@ -845,8 +784,8 @@ impl ClipInstance {
self
}
/// Set timeline position (beats).
pub fn with_timeline_start(mut self, timeline_start: Beats) -> Self {
/// Set timeline position
pub fn with_timeline_start(mut self, timeline_start: f64) -> Self {
self.timeline_start = timeline_start;
self
}
@ -871,78 +810,79 @@ impl ClipInstance {
}
/// Set explicit timeline duration (in beats) by directly setting `timeline_duration`.
pub fn with_timeline_duration(mut self, duration_beats: Beats) -> Self {
pub fn with_timeline_duration(mut self, duration_beats: f64) -> Self {
self.timeline_duration = Some(duration_beats);
self
}
/// Content window size in seconds: `trim_end - trim_start`.
/// Used for internal looping calculations.
pub fn content_window_secs(&self, clip_duration_secs: Seconds) -> Seconds {
let end = self.trim_end.unwrap_or(clip_duration_secs.seconds_to_f64());
Seconds((end - self.trim_start).max(0.0))
pub fn content_window_secs(&self, clip_duration_secs: f64) -> f64 {
let end = self.trim_end.unwrap_or(clip_duration_secs);
(end - self.trim_start).max(0.0)
}
/// How long this instance appears on the timeline, in **beats**.
///
/// If `timeline_duration` is set, returns that (enabling content looping).
/// Otherwise converts the content window from seconds to beats using the tempo map.
pub fn effective_duration_beats(&self, clip_duration_secs: Seconds, tempo_map: &crate::tempo_map::TempoMap) -> Beats {
pub fn effective_duration_beats(&self, clip_duration_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> f64 {
if let Some(td) = self.timeline_duration {
return td;
}
let window = self.content_window_secs(clip_duration_secs);
let start_secs = tempo_map.beats_to_seconds(self.timeline_start);
tempo_map.seconds_to_beats(start_secs + window) - self.timeline_start
let window_secs = self.content_window_secs(clip_duration_secs);
let start_secs = tempo_map.transform(self.timeline_start);
tempo_map.inverse_transform(start_secs + window_secs) - self.timeline_start
}
/// Left edge of the clip's visual extent on the timeline, in **beats**.
pub fn effective_start(&self) -> Beats {
self.timeline_start - self.loop_before.unwrap_or(Beats::ZERO)
pub fn effective_start(&self) -> f64 {
self.timeline_start - self.loop_before.unwrap_or(0.0)
}
/// Total visual duration (loop_before + effective_duration), in **beats**.
pub fn total_duration(&self, clip_duration_secs: Seconds, tempo_map: &crate::tempo_map::TempoMap) -> Beats {
self.loop_before.unwrap_or(Beats::ZERO) + self.effective_duration_beats(clip_duration_secs, tempo_map)
pub fn total_duration(&self, clip_duration_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> f64 {
self.loop_before.unwrap_or(0.0) + self.effective_duration_beats(clip_duration_secs, tempo_map)
}
/// Map a playback time (in **seconds**) to clip-local content time (in **seconds**).
///
/// Returns `None` if the clip instance is not active at `time_secs`.
pub fn remap_time_secs(&self, time: Seconds, clip_duration_secs: Seconds, tempo_map: &crate::tempo_map::TempoMap) -> Option<Seconds> {
let start_secs = tempo_map.beats_to_seconds(self.timeline_start);
pub fn remap_time_secs(&self, time_secs: f64, clip_duration_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> Option<f64> {
let start_secs = tempo_map.transform(self.timeline_start);
let dur_beats = self.effective_duration_beats(clip_duration_secs, tempo_map);
let end_secs = tempo_map.beats_to_seconds(self.timeline_start + dur_beats);
let end_secs = tempo_map.transform(self.timeline_start + dur_beats);
if time < start_secs || time >= end_secs {
if time_secs < start_secs || time_secs >= end_secs {
return None;
}
let content_time = (time - start_secs) * self.playback_speed;
let relative_secs = time_secs - start_secs;
let content_time = relative_secs * self.playback_speed;
let content_window = self.content_window_secs(clip_duration_secs);
if content_window == Seconds::ZERO {
return Some(Seconds(self.trim_start));
if content_window == 0.0 {
return Some(self.trim_start);
}
let looped = if content_time > content_window {
let looped_time = if content_time > content_window {
content_time % content_window
} else {
content_time
};
Some(Seconds(self.trim_start) + looped)
Some(self.trim_start + looped_time)
}
/// Alias for `remap_time_secs`.
#[inline]
pub fn remap_time(&self, time: Seconds, clip_duration_secs: Seconds, tempo_map: &crate::tempo_map::TempoMap) -> Option<Seconds> {
self.remap_time_secs(time, clip_duration_secs, tempo_map)
pub fn remap_time(&self, time_secs: f64, clip_duration_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> Option<f64> {
self.remap_time_secs(time_secs, clip_duration_secs, tempo_map)
}
/// Alias for `effective_duration_beats`.
#[inline]
pub fn effective_duration(&self, clip_duration_secs: Seconds, tempo_map: &crate::tempo_map::TempoMap) -> Beats {
pub fn effective_duration(&self, clip_duration_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> f64 {
self.effective_duration_beats(clip_duration_secs, tempo_map)
}
@ -1019,7 +959,7 @@ mod tests {
assert_eq!(instance.clip_id, clip_id);
assert_eq!(instance.opacity, 1.0);
assert_eq!(instance.timeline_start, Beats::ZERO);
assert_eq!(instance.timeline_start, 0.0);
assert_eq!(instance.trim_start, 0.0);
assert_eq!(instance.trim_end, None);
assert_eq!(instance.playback_speed, 1.0);
@ -1037,7 +977,7 @@ mod tests {
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
// beats-domain effective duration equals the seconds content window.
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(Seconds(10.0), &tempo_map), Beats(6.0));
assert_eq!(instance.effective_duration(10.0, &tempo_map), 6.0);
}
#[test]
@ -1050,7 +990,7 @@ mod tests {
assert_eq!(instance.trim_end, None);
// At 60 BPM the tempo map is identity (1 beat == 1 second).
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(Seconds(10.0), &tempo_map), Beats(8.0));
assert_eq!(instance.effective_duration(10.0, &tempo_map), 8.0);
}
#[test]

156
lightningbeam-ui/lightningbeam-core/src/document.rs
View File

@ -5,7 +5,6 @@
use crate::asset_folder::AssetFolderTree;
use crate::clip::{AudioClip, ClipInstance, ImageAsset, VideoClip, VectorClip};
use daw_backend::{Beats, Seconds};
use crate::effect::EffectDefinition;
use crate::layer::{AnyLayer, GroupLayer};
use crate::script::ScriptDefinition;
@ -439,24 +438,20 @@ impl Document {
/// Returns the end time of the last clip instance across all layers,
/// or the document's duration if no clips are found.
pub fn calculate_timeline_endpoint(&self) -> f64 {
let tempo_map = self.tempo_map();
// Accumulated in **beats** (as f64, to keep the recursive helper's `Fn(_, f64) -> f64`
// signature); converted to seconds once at the return.
let mut max_end_time: f64 = 0.0;
// End position of a clip instance, in **beats**. Its trimmed content window is in seconds
// (scaled by playback speed), so convert that seconds span to beats via the tempo map — the
// old code added a seconds duration straight onto the beats start.
// Helper function to calculate the end time of a clip instance
let calculate_instance_end = |instance: &ClipInstance, clip_duration: f64| -> f64 {
let end_beats: Beats = if let Some(timeline_duration) = instance.timeline_duration {
instance.timeline_start + timeline_duration
let effective_duration = if let Some(timeline_duration) = instance.timeline_duration {
// Explicit timeline duration set (may include looping)
timeline_duration
} else {
// Calculate from trim points
let trim_end = instance.trim_end.unwrap_or(clip_duration);
let trimmed_secs = ((trim_end - instance.trim_start) / instance.playback_speed).max(0.0);
let start_secs = tempo_map.beats_to_seconds(instance.timeline_start);
tempo_map.seconds_to_beats(start_secs + Seconds(trimmed_secs))
let trimmed_duration = trim_end - instance.trim_start;
trimmed_duration / instance.playback_speed // Adjust for playback speed
};
end_beats.beats_to_f64()
instance.timeline_start + effective_duration
};
// Iterate through all layers to find the maximum end time
@ -472,10 +467,8 @@ impl Document {
}
crate::layer::AnyLayer::Audio(audio_layer) => {
for instance in &audio_layer.clip_instances {
// get_clip_duration yields seconds (converting MIDI's beats duration),
// which is what the closure expects.
if let Some(clip_duration) = self.get_clip_duration(&instance.clip_id) {
let end_time = calculate_instance_end(instance, clip_duration.seconds_to_f64());
if let Some(clip) = self.audio_clips.get(&instance.clip_id) {
let end_time = calculate_instance_end(instance, clip.duration);
max_end_time = max_end_time.max(end_time);
}
}
@ -491,7 +484,7 @@ impl Document {
crate::layer::AnyLayer::Effect(effect_layer) => {
for instance in &effect_layer.clip_instances {
if let Some(clip_duration) = self.get_clip_duration(&instance.clip_id) {
let end_time = calculate_instance_end(instance, clip_duration.seconds_to_f64());
let end_time = calculate_instance_end(instance, clip_duration);
max_end_time = max_end_time.max(end_time);
}
}
@ -518,8 +511,8 @@ impl Document {
}
crate::layer::AnyLayer::Audio(al) => {
for inst in &al.clip_instances {
if let Some(clip_duration) = doc.get_clip_duration(&inst.clip_id) {
*max_end = max_end.max(calc_end(inst, clip_duration.seconds_to_f64()));
if let Some(clip) = doc.audio_clips.get(&inst.clip_id) {
*max_end = max_end.max(calc_end(inst, clip.duration));
}
}
}
@ -533,7 +526,7 @@ impl Document {
crate::layer::AnyLayer::Effect(el) => {
for inst in &el.clip_instances {
if let Some(dur) = doc.get_clip_duration(&inst.clip_id) {
*max_end = max_end.max(calc_end(inst, dur.seconds_to_f64()));
*max_end = max_end.max(calc_end(inst, dur));
}
}
}
@ -551,10 +544,9 @@ impl Document {
}
}
// Return the max end (converting the beats accumulator to seconds), or the document
// duration (already seconds) if no clips were found.
// Return the maximum end time, or document duration if no clips found
if max_end_time > 0.0 {
tempo_map.beats_to_seconds(Beats(max_end_time)).seconds_to_f64()
max_end_time
} else {
self.duration
}
@ -898,19 +890,19 @@ impl Document {
/// Searches through all clip libraries to find the clip and return its duration.
/// For effect definitions, returns `EFFECT_DURATION` (f64::MAX) since effects
/// have infinite internal duration.
pub fn get_clip_duration(&self, clip_id: &Uuid) -> Option<Seconds> {
pub fn get_clip_duration(&self, clip_id: &Uuid) -> Option<f64> {
if let Some(clip) = self.vector_clips.get(clip_id) {
if clip.is_group {
Some(Seconds(clip.duration))
Some(clip.duration)
} else {
let tempo_map = self.tempo_map();
Some(Seconds(clip.content_duration_with(self.framerate, tempo_map, |id| {
Some(clip.content_duration_with(self.framerate, tempo_map, |id| {
// Resolve nested clip durations (audio, video, other vector clips)
if let Some(vc) = self.vector_clips.get(id) {
// Avoid deep recursion — use stored duration for nested vector clips
Some(vc.content_duration(self.framerate, tempo_map))
} else if let Some(ac) = self.audio_clips.get(id) {
Some(ac.content_duration().to_seconds(tempo_map).seconds_to_f64())
Some(ac.duration)
} else if let Some(vc) = self.video_clips.get(id) {
Some(vc.duration)
} else if self.effect_definitions.contains_key(id) {
@ -918,25 +910,23 @@ impl Document {
} else {
None
}
})))
}))
}
} else if let Some(clip) = self.video_clips.get(clip_id) {
Some(Seconds(clip.duration))
Some(clip.duration)
} else if let Some(clip) = self.audio_clips.get(clip_id) {
// Interpret the clip's native-domain duration as wall-clock seconds (MIDI stores
// beats, sampled stores seconds — content_duration keeps that straight).
Some(clip.content_duration().to_seconds(self.tempo_map()))
Some(clip.duration)
} else if self.effect_definitions.contains_key(clip_id) {
// Effects have infinite internal duration - their timeline length
// is controlled by ClipInstance.trim_end
Some(Seconds(crate::effect::EFFECT_DURATION))
Some(crate::effect::EFFECT_DURATION)
} else {
None
}
}
/// Calculate the end position of a clip instance on the timeline, in **beats**.
pub fn get_clip_instance_end_time(&self, layer_id: &Uuid, instance_id: &Uuid) -> Option<Beats> {
/// Calculate the end time of a clip instance on the timeline
pub fn get_clip_instance_end_time(&self, layer_id: &Uuid, instance_id: &Uuid) -> Option<f64> {
let layer = self.get_layer(layer_id)?;
// Find the clip instance
@ -952,8 +942,12 @@ impl Document {
let instance = instances.iter().find(|inst| &inst.id == instance_id)?;
let clip_duration = self.get_clip_duration(&instance.clip_id)?;
// End position on the timeline, in beats (convert the seconds content window via tempo map).
Some(instance.timeline_start + instance.effective_duration_beats(clip_duration, self.tempo_map()))
let trim_start = instance.trim_start;
let trim_end = instance.trim_end.unwrap_or(clip_duration);
let effective_duration = trim_end - trim_start;
Some(instance.timeline_start + effective_duration)
}
/// Check if a time range overlaps with any existing clip on the layer
@ -964,8 +958,8 @@ impl Document {
pub fn check_overlap_on_layer(
&self,
layer_id: &Uuid,
start_time: Beats,
end_time: Beats,
start_time: f64,
end_time: f64,
exclude_instance_ids: &[Uuid],
) -> (bool, Option<Uuid>) {
let Some(layer) = self.get_layer(layer_id) else {
@ -1018,14 +1012,14 @@ impl Document {
pub fn find_nearest_valid_position(
&self,
layer_id: &Uuid,
desired_start: Beats,
clip_duration: Beats,
desired_start: f64,
clip_duration: f64,
exclude_instance_ids: &[Uuid],
) -> Option<Beats> {
) -> Option<f64> {
let layer = self.get_layer(layer_id)?;
// Clamp to timeline start (can't go before 0)
let desired_start = desired_start.max(Beats::ZERO);
let desired_start = desired_start.max(0.0);
// Vector layers don't need overlap adjustment, but still respect timeline start
if matches!(layer, AnyLayer::Vector(_) | AnyLayer::Group(_)) {
@ -1050,7 +1044,7 @@ impl Document {
AnyLayer::Text(_) => return Some(desired_start), // Text layers don't have own clips
};
let mut occupied_ranges: Vec<(Beats, Beats, Uuid)> = Vec::new();
let mut occupied_ranges: Vec<(f64, f64, Uuid)> = Vec::new();
for instance in instances {
if exclude_instance_ids.contains(&instance.id) {
continue;
@ -1069,7 +1063,7 @@ impl Document {
// Find the clip we're overlapping with and try both sides, pick nearest
for (occupied_start, occupied_end, _) in &occupied_ranges {
if desired_start < *occupied_end && *occupied_start < desired_end {
let mut candidates: Vec<Beats> = Vec::new();
let mut candidates: Vec<f64> = Vec::new();
// Try snapping to the right (after this clip)
let snap_right = *occupied_end;
@ -1085,8 +1079,8 @@ impl Document {
}
// Try snapping to the left (before this clip)
let snap_left = *occupied_start - clip_duration;
if snap_left >= Beats::ZERO {
let snap_left = occupied_start - clip_duration;
if snap_left >= 0.0 {
let (overlaps_left, _) = self.check_overlap_on_layer(
layer_id,
snap_left,
@ -1101,8 +1095,8 @@ impl Document {
// Pick the candidate closest to desired_start
if !candidates.is_empty() {
candidates.sort_by(|a, b| {
let dist_a = (*a - desired_start).abs();
let dist_b = (*b - desired_start).abs();
let dist_a = (a - desired_start).abs();
let dist_b = (b - desired_start).abs();
dist_a.partial_cmp(&dist_b).unwrap_or(std::cmp::Ordering::Equal)
});
return Some(candidates[0]);
@ -1112,7 +1106,7 @@ impl Document {
// If no gap found, try placing at timeline start
if occupied_ranges.is_empty() || occupied_ranges[0].0 >= clip_duration {
return Some(Beats::ZERO);
return Some(0.0);
}
// No valid position found
@ -1124,9 +1118,9 @@ impl Document {
pub fn clamp_group_move_offset(
&self,
layer_id: &Uuid,
group: &[(Uuid, Beats, Beats)], // (instance_id, timeline_start, effective_duration) in beats
desired_offset: Beats,
) -> Beats {
group: &[(Uuid, f64, f64)], // (instance_id, timeline_start, effective_duration)
desired_offset: f64,
) -> f64 {
let Some(layer) = self.get_layer(layer_id) else {
return desired_offset;
};
@ -1146,8 +1140,8 @@ impl Document {
AnyLayer::Text(_) => &[],
};
// Collect non-group clip ranges (beats)
let mut non_group: Vec<(Beats, Beats)> = Vec::new();
// Collect non-group clip ranges
let mut non_group: Vec<(f64, f64)> = Vec::new();
for inst in instances {
if group_ids.contains(&inst.id) {
continue;
@ -1169,12 +1163,12 @@ impl Document {
// Check against non-group clips
for &(ns, ne) in &non_group {
if clamped < Beats::ZERO {
if clamped < 0.0 {
// Moving left: if non-group clip end is between our destination and current start
if ne <= start && ne > start + clamped {
clamped = clamped.max(ne - start);
}
} else if clamped > Beats::ZERO {
} else if clamped > 0.0 {
// Moving right: if non-group clip start is between our current end and destination
if ns >= end && ns < end + clamped {
clamped = clamped.min(ns - end);
@ -1190,25 +1184,23 @@ impl Document {
///
/// Returns the distance to the nearest clip to the left, or the distance to
/// timeline start (0.0) if no clips exist to the left.
/// Returns the max leftward trim extension as a content-seconds span (the wall-clock
/// length of the timeline gap to the previous clip); the trim domain is seconds.
pub fn find_max_trim_extend_left(
&self,
layer_id: &Uuid,
instance_id: &Uuid,
current_timeline_start: Beats,
) -> Seconds {
current_timeline_start: f64,
) -> f64 {
let Some(layer) = self.get_layer(layer_id) else {
return self.tempo_map().beats_to_seconds(current_timeline_start); // No limit if layer not found
return current_timeline_start; // No limit if layer not found
};
// Only check audio, video, and effect layers
if matches!(layer, AnyLayer::Vector(_) | AnyLayer::Group(_)) {
return self.tempo_map().beats_to_seconds(current_timeline_start); // No limit for vector/group layers
return current_timeline_start; // No limit for vector/group layers
};
// Find the nearest clip to the left
let mut nearest_end = Beats::ZERO; // Can extend to timeline start by default
let mut nearest_end = 0.0; // Can extend to timeline start by default
let instances: &[ClipInstance] = match layer {
AnyLayer::Audio(audio) => &audio.clip_instances,
@ -1228,7 +1220,6 @@ impl Document {
// Calculate other clip's extent (accounting for loop_before)
if let Some(clip_duration) = self.get_clip_duration(&other.clip_id) {
let other_end = other.timeline_start + other.effective_duration(clip_duration, self.tempo_map());
// (clip_duration is Seconds via get_clip_duration; effective_duration converts.)
// If this clip is to the left and closer than current nearest
if other_end <= current_timeline_start && other_end > nearest_end {
@ -1237,27 +1228,27 @@ impl Document {
}
}
self.tempo_map().beats_to_seconds(current_timeline_start) - self.tempo_map().beats_to_seconds(nearest_end)
current_timeline_start - nearest_end
}
/// Find the maximum amount we can extend a clip to the right without overlapping.
/// Find the maximum amount we can extend a clip to the right without overlapping
///
/// Returns the content-seconds span of the timeline gap to the nearest clip on the
/// right, or Seconds(f64::MAX) if none. `current_effective_duration` is beats (timeline).
/// Returns the distance to the nearest clip to the right, or f64::MAX if no
/// clips exist to the right.
pub fn find_max_trim_extend_right(
&self,
layer_id: &Uuid,
instance_id: &Uuid,
current_timeline_start: Beats,
current_effective_duration: Beats,
) -> Seconds {
current_timeline_start: f64,
current_effective_duration: f64,
) -> f64 {
let Some(layer) = self.get_layer(layer_id) else {
return Seconds(f64::MAX); // No limit if layer not found
return f64::MAX; // No limit if layer not found
};
// Only check audio, video, and effect layers
if matches!(layer, AnyLayer::Vector(_) | AnyLayer::Group(_)) {
return Seconds(f64::MAX); // No limit for vector/group layers
return f64::MAX; // No limit for vector/group layers
}
let instances: &[ClipInstance] = match layer {
@ -1270,7 +1261,7 @@ impl Document {
AnyLayer::Text(_) => &[],
};
let mut nearest_start = Beats(f64::MAX);
let mut nearest_start = f64::MAX;
let current_end = current_timeline_start + current_effective_duration;
for other in instances {
@ -1285,11 +1276,10 @@ impl Document {
}
}
if nearest_start == Beats(f64::MAX) {
Seconds(f64::MAX) // No clip to the right, can extend freely
if nearest_start == f64::MAX {
f64::MAX // No clip to the right, can extend freely
} else {
// Gap between our end and next clip's start, as content seconds.
(self.tempo_map().beats_to_seconds(nearest_start) - self.tempo_map().beats_to_seconds(current_end)).max(Seconds::ZERO)
(nearest_start - current_end).max(0.0) // Gap between our end and next clip's start
}
}
/// Find the maximum amount we can extend loop_before to the left without overlapping.
@ -1299,8 +1289,8 @@ impl Document {
&self,
layer_id: &Uuid,
instance_id: &Uuid,
current_effective_start: Beats,
) -> Beats {
current_effective_start: f64,
) -> f64 {
let Some(layer) = self.get_layer(layer_id) else {
return current_effective_start;
};
@ -1319,7 +1309,7 @@ impl Document {
AnyLayer::Text(_) => &[],
};
let mut nearest_end = Beats::ZERO;
let mut nearest_end = 0.0;
for other in instances {
if &other.id == instance_id {

2
lightningbeam-ui/lightningbeam-core/src/effect.rs
View File

@ -351,7 +351,7 @@ impl EffectDefinition {
///
/// * `timeline_start` - When the effect starts on the timeline (seconds)
/// * `duration` - How long the effect appears on the timeline (seconds)
pub fn create_instance(&self, timeline_start: daw_backend::Beats, duration: daw_backend::Beats) -> ClipInstance {
pub fn create_instance(&self, timeline_start: f64, duration: f64) -> ClipInstance {
ClipInstance::new(self.id)
.with_timeline_start(timeline_start)
.with_timeline_duration(duration)

14
lightningbeam-ui/lightningbeam-core/src/effect_layer.rs
View File

@ -148,11 +148,11 @@ impl EffectLayer {
/// `timeline_start` (beats) to seconds for comparison.
pub fn active_clip_instances_at(&self, time_secs: f64, tempo_map: &crate::tempo_map::TempoMap) -> Vec<&ClipInstance> {
use crate::effect::EFFECT_DURATION;
let time_beats = tempo_map.seconds_to_beats(daw_backend::Seconds(time_secs));
let time_beats = tempo_map.inverse_transform(time_secs);
self.clip_instances
.iter()
.filter(|e| {
let end = e.timeline_start + e.effective_duration(daw_backend::Seconds(EFFECT_DURATION), tempo_map);
let end = e.timeline_start + e.effective_duration(EFFECT_DURATION, tempo_map);
time_beats >= e.timeline_start && time_beats < end
})
.collect()
@ -218,7 +218,7 @@ mod tests {
fn test_add_effect() {
let mut layer = EffectLayer::new("Effects");
let def = create_test_effect_def();
let effect = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let effect = def.create_instance(0.0, 10.0);
let effect_id = effect.id;
let id = layer.add_clip_instance(effect);
@ -233,11 +233,11 @@ mod tests {
let def = create_test_effect_def();
// Effect 1: active from 0 to 5
let effect1 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(5.0));
let effect1 = def.create_instance(0.0, 5.0);
layer.add_clip_instance(effect1);
// Effect 2: active from 3 to 10
let effect2 = def.create_instance(daw_backend::Beats(3.0), daw_backend::Beats(7.0)); // 3.0 + 7.0 = 10.0 end
let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end
layer.add_clip_instance(effect2);
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
@ -259,11 +259,11 @@ mod tests {
let mut layer = EffectLayer::new("Effects");
let def = create_test_effect_def();
let effect1 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let effect1 = def.create_instance(0.0, 10.0);
let id1 = effect1.id;
layer.add_clip_instance(effect1);
let effect2 = def.create_instance(daw_backend::Beats(0.0), daw_backend::Beats(10.0));
let effect2 = def.create_instance(0.0, 10.0);
let id2 = effect2.id;
layer.add_clip_instance(effect2);

12
lightningbeam-ui/lightningbeam-core/src/hit_test.rs
View File

@ -260,15 +260,15 @@ pub fn hit_test_clip_instances(
for clip_instance in clip_instances.iter().rev() {
// Check time bounds: skip clip instances not active at this time
// timeline_start/instance_end are in beats; convert timeline_time (seconds) to beats.
let clip_duration = document.get_clip_duration(&clip_instance.clip_id).unwrap_or(daw_backend::Seconds::ZERO);
let clip_duration = document.get_clip_duration(&clip_instance.clip_id).unwrap_or(0.0);
let instance_end = clip_instance.timeline_start + clip_instance.effective_duration(clip_duration, tempo_map);
let timeline_beats = tempo_map.seconds_to_beats(daw_backend::Seconds(timeline_time));
let timeline_beats = tempo_map.inverse_transform(timeline_time);
if timeline_beats < clip_instance.timeline_start || timeline_beats >= instance_end {
continue;
}
// clip_time is in seconds; offset from clip start (in seconds) + trim_start (seconds)
let start_secs = tempo_map.beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let start_secs = tempo_map.transform(clip_instance.timeline_start);
let clip_time = ((timeline_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
let content_bounds = if let Some(vector_clip) = document.get_vector_clip(&clip_instance.clip_id) {
@ -304,14 +304,14 @@ pub fn hit_test_clip_instances_in_rect(
for clip_instance in clip_instances {
// Check time bounds: skip clip instances not active at this time
// timeline_start/instance_end are in beats; convert timeline_time (seconds) to beats.
let clip_duration = document.get_clip_duration(&clip_instance.clip_id).unwrap_or(daw_backend::Seconds::ZERO);
let clip_duration = document.get_clip_duration(&clip_instance.clip_id).unwrap_or(0.0);
let instance_end = clip_instance.timeline_start + clip_instance.effective_duration(clip_duration, tempo_map);
let timeline_beats = tempo_map.seconds_to_beats(daw_backend::Seconds(timeline_time));
let timeline_beats = tempo_map.inverse_transform(timeline_time);
if timeline_beats < clip_instance.timeline_start || timeline_beats >= instance_end {
continue;
}
let start_secs = tempo_map.beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let start_secs = tempo_map.transform(clip_instance.timeline_start);
let clip_time = ((timeline_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
let content_bounds = if let Some(vector_clip) = document.get_vector_clip(&clip_instance.clip_id) {

29
lightningbeam-ui/lightningbeam-core/src/renderer.rs
View File

@ -11,7 +11,6 @@
use crate::animation::TransformProperty;
use crate::clip::{ClipInstance, ImageAsset};
use crate::document::Document;
use daw_backend::Seconds;
use crate::gpu::BlendMode;
use crate::layer::{AnyLayer, LayerTrait, VectorLayer};
use kurbo::Affine;
@ -568,8 +567,7 @@ pub fn render_layer_isolated(
let tempo_map = document.tempo_map();
for clip_instance in &video_layer.clip_instances {
let Some(video_clip) = document.video_clips.get(&clip_instance.clip_id) else { continue };
let Some(clip_time) = clip_instance.remap_time(Seconds(time), Seconds(video_clip.duration), tempo_map) else { continue };
let clip_time = clip_time.seconds_to_f64();
let Some(clip_time) = clip_instance.remap_time(time, video_clip.duration, tempo_map) else { continue };
let Some(frame) = video_mgr.get_frame(&clip_instance.clip_id, clip_time, target_w, target_h) else { continue };
// Evaluate animated transform properties.
@ -977,7 +975,7 @@ pub fn render_single_clip_instance(
.filter(|vc| vc.is_group)
.map(|_| {
let frame_duration = 1.0 / document.framerate;
vector_layer.group_visibility_end(&clip_instance.id, document.tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64(), frame_duration)
vector_layer.group_visibility_end(&clip_instance.id, clip_instance.timeline_start, frame_duration)
});
render_clip_instance(
@ -1012,18 +1010,18 @@ fn render_clip_instance(
let clip_time = if vector_clip.is_group {
// Groups are static — visible from timeline_start to the next keyframe boundary.
// timeline_start is in beats; group_end_time is in seconds (render time).
let start_secs = tempo_map.beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let start_secs = tempo_map.transform(clip_instance.timeline_start);
let end = group_end_time.unwrap_or(start_secs);
if time < start_secs || time >= end {
return;
}
0.0
} else {
let clip_dur = document.get_clip_duration(&vector_clip.id).unwrap_or(Seconds(vector_clip.duration));
let Some(t) = clip_instance.remap_time(Seconds(time), clip_dur, tempo_map) else {
let clip_dur = document.get_clip_duration(&vector_clip.id).unwrap_or(vector_clip.duration);
let Some(t) = clip_instance.remap_time(time, clip_dur, tempo_map) else {
return; // Clip instance not active at this time
};
t.seconds_to_f64()
t
};
// Evaluate animated transform properties
@ -1174,10 +1172,9 @@ fn render_video_layer(
// Remap timeline time to clip's internal time
let tempo_map = document.tempo_map();
let Some(clip_time) = clip_instance.remap_time(Seconds(time), Seconds(video_clip.duration), tempo_map) else {
let Some(clip_time) = clip_instance.remap_time(time, video_clip.duration, tempo_map) else {
continue; // Clip instance not active at this time
};
let clip_time = clip_time.seconds_to_f64();
// Get video frame from VideoManager at the output (export/preview) resolution.
let (target_w, target_h) = video_decode_target(document, base_transform);
@ -1571,7 +1568,7 @@ fn render_vector_layer(
.filter(|vc| vc.is_group)
.map(|_| {
let frame_duration = 1.0 / document.framerate;
layer.group_visibility_end(&clip_instance.id, document.tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64(), frame_duration)
layer.group_visibility_end(&clip_instance.id, clip_instance.timeline_start, frame_duration)
});
render_clip_instance(document, time, clip_instance, layer_opacity, scene, base_transform, &layer.layer.animation_data, image_cache, video_manager, group_end_time, extract.as_deref_mut());
}
@ -1880,7 +1877,7 @@ fn render_vector_layer_cpu(
.filter(|vc| vc.is_group)
.map(|_| {
let frame_duration = 1.0 / document.framerate;
layer.group_visibility_end(&clip_instance.id, document.tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64(), frame_duration)
layer.group_visibility_end(&clip_instance.id, clip_instance.timeline_start, frame_duration)
});
render_clip_instance_cpu(
document, time, clip_instance, layer_opacity, pixmap, base_transform,
@ -1905,14 +1902,14 @@ fn render_clip_instance_cpu(
let tempo_map = document.tempo_map();
let clip_time = if vector_clip.is_group {
let start_secs = tempo_map.beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let start_secs = tempo_map.transform(clip_instance.timeline_start);
let end = group_end_time.unwrap_or(start_secs);
if time < start_secs || time >= end { return; }
0.0
} else {
let clip_dur = document.get_clip_duration(&vector_clip.id).unwrap_or(Seconds(vector_clip.duration));
let Some(t) = clip_instance.remap_time(Seconds(time), clip_dur, tempo_map) else { return };
t.seconds_to_f64()
let clip_dur = document.get_clip_duration(&vector_clip.id).unwrap_or(vector_clip.duration);
let Some(t) = clip_instance.remap_time(time, clip_dur, tempo_map) else { return };
t
};
let transform = &clip_instance.transform;

2
lightningbeam-ui/lightningbeam-editor/Cargo.toml
View File

@ -1,6 +1,6 @@
[package]
name = "lightningbeam-editor"
version = "1.0.9-alpha"
version = "1.0.8-alpha"
edition = "2021"
description = "Multimedia editor for audio, video and 2D animation"
license = "GPL-3.0-or-later"

7
lightningbeam-ui/lightningbeam-editor/src/export/video_exporter.rs
View File

@ -1062,13 +1062,10 @@ fn composite_document_to_hdr(
let success = gpu_resources.effect_processor.compile_effect(device, effect_def);
if !success { eprintln!("Failed to compile effect: {}", effect_def.name); continue; }
}
let tempo_map = document.tempo_map();
let effect_end_beats = effect_instance.timeline_start
+ effect_instance.effective_duration(daw_backend::Seconds(lightningbeam_core::effect::EFFECT_DURATION), tempo_map);
let effect_inst = lightningbeam_core::effect::EffectInstance::new(
effect_def,
tempo_map.beats_to_seconds(effect_instance.timeline_start).seconds_to_f64(),
tempo_map.beats_to_seconds(effect_end_beats).seconds_to_f64(),
effect_instance.timeline_start,
effect_instance.timeline_start + effect_instance.effective_duration(lightningbeam_core::effect::EFFECT_DURATION, document.tempo_map()),
);
let effect_output_handle = gpu_resources.buffer_pool.acquire(device, hdr_spec);
if let Some(effect_output_view) = gpu_resources.buffer_pool.get_view(effect_output_handle) {

114
lightningbeam-ui/lightningbeam-editor/src/main.rs
View File

@ -1,8 +1,6 @@
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
use eframe::egui;
use daw_backend::{Beats, Seconds};
use lightningbeam_core::clip::ClipDuration;
use lightningbeam_core::layer::{AnyLayer, AudioLayer};
use lightningbeam_core::layout::{LayoutDefinition, LayoutNode};
use lightningbeam_core::pane::PaneType;
@ -2354,8 +2352,7 @@ impl EditorApp {
use lightningbeam_core::instance_group::InstanceGroup;
use std::collections::HashSet;
// Split position as a beats timeline position (playback_time is seconds).
let split_time = self.action_executor.document().tempo_map().seconds_to_beats(Seconds(self.playback_time));
let split_time = self.playback_time;
let active_layer_id = match self.active_layer_id {
Some(id) => id,
None => return, // No active layer, nothing to split
@ -2366,7 +2363,7 @@ impl EditorApp {
// Helper to find clips that span the playhead in a specific layer
fn find_splittable_clips(
clip_instances: &[lightningbeam_core::clip::ClipInstance],
split_time: Beats,
split_time: f64,
document: &lightningbeam_core::document::Document,
) -> Vec<uuid::Uuid> {
let mut result = Vec::new();
@ -2375,9 +2372,9 @@ impl EditorApp {
let effective_duration = instance.effective_duration(clip_duration, document.tempo_map());
let timeline_end = instance.timeline_start + effective_duration;
let epsilon = Beats(0.001);
if split_time > instance.timeline_start + epsilon
&& split_time < timeline_end - epsilon
const EPSILON: f64 = 0.001;
if split_time > instance.timeline_start + EPSILON
&& split_time < timeline_end - EPSILON
{
result.push(instance.id);
}
@ -3054,11 +3051,10 @@ impl EditorApp {
let min_start = instances
.iter()
.map(|i| i.timeline_start)
.fold(Beats(f64::INFINITY), |a, b| a.min(b));
let playhead_beats = self.action_executor.document().tempo_map().seconds_to_beats(Seconds(self.playback_time));
let offset = playhead_beats - min_start;
.fold(f64::INFINITY, f64::min);
let offset = self.playback_time - min_start;
for inst in &mut instances {
inst.timeline_start = (inst.timeline_start + offset).max(Beats::ZERO);
inst.timeline_start = (inst.timeline_start + offset).max(0.0);
}
}
@ -3260,7 +3256,7 @@ impl EditorApp {
let duplicates: Vec<lightningbeam_core::clip::ClipInstance> = clips_to_duplicate.iter().map(|original| {
let mut duplicate = original.clone();
duplicate.id = uuid::Uuid::new_v4();
let clip_duration = document.get_clip_duration(&original.clip_id).unwrap_or(Seconds(1.0));
let clip_duration = document.get_clip_duration(&original.clip_id).unwrap_or(1.0);
let effective_duration = original.effective_duration(clip_duration, document.tempo_map());
duplicate.timeline_start = original.timeline_start + effective_duration;
if let Some((new_clip_def_id, _)) = midi_clip_replacements.get(&original.clip_id) {
@ -5557,7 +5553,6 @@ impl EditorApp {
use lightningbeam_core::layer::*;
let drop_time = self.playback_time;
let drop_beats = self.action_executor.document().tempo_map().seconds_to_beats(Seconds(drop_time));
// Find or create a compatible layer
let document = self.action_executor.document();
@ -5650,7 +5645,7 @@ impl EditorApp {
} else {
// For clips, create a clip instance
let mut clip_instance = ClipInstance::new(asset_info.clip_id)
.with_timeline_start(drop_beats);
.with_timeline_start(drop_time);
// For video clips, scale to fit and center in document
if asset_info.clip_type == panes::DragClipType::Video {
@ -5711,7 +5706,7 @@ impl EditorApp {
// Create audio clip instance at same timeline position
let audio_instance = ClipInstance::new(linked_audio_clip_id)
.with_timeline_start(drop_beats);
.with_timeline_start(drop_time);
let audio_instance_id = audio_instance.id;
// Execute audio action with backend sync
@ -5753,7 +5748,7 @@ impl EditorApp {
// Find the video clip instance in the document
let document = self.action_executor.document();
let mut video_instance_info: Option<(uuid::Uuid, Beats, bool)> = None; // (layer_id, timeline_start [beats], already_in_group)
let mut video_instance_info: Option<(uuid::Uuid, f64, bool)> = None; // (layer_id, timeline_start, already_in_group)
// Search root layers for a video clip instance with matching clip_id
for layer in &document.root.children {
@ -5880,7 +5875,7 @@ impl EditorApp {
// Get audio clip duration for logging
let duration = self.action_executor.document().audio_clips
.get(&audio_clip_id)
.map(|c| c.content_duration().native())
.map(|c| c.duration)
.unwrap_or(0.0);
println!("✅ Extracted audio from '{}' ({:.1}s, {}ch, {}Hz) - AudioClip ID: {}",
@ -6423,10 +6418,9 @@ impl eframe::App for EditorApp {
let clip = AudioClip::new_recording("Recording...");
let doc_clip_id = self.action_executor.document_mut().add_audio_clip(clip);
// Create clip instance on the layer (recording_start_time is seconds)
let rec_start_beats = self.action_executor.document().tempo_map().seconds_to_beats(Seconds(self.recording_start_time));
// Create clip instance on the layer
let clip_instance = ClipInstance::new(doc_clip_id)
.with_timeline_start(rec_start_beats);
.with_timeline_start(self.recording_start_time);
let clip_instance_id = clip_instance.id;
@ -6478,7 +6472,7 @@ impl eframe::App for EditorApp {
if let Some(doc_clip_id) = doc_clip_id {
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
if clip.is_recording() {
clip.set_content_duration(ClipDuration::Seconds(duration));
clip.duration = duration.seconds_to_f64();
}
}
}
@ -6543,12 +6537,14 @@ impl eframe::App for EditorApp {
None
}
})
.unwrap_or((uuid::Uuid::nil(), uuid::Uuid::nil(), Beats::ZERO, 0.0))
.unwrap_or((uuid::Uuid::nil(), uuid::Uuid::nil(), 0.0, 0.0))
};
if !clip_id.is_nil() {
// Finalize the clip (update pool_index and duration).
// Finalize the clip (update pool_index and duration)
// A finished recording (samples in the pool) needs capturing.
self.autosave.pending_event = true;
self.media_modified = true;
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&clip_id) {
if clip.finalize_recording(pool_index, duration) {
clip.name = format!("Recording {}", pool_index);
@ -6561,31 +6557,11 @@ impl eframe::App for EditorApp {
// Map the document instance_id → the existing backend clip so that
// delete/move/trim actions can reference it correctly.
// DO NOT call AddAudioClipSync — that would create a duplicate clip.
let backend_id = lightningbeam_core::action::BackendClipInstanceId::Audio(_backend_clip_id);
self.clip_instance_to_backend_map.insert(instance_id, backend_id);
self.clip_instance_to_backend_map.insert(
instance_id,
lightningbeam_core::action::BackendClipInstanceId::Audio(_backend_clip_id),
);
eprintln!("[AUDIO] Mapped doc instance {} → backend clip {}", instance_id, _backend_clip_id);
// Register the finished recording as an already-applied action so
// it bumps the epoch (marks the document modified for save-on-close
// and autosave) and can be undone/redone like any other edit.
let clip_instance = self.layer_to_track_map.get(&layer_id).copied().and_then(|track_id| {
self.action_executor.document()
.get_layer(&layer_id)
.and_then(|l| if let AnyLayer::Audio(al) = l {
al.clip_instances.iter().find(|ci| ci.id == instance_id).cloned()
} else { None })
.map(|ci| (track_id, ci))
});
if let Some((track_id, clip_instance)) = clip_instance {
let action = lightningbeam_core::actions::AddClipInstanceAction::already_applied(
layer_id, clip_instance, track_id, backend_id,
);
self.action_executor.push_applied(Box::new(action));
} else {
// Couldn't build the action; still mark modified so the recording
// isn't silently lost on close.
self.media_modified = true;
}
}
}
@ -6669,7 +6645,7 @@ impl eframe::App for EditorApp {
}
// Update the clip's duration so the timeline bar grows
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
clip.set_content_duration(ClipDuration::Beats(duration));
clip.duration = duration.beats_to_f64();
}
}
}
@ -6705,7 +6681,7 @@ impl eframe::App for EditorApp {
.map(|(id, _)| id);
if let Some(doc_clip_id) = doc_clip_id {
if let Some(clip) = self.action_executor.document_mut().audio_clips.get_mut(&doc_clip_id) {
clip.set_content_duration(ClipDuration::Beats(Beats(midi_clip_data.duration)));
clip.duration = midi_clip_data.duration;
clip.name = format!("MIDI Recording {}", clip_id);
}
}
@ -6720,33 +6696,9 @@ impl eframe::App for EditorApp {
}
}
// Register the finished MIDI recording as an already-applied action so it
// marks the document modified (save-on-close / autosave) and is undoable,
// like the audio path. The backend instance id was mapped during
// MidiRecordingProgress.
if let Some(&layer_id) = self.track_to_layer_map.get(&track_id) {
let doc_clip_id = self.action_executor.document()
.audio_clip_by_midi_clip_id(clip_id).map(|(id, _)| id);
if let Some(doc_clip_id) = doc_clip_id {
let instance = self.action_executor.document()
.get_layer(&layer_id)
.and_then(|l| if let AnyLayer::Audio(al) = l {
al.clip_instances.iter().find(|ci| ci.clip_id == doc_clip_id).cloned()
} else { None });
if let Some(instance) = instance {
if let Some(&backend_id) = self.clip_instance_to_backend_map.get(&instance.id) {
let action = lightningbeam_core::actions::AddClipInstanceAction::already_applied(
layer_id, instance, track_id, backend_id,
);
self.action_executor.push_applied(Box::new(action));
} else {
// No backend mapping (e.g. snapshot lookup missed); still mark
// modified so the recording isn't silently lost on close.
self.media_modified = true;
}
}
}
}
// TODO: Store clip_instance_to_backend_map entry for this MIDI clip.
// The backend created the instance in create_midi_clip(), but doesn't
// report the instance_id back. Needed for move/trim operations later.
// Remove this MIDI layer from active recordings
if let Some(&layer_id) = self.track_to_layer_map.get(&track_id) {
@ -7542,13 +7494,9 @@ impl eframe::App for EditorApp {
let duration = clip.duration;
self.action_executor.document_mut().video_clips.insert(clip_id, clip);
// recording_start_time and duration are seconds; convert to beats.
let tempo_map = self.action_executor.document().tempo_map();
let rec_start_beats = tempo_map.seconds_to_beats(Seconds(self.recording_start_time));
let dur_beats = tempo_map.seconds_to_beats(tempo_map.beats_to_seconds(rec_start_beats) + Seconds(duration)) - rec_start_beats;
let mut clip_instance = ClipInstance::new(clip_id)
.with_timeline_start(rec_start_beats)
.with_timeline_duration(dur_beats);
.with_timeline_start(self.recording_start_time)
.with_timeline_duration(duration);
// Scale to fit document and center (like drag-dropped videos)
{

5
lightningbeam-ui/lightningbeam-editor/src/mobile/transport.rs
View File

@ -2,7 +2,6 @@
//! project scrub. Wired to the audio controller exactly like `TimelinePane`'s header.
use eframe::egui;
use daw_backend::Seconds;
use super::{icons, Palette};
use crate::panes::SharedPaneState;
@ -33,7 +32,7 @@ pub fn render(ui: &mut egui::Ui, rect: egui::Rect, shared: &mut SharedPaneState,
if let Some(controller_arc) = shared.audio_controller {
let mut controller = controller_arc.lock().unwrap();
if *shared.is_playing {
controller.seek(Seconds(*shared.playback_time));
controller.seek(*shared.playback_time);
controller.play();
} else {
controller.pause();
@ -102,7 +101,7 @@ pub fn render(ui: &mut egui::Ui, rect: egui::Rect, shared: &mut SharedPaneState,
*shared.playback_time = new_time;
if let Some(controller_arc) = shared.audio_controller {
let mut controller = controller_arc.lock().unwrap();
controller.seek(Seconds(new_time));
controller.seek(new_time);
}
}
}

12
lightningbeam-ui/lightningbeam-editor/src/panes/asset_library.rs
View File

@ -932,7 +932,7 @@ impl AssetLibraryPane {
name: clip.name.clone(),
category: AssetCategory::Audio,
drag_clip_type,
duration: clip.content_duration().native(),
duration: clip.duration,
dimensions: None,
extra_info,
is_builtin: false,
@ -1136,7 +1136,7 @@ impl AssetLibraryPane {
name: clip.name.clone(),
category: AssetCategory::Audio,
drag_clip_type,
duration: clip.content_duration().native(),
duration: clip.duration,
dimensions: None,
extra_info,
is_builtin: false,
@ -1802,7 +1802,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color))
} else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
}
@ -2358,7 +2358,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color))
} else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
}
@ -2495,7 +2495,7 @@ impl AssetLibraryPane {
AudioClipType::Midi { midi_clip_id } => {
let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color))
} else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
}
@ -2858,7 +2858,7 @@ impl AssetLibraryPane {
let note_color = egui::Color32::from_rgb(100, 200, 100);
if let Some(events) = shared.midi_event_cache.get(midi_clip_id) {
Some(generate_midi_thumbnail(events, clip.content_duration().native(), bg_color, note_color))
Some(generate_midi_thumbnail(events, clip.duration, bg_color, note_color))
} else {
Some(generate_placeholder_thumbnail(AssetCategory::Audio, 200))
}

10
lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs
View File

@ -1609,17 +1609,15 @@ impl InfopanelPane {
ui.horizontal(|ui| {
ui.label("Start:");
ui.label(format!("{:.2}s", document.tempo_map().beats_to_seconds(ci.effective_start()).seconds_to_f64()));
ui.label(format!("{:.2}s", ci.effective_start()));
});
let clip_dur = document.get_clip_duration(&ci.clip_id)
.unwrap_or_else(|| daw_backend::Seconds(ci.trim_end.unwrap_or(1.0) - ci.trim_start));
.unwrap_or_else(|| ci.trim_end.unwrap_or(1.0) - ci.trim_start);
let total_dur = ci.total_duration(clip_dur, document.tempo_map());
let total_dur_secs = (document.tempo_map().beats_to_seconds(ci.effective_start() + total_dur)
- document.tempo_map().beats_to_seconds(ci.effective_start())).seconds_to_f64();
ui.horizontal(|ui| {
ui.label("Duration:");
ui.label(format!("{:.2}s", total_dur_secs));
ui.label(format!("{:.2}s", total_dur));
});
if ci.trim_start > 0.0 {
@ -1808,7 +1806,7 @@ impl InfopanelPane {
});
ui.horizontal(|ui| {
ui.label("Duration:");
ui.label(format!("{:.2}s", clip.content_duration().to_seconds(document.tempo_map()).seconds_to_f64()));
ui.label(format!("{:.2}s", clip.duration));
});
} else {
// Could be an image asset or effect — show ID

18
lightningbeam-ui/lightningbeam-editor/src/panes/piano_roll.rs
View File

@ -5,7 +5,6 @@
/// When a sampled audio layer is selected, shows a GPU-rendered spectrogram.
use eframe::egui;
use daw_backend::Seconds;
use egui::{pos2, vec2, Align2, Color32, FontId, Rect, Stroke, StrokeKind};
use std::collections::{HashMap, HashSet};
use uuid::Uuid;
@ -466,8 +465,8 @@ impl PianoRollPane {
for instance in &audio_layer.clip_instances {
if let Some(clip) = document.audio_clips.get(&instance.clip_id) {
if let AudioClipType::Midi { midi_clip_id } = clip.clip_type {
let duration = instance.effective_duration(clip.content_duration().to_seconds(document.tempo_map()), document.tempo_map());
clip_data.push((midi_clip_id, instance.timeline_start.beats_to_f64(), instance.trim_start, duration.beats_to_f64(), instance.id));
let duration = instance.effective_duration(clip.duration, document.tempo_map());
clip_data.push((midi_clip_id, instance.timeline_start, instance.trim_start, duration, instance.id));
}
}
}
@ -651,7 +650,7 @@ impl PianoRollPane {
*shared.playback_time = nt;
if let Some(ctrl) = shared.audio_controller.as_ref() {
if let Ok(mut c) = ctrl.lock() {
c.seek(Seconds(nt));
c.seek(nt);
}
}
}
@ -1764,7 +1763,7 @@ impl PianoRollPane {
let seek_time = snap_to_value(time.max(0.0), self.snap_value, tempo_map);
*shared.playback_time = seek_time;
if let Some(ctrl) = shared.audio_controller.as_ref() {
if let Ok(mut c) = ctrl.lock() { c.seek(Seconds(seek_time)); }
if let Ok(mut c) = ctrl.lock() { c.seek(seek_time); }
}
}
}
@ -2455,15 +2454,10 @@ impl PianoRollPane {
for instance in &audio_layer.clip_instances {
if let Some(clip) = document.audio_clips.get(&instance.clip_id) {
if let AudioClipType::Sampled { audio_pool_index } = clip.clip_type {
// Duration in beats: explicit timeline_duration, else the clip's content
// length converted to beats at the clip's start.
let duration = instance.timeline_duration.unwrap_or_else(|| {
let tmap = document.tempo_map();
tmap.seconds_to_beats(tmap.beats_to_seconds(instance.timeline_start) + clip.content_duration().to_seconds(tmap)) - instance.timeline_start
});
let duration = instance.timeline_duration.unwrap_or(clip.duration);
// Get sample rate from raw_audio_cache
if let Some((_samples, sr, _ch)) = shared.raw_audio_cache.get(&audio_pool_index) {
clip_infos.push((audio_pool_index, instance.timeline_start.beats_to_f64(), instance.trim_start, duration.beats_to_f64(), *sr));
clip_infos.push((audio_pool_index, instance.timeline_start, instance.trim_start, duration, *sr));
}
}
}

63
lightningbeam-ui/lightningbeam-editor/src/panes/stage.rs
View File

@ -4,7 +4,6 @@
/// Supports HDR compositing pipeline with per-layer buffers and effects.
use eframe::egui;
use daw_backend::Seconds;
use lightningbeam_core::action::Action;
use lightningbeam_core::clip::ClipInstance;
use lightningbeam_core::gpu::{BufferPool, BufferFormat, BufferSpec, Compositor, EffectProcessor, SrgbToLinearConverter};
@ -1852,13 +1851,10 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
// Create EffectInstance from ClipInstance for the processor
// For now, create a simple effect instance with default parameters
let tempo_map = self.ctx.document.tempo_map();
let effect_end_beats = effect_instance.timeline_start
+ effect_instance.effective_duration(Seconds(lightningbeam_core::effect::EFFECT_DURATION), tempo_map);
let effect_inst = lightningbeam_core::effect::EffectInstance::new(
effect_def,
tempo_map.beats_to_seconds(effect_instance.timeline_start).seconds_to_f64(),
tempo_map.beats_to_seconds(effect_end_beats).seconds_to_f64(),
effect_instance.timeline_start,
effect_instance.timeline_start + effect_instance.effective_duration(lightningbeam_core::effect::EFFECT_DURATION, self.ctx.document.tempo_map()),
);
// Acquire temp buffer for effect output (HDR format)
@ -2208,8 +2204,7 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
let combined_transform = overlay_transform * clip_transform;
// Calculate clip bounds for preview
let start_secs = self.ctx.document.tempo_map().beats_to_seconds(clip_inst.timeline_start).seconds_to_f64();
let clip_time = ((self.ctx.playback_time - start_secs) * clip_inst.playback_speed) + clip_inst.trim_start;
let clip_time = ((self.ctx.playback_time - clip_inst.timeline_start) * clip_inst.playback_speed) + clip_inst.trim_start;
let content_bounds = if let Some(vector_clip) = self.ctx.document.get_vector_clip(&clip_inst.clip_id) {
vector_clip.calculate_content_bounds(&self.ctx.document, clip_time)
} else if let Some(video_clip) = self.ctx.document.get_video_clip(&clip_inst.clip_id) {
@ -2298,19 +2293,15 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
// Also draw selection outlines for clip instances
for &clip_id in self.ctx.selection.clip_instances() {
if let Some(clip_instance) = vector_layer.clip_instances.iter().find(|ci| ci.id == clip_id) {
// Skip clip instances not active at current time (compare in seconds).
let clip_dur = self.ctx.document.get_clip_duration(&clip_instance.clip_id).unwrap_or(Seconds::ZERO);
let tempo_map = self.ctx.document.tempo_map();
let start_secs = tempo_map.beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let instance_end = tempo_map.beats_to_seconds(
clip_instance.timeline_start + clip_instance.effective_duration(clip_dur, tempo_map)
).seconds_to_f64();
if self.ctx.playback_time < start_secs || self.ctx.playback_time >= instance_end {
// Skip clip instances not active at current time
let clip_dur = self.ctx.document.get_clip_duration(&clip_instance.clip_id).unwrap_or(0.0);
let instance_end = clip_instance.timeline_start + clip_instance.effective_duration(clip_dur, self.ctx.document.tempo_map());
if self.ctx.playback_time < clip_instance.timeline_start || self.ctx.playback_time >= instance_end {
continue;
}
// Calculate clip-local time
let clip_time = ((self.ctx.playback_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
let clip_time = ((self.ctx.playback_time - clip_instance.timeline_start) * clip_instance.playback_speed) + clip_instance.trim_start;
// Get dynamic clip bounds from content at current time
let bbox = if let Some(vector_clip) = self.ctx.document.get_vector_clip(&clip_instance.clip_id) {
@ -2680,11 +2671,9 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
// Find clip instance visible at playback time
let visible_clip = video_layer.clip_instances.iter().find(|inst| {
let clip_duration = self.ctx.document.get_clip_duration(&inst.clip_id).unwrap_or(Seconds::ZERO);
let tempo_map = self.ctx.document.tempo_map();
let start_secs = tempo_map.beats_to_seconds(inst.timeline_start).seconds_to_f64();
let end_secs = tempo_map.beats_to_seconds(inst.timeline_start + inst.effective_duration(clip_duration, tempo_map)).seconds_to_f64();
playback_time >= start_secs && playback_time < end_secs
let clip_duration = self.ctx.document.get_clip_duration(&inst.clip_id).unwrap_or(0.0);
let effective_duration = inst.effective_duration(clip_duration, self.ctx.document.tempo_map());
playback_time >= inst.timeline_start && playback_time < inst.timeline_start + effective_duration
});
if let Some(clip_inst) = visible_clip {
@ -10141,8 +10130,7 @@ impl StagePane {
for &clip_id in shared.selection.clip_instances() {
if let Some(clip_instance) = vector_layer.clip_instances.iter().find(|ci| ci.id == clip_id) {
// Calculate clip-local time
let start_secs = shared.action_executor.document().tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let clip_time = ((*shared.playback_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
let clip_time = ((*shared.playback_time - clip_instance.timeline_start) * clip_instance.playback_speed) + clip_instance.trim_start;
// Get dynamic clip bounds from content at current time
use vello::kurbo::Rect as KurboRect;
@ -10342,8 +10330,7 @@ impl StagePane {
// Try clip instance
if let Some(clip_instance) = vector_layer.clip_instances.iter().find(|ci| ci.id == object_id) {
// Calculate clip-local time
let start_secs = shared.action_executor.document().tempo_map().beats_to_seconds(clip_instance.timeline_start).seconds_to_f64();
let clip_time = ((*shared.playback_time - start_secs) * clip_instance.playback_speed) + clip_instance.trim_start;
let clip_time = ((*shared.playback_time - clip_instance.timeline_start) * clip_instance.playback_speed) + clip_instance.trim_start;
// Get dynamic clip bounds from content at current time
let local_bbox = if let Some(vector_clip) = shared.action_executor.document().get_vector_clip(&clip_instance.clip_id) {
@ -11059,11 +11046,9 @@ impl StagePane {
let document = shared.action_executor.document();
if let Some(AnyLayer::Video(video_layer)) = document.get_layer(layer_id) {
video_layer.clip_instances.iter().find(|inst| {
let clip_duration = document.get_clip_duration(&inst.clip_id).unwrap_or(Seconds::ZERO);
let tempo_map = document.tempo_map();
let start_secs = tempo_map.beats_to_seconds(inst.timeline_start).seconds_to_f64();
let end_secs = tempo_map.beats_to_seconds(inst.timeline_start + inst.effective_duration(clip_duration, tempo_map)).seconds_to_f64();
playback_time >= start_secs && playback_time < end_secs
let clip_duration = document.get_clip_duration(&inst.clip_id).unwrap_or(0.0);
let effective_duration = inst.effective_duration(clip_duration, document.tempo_map());
playback_time >= inst.timeline_start && playback_time < inst.timeline_start + effective_duration
}).map(|inst| inst.id)
} else {
None
@ -12502,14 +12487,8 @@ impl PaneRenderer for StagePane {
let canvas_pos = pointer_pos - rect.min;
let world_pos = (canvas_pos - self.pan_offset) / self.zoom;
// Use playhead time (seconds); the beats placement position for clips.
// Use playhead time
let drop_time = *shared.playback_time;
let drop_beats = shared.action_executor.document().tempo_map().seconds_to_beats(Seconds(drop_time));
// 5-second default effect duration as a beats span at the drop point.
let effect_dur_beats = {
let tmap = shared.action_executor.document().tempo_map();
tmap.seconds_to_beats(tmap.beats_to_seconds(drop_beats) + Seconds(5.0)) - drop_beats
};
// Find or create a compatible layer
let document = shared.action_executor.document();
@ -12580,8 +12559,8 @@ impl PaneRenderer for StagePane {
// Create clip instance for effect with 5 second default duration
let clip_instance = ClipInstance::new(def.id)
.with_timeline_start(drop_beats)
.with_timeline_duration(effect_dur_beats);
.with_timeline_start(drop_time)
.with_timeline_duration(5.0);
// Use AddEffectAction for effect layers
let action = lightningbeam_core::actions::AddEffectAction::new(
@ -12593,7 +12572,7 @@ impl PaneRenderer for StagePane {
} else {
// For clips, create a clip instance
let mut clip_instance = ClipInstance::new(dragging.clip_id)
.with_timeline_start(drop_beats);
.with_timeline_start(drop_time);
// For video clips, scale to fit and center in document
if dragging.clip_type == DragClipType::Video {
@ -12663,7 +12642,7 @@ impl PaneRenderer for StagePane {
// Create audio clip instance at same timeline position
let audio_instance = ClipInstance::new(linked_audio_clip_id)
.with_timeline_start(drop_beats);
.with_timeline_start(drop_time);
let audio_instance_id = audio_instance.id;
eprintln!("DEBUG STAGE: Created audio instance: {} for clip: {}", audio_instance_id, linked_audio_clip_id);

700
lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs
View File

File diff suppressed because it is too large Load Diff