skyler
/
Lightningbeam
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Release 1.0.5-alpha

This commit is contained in:
Skyler Lehmkuhl 2026-06-21 16:14:44 -04:00
parent 327e40026a 3c3a482e2e
commit f0929e2b6d
92 changed files with 10788 additions and 2411 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
Changelog.md
View File

@ -1,3 +1,29 @@
# 1.0.5-alpha:
Changes:
- Add shape tweens (morph vector geometry between keyframes)
- Add motion tweens for groups and movie clips
- Group geometry and Convert to Movie Clip now work on DCEL vector shapes
- Region/lasso select now cuts the shape and feeds the normal selection, so Group, Convert, Delete and Properties all work from a lasso (hold shift to add to the selection)
- Clip instances now draw on top of a layer's loose shapes
- Add onion skinning for raster and vector layers, with tinted ghosts and settings in the Info Panel
- Images can now fill vector shapes (None / Solid / Gradient / Image fill types)
- Imported images can be placed on the canvas
- Add a raster keyframe timeline UI with explicit keyframe creation; click a keyframe diamond to snap the playhead to it
- Stream audio, video and images to and from the project file instead of holding them in memory, supporting arbitrarily long media
- Persist (and resume) waveforms and video thumbnails in the project file
- Use low-res proxies for fast cold scrubbing of raster frames
- Bound memory use for raster pixels, GPU textures, video frames and decoded images on large projects
- Video export is roughly 4x faster
- Downmix surround video audio to stereo
Bugfixes:
- Fix video export resolution scaling and a post-export UI hang
- Fix gamma handling and improve brush canvas performance
- Fix a save crash on projects with zero or sparse audio
- Fix raster strokes vanishing when committed
- Fix image fill mapping (anchor to the fill's bounding box)
- Fix video thumbnail strip bugs
# 1.0.4-alpha:
Changes:
- Beats are now the canonical time representation (replacing seconds)

853
STREAMING_TO_DISK_PLAN.md Normal file
View File

@ -0,0 +1,853 @@
# Streaming Media To/From Disk — Plan
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
and to disk when the data is too large to fit comfortably in memory, rather than loading
the entire file regardless of size.
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
is impossible while the container forces a full decode, so the container decision (below)
is now part of this plan.
## Deferred bugs (do at the end)
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
## Raster-keyframe-UI bugs — **[DONE]** (built the raster keyframe timeline UI, 2026-06-20)
Both resolved by the raster-keyframe-timeline-UI work: timeline now draws a diamond per
`RasterKeyframe` (mirrors vector), `K`/New Keyframe inserts a blank cel via `AddRasterKeyframeAction`
(canvas refreshes), paint tools edit the active keyframe instead of lazily creating, diamonds are
click-to-seek (pointing-hand cursor), playback prefetches frames, and onion skinning (raster+vector,
tinted, Info-Panel settings) is in. (a) canvas-refresh-on-new-keyframe and (b) keyframes-on-timeline
are both fixed.
## Noted enhancements (later, after the phases)
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
Native multichannel support remains a separate, larger project.
- **Export speed (audited 2026-06-21):** a 1:14 1080p MP4 took ~9:06 (~7.4x realtime, ~135 ms/frame).
Audit **refuted** the per-frame-seek theory — export decodes the source *sequentially*
(`video.rs` `need_seek` is false once advancing forward), and readback is already async +
triple-buffered. Real hotspots:
- **[DONE] #1 — per-frame renderer rebuild.** The export pump built a fresh `vello::Renderer`
(full wgpu pipeline init) + empty `ImageCache` *every egui repaint* (`main.rs` ~6218). Now built
once per export and reused; also fixed lazy-image export (the throwaway cache had no container
path). **Expected the dominant win.**
- **[DONE] #2a — encode swscale rebuilt per frame.** `CpuYuvConverter::convert` now caches the
RGBA→YUV420p `scaling::Context` + frames in `new()` instead of per call.
- **[TODO] #2b — decode swscale + stride-repack** per frame in `video.rs:294-320` (shared with
scrubbing; cache the YUV→RGBA scaler on the decoder). Small win, modest risk.
- **Result of #1+#2a (measured):** ~7.4x → **~1.74x realtime** (130.7 s for 4488 frames @ 60 fps;
34 fps). Per-stage avg: Render(CPU build) 15 ms, **Readback(GPU latency) 42 ms**, Extract 1.3 ms,
Convert 5.7 ms.
- **Now GPU-bound.** Per ~87 ms poll cycle the CPU does ~66 ms (3× build 45 + convert 17 + extract 4)
but the GPU does ~87 ms (3 × ~29 ms composite) → GPU saturated at ~29 ms/frame; "Readback 42 ms" is
queue latency, not transfer (8 MB is sub-ms).
- **[SKIP] #3 GPU YUV / #5 pacing** — both only trim the CPU side, which is already *under* the GPU.
Won't move a GPU-bound throughput.
- **[TODO, big] Reduce the GPU composite (~29 ms/frame).** The per-layer HDR pipeline (Vello render →
linear → composite, ×layers) is the wall, shared with live rendering. Options: batch composite
passes; a fast-path skipping HDR compositing for simple single-layer/no-blend docs; cache unchanged
layers' scenes (CPU-side, only helps if it later becomes CPU-bound). Render-architecture project.
- Non-issues: per-frame seek, blocking readback, audio. (`video.rs:237` container-reopen-on-seek is
a latent cost but doesn't fire on forward export.)
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
decode) still worth chasing if it recurs.
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
2 h video keeps that work and continues from there on reload.
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
complete clips; a mid-generation save resumes from where it left off on reload.)*
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
absent). The 5 s interval already bounds count; no extra budget needed.
- **Progressive waveform on first import:** generation streams the whole file before the
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
the clip left-to-right instead of appearing all at once. Persistence saves only the final
complete pyramid.
## Guiding principle
Three subsystems already have the right streaming primitive; most of the work is wiring,
bounding caches, and adding a residency window. The recurring pattern:
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
> stable ID, and evict everything outside a window around the playhead.
---
## Audit summary (where we stand today)
### Correctly streaming / bounded
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` —
keyframe-index seek + decode-until-target, one frame resident).
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
frames in flight).
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
### Fully-loaded, unbounded by file length (the problems)
| Site | Issue |
|---|---|
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
---
## Container format decision: `.beam` → SQLite *(DECIDED)*
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
`.beam` extension). This is the foundation the rest of the plan builds on.
### Why
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
mutation** — every save and every raster frame write-back rewrites the whole archive — and
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
SQLite dissolves the single-file-vs-performance tension:
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
confusion; we have no bundle magic on Linux/Windows).
- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
atomicity).
- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly.
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
reads (or external paths); raster frames live in blob rows and write back transactionally.
### Large-media policy: packed OR referenced
Two storage modes per media item, both supported:
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
on a network drive, or media too large/volatile to pack).
**Default-mode preference for files over the per-blob limit (~2GB):**
- A user preference `large_media_default: Pack | Reference` controls what happens to
imports above the threshold.
- The **first time** the user imports a media file over the limit, **prompt** them
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
large imports (changeable later in settings).
- Files under the limit are packed by default (chunked only if needed).
### Schema sketch
```
media(
id BLOB PRIMARY KEY, -- stable Uuid
kind INTEGER, -- audio | video | raster | image-asset
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
storage INTEGER, -- 0 = packed, 1 = referenced
ext_path TEXT, -- set when storage = referenced
total_len INTEGER, -- bytes (packed) for chunk math
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
)
media_chunk(
media_id BLOB, chunk_index INTEGER, bytes BLOB,
PRIMARY KEY (media_id, chunk_index)
)
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
```
`project.json` stays the same serialized `BeamProject` for now — only its container and the
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
first open/save.
### Streaming reads from packed media
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
plain `File` exactly as `do_import_audio` already does for originals today.
---
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
### 1a. Turn on the compressed-audio disk reader
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
assign `AudioClip::read_ahead`.
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
from the ring buffer, not RAM.
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
already distinguishes.
### 1b. Stream on project load *(depends on the SQLite container)*
Three coupled changes (none works alone):
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
for them. After the engine installs the project, walk all audio clips and
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
`BlobReader` instead of a path for packed media.)
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
table instead of base64 `embedded_data`.
### 1c. Video's embedded audio track — stream from the video via ffmpeg
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
save/reload. **Superseded by the design below.**
**Proper design — stream the video's audio track on demand, never materialized.**
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
interface.
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
`sample_rate`/`channels`/`total_frames`.
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
entry `path` = the video file.
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
One ffmpeg context + 3 s buffer per active clip instance.
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` →
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
cleanup.
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
frames before target; for the frame straddling target, drop its leading samples). After
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
while here, or at least flag.
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
seek (both readers) → waveform-peaks pass.
---
## Phase 2 — Video: bound the caches *(small, isolated)*
### 2a. Bound `VideoManager.frame_cache`
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
### 2b. Stream the export mux
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
---
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
the container is opened only at load/save (no live handle).
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
### Drive-by (Arc pixels): DROPPED
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
clone). Not worth 64 edits. Start at 3a.
### 3a. Lazy async fault-in + image proxy
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
results applied at top of `update()`. No UI block on cold scrub.
- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written
beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`.
Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to
the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident.
*(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge
paint projects.)*
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
leave full `raw_pixels` empty.
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
upscaled proxy. Reused by the exporter (already frame-by-frame).
### 3b. Residency window + eviction **[DONE]**
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
rows via `media_exists` (no data loss) and walks all layers to match load.
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
### 3c. Bound the GPU cache **[DONE for raster_layer_cache]**
`raster_layer_cache` (`gpu_brush.rs`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
≈ `w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in
`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay
now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom
if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello
`ImageCache` is image *assets* → Phase 4.)*
### 3d. Undo memory **[DONE]**
`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames.
Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in
`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor
faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`),
correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒
skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose
bbox is the whole frame.)*
### 3e. Prefetch frames **[DONE for playback]**
Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4`
upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so
full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during
playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a
prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction
prefetch still TODO.)*
Original note: *(future, after 3d — pure latency win, no correctness need)*
Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the
proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**:
on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of
playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker +
`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap
concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU
texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res
frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up.
### Build order & tests
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
---
## Phase 3.5 — Image textures in vector scenes **[DONE 2026-06-21]** *(prereq for Phase 4; fixed DCEL-broken image import)*
**Done:** 3.5a — import/drop places an image as a borderless image-filled rectangle
(`AddShapeAction::image_rect`), centered (direct import) or at the drop point (library drag);
renderer now maps the image brush onto the fill's bounding box (was anchored at world origin →
only a corner showed); `SetImageFillAction` + an **Image** fill-type tab (None|Solid|Gradient|Image)
with an asset picker in the Info Panel. 3.5b — image bytes persist as `MediaKind::ImageAsset` rows in
the `.beam` (kept-in-place; `ImageAsset.data` is `skip_serializing` + container-backed; old base64
projects migrate on re-save); eager-read on load. *(ImageCache still unbounded — Phase 4 adds the
usage-based LRU/lazy paging.)*
### (original plan below)
## Phase 3.5 — Image textures in vector scenes *(prereq for testing Phase 4; fixes DCEL-broken image import)*
**Why:** Phase 4 pages *image assets*, but there's currently no way to get an image asset into a
vector scene — so nothing to page. This also repairs image import, half-broken since the DCEL switch.
**Current state (audited 2026-06-21):**
- *Works:* `import_image` (`main.rs`) decodes dims + creates an `ImageAsset` (raw bytes embedded in
`Document::image_assets`, serialized as **base64 in project JSON**). The renderer's image-fill paths
are **complete** — GPU/Vello (`renderer.rs:~1160`, `ImageBrush` via `ImageCache.get_or_decode`) and
CPU/tiny-skia (`renderer.rs:~1486`). `Fill::image_fill` (`vector_graph/mod.rs:110`) and
`Face::image_fill` (`dcel2/mod.rs:117`) fields exist and render when set.
- *Broken/missing (the workflow):*
1. **Drop image → canvas is stubbed:** `stage.rs:~11782` and `main.rs:~4924` both just print
"Image drag to stage not yet supported with DCEL backend". Nothing is added to the scene.
2. **No way to assign an image fill:** no `SetImageFillAction` (only `SetFillPaintAction` for
color/gradient); no Info-Panel picker. `Fill`/`Face.image_fill` are never populated.
3. **DCEL faces never get `image_fill`** (`dcel2/import.rs:275` always `None`; topology copies from
parent which is also `None`).
4. **Not in the container:** `MediaKind::ImageAsset` exists but is **dead** — image bytes live only
as base64 in project JSON. Not chunked, not pageable (so Phase 4 can't page them).
**Tasks:**
- **3.5a — Place + assign.** Replace the two drop stubs: dropping an image onto a vector layer creates
a rectangle face sized to the image at the drop point with `image_fill = asset_id`. Add
`SetImageFillAction` (set/clear an image fill on the selected face/shape; mirrors `SetFillPaintAction`)
+ an Info-Panel image-asset picker for the selected shape's fill. Populate `Face.image_fill` in DCEL
(and keep it through topology ops — already copied from parent).
- **3.5b — Persist in the container.** Write image assets as `MediaKind::ImageAsset` rows in the `.beam`
SQLite (like raster/audio: write on save kept-in-place on re-save; read on load), keyed by asset id;
drop the base64-in-JSON embedding (or keep a tiny ref). This is the storage Phase 4 pages from.
- **3.5c — Lazy decode hook.** Image bytes load from the container into `ImageCache` on first render
(decode → `ImageBrush`/`Pixmap`). Leave `ImageCache` **unbounded for now**; Phase 4 adds the
usage-based LRU/eviction (this phase just makes there *be* real, container-backed image assets to page).
- **Tests:** import→drop→render round-trip; save/reload preserves the image fill + reads bytes from the
container (not JSON); CPU and GPU render paths both show the image.
---
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
**Data model.**
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
`Pixmap` map, keyed by asset id, **unbounded**.
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
**No asset→frame or frame→asset index exists today.**
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
the heavy one).
**Progress (2026-06-21):**
- **[DONE] Tier 2 — bound the decoded `ImageCache`.** 256 MB **usage-LRU**: every
`get_or_decode`/`_cpu` bumps the asset's recency; inserts past budget evict the least-recently-used
(a miss re-decodes from `asset.data`). Achieves usage-based eviction via render-access recency
(simpler than the frame→asset enumeration below; that enumeration is only needed for *prefetch*).
- **[DONE] Tier 1 — lazy compressed bytes.** `ImageCache` holds the container path (threaded
App.current_file_path → SharedPaneState → VelloRenderContext) and pages bytes on a decode miss via
`read_packed_media_readonly`; `load_beam_sqlite` no longer eager-reads → instant load, compressed
bytes don't accumulate. `asset.data` is still used when resident (fresh import / old base64 project).
*(Refinement: persistent read connection vs open-per-miss.)*
- **[DONE] Prefetch.** `assets_needed_at(document, time)` enumerates image ids in the visible vector
layers' active keyframes; during playback the stage decodes the ~0.5s-ahead set into the cache.
*(Refinements: nested clip-instance recursion; background-thread decode.)*
**Phase 4 = DONE** (image asset paging by usage + LRU).
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
### 4b. Usage bookkeeping (the multi-frame problem)
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
paths in `vector_graph` and the relevant actions).
- count 0 → dead, fully evictable / GC candidate.
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
playhead**, not by count (a high-count asset far from the playhead is still evicted).
Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
Eviction never touches an asset in needed-now.
### 4c. Bound the decoded tier
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
image-texture cache the same way, keyed to the residency window.
### Nested-clip prefetch (important)
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
each of which can reference its own image assets. Prefetch must therefore:
- Recurse through clip instances when computing both needed-now and needed-soon.
- Map outer playhead time → each nested clip's local time, and look ahead along the
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
keyframe are loaded before the nested clip reaches it.
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
once); the usage index handles refcounting.
---
## Cross-cutting: a shared residency abstraction
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
disk backing, residency = window/needed-set around playhead + LRU byte budget:
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|---|---|---|---|---|
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
enumeration + usage index is unique to Phase 4.
---
## Sequencing
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
2. **Phase 2** — small, isolated, independently shippable; container-independent.
3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy +
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
5. **Phase 1c** — depends on 1b's pool path.
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
---
## Status
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
buffer (was effectively silent before, since `read_ahead` was always `None`).
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
(bundled) — `lightningbeam-core/src/beam_archive.rs`
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
referenced-path, overwrite-replaces-chunks.
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
changed rows written; unchanged large media never rewritten); orphan cleanup via
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
must NOT copy+rename for existing SQLite files.
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
deleted (now packs original bytes with their codec).
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
`Ask` behaves as `Reference` at save time.
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
import_audio/video/image) queues a one-time modal; choice persists to config.
Threshold shown in the modal is derived from the constant.
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
skip unchanged frames on in-place save (Phase 3).
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
> build the lib in normal mode. Worth fixing separately.
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
+ proper video-audio reload remain as noted follow-ups.
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** —
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
violation; referenced-only-first = leaves packed <2GB in RAM.)
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
never fires for loaded clips (they bypass `AddAudioClip`).
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory`
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
by the editor between `load_audio_pool` and `set_project`.
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` —
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
from the full in-RAM decode, which no longer happens). Added
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
overview now fills in shortly after load.
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
then instantly on subsequent loads once saved; RAM stays flat on a big project.
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
→ `from_compressed` placeholder). Higher risk (touches the working referenced path); packed
covers the common <2GB case first.
- [ ] **Deferred (follow-up): packed video streaming.** Let small videos be packed into the `.beam`
(a `MediaKind::Video` blob, `VideoClip` referencing it by id) and stream **both frames and audio**
from the DB blob via FFmpeg. ffmpeg-next has no custom-I/O wrapper, so this needs an
`AVIOContext`-over-`BlobReader` shim via raw FFI. **Decision (user):** that FFI wrapper lives in
its **own crate, version-pinned to the ffmpeg version**, isolating the unsafe + the ABI coupling.
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
small temp partitions) and the temp path doesn't survive reload.
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
reference survives save/load.
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
empty channel layout → must `set_channel_layout` before resampling, else swr returns
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
until wired. `VideoAudioReader` made `pub` for the integration test.
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
daw-backend + editor compile clean; VideoAudioReader tests still pass.
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
regression) and that an activated VideoAudio clip produces sound.
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` →
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
**→ Feature is live end-to-end; ready for in-app testing.**
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
whole packet is discarded). So compressed clips no longer drift vs video audio after
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` —
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
Ragged last buckets reduce over available children (no value padding). Bounded
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
ordering, stereo channels, size bound, chunk-agnostic.
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
memory (one chunk + the pyramid). Test: envelope matches the signal through
both backends.
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
- [ ] Orchestration (with 1c).
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
- [x] Generation: on video-audio import Success, the same bg thread streams
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
main.rs construction) → `render_layers` → **both** render sites (collapsed-group
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
features hold position at every zoom and line up with playback.
See memory `waveform-shader-fractional-mip-offset`.
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
`register_loaded_videos` only loads frames (not audio), so there is no redundant
regeneration to suppress. Compiles clean across all three crates.
### Waveform LOD pyramid design (step 6)
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
in → per-sample; seamless between.
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
stored.
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
temp before first save). `B` is stored with it (preference is just the default for new gen).
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
cleared only after its children") — so rendering can always walk up to a resident ancestor;
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 —
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
above the floor it's a disk read; below it the span is already small.
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
viewports of fine tiles; persist pyramid in `.beam`.
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
in-app export to confirm A/V sync.)*
- [x] Phase 3a — lazy + async raster fault-in (`RasterStore` + background thread + image proxy)
- [x] Phase 3b — raster residency LRU + eviction (dirty-flag data-loss safety)
- [x] Phase 3c — bound raster GPU texture cache (recency LRU + F3 VRAM readout)
- [x] Phase 3d — raster undo dirty-rect diffs (+ fault-in-before-undo)
- [x] Phase 3.5 — image textures in vector scenes (fixed DCEL-broken image import; image-fill tab + picker; container-persisted)
- [x] Phase 4 — image asset paging: Tier 2 decoded-cache byte-LRU, Tier 1 lazy container bytes, playback prefetch
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.

305
TODO.md Normal file
View File

@ -0,0 +1,305 @@
# Lightningbeam TODO
> ⚠️ **Stale entries:** Lightningbeam was rewritten from JavaScript to Rust. Any entry below
> that cites `src/*.js` / `main.js` / `animation.js` predates that migration — the *issue* may
> or may not still exist in the Rust codebase, but the file/line references are obsolete.
> **Re-verify against the current Rust code before acting** (this covers the "Animation System
> Refactoring" section and the JS-referencing "Known Issues" entries — node editor, default
> interpolation, etc.). Items with no `.js` references are current.
## Animation System Refactoring *(STALE — JS-era migration notes; superseded by the Rust DCEL/keyframe system)*
### Completed
- ✅ Implement AnimationData curve-based system (Keyframe, AnimationCurve, AnimationData classes)
- ✅ Add GraphicsObject.currentTime property
- ✅ Migrate shape rendering to use AnimationData curves (exists, zOrder)
- ✅ Binary search optimization for keyframe lookups
### In Progress
- Migrating from Frame-based to AnimationData curve-based system throughout codebase
### Pending Features
#### Animation Curve Enhancements
- [ ] Implement extrapolation modes (separate for start vs end):
- "hold" (default) - hold value at first/last keyframe
- "extend" - linearly extend the curve beyond keyframes
- "repeat" - repeat the animation
- "decay" - exponential decay to a target value
- [ ] Add position, scale, rotation animation curves for shapes
- [ ] Add shape morphing/tweening between keyframes
#### Keyframing Behavior
- [ ] Add user preference for keyframing behavior when editing objects:
- Auto-keyframe (current default): create/update keyframe at current time
- Edit previous (Flash-style): update most recent keyframe before current time
- Ephemeral (Blender-style): changes don't persist without manual keyframe
- Optional: Add modifier key (e.g. Shift) to toggle between modes
#### Shape Ordering
- [ ] Add "Bring Forward" menu option (swap zOrder with shape in front)
- [ ] Add "Send Backward" menu option (swap zOrder with shape behind)
- [ ] Add "Bring to Front" menu option (set zOrder to max + 1)
- [ ] Add "Send to Back" menu option (set zOrder to min - 1)
#### Code Cleanup
- [ ] Remove all remaining references to Frame-based system
- [ ] Remove legacy Frame class once migration is complete
- [ ] Clean up GraphicsObject.shapes[] array (shapes should only live in Layers)
## Known Issues / Platform Limitations
### Animation: Tweens are broken (Rust codebase) — LOW PRIORITY
- **Issue**: Animation tweening between keyframes (shape/vector interpolation, and the
`tween_after` behavior on keyframes) does not work correctly in the current Rust app.
Needs investigation + fix. Not urgent — revisit later.
- (Older JS-codebase animation entries below reference `src/*.js` and are stale.)
### Audio: Oscillator Timbre Drift (Phase Accumulation Error)
- **Issue**: Oscillators exhibit timbre changes over time due to floating-point phase accumulation errors
- **Affected Files**:
- `daw-backend/src/effects/synth.rs:117-120` (SimpleSynth)
- `daw-backend/src/audio/node_graph/nodes/oscillator.rs:167-170` (OscillatorNode)
- **Root Cause**: Current phase wrapping uses conditional subtraction (`if phase >= 1.0 { phase -= 1.0 }`), which accumulates f32 rounding errors over time, especially for long-playing notes
- **Current Code**:
```rust
self.phase += frequency / sample_rate;
if self.phase >= 1.0 {
self.phase -= 1.0;
}
```
- **Recommended Fix**: Replace with `.fract()` for numerically stable wraparound:
```rust
self.phase += frequency / sample_rate;
self.phase = self.phase.fract();
```
- **Impact**: Medium - affects audio quality for sustained notes, becomes noticeable after several seconds
- **Priority**: Medium - should be addressed before production use
### UI: Node Connections Render Behind VoiceAllocator Child Nodes
- **Issue**: Connection lines (SVG paths) inside expanded VoiceAllocator nodes render behind child nodes due to z-index stacking
- **Affected File**: `src/styles.css:1128`
- **Root Cause**: Child nodes have `z-index: 10` while connection SVG paths have default/lower z-index
- **Current Code**:
```css
.drawflow .drawflow-node.child-node {
opacity: 0.9;
border: 1px solid #5a5aaa !important;
box-shadow: 0 2px 8px rgba(90, 90, 170, 0.3);
z-index: 10;
}
```
- **Recommended Fix**: Either:
1. Remove `z-index: 10` from `.child-node` (simplest), or
2. Add higher z-index to connection SVG paths, or
3. Use CSS `isolation: isolate` on the VoiceAllocator contents area to create a new stacking context
- **Impact**: Low - visual issue only, connections still function but appear to go "behind" nodes
- **Priority**: Low - cosmetic issue that doesn't affect functionality
### UI: VoiceAllocator Child Nodes Don't Move with Parent
- **Issue**: When a VoiceAllocator node is moved, its child nodes remain in their original positions instead of moving with the parent
- **Affected File**: `src/main.js:6202-6207`
- **Root Cause**: The `nodeMoved` event handler only handles the case where a child node is moved (resizes parent), but doesn't handle when the VoiceAllocator itself is moved
- **Current Code**:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
});
```
- **Recommended Fix**: Add logic to detect when a VoiceAllocator is moved and update all child node positions:
```javascript
editor.on("nodeMoved", (nodeId) => {
const node = editor.getNodeFromId(nodeId);
// Case 1: A child node was moved - resize parent
if (node && node.data.parentNodeId) {
resizeVoiceAllocatorToFit(node.data.parentNodeId);
}
// Case 2: A VoiceAllocator was moved - move all children
if (node && node.data.nodeType === 'VoiceAllocator') {
// Calculate delta from previous position (need to track)
// Update all child node positions by the delta
// Call editor.updateConnectionNodes() for parent and all children
}
});
```
- **Impact**: High - child nodes become disconnected from parent visually
- **Priority**: High - breaks expected behavior of grouped nodes
### UI: VoiceAllocator Expansion Doesn't Update Connection Positions
- **Issue**: When expanding/collapsing a VoiceAllocator, connection endpoints don't update to match the new port positions
- **Affected File**: `src/main.js:6496-6555` (handleNodeDoubleClick function)
- **Root Cause**: The expand/collapse logic shows/hides child nodes and resizes the container, but never calls `editor.updateConnectionNodes()` to refresh connection positions
- **Current Code**: In `handleNodeDoubleClick()`, after expanding or collapsing:
```javascript
// Expand
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
nodeElement.style.width = '600px';
nodeElement.style.height = '400px';
// ... shows child nodes ...
// Missing: editor.updateConnectionNodes(`node-${nodeId}`)
```
- **Recommended Fix**: Call `editor.updateConnectionNodes()` after resizing:
```javascript
// After expanding
expandedNodes.add(nodeId);
nodeElement.classList.add('expanded');
// ... resize and show children ...
// Update connection positions for VoiceAllocator and all children
editor.updateConnectionNodes(`node-${nodeId}`);
for (const [childId, parentId] of nodeParents.entries()) {
if (parentId === nodeId) {
editor.updateConnectionNodes(`node-${childId}`);
}
}
```
- **Impact**: Medium - connections appear in wrong positions until manually moved
- **Priority**: Medium - visual issue that affects usability
### UI: Node Editor Allows Editing Without MIDI Layer Selected
- **Issue**: The node editor pane allows adding/editing instrument nodes even when no MIDI layer is selected, and always uses hardcoded `trackId: 0`
- **Affected File**: `src/main.js:6045-6920` (nodeEditor function)
- **Root Cause**: The node editor never checks if `context.activeObject.activeLayer` exists or is a MIDI track, and all backend commands use hardcoded `trackId: 0`
- **Current Code**: All graph commands hardcode track 0:
```javascript
const commandArgs = parentNodeId
? {
trackId: 0, // HARDCODED!
voiceAllocatorId: editor.getNodeFromId(parentNodeId).data.backendId,
nodeType: nodeType,
x: x,
y: y
}
: {
trackId: 0, // HARDCODED!
nodeType: nodeType,
x: x,
y: y
};
```
- **Recommended Fix**:
1. Check if activeLayer is a MIDI track before allowing edits:
```javascript
function getSelectedMidiTrack() {
const activeLayer = context.activeObject?.activeLayer;
if (!activeLayer || activeLayer.type !== 'midi') {
return null;
}
return activeLayer;
}
```
2. Show placeholder when no MIDI track selected:
```javascript
function nodeEditor() {
const container = document.createElement("div");
const midiTrack = getSelectedMidiTrack();
if (!midiTrack) {
container.innerHTML = '<div class="placeholder">Select a MIDI layer to edit instruments</div>';
return container;
}
// ... rest of node editor code ...
}
```
3. Use actual track ID instead of hardcoded 0:
```javascript
const trackId = midiTrack.audioTrackId || 0;
const commandArgs = { trackId, nodeType, x, y };
```
4. Add listener to refresh node editor when layer selection changes
- **Impact**: High - allows editing wrong track's instrument graph, data corruption risk
- **Priority**: High - can cause confusion and data loss
### Animation: Wrong Default Interpolation for Shape and Object Keyframes
- **Issue**: Shape index and object transform keyframes default to "linear" interpolation but should default to "hold" (step function), and there's no UI to change interpolation after creation
- **Affected Files**:
- `src/models/animation.js:124` (Keyframe constructor defaults to "linear")
- `src/main.js:2161` (shapeIndex keyframes default to "linear")
- `src/main.js:2198` (object position/rotation/scale keyframes default to "linear")
- `src/main.js:5910` (Timeline menu - missing tween options)
- **Root Cause**:
1. The Keyframe constructor defaults interpolation to "linear"
2. Shape index keyframes preserve existing interpolation or default to "linear"
3. Object transform keyframes explicitly use "linear"
4. No menu options exist to change interpolation mode after keyframe creation
- **Current Code**:
- Keyframe constructor (animation.js:124):
```javascript
constructor(time, value, interpolation = "linear", uuid = undefined) {
```
- Shape index keyframes (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'linear';
const shapeIndexKeyframe = new Keyframe(currentTime, newShapeIndex, interpolationType);
```
- Object keyframes (main.js:2198):
```javascript
const newKeyframe = new Keyframe(
currentTime,
currentValue,
'linear' // Default to linear interpolation
);
```
- **Expected Behavior**:
- Shape index keyframes should default to "hold" (shapes shouldn't morph between versions)
- Object transforms should default to "hold" (objects shouldn't move/rotate/scale between keyframes unless explicitly tweened)
- Timeline menu should have options to convert between interpolation modes
- **Recommended Fix**:
1. Change shapeIndex default to "hold" (main.js:2161):
```javascript
const interpolationType = existingShapeIndexKf ? existingShapeIndexKf.interpolation : 'hold';
```
2. Change object keyframe default to "hold" (main.js:2198):
```javascript
const newKeyframe = new Keyframe(currentTime, currentValue, 'hold');
```
3. Add Timeline menu options (main.js:5910, in timelineSubmenu):
```javascript
{
text: "Add Shape Tween",
enabled: /* check if shape is selected and has keyframes */,
action: () => {
// Find shapeIndex curve for selected shape
// Change interpolation between keyframes to "linear"
}
},
{
text: "Add Motion Tween",
enabled: /* check if object is selected and has transform keyframes */,
action: () => {
// Find position/rotation/scale curves for selected object
// Change interpolation between keyframes to "linear" or "bezier"
}
}
```
- **Note**: exists and zOrder keyframes already correctly use "hold" (main.js:2139, 2150)
- **Impact**: High - causes unwanted interpolation, shapes morph unexpectedly, objects move when they shouldn't
- **Priority**: High - fundamental animation behavior is incorrect
### Tauri Pinch-Zoom on Linux
- **Issue**: Two-finger pinch gestures zoom the entire Tauri window instead of individual canvases
- **Status**: Known Tauri limitation on Linux/GTK with no cross-platform solution
- **Tracking**: https://github.com/tauri-apps/tauri/discussions/3843
- **Workaround attempts**: Tried `zoomHotkeysEnabled: false`, `touch-action: none`, viewport meta tags - none worked
- **Resolution**: Monitor Tauri releases for official fix
## Notes
### Architecture
- **GraphicsObject** contains Layers and has `currentTime` (continuous time)
- **Layer** contains `shapes[]` array and `animationData` (AnimationData instance)
- **AnimationData** contains curves dictionary, each curve identified by parameter name
- Shape curves: `shape.{uuid}.exists`, `shape.{uuid}.zOrder`
- Future: `shape.{uuid}.x`, `shape.{uuid}.y`, `shape.{uuid}.rotation`, etc.
- **Shapes render based on curves**: Layer.draw checks exists > 0, sorts by zOrder, draws in order
### Interpolation Types
- `linear` - Linear interpolation between keyframes
- `bezier` - Cubic Bezier with easing control points
- `step`/`hold` - Step function (jumps to next value)

58
daw-backend/src/audio/automation.rs
View File

@ -189,22 +189,22 @@ mod tests {
fn test_add_points_sorted() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(2.0, 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(3.0, 0.8, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(2.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(3.0), 0.8, CurveType::Linear));
assert_eq!(lane.points().len(), 3);
assert_eq!(lane.points()[0].time, 1.0);
assert_eq!(lane.points()[1].time, 2.0);
assert_eq!(lane.points()[2].time, 3.0);
assert_eq!(lane.points()[0].time, Beats(1.0));
assert_eq!(lane.points()[1].time, Beats(2.0));
assert_eq!(lane.points()[2].time, Beats(3.0));
}
#[test]
fn test_replace_point_at_same_time() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
assert_eq!(lane.points().len(), 1);
assert_eq!(lane.points()[0].value, 0.5);
@ -214,59 +214,59 @@ mod tests {
fn test_linear_interpolation() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.0, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(0.0), 0.0, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Linear));
assert_eq!(lane.evaluate(0.0), Some(0.0));
assert_eq!(lane.evaluate(0.5), Some(0.5));
assert_eq!(lane.evaluate(1.0), Some(1.0));
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.0));
assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
}
#[test]
fn test_step_interpolation() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Step));
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Step));
lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Step));
lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Step));
assert_eq!(lane.evaluate(0.0), Some(0.5));
assert_eq!(lane.evaluate(0.5), Some(0.5));
assert_eq!(lane.evaluate(0.99), Some(0.5));
assert_eq!(lane.evaluate(1.0), Some(1.0));
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
assert_eq!(lane.evaluate(Beats(0.99)), Some(0.5));
assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
}
#[test]
fn test_evaluate_outside_range() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(2.0, 1.0, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(2.0), 1.0, CurveType::Linear));
// Before first point
assert_eq!(lane.evaluate(0.0), Some(0.5));
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
// After last point
assert_eq!(lane.evaluate(3.0), Some(1.0));
assert_eq!(lane.evaluate(Beats(3.0)), Some(1.0));
}
#[test]
fn test_disabled_lane() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Linear));
lane.enabled = false;
assert_eq!(lane.evaluate(0.0), None);
assert_eq!(lane.evaluate(Beats(0.0)), None);
}
#[test]
fn test_remove_point() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(2.0, 0.8, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(Beats(2.0), 0.8, CurveType::Linear));
assert!(lane.remove_point_at_time(1.0, 0.001));
assert!(lane.remove_point_at_time(Beats(1.0), Beats(0.001)));
assert_eq!(lane.points().len(), 1);
assert_eq!(lane.points()[0].time, 2.0);
assert_eq!(lane.points()[0].time, Beats(2.0));
}
}

530
daw-backend/src/audio/disk_reader.rs
View File

@ -307,23 +307,96 @@ impl ReadAheadBuffer {
// ---------------------------------------------------------------------------
/// Wraps a Symphonia decoder for streaming a single compressed audio file.
struct CompressedReader {
///
/// Public (like [`VideoAudioReader`]) only so integration tests can exercise it
/// directly; treat it as crate-internal.
pub struct CompressedReader {
format_reader: Box<dyn symphonia::core::formats::FormatReader>,
decoder: Box<dyn symphonia::core::codecs::Decoder>,
track_id: u32,
/// Current decoder position in frames.
current_frame: u64,
/// Frames still to drop from the front of decoded output, so that after a
/// (coarse) seek the next emitted sample lands exactly on the target frame.
pending_discard: u64,
sample_rate: u32,
channels: u32,
#[allow(dead_code)]
total_frames: u64,
/// Temporary decode buffer.
sample_buf: Option<SampleBuffer<f32>>,
}
/// A seekable byte stream for packed media held in the host's project container.
///
/// `daw-backend` stays container-agnostic: it never references the `.beam` SQLite
/// store directly. Instead the host (lightningbeam-core) implements this trait over
/// its incremental blob reader and installs a factory ([`AudioBlobSourceFactory`])
/// into the engine, so packed compressed audio can be stream-decoded without ever
/// being fully loaded into RAM.
pub trait MediaByteSource: std::io::Read + std::io::Seek + Send + Sync {
/// Total length of the stream in bytes (Symphonia needs this for seeking).
fn byte_len(&self) -> u64;
}
/// Opens fresh byte streams for packed media by id. Installed into the engine by
/// the host; invoked when activating a clip backed by container-packed audio.
/// (`Debug` so it can ride in the `Query` enum, which derives `Debug`.)
pub trait AudioBlobSourceFactory: Send + Sync + std::fmt::Debug {
/// Open a new independent reader for the packed media item `media_id`
/// (the UUID string stored on the audio pool entry).
fn open(&self, media_id: &str) -> Result<Box<dyn MediaByteSource>, String>;
}
/// Adapts a [`MediaByteSource`] to Symphonia's `MediaSource` (adds the seekable +
/// byte-length metadata Symphonia's probe/seek require).
struct SymphoniaByteSource(Box<dyn MediaByteSource>);
impl std::io::Read for SymphoniaByteSource {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.0.read(buf)
}
}
impl std::io::Seek for SymphoniaByteSource {
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
self.0.seek(pos)
}
}
impl symphonia::core::io::MediaSource for SymphoniaByteSource {
fn is_seekable(&self) -> bool {
true
}
fn byte_len(&self) -> Option<u64> {
Some(self.0.byte_len())
}
}
/// How to open a streaming audio source: a filesystem path (referenced media or a
/// video file) or a host-provided byte stream (container-packed media).
pub enum StreamOpen {
Path(PathBuf),
Source {
src: Box<dyn MediaByteSource>,
/// Codec/extension hint for the Symphonia probe (e.g. `"mp3"`, `"flac"`).
ext: Option<String>,
},
}
impl CompressedReader {
pub fn sample_rate(&self) -> u32 {
self.sample_rate
}
pub fn channels(&self) -> u32 {
self.channels
}
/// Total frames from the codec header (0 if the format doesn't report it).
pub fn total_frames(&self) -> u64 {
self.total_frames
}
/// Open a compressed audio file and prepare for streaming decode.
fn open(path: &Path) -> Result<Self, String> {
pub fn open(path: &Path) -> Result<Self, String> {
let file =
std::fs::File::open(path).map_err(|e| format!("Failed to open file: {}", e))?;
let mss = MediaSourceStream::new(Box::new(file), Default::default());
@ -332,7 +405,21 @@ impl CompressedReader {
if let Some(ext) = path.extension().and_then(|e| e.to_str()) {
hint.with_extension(ext);
}
Self::from_mss(mss, hint)
}
/// Open a compressed stream from a host-provided byte source (packed media).
pub fn open_source(src: Box<dyn MediaByteSource>, ext: Option<&str>) -> Result<Self, String> {
let mss = MediaSourceStream::new(Box::new(SymphoniaByteSource(src)), Default::default());
let mut hint = Hint::new();
if let Some(ext) = ext {
hint.with_extension(ext);
}
Self::from_mss(mss, hint)
}
/// Shared probe + decoder setup over an already-built media stream.
fn from_mss(mss: MediaSourceStream, hint: Hint) -> Result<Self, String> {
let probed = symphonia::default::get_probe()
.format(
&hint,
@ -368,6 +455,7 @@ impl CompressedReader {
decoder,
track_id,
current_frame: 0,
pending_discard: 0,
sample_rate,
channels,
total_frames,
@ -375,9 +463,17 @@ impl CompressedReader {
})
}
/// Seek to a specific frame. Returns the actual frame reached (may differ
/// for compressed formats that can only seek to keyframes).
fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
/// Seek to `target_frame`, **sample-accurately**. Uses `SeekMode::Accurate`:
/// for an elementary stream like MP3 a *coarse* seek byte-estimates the
/// position and seeds the timestamp from that estimate — which for VBR (or a
/// file whose header padding the estimate ignores) lands off by up to ~1s.
/// Accurate mode instead counts frame *headers* (no decode) from a true anchor
/// (the current position, or a rewind to the start for backward seeks), so the
/// returned `actual_ts` is exact; the small residual to `target_frame` is then
/// dropped in `decode_next`. Container formats with seek tables (FLAC/OGG) seek
/// cheaply; a long MP3 walks headers from the anchor (I/O, not decode) — a
/// per-file seek index would make that O(1) (future work).
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
let seek_to = SeekTo::TimeStamp {
ts: target_frame,
track_id: self.track_id,
@ -385,21 +481,23 @@ impl CompressedReader {
let seeked = self
.format_reader
.seek(SeekMode::Coarse, seek_to)
.seek(SeekMode::Accurate, seek_to)
.map_err(|e| format!("Seek failed: {}", e))?;
let actual_frame = seeked.actual_ts;
self.current_frame = actual_frame;
// Drop the frames between where the coarse seek landed and the target.
self.pending_discard = target_frame.saturating_sub(actual_frame);
// Reset the decoder after seeking.
self.decoder.reset();
Ok(actual_frame)
Ok(target_frame)
}
/// Decode the next chunk of audio into `out`. Returns the number of frames
/// decoded. Returns `Ok(0)` at end-of-file.
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
out.clear();
loop {
@ -428,10 +526,22 @@ impl CompressedReader {
if let Some(ref mut buf) = self.sample_buf {
buf.copy_interleaved_ref(decoded);
let samples = buf.samples();
out.extend_from_slice(samples);
let frames = samples.len() / self.channels as usize;
self.current_frame += frames as u64;
return Ok(frames);
let ch = self.channels as usize;
let frames = samples.len() / ch;
// Drop leading frames for sample-accurate seek alignment.
let discard = self.pending_discard.min(frames as u64) as usize;
self.pending_discard -= discard as u64;
out.extend_from_slice(&samples[discard * ch..]);
let emitted = frames - discard;
self.current_frame += emitted as u64;
if emitted > 0 {
return Ok(emitted);
}
// Whole packet discarded for alignment — keep decoding so
// we never falsely report EOF (Ok(0)).
continue;
}
return Ok(0);
@ -445,16 +555,383 @@ impl CompressedReader {
}
}
// ---------------------------------------------------------------------------
// VideoAudioReader
// ---------------------------------------------------------------------------
/// Streams the audio track out of a media file (a video container, or any audio
/// file) using FFmpeg, decoding on demand. Mirrors [`CompressedReader`]'s
/// interface so the disk reader can drive either through [`StreamSource`].
///
/// Seeking is **sample-accurate**: after `seek(target)`, the next `decode_next`
/// yields samples beginning at exactly `target`. FFmpeg's container seek only
/// lands at-or-before the target, so we decode forward and discard the leading
/// samples to hit the frame precisely — this keeps video audio frame-synced with
/// other (mmap/in-memory) clips.
///
/// Public (vs. the private `CompressedReader`) only so integration tests can
/// exercise it directly; treat it as crate-internal.
pub struct VideoAudioReader {
input: ffmpeg_next::format::context::Input,
decoder: ffmpeg_next::decoder::Audio,
/// Built lazily from the first decoded frame's format/layout → interleaved f32.
resampler: Option<ffmpeg_next::software::resampling::Context>,
stream_index: usize,
/// Seconds per stream-timestamp unit.
time_base: f64,
sample_rate: u32,
channels: u32,
total_frames: u64,
/// Absolute frame index of the next sample `decode_next` will output.
current_frame: u64,
/// Frames still to drop from the front of decoded output (seek alignment).
pending_discard: u64,
/// When set, the next decoded frame establishes the discard needed to land on
/// this absolute target frame (sample-accurate seek).
align_to: Option<u64>,
}
impl VideoAudioReader {
pub fn sample_rate(&self) -> u32 {
self.sample_rate
}
pub fn channels(&self) -> u32 {
self.channels
}
/// Estimated total audio frames (from the stream/container duration).
pub fn total_frames(&self) -> u64 {
self.total_frames
}
pub fn open(path: &Path) -> Result<Self, String> {
ffmpeg_next::init().map_err(|e| e.to_string())?;
let input = ffmpeg_next::format::input(&path)
.map_err(|e| format!("Failed to open media: {}", e))?;
// Pull stream scalars + build the decoder inside a scope so the stream
// borrow of `input` ends before we use `input` again.
let (stream_index, time_base, stream_duration, decoder) = {
let stream = input
.streams()
.best(ffmpeg_next::media::Type::Audio)
.ok_or_else(|| "No audio stream found".to_string())?;
let stream_index = stream.index();
let time_base = f64::from(stream.time_base());
let stream_duration = stream.duration();
let ctx = ffmpeg_next::codec::context::Context::from_parameters(stream.parameters())
.map_err(|e| e.to_string())?;
let decoder = ctx.decoder().audio().map_err(|e| e.to_string())?;
(stream_index, time_base, stream_duration, decoder)
};
let sample_rate = decoder.rate();
let channels = decoder.channels() as u32;
let duration_secs = if stream_duration > 0 {
stream_duration as f64 * time_base
} else if input.duration() > 0 {
input.duration() as f64 / f64::from(ffmpeg_next::ffi::AV_TIME_BASE)
} else {
0.0
};
let total_frames = (duration_secs * sample_rate as f64).max(0.0) as u64;
Ok(Self {
input,
decoder,
resampler: None,
stream_index,
time_base,
sample_rate,
channels,
total_frames,
current_frame: 0,
pending_discard: 0,
align_to: None,
})
}
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
let seconds = target_frame as f64 / self.sample_rate.max(1) as f64;
let ts_av = (seconds * f64::from(ffmpeg_next::ffi::AV_TIME_BASE)) as i64;
// Seek to at-or-before the target (max_ts = ts_av) so we can decode
// forward to it exactly. ffmpeg-next's `seek` wants a bounded range.
self.input
.seek(ts_av, 0..(ts_av + 1))
.map_err(|e| format!("Seek failed: {}", e))?;
self.decoder.flush();
self.pending_discard = 0;
self.align_to = Some(target_frame);
self.current_frame = target_frame;
// We align to the exact frame below, so the effective position IS target.
Ok(target_frame)
}
pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
out.clear();
loop {
// Drain a decoded frame if one is ready.
let mut decoded = ffmpeg_next::frame::Audio::empty();
if self.decoder.receive_frame(&mut decoded).is_ok() {
self.ensure_layout(&mut decoded);
let n = self.emit(&decoded, out);
if n > 0 {
return Ok(n);
}
continue; // frame fully discarded by seek-alignment; keep going
}
// Read one packet (owned), releasing the `input` borrow before decoding.
let packet = self.input.packets().next().map(|(_, p)| p);
match packet {
Some(packet) => {
if packet.stream() == self.stream_index {
self.decoder
.send_packet(&packet)
.map_err(|e| e.to_string())?;
}
}
None => {
// EOF: flush and drain whatever remains.
let _ = self.decoder.send_eof();
let mut decoded = ffmpeg_next::frame::Audio::empty();
if self.decoder.receive_frame(&mut decoded).is_ok() {
self.ensure_layout(&mut decoded);
return Ok(self.emit(&decoded, out));
}
return Ok(0);
}
}
}
}
/// Decoders for some formats (e.g. raw mono WAV) leave the frame's channel
/// layout unset. The resampler needs a concrete layout that matches the
/// frame, so fill one in from the channel count when it's missing.
fn ensure_layout(&self, frame: &mut ffmpeg_next::frame::Audio) {
if frame.channel_layout().is_empty() {
frame.set_channel_layout(
ffmpeg_next::channel_layout::ChannelLayout::default(self.channels as i32),
);
}
}
/// Resample one decoded frame to interleaved f32, apply any pending
/// seek-alignment discard, append to `out`, return frames emitted.
fn emit(&mut self, frame: &ffmpeg_next::frame::Audio, out: &mut Vec<f32>) -> usize {
// `frame` already has a non-empty channel layout (set by `ensure_layout`
// before this call), so the resampler config and the actual frame agree
// — otherwise swr fails with AVERROR_INPUT_CHANGED.
if self.resampler.is_none() {
match ffmpeg_next::software::resampling::Context::get(
frame.format(),
frame.channel_layout(),
self.sample_rate,
ffmpeg_next::format::Sample::F32(ffmpeg_next::format::sample::Type::Packed),
frame.channel_layout(),
self.sample_rate,
) {
Ok(r) => self.resampler = Some(r),
Err(_) => return 0,
}
}
let mut resampled = ffmpeg_next::frame::Audio::empty();
if self
.resampler
.as_mut()
.unwrap()
.run(frame, &mut resampled)
.is_err()
{
return 0;
}
// The output is packed (interleaved) f32. Read it from the raw byte plane
// `data(0)` — its length is correct (`frames * channels * 4`), whereas
// `plane::<f32>(0)` is a known ffmpeg-next footgun that reports only
// `samples()` elements (ignoring channels) and would slice out of range
// for multi-channel audio.
let ch = self.channels.max(1) as usize;
let bytes = resampled.data(0);
let n_frames = (bytes.len() / 4) / ch;
if n_frames == 0 {
return 0;
}
// On the first frame after a seek, compute how many leading frames to
// drop so output begins exactly at the seek target.
if let Some(target) = self.align_to.take() {
let frame_start = self.pts_to_frame(frame.pts());
self.pending_discard = target.saturating_sub(frame_start);
}
let discard = (self.pending_discard.min(n_frames as u64)) as usize;
self.pending_discard -= discard as u64;
let start_byte = discard * ch * 4;
let end_byte = n_frames * ch * 4;
out.extend(
bytes[start_byte..end_byte]
.chunks_exact(4)
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]])),
);
let emitted = n_frames - discard;
self.current_frame += emitted as u64;
emitted
}
/// Convert a stream PTS to an absolute audio frame index.
fn pts_to_frame(&self, pts: Option<i64>) -> u64 {
match pts {
Some(p) if p >= 0 => {
((p as f64 * self.time_base) * self.sample_rate as f64).round() as u64
}
_ => self.current_frame,
}
}
}
// ---------------------------------------------------------------------------
// StreamSource — dispatches the disk reader over either decoder backend.
// (Wired into the reader thread in a later step.)
// ---------------------------------------------------------------------------
/// Which decoder backend a streaming source uses.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SourceKind {
/// Symphonia, for compressed audio files (MP3, FLAC, OGG, …).
CompressedAudio,
/// FFmpeg, for the audio track of a video container.
VideoAudio,
}
/// A streaming audio source backing one active clip: either Symphonia
/// ([`CompressedReader`]) or FFmpeg ([`VideoAudioReader`]).
enum StreamSource {
Compressed(CompressedReader),
Video(VideoAudioReader),
}
impl StreamSource {
fn open(open: StreamOpen, kind: SourceKind) -> Result<Self, String> {
match (kind, open) {
(SourceKind::CompressedAudio, StreamOpen::Path(p)) => {
Ok(StreamSource::Compressed(CompressedReader::open(&p)?))
}
(SourceKind::CompressedAudio, StreamOpen::Source { src, ext }) => {
Ok(StreamSource::Compressed(CompressedReader::open_source(src, ext.as_deref())?))
}
(SourceKind::VideoAudio, StreamOpen::Path(p)) => {
Ok(StreamSource::Video(VideoAudioReader::open(&p)?))
}
(SourceKind::VideoAudio, StreamOpen::Source { .. }) => {
Err("VideoAudio cannot be opened from a packed byte source".to_string())
}
}
}
fn sample_rate(&self) -> u32 {
match self {
StreamSource::Compressed(r) => r.sample_rate,
StreamSource::Video(r) => r.sample_rate,
}
}
fn channels(&self) -> u32 {
match self {
StreamSource::Compressed(r) => r.channels,
StreamSource::Video(r) => r.channels,
}
}
fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
match self {
StreamSource::Compressed(r) => r.seek(target_frame),
StreamSource::Video(r) => r.seek(target_frame),
}
}
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
match self {
StreamSource::Compressed(r) => r.decode_next(out),
StreamSource::Video(r) => r.decode_next(out),
}
}
fn total_frames(&self) -> u64 {
match self {
StreamSource::Compressed(r) => r.total_frames(),
StreamSource::Video(r) => r.total_frames(),
}
}
}
/// Decode a media source end-to-end and build its [`WaveformPyramid`] overview,
/// streaming — only one decode chunk plus the (bounded) pyramid are ever in
/// memory, never the full sample buffer. `floor_samples_per_texel` is the
/// finest-level resolution (see [`crate::audio::waveform_pyramid`]).
pub fn build_waveform_pyramid(
path: &Path,
kind: SourceKind,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
let src = StreamSource::open(StreamOpen::Path(path.to_path_buf()), kind)?;
build_pyramid_from_streamsource(src, floor_samples_per_texel)
}
/// Build a waveform pyramid from a host-provided byte source (container-packed
/// compressed audio) — the load-time counterpart of [`build_waveform_pyramid`]
/// for media that has no filesystem path.
pub fn build_waveform_pyramid_from_source(
src: Box<dyn MediaByteSource>,
ext: Option<&str>,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
let src = StreamSource::open(
StreamOpen::Source { src, ext: ext.map(|s| s.to_string()) },
SourceKind::CompressedAudio,
)?;
build_pyramid_from_streamsource(src, floor_samples_per_texel)
}
fn build_pyramid_from_streamsource(
mut src: StreamSource,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
use crate::audio::waveform_pyramid::WaveformPyramidBuilder;
let channels = src.channels();
let mut builder = WaveformPyramidBuilder::new(channels, floor_samples_per_texel);
builder.reserve_for_frames(src.total_frames());
let mut buf = Vec::new();
loop {
let frames = src.decode_next(&mut buf)?;
if frames == 0 {
break;
}
builder.push_interleaved(&buf);
}
Ok(builder.finish())
}
// ---------------------------------------------------------------------------
// DiskReaderCommand
// ---------------------------------------------------------------------------
/// Commands sent from the engine to the disk reader thread.
pub enum DiskReaderCommand {
/// Start streaming a compressed file for a clip instance.
/// Start streaming a file for a clip instance, using the decoder backend
/// selected by `kind` (compressed audio vs. a video's audio track). `open`
/// is either a filesystem path (referenced media / video) or a host-provided
/// byte stream (container-packed media).
ActivateFile {
reader_id: u64,
path: PathBuf,
open: StreamOpen,
kind: SourceKind,
buffer: Arc<ReadAheadBuffer>,
},
/// Stop streaming for a clip instance.
@ -529,7 +1006,7 @@ impl DiskReader {
mut command_rx: rtrb::Consumer<DiskReaderCommand>,
running: Arc<AtomicBool>,
) {
let mut active_files: HashMap<u64, (CompressedReader, Arc<ReadAheadBuffer>)> =
let mut active_files: HashMap<u64, (StreamSource, Arc<ReadAheadBuffer>)> =
HashMap::new();
let mut decode_buf = Vec::with_capacity(8192);
@ -539,18 +1016,19 @@ impl DiskReader {
match cmd {
DiskReaderCommand::ActivateFile {
reader_id,
path,
open,
kind,
buffer,
} => match CompressedReader::open(&path) {
} => match StreamSource::open(open, kind) {
Ok(reader) => {
eprintln!("[DiskReader] Activated reader={}, ch={}, sr={}, path={:?}",
reader_id, reader.channels, reader.sample_rate, path);
eprintln!("[DiskReader] Activated reader={}, kind={:?}, ch={}, sr={}",
reader_id, kind, reader.channels(), reader.sample_rate());
active_files.insert(reader_id, (reader, buffer));
}
Err(e) => {
eprintln!(
"[DiskReader] Failed to open compressed file {:?}: {}",
path, e
"[DiskReader] Failed to open reader={} ({:?}): {}",
reader_id, kind, e
);
}
},
@ -588,7 +1066,7 @@ impl DiskReader {
// If the target has jumped behind or far ahead of the buffer,
// seek the decoder and reset.
if target < buf_start || target > buf_end + reader.sample_rate as u64 {
if target < buf_start || target > buf_end + reader.sample_rate() as u64 {
buffer.reset(target);
let _ = reader.seek(target);
continue;
@ -607,7 +1085,7 @@ impl DiskReader {
let buf_valid = buffer.valid_frames_count();
let buf_end = buf_start + buf_valid;
let prefetch_target =
target + (PREFETCH_SECONDS * reader.sample_rate as f64) as u64;
target + (PREFETCH_SECONDS * reader.sample_rate() as f64) as u64;
if buf_end >= prefetch_target {
continue; // Already filled far enough ahead.
@ -649,3 +1127,7 @@ impl Drop for DiskReader {
}
}
}
// Tests for VideoAudioReader live in `daw-backend/tests/video_audio_stream.rs`
// (integration tests) so they build the lib in normal mode, independent of
// pre-existing breakage in the crate's `#[cfg(test)]` unit tests (automation.rs).

252
daw-backend/src/audio/engine.rs
View File

@ -91,6 +91,10 @@ pub struct Engine {
// Disk reader for streaming playback of compressed files
disk_reader: Option<crate::audio::disk_reader::DiskReader>,
// Host-installed factory for opening container-packed audio as a byte stream
// (set on load, before the project's clips are bulk-activated for streaming).
blob_source_factory: Option<Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>>,
// Input monitoring and metering
input_monitoring: bool,
input_gain: f32,
@ -176,6 +180,7 @@ impl Engine {
metronome: Metronome::new(sample_rate),
recording_sample_buffer: Vec::with_capacity(4096),
disk_reader: Some(disk_reader),
blob_source_factory: None,
input_monitoring: false,
input_gain: 1.0,
input_level_peak: 0.0,
@ -273,6 +278,97 @@ impl Engine {
/// Rebuild the clip snapshot from the current project state.
/// Call this after any command that adds, removes, or modifies clip instances.
/// Set up disk streaming for a clip backed by a streaming pool entry
/// (Compressed or VideoAudio). Sends `ActivateFile` to the disk reader and
/// returns the read-ahead buffer to attach to the clip. Returns `None` if the
/// entry isn't streamed, or a packed source can't be opened.
fn activate_streaming_for(
&mut self,
reader_id: u64,
pool_index: usize,
) -> Option<Arc<crate::audio::disk_reader::ReadAheadBuffer>> {
use crate::audio::pool::AudioStorage;
use crate::audio::disk_reader::{DiskReader, DiskReaderCommand, SourceKind, StreamOpen};
// Decide how to open the source. `Packed` ⇒ via the host factory (bytes in
// the container); otherwise stream from the file path. Extract owned values
// first so the immutable pool borrow ends before we touch the disk reader.
enum OpenDesc {
Path(std::path::PathBuf),
Packed { media_id: String, ext: Option<String> },
}
let (kind, sample_rate, channels, desc) = {
let file = self.audio_pool.get_file(pool_index)?;
let kind = match file.storage {
AudioStorage::Compressed { .. } => SourceKind::CompressedAudio,
AudioStorage::VideoAudio { .. } => SourceKind::VideoAudio,
_ => return None,
};
let desc = match &file.packed_media_id {
Some(id) => OpenDesc::Packed { media_id: id.clone(), ext: file.original_format.clone() },
None => OpenDesc::Path(file.path.clone()),
};
(kind, file.sample_rate, file.channels, desc)
};
let open = match desc {
OpenDesc::Path(p) => StreamOpen::Path(p),
OpenDesc::Packed { media_id, ext } => {
let factory = match self.blob_source_factory.as_ref() {
Some(f) => f,
None => {
eprintln!("[Engine] packed audio (pool {}) but no blob factory installed", pool_index);
return None;
}
};
match factory.open(&media_id) {
Ok(src) => StreamOpen::Source { src, ext },
Err(e) => {
eprintln!("[Engine] blob factory open({}) failed: {}", media_id, e);
return None;
}
}
}
};
let buffer = DiskReader::create_buffer(sample_rate, channels);
if let Some(ref mut dr) = self.disk_reader {
dr.send(DiskReaderCommand::ActivateFile { reader_id, open, kind, buffer: buffer.clone() });
}
Some(buffer)
}
/// Activate disk streaming for every loaded clip backed by a streaming pool
/// entry. Called after `SetProject` since loaded clips bypass `AddAudioClip`.
fn activate_all_streaming_clips(&mut self) {
use crate::audio::track::TrackNode;
// Collect (track, clip, pool) first via an immutable walk, then activate
// (needs &mut self) and attach the buffer back to the clip.
let targets: Vec<(TrackId, u64, usize)> = self
.project
.track_iter()
.filter_map(|(track_id, node)| match node {
TrackNode::Audio(t) => Some((track_id, t)),
_ => None,
})
.flat_map(|(track_id, t)| {
t.clips
.iter()
.map(move |c| (track_id, c.id as u64, c.audio_pool_index))
})
.collect();
for (track_id, clip_id, pool_index) in targets {
if let Some(buffer) = self.activate_streaming_for(clip_id, pool_index) {
if let Some(TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
if let Some(clip) = track.clips.iter_mut().find(|c| c.id as u64 == clip_id) {
clip.read_ahead = Some(buffer);
}
}
}
}
}
fn refresh_clip_snapshot(&self) {
let mut snap = self.clip_snapshot.write().unwrap();
snap.audio.clear();
@ -914,7 +1010,7 @@ impl Engine {
let start_secs = self.tempo_map.beats_to_seconds(start_beats);
let end_secs = self.tempo_map.beats_to_seconds(end_beats);
let content_dur_secs = (end_secs - start_secs).seconds_to_f64();
let clip = AudioClipInstance::new(
let mut clip = AudioClipInstance::new(
clip_id,
pool_index,
Seconds(offset),
@ -923,6 +1019,13 @@ impl Engine {
Beats(duration),
);
// If the source is streamed (a compressed audio file, or a video's
// audio track), set up disk streaming instead of an in-memory decode.
// Each clip instance gets its own read-ahead buffer keyed by clip_id.
if let Some(buffer) = self.activate_streaming_for(clip_id as u64, pool_index) {
clip.read_ahead = Some(buffer);
}
// Add clip to track
if let Some(crate::audio::track::TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
track.clips.push(clip);
@ -2313,6 +2416,34 @@ impl Engine {
}
}
/// Add a video file's audio track as a streaming pool entry (FFmpeg, decoded
/// on demand — no extraction). Probes the audio track for channels/rate/frames
/// without decoding, and returns the pool index. Playback activation
/// (per-clip read-ahead) is wired separately when a clip references it.
fn do_add_video_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
use crate::audio::disk_reader::VideoAudioReader;
let reader = VideoAudioReader::open(path)?;
let channels = reader.channels();
let sample_rate = reader.sample_rate();
let total_frames = reader.total_frames();
drop(reader);
let audio_file = crate::audio::pool::AudioFile::from_video_audio(
path.to_path_buf(),
channels,
sample_rate,
total_frames,
);
let pool_index = self.audio_pool.add_file(audio_file);
eprintln!(
"[ENGINE] AddVideoAudio: ch={}, sr={}, total_frames={}, pool_index={}, path={:?}",
channels, sample_rate, total_frames, pool_index, path
);
Ok(pool_index)
}
/// Import an audio file into the pool: mmap for PCM, streaming for compressed.
/// Returns the pool index on success. Emits AudioFileReady event.
fn do_import_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
@ -2738,10 +2869,14 @@ impl Engine {
Query::GetPoolAudioSamples(pool_index) => {
match self.audio_pool.get_file(pool_index) {
Some(file) => {
// For Compressed storage, return decoded_for_waveform if available
// For streamed (Compressed/VideoAudio) storage, return the
// progressively-decoded waveform overview if available.
let samples = match &file.storage {
crate::audio::pool::AudioStorage::Compressed {
decoded_for_waveform, decoded_frames, ..
}
| crate::audio::pool::AudioStorage::VideoAudio {
decoded_for_waveform, decoded_frames, ..
} if *decoded_frames > 0 => {
decoded_for_waveform.clone()
}
@ -2794,77 +2929,12 @@ impl Engine {
self.refresh_clip_snapshot();
result
}
Query::AddAudioFileSync(path, data, channels, sample_rate) => {
// Add audio file to pool and return the pool index
// Detect original format from file extension
let path_buf = std::path::PathBuf::from(&path);
let original_format = path_buf.extension()
.and_then(|ext| ext.to_str())
.map(|s| s.to_lowercase());
// Create AudioFile and add to pool
let audio_file = crate::audio::pool::AudioFile::with_format(
path_buf.clone(),
data.clone(), // Clone data for background thread
channels,
sample_rate,
original_format,
);
let pool_index = self.audio_pool.add_file(audio_file);
// Generate Level 0 (overview) waveform chunks asynchronously in background thread
let chunk_tx = self.chunk_generation_tx.clone();
let duration = data.len() as f64 / (sample_rate as f64 * channels as f64);
println!("🔄 [ENGINE] Spawning background thread to generate Level 0 chunks for pool {}", pool_index);
std::thread::spawn(move || {
// Create temporary AudioFile for chunk generation
let temp_audio_file = crate::audio::pool::AudioFile::with_format(
path_buf,
data,
channels,
sample_rate,
None,
);
// Generate Level 0 chunks
let chunk_count = crate::audio::waveform_cache::WaveformCache::calculate_chunk_count(duration, 0);
println!("🔄 [BACKGROUND] Generating {} Level 0 chunks for pool {}", chunk_count, pool_index);
let chunks = crate::audio::waveform_cache::WaveformCache::generate_chunks(
&temp_audio_file,
pool_index,
0, // Level 0 (overview)
&(0..chunk_count).collect::<Vec<_>>(),
);
// Send chunks via MPSC channel (will be forwarded by audio thread)
if !chunks.is_empty() {
println!("📤 [BACKGROUND] Generated {} chunks, sending to audio thread (pool {})", chunks.len(), pool_index);
let event_chunks: Vec<(u32, (f64, f64), Vec<crate::io::WaveformPeak>)> = chunks
.into_iter()
.map(|chunk| (chunk.chunk_index, chunk.time_range, chunk.peaks))
.collect();
match chunk_tx.send(AudioEvent::WaveformChunksReady {
pool_index,
detail_level: 0,
chunks: event_chunks,
}) {
Ok(_) => println!("✅ [BACKGROUND] Chunks sent successfully for pool {}", pool_index),
Err(e) => eprintln!("❌ [BACKGROUND] Failed to send chunks: {}", e),
}
} else {
eprintln!("⚠️ [BACKGROUND] No chunks generated for pool {}", pool_index);
}
});
// Notify UI about the new audio file (for event listeners)
let _ = self.event_tx.push(AudioEvent::AudioFileAdded(pool_index, path));
QueryResponse::AudioFileAddedSync(Ok(pool_index))
}
Query::ImportAudioSync(path) => {
QueryResponse::AudioImportedSync(self.do_import_audio(&path))
}
Query::AddVideoAudioSync(path) => {
QueryResponse::AudioImportedSync(self.do_add_video_audio(&path))
}
Query::GetProject => {
// Save graph presets before cloning — AudioTrack::clone() creates
// a fresh default graph (not a copy), so the preset must be populated
@ -2879,11 +2949,19 @@ impl Engine {
match project.rebuild_audio_graphs(self.buffer_pool.buffer_size()) {
Ok(()) => {
self.project = project;
// Loaded clips bypass AddAudioClip, so their disk streaming was
// never activated — do it now for every streaming-backed clip.
self.activate_all_streaming_clips();
self.refresh_clip_snapshot();
QueryResponse::ProjectSet(Ok(()))
}
Err(e) => QueryResponse::ProjectSet(Err(format!("Failed to rebuild audio graphs: {}", e))),
}
}
Query::SetBlobSourceFactory(factory) => {
self.blob_source_factory = Some(factory);
QueryResponse::BlobSourceFactorySet(Ok(()))
}
Query::DuplicateMidiClipSync(clip_id) => {
match self.project.midi_clip_pool.duplicate_clip(clip_id) {
Some(new_id) => QueryResponse::MidiClipDuplicated(Ok(new_id)),
@ -3395,16 +3473,6 @@ impl EngineController {
}
}
/// Add an audio file to the pool synchronously and get the pool index
/// Returns the pool index where the audio file was added
pub fn add_audio_file_sync(&mut self, path: String, data: Vec<f32>, channels: u32, sample_rate: u32) -> Result<usize, String> {
let query = Query::AddAudioFileSync(path, data, channels, sample_rate);
match self.send_query(query)? {
QueryResponse::AudioFileAddedSync(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Import an audio file asynchronously. The engine will memory-map WAV/AIFF
/// files for instant availability, or set up stream decoding for compressed
/// formats. Listen for `AudioEvent::AudioFileReady` to get the pool index.
@ -3427,6 +3495,30 @@ impl EngineController {
}
}
/// Add a video file's audio track as a streaming pool entry (decoded on
/// demand via FFmpeg — no extraction to disk or RAM). Probes the audio track
/// and returns the pool index. Use this for a video clip's embedded audio.
pub fn add_video_audio_sync(&mut self, path: std::path::PathBuf) -> Result<usize, String> {
let query = Query::AddVideoAudioSync(path);
match self.send_query(query)? {
QueryResponse::AudioImportedSync(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Install the host's packed-media byte-source factory. Must be called before
/// loading a project so its container-packed audio can be streamed (the disk
/// reader opens packed entries through this factory).
pub fn set_blob_source_factory(
&mut self,
factory: std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>,
) -> Result<(), String> {
match self.send_query(Query::SetBlobSourceFactory(factory))? {
QueryResponse::BlobSourceFactorySet(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Generate the next unique audio clip instance ID (atomic, thread-safe)
pub fn next_audio_clip_id(&self) -> AudioClipInstanceId {
self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed)

29
daw-backend/src/audio/export.rs
View File

@ -671,14 +671,39 @@ fn convert_chunk_to_planar_i16(interleaved: &[f32], channels: u32) -> Vec<Vec<i1
planar
}
/// Convert a chunk of interleaved f32 samples to planar f32 format
/// Convert a chunk of interleaved f32 samples to planar f32 format.
///
/// Non-finite samples (NaN/±Inf) are replaced with `0.0` and finite samples are
/// clamped to `[-1.0, 1.0]`: the float encoders (e.g. AAC, which takes `fltp`)
/// reject a frame outright on "(near) NaN/+-Inf", failing the whole export, so we
/// sanitize here exactly as the integer paths already clamp.
fn convert_chunk_to_planar_f32(interleaved: &[f32], channels: u32) -> Vec<Vec<f32>> {
let num_frames = interleaved.len() / channels as usize;
let mut planar = vec![vec![0.0f32; num_frames]; channels as usize];
let mut non_finite = 0u64;
for (i, chunk) in interleaved.chunks(channels as usize).enumerate() {
for (ch, &sample) in chunk.iter().enumerate() {
planar[ch][i] = sample;
planar[ch][i] = if sample.is_finite() {
sample.clamp(-1.0, 1.0)
} else {
non_finite += 1;
0.0
};
}
}
if non_finite > 0 {
// One-time warning: we sanitized rather than failed, but a non-finite
// sample reaching here means something upstream (an effect, automation,
// or a source decode) produced NaN/Inf — worth chasing if audio is wrong.
use std::sync::atomic::{AtomicBool, Ordering};
static WARNED: AtomicBool = AtomicBool::new(false);
if !WARNED.swap(true, Ordering::Relaxed) {
eprintln!(
"⚠️ [EXPORT] sanitized {} non-finite (NaN/Inf) audio sample(s) in a chunk — \
check effects/automation/source decode",
non_finite
);
}
}

2
daw-backend/src/audio/mod.rs
View File

@ -15,10 +15,12 @@ pub mod recording;
pub mod sample_loader;
pub mod track;
pub mod waveform_cache;
pub mod waveform_pyramid;
pub use automation::{AutomationLane, AutomationLaneId, AutomationPoint, CurveType, ParameterId};
pub use buffer_pool::BufferPool;
pub use clip::{AudioClipInstance, AudioClipInstanceId, Clip, ClipId};
pub use disk_reader::{AudioBlobSourceFactory, MediaByteSource};
pub use engine::{AudioClipSnapshot, Engine, EngineController};
pub use export::{export_audio, ExportFormat, ExportSettings};
pub use metronome::Metronome;

319
daw-backend/src/audio/pool.rs
View File

@ -4,6 +4,46 @@ use std::f32::consts::PI;
use serde::{Deserialize, Serialize};
use crate::time::Seconds;
/// Per-output-channel mix coefficients to fold a multichannel source down to
/// stereo, indexed `[out_channel(0=L,1=R)][src_channel]`.
///
/// Assumes the conventional interleave order for each channel count (FL, FR, FC,
/// LFE, BL, BR, SL, SR …). Uses standard ITU/AC-3-style coefficients: full level
/// for the matching front channel, `1/√2` (≈ −3 dB) for centre and each surround,
/// LFE dropped. Each row is then normalized so its absolute-coefficient sum ≤ 1,
/// which prevents clipping (matching FFmpeg's default `normalize` behaviour).
///
/// Returns `None` for layouts we don't special-case (caller falls back to taking
/// the front L/R pair).
fn stereo_downmix_matrix(src_channels: usize) -> Option<[Vec<f32>; 2]> {
const C: f32 = std::f32::consts::FRAC_1_SQRT_2; // ≈ 0.7071
// (L row, R row); each entry is the gain applied to that source channel.
let (l, r): (Vec<f32>, Vec<f32>) = match src_channels {
3 => (vec![1.0, 0.0, C], vec![0.0, 1.0, C]), // FL FR FC
4 => (vec![1.0, 0.0, C, 0.0], vec![0.0, 1.0, 0.0, C]), // quad: FL FR BL BR
5 => (vec![1.0, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C]), // FL FR FC BL BR
// 5.1: FL FR FC LFE BL BR (LFE dropped)
6 => (vec![1.0, 0.0, C, 0.0, C, 0.0], vec![0.0, 1.0, C, 0.0, 0.0, C]),
// 6.1: FL FR FC LFE BC SL SR (BC → both)
7 => (vec![1.0, 0.0, C, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C, 0.0, C]),
// 7.1: FL FR FC LFE BL BR SL SR
8 => (
vec![1.0, 0.0, C, 0.0, C, 0.0, C, 0.0],
vec![0.0, 1.0, C, 0.0, 0.0, C, 0.0, C],
),
_ => return None,
};
let normalize = |row: Vec<f32>| -> Vec<f32> {
let sum: f32 = row.iter().map(|c| c.abs()).sum();
if sum > 1.0 {
row.into_iter().map(|c| c / sum).collect()
} else {
row
}
};
Some([normalize(l), normalize(r)])
}
/// Windowed sinc interpolation for high-quality time stretching
/// This is stateless and can handle arbitrary fractional positions
#[inline]
@ -83,6 +123,16 @@ pub enum AudioStorage {
decoded_frames: u64,
total_frames: u64,
},
/// Audio track of a video container, decoded on demand via FFmpeg
/// (`VideoAudioReader`). The source video file is `AudioFile::path`. Like
/// `Compressed`, playback is streamed through the disk reader and
/// `decoded_for_waveform` is filled progressively for the overview.
VideoAudio {
decoded_for_waveform: Vec<f32>,
decoded_frames: u64,
total_frames: u64,
},
}
/// Audio file stored in the pool
@ -98,6 +148,10 @@ pub struct AudioFile {
pub original_format: Option<String>,
/// Original compressed file bytes (preserved across save/load to avoid re-encoding)
pub original_bytes: Option<Vec<u8>>,
/// When `Some`, this entry's bytes are packed in the project container (not on
/// disk at `path`); the disk reader opens them via the host's
/// `AudioBlobSourceFactory` using this media id. `None` ⇒ stream from `path`.
pub packed_media_id: Option<String>,
}
impl AudioFile {
@ -112,6 +166,7 @@ impl AudioFile {
frames,
original_format: None,
original_bytes: None,
packed_media_id: None,
}
}
@ -126,6 +181,7 @@ impl AudioFile {
frames,
original_format,
original_bytes: None,
packed_media_id: None,
}
}
@ -158,6 +214,7 @@ impl AudioFile {
frames: total_frames,
original_format: Some("wav".to_string()),
original_bytes: None,
packed_media_id: None,
}
}
@ -181,6 +238,32 @@ impl AudioFile {
frames: total_frames,
original_format,
original_bytes: None,
packed_media_id: None,
}
}
/// Create a placeholder AudioFile for a video's audio track. `path` is the
/// source video file; the audio is streamed on demand by the disk reader's
/// FFmpeg-backed `VideoAudioReader`.
pub fn from_video_audio(
path: PathBuf,
channels: u32,
sample_rate: u32,
total_frames: u64,
) -> Self {
Self {
path,
storage: AudioStorage::VideoAudio {
decoded_for_waveform: Vec::new(),
decoded_frames: 0,
total_frames,
},
channels,
sample_rate,
frames: total_frames,
original_format: None,
original_bytes: None,
packed_media_id: None,
}
}
@ -274,8 +357,8 @@ impl AudioFile {
}
written
}
AudioStorage::Compressed { .. } => {
// Compressed files are read through the disk reader
AudioStorage::Compressed { .. } | AudioStorage::VideoAudio { .. } => {
// Streamed through the disk reader, not via read_samples().
0
}
}
@ -537,6 +620,15 @@ impl AudioClipPool {
let dst_channels = engine_channels as usize;
let output_frames = output.len() / dst_channels;
// Fold a multichannel source (5.1, 7.1, …) down to stereo with proper
// coefficients (centre + surrounds mixed in, LFE dropped) instead of just
// taking the front L/R pair. `None` ⇒ no downmix needed / unknown layout.
let downmix = if dst_channels == 2 && src_channels > 2 {
stereo_downmix_matrix(src_channels)
} else {
None
};
let src_start_position = start_time_seconds * audio_file.sample_rate as f64;
// Tell the disk reader where we're reading so it buffers the right region.
@ -582,6 +674,15 @@ impl AudioClipPool {
sum += get_sample!(sf, src_ch);
}
sum / src_channels as f32
} else if let Some(ref mat) = downmix {
// Surround → stereo with proper coefficients.
let mut s = 0.0f32;
for (src_ch, &c) in mat[dst_ch].iter().enumerate() {
if c != 0.0 {
s += c * get_sample!(sf, src_ch);
}
}
s
} else {
get_sample!(sf, dst_ch % src_channels)
};
@ -606,39 +707,45 @@ impl AudioClipPool {
break;
}
// Sinc-interpolate a single source channel at the current position.
macro_rules! sinc_ch {
($ch:expr) => {{
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, $ch);
}
}
windowed_sinc_interpolate(&channel_samples, frac)
}};
}
for dst_ch in 0..dst_channels {
let sample = if let Some(ref mat) = downmix {
// Surround → stereo: interpolate each contributing channel.
let mut s = 0.0f32;
for (ch, &c) in mat[dst_ch].iter().enumerate() {
if c != 0.0 {
s += c * sinc_ch!(ch);
}
}
s
} else if dst_channels == 1 {
let mut sum = 0.0;
for ch in 0..src_channels {
sum += sinc_ch!(ch);
}
sum / src_channels as f32
} else {
let src_ch = if src_channels == dst_channels {
dst_ch
} else if src_channels == 1 {
0
} else if dst_channels == 1 {
usize::MAX // sentinel: average all channels below
} else {
dst_ch % src_channels
};
let sample = if src_ch == usize::MAX {
let mut sum = 0.0;
for ch in 0..src_channels {
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, ch);
}
}
sum += windowed_sinc_interpolate(&channel_samples, frac);
}
sum / src_channels as f32
} else {
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, src_ch);
}
}
windowed_sinc_interpolate(&channel_samples, frac)
sinc_ch!(src_ch)
};
output[output_frame * dst_channels + dst_ch] += sample * gain;
@ -786,6 +893,25 @@ pub struct AudioPoolEntry {
pub channels: u32,
/// Embedded audio data (for files < 10MB)
pub embedded_data: Option<EmbeddedAudioData>,
/// Stable media id (UUID string) for the SQLite `.beam` container. When set,
/// the audio bytes live in the container's `media` table keyed by this id
/// (packed storage). `None` for referenced entries (use `relative_path`) or
/// legacy ZIP-loaded entries. Populated by the file_io save/load layer.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub media_id: Option<String>,
/// Transient carrier for this entry's serialized waveform-pyramid blob (LBWF
/// bytes). Never serialized into project.json — the bytes live in the
/// container's `media` table (kind `Waveform`). Set by the file_io save layer
/// (in) and load layer (out); `None` everywhere else.
#[serde(skip)]
pub waveform_blob: Option<Vec<u8>>,
/// This entry is a video container's audio track (`relative_path` points at the
/// video file). It is always stored as a path reference (never packed/embedded
/// — the `VideoClip` already references the file) and reloaded by re-probing
/// the video via FFmpeg, so multichannel (5.1/7.1) audio survives the round-trip
/// (Symphonia reconstitution would otherwise collapse it).
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub is_video_audio: bool,
}
impl AudioClipPool {
@ -800,6 +926,66 @@ impl AudioClipPool {
let mut entries = Vec::new();
for (index, file) in self.files.iter().enumerate() {
// Skip placeholder pool slots: `load_from_serialized` resizes the pool to
// `max_index + 1` filled with empty `AudioFile::new(PathBuf::new(), …)` to
// cover index gaps (and creates one even when there are no entries at all).
// Such a slot has an empty path and no packed media — there's nothing to
// persist, and emitting it yields an entry whose empty `relative_path`
// resolves to the project directory itself (unreadable on the next save).
if file.path.as_os_str().is_empty() && file.packed_media_id.is_none() {
continue;
}
// Video's audio track: reference the video file (it's also referenced
// by the VideoClip) and re-probe it via FFmpeg on load. Never pack or
// embed it as audio media — that both wastes space and loses the 5.1+
// layout when Symphonia later decodes it.
if matches!(file.storage, AudioStorage::VideoAudio { .. }) {
let relative_path = pathdiff::diff_paths(&file.path, project_dir)
.map(|r| r.to_string_lossy().to_string())
.or_else(|| Some(file.path.to_string_lossy().to_string()));
entries.push(AudioPoolEntry {
pool_index: index,
is_video_audio: true,
waveform_blob: None,
name: file
.path
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| format!("file_{}", index)),
relative_path,
duration: file.duration_seconds(),
sample_rate: file.sample_rate,
channels: file.channels,
embedded_data: None,
media_id: None,
});
continue;
}
// Packed-in-container streaming entry: its bytes already live in the
// `.beam` media table (kept in place across re-saves). Emit just the
// media id — no path, no embedded bytes, nothing to decode.
if let Some(media_id) = &file.packed_media_id {
entries.push(AudioPoolEntry {
pool_index: index,
is_video_audio: false,
waveform_blob: None,
name: file
.path
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| format!("file_{}", index)),
relative_path: None,
duration: file.duration_seconds(),
sample_rate: file.sample_rate,
channels: file.channels,
embedded_data: None,
media_id: Some(media_id.clone()),
});
continue;
}
let file_path = &file.path;
let file_path_str = file_path.to_string_lossy();
@ -830,6 +1016,8 @@ impl AudioClipPool {
let entry = AudioPoolEntry {
pool_index: index,
is_video_audio: false,
waveform_blob: None,
name: file_path
.file_name()
.map(|n| n.to_string_lossy().to_string())
@ -839,6 +1027,7 @@ impl AudioClipPool {
sample_rate: file.sample_rate,
channels: file.channels,
embedded_data,
media_id: None,
};
entries.push(entry);
@ -962,22 +1151,86 @@ impl AudioClipPool {
self.files.clear();
eprintln!("📊 [LOAD_SERIALIZED] Clear pool took {:.2}ms", clear_start.elapsed().as_secs_f64() * 1000.0);
// Find the maximum pool index to determine required size
let max_index = entries.iter()
.map(|e| e.pool_index)
// Size the pool to hold the highest pool_index (slots are addressed by index,
// so gaps are filled with placeholders). No entries → length 0, NOT 1: the old
// `max().unwrap_or(0) + 1` produced a spurious placeholder for an empty pool.
let pool_size = entries.iter()
.map(|e| e.pool_index + 1)
.max()
.unwrap_or(0);
// Ensure we have space for all entries
let resize_start = std::time::Instant::now();
self.files.resize(max_index + 1, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100));
eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", max_index + 1, resize_start.elapsed().as_secs_f64() * 1000.0);
self.files.resize(pool_size, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100));
eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", pool_size, resize_start.elapsed().as_secs_f64() * 1000.0);
for (i, entry) in entries.iter().enumerate() {
let entry_start = std::time::Instant::now();
eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name);
let success = if let Some(ref embedded) = entry.embedded_data {
let success = if entry.is_video_audio {
// Re-probe the video's audio track via FFmpeg → a streaming
// VideoAudio entry (keeps full 5.1/7.1; no decode-to-RAM).
match entry.relative_path.as_ref() {
Some(rel) => {
let full = if std::path::Path::new(rel).is_absolute() {
PathBuf::from(rel)
} else {
project_dir.join(rel)
};
if full.exists() {
match crate::audio::disk_reader::VideoAudioReader::open(&full) {
Ok(reader) => {
let file = AudioFile::from_video_audio(
full,
reader.channels(),
reader.sample_rate(),
reader.total_frames(),
);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;
true
} else {
false
}
}
Err(e) => {
eprintln!("[AudioPool] Failed to reopen video audio {:?}: {}", full, e);
false
}
}
} else {
eprintln!("[AudioPool] Video file not found for audio: {:?}", full);
false
}
}
None => false,
}
} else if entry.media_id.is_some() && entry.embedded_data.is_none() {
// Packed-in-container streaming entry: build a Compressed placeholder
// backed by the host blob factory (opened at clip-activation time).
// No decode here — playback streams through the disk reader.
let media_id = entry.media_id.clone().unwrap();
let ext = std::path::Path::new(&entry.name)
.extension()
.and_then(|e| e.to_str())
.map(|s| s.to_lowercase());
let total_frames = (entry.duration * entry.sample_rate as f64).ceil() as u64;
let mut file = AudioFile::from_compressed(
PathBuf::from(&entry.name),
entry.channels,
entry.sample_rate,
total_frames,
ext,
);
file.packed_media_id = Some(media_id);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;
true
} else {
false
}
} else if let Some(ref embedded) = entry.embedded_data {
// Load from embedded data
eprintln!("📊 [LOAD_SERIALIZED] Entry has embedded data (format: {})", embedded.format);
match Self::load_from_embedded_into_pool(self, entry.pool_index, embedded.clone(), &entry.name) {

292
daw-backend/src/audio/waveform_pyramid.rs Normal file
View File

@ -0,0 +1,292 @@
//! Streaming min/max waveform LOD pyramid.
//!
//! A waveform pyramid is a tree of zoom levels. **Index = tree depth:**
//! `levels[0]` is the **root** (a single texel — the min/max envelope of the
//! whole file, lowest resolution); each deeper level is `BRANCH`× finer, and
//! `levels.last()` is the **floor** (one texel per `floor_samples_per_texel`
//! source frames — the finest *persisted* level). A node's children live at
//! `index + 1`, so the residency invariant ("a node is cleared only after its
//! children") reads straight off the index.
//!
//! Below the floor (finer than the floor bucket) is *not* stored; the caller
//! re-decodes the source window on demand for true per-sample detail.
//!
//! The builder is **streaming**: samples are pushed once, in order, and only the
//! finest level is accumulated (~`total_frames / floor` texels); the coarser
//! levels are derived by repeated `BRANCH:1` min/max reduction in [`finish`].
//! This yields the identical pyramid to an in-stream cascade (each parent = the
//! min/max of its children) without ever holding the full sample buffer.
//!
//! **Ragged edges are handled by reducing over available children:** a bucket
//! whose group is partial (1..BRANCH children, or `< floor` samples at the floor)
//! simply takes the min/max of what's there — no value padding. Padding to a
//! regular shape, if ever needed, is a GPU-texture/tile concern, not the data's.
//!
//! Each texel carries per-channel min/max for up to two channels
//! (`Lmin,Lmax,Rmin,Rmax`), matching the GPU waveform texture; mono mirrors the
//! left channel into the right.
//!
//! [`finish`]: WaveformPyramidBuilder::finish
/// Reduction factor between adjacent pyramid levels.
pub const BRANCH: u32 = 4;
/// Default finest-level resolution (source frames per floor texel). Trades
/// on-disk pyramid size against how soon zoom-in must re-decode the source.
pub const DEFAULT_FLOOR_SAMPLES_PER_TEXEL: u32 = 256;
/// One waveform texel: per-channel min/max (stereo; mono duplicates left→right).
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Texel {
pub l_min: f32,
pub l_max: f32,
pub r_min: f32,
pub r_max: f32,
}
impl Texel {
const EMPTY: Texel = Texel {
l_min: f32::INFINITY,
l_max: f32::NEG_INFINITY,
r_min: f32::INFINITY,
r_max: f32::NEG_INFINITY,
};
#[inline]
fn include_sample(&mut self, l: f32, r: f32) {
self.l_min = self.l_min.min(l);
self.l_max = self.l_max.max(l);
self.r_min = self.r_min.min(r);
self.r_max = self.r_max.max(r);
}
#[inline]
fn include_texel(&mut self, c: &Texel) {
self.l_min = self.l_min.min(c.l_min);
self.l_max = self.l_max.max(c.l_max);
self.r_min = self.r_min.min(c.r_min);
self.r_max = self.r_max.max(c.r_max);
}
}
/// A built min/max LOD pyramid, **root-first**: `levels[0]` is the coarsest
/// (whole-file envelope), `levels.last()` is the finest persisted (floor).
#[derive(Clone, Debug)]
pub struct WaveformPyramid {
pub floor_samples_per_texel: u32,
pub branch: u32,
pub channels: u32,
pub total_frames: u64,
pub levels: Vec<Vec<Texel>>,
}
impl WaveformPyramid {
/// Coarsest level — a single texel (whole-file envelope), or empty if no
/// samples were pushed.
pub fn root(&self) -> &[Texel] {
self.levels.first().map_or(&[][..], |v| v)
}
/// Finest persisted level (`floor_samples_per_texel` frames per texel).
pub fn floor(&self) -> &[Texel] {
self.levels.last().map_or(&[][..], |v| v)
}
/// Number of levels (tree depth + 1).
pub fn depth(&self) -> usize {
self.levels.len()
}
/// Serialize to a compact binary blob (for persisting in the `.beam`
/// container). Header carries `B`/branch/channels/total_frames + per-level
/// lengths, then root-first texel data (`f32` min/max).
pub fn to_bytes(&self) -> Vec<u8> {
let total_texels: usize = self.levels.iter().map(|l| l.len()).sum();
let mut out = Vec::with_capacity(32 + self.levels.len() * 4 + total_texels * 16);
out.extend_from_slice(b"LBWF");
out.extend_from_slice(&1u32.to_le_bytes()); // format version
out.extend_from_slice(&self.floor_samples_per_texel.to_le_bytes());
out.extend_from_slice(&self.branch.to_le_bytes());
out.extend_from_slice(&self.channels.to_le_bytes());
out.extend_from_slice(&self.total_frames.to_le_bytes());
out.extend_from_slice(&(self.levels.len() as u32).to_le_bytes());
for level in &self.levels {
out.extend_from_slice(&(level.len() as u32).to_le_bytes());
}
for level in &self.levels {
for t in level {
out.extend_from_slice(&t.l_min.to_le_bytes());
out.extend_from_slice(&t.l_max.to_le_bytes());
out.extend_from_slice(&t.r_min.to_le_bytes());
out.extend_from_slice(&t.r_max.to_le_bytes());
}
}
out
}
/// Reconstruct from [`WaveformPyramid::to_bytes`].
pub fn from_bytes(data: &[u8]) -> Result<WaveformPyramid, String> {
let mut r = ByteReader::new(data);
if r.take(4)? != b"LBWF" {
return Err("Not a waveform pyramid blob".to_string());
}
let version = r.u32()?;
if version != 1 {
return Err(format!("Unsupported waveform pyramid version {}", version));
}
let floor_samples_per_texel = r.u32()?;
let branch = r.u32()?;
let channels = r.u32()?;
let total_frames = r.u64()?;
let num_levels = r.u32()? as usize;
let mut level_lens = Vec::with_capacity(num_levels);
for _ in 0..num_levels {
level_lens.push(r.u32()? as usize);
}
let mut levels = Vec::with_capacity(num_levels);
for &len in &level_lens {
let mut level = Vec::with_capacity(len);
for _ in 0..len {
level.push(Texel {
l_min: r.f32()?,
l_max: r.f32()?,
r_min: r.f32()?,
r_max: r.f32()?,
});
}
levels.push(level);
}
Ok(WaveformPyramid {
floor_samples_per_texel,
branch,
channels,
total_frames,
levels,
})
}
}
/// Minimal little-endian byte cursor for [`WaveformPyramid::from_bytes`].
struct ByteReader<'a> {
data: &'a [u8],
pos: usize,
}
impl<'a> ByteReader<'a> {
fn new(data: &'a [u8]) -> Self {
Self { data, pos: 0 }
}
fn take(&mut self, n: usize) -> Result<&'a [u8], String> {
let end = self.pos.checked_add(n).ok_or("overflow")?;
if end > self.data.len() {
return Err("Truncated waveform pyramid blob".to_string());
}
let s = &self.data[self.pos..end];
self.pos = end;
Ok(s)
}
fn u32(&mut self) -> Result<u32, String> {
Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap()))
}
fn u64(&mut self) -> Result<u64, String> {
Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap()))
}
fn f32(&mut self) -> Result<f32, String> {
Ok(f32::from_le_bytes(self.take(4)?.try_into().unwrap()))
}
}
/// Streaming builder for a [`WaveformPyramid`]. See the module docs.
pub struct WaveformPyramidBuilder {
floor: u32,
branch: u32,
channels: u32,
total_frames: u64,
floor_level: Vec<Texel>,
acc: Texel,
acc_count: u32,
}
impl WaveformPyramidBuilder {
pub fn new(channels: u32, floor_samples_per_texel: u32) -> Self {
Self {
floor: floor_samples_per_texel.max(1),
branch: BRANCH,
channels: channels.max(1),
total_frames: 0,
floor_level: Vec::new(),
acc: Texel::EMPTY,
acc_count: 0,
}
}
/// Pre-reserve the floor `Vec` from an estimated total frame count (e.g. the
/// probe's `total_frames`), to avoid reallocations during streaming. Purely a
/// hint — the final size is set by the actual number of frames pushed.
pub fn reserve_for_frames(&mut self, estimated_frames: u64) {
let est_texels = (estimated_frames / self.floor as u64).saturating_add(1);
self.floor_level.reserve(est_texels.min(usize::MAX as u64) as usize);
}
/// Push a block of interleaved samples (`channels` per frame). Partial
/// trailing frames (fewer than `channels`) are ignored.
pub fn push_interleaved(&mut self, samples: &[f32]) {
let ch = self.channels as usize;
for frame in samples.chunks_exact(ch) {
let l = frame[0];
let r = if ch >= 2 { frame[1] } else { l };
self.push_frame(l, r);
}
}
#[inline]
fn push_frame(&mut self, l: f32, r: f32) {
self.total_frames += 1;
self.acc.include_sample(l, r);
self.acc_count += 1;
if self.acc_count >= self.floor {
self.floor_level.push(std::mem::replace(&mut self.acc, Texel::EMPTY));
self.acc_count = 0;
}
}
/// Flush the trailing partial bucket and reduce up to the root.
pub fn finish(mut self) -> WaveformPyramid {
if self.acc_count > 0 {
self.floor_level.push(self.acc);
}
// Build finest-first by repeated BRANCH:1 reduction until one texel.
// The shape is fully determined by the floor texel count; the last group
// at each level may be ragged (1..BRANCH children) and reduces over what
// it has.
let mut levels = vec![std::mem::take(&mut self.floor_level)];
let branch = self.branch as usize;
while levels.last().map_or(0, |l| l.len()) > 1 {
let prev = levels.last().unwrap();
let mut next = Vec::with_capacity(prev.len().div_ceil(branch));
for chunk in prev.chunks(branch) {
let mut t = Texel::EMPTY;
for c in chunk {
t.include_texel(c);
}
next.push(t);
}
levels.push(next);
}
// Output is root-first (convention B): levels[0] = root, last = floor.
levels.reverse();
WaveformPyramid {
floor_samples_per_texel: self.floor,
branch: self.branch,
channels: self.channels,
total_frames: self.total_frames,
levels,
}
}
}
// Tests live in `daw-backend/tests/waveform_pyramid.rs` (integration tests) so
// they build the lib in normal mode, independent of the crate's pre-existing
// broken `#[cfg(test)]` unit tests (automation.rs).

14
daw-backend/src/command/types.rs
View File

@ -430,8 +430,6 @@ pub enum Query {
/// Add a MIDI clip instance to a track synchronously (track_id, instance) - returns instance ID
/// The clip must already exist in the MidiClipPool
AddMidiClipInstanceSync(TrackId, crate::audio::midi::MidiClipInstance),
/// Add an audio file to the pool synchronously (path, data, channels, sample_rate) - returns pool index
AddAudioFileSync(String, Vec<f32>, u32, u32),
/// Import an audio file synchronously (path) - returns pool index.
/// Does the same work as Command::ImportAudio (mmap for PCM, streaming
/// setup for compressed) but returns the real pool index in the response.
@ -440,12 +438,20 @@ pub enum Query {
/// problem for very large files, switch to async import with event-based
/// pool index reconciliation.
ImportAudioSync(std::path::PathBuf),
/// Add the audio track of a video file as a streaming pool entry (FFmpeg,
/// decoded on demand — no extraction). Probes the audio track and returns
/// the pool index. Response: `AudioImportedSync`.
AddVideoAudioSync(std::path::PathBuf),
/// Get raw audio samples from pool (pool_index) - returns (samples, sample_rate, channels)
GetPoolAudioSamples(usize),
/// Get a clone of the current project for serialization
GetProject,
/// Set the project (replaces current project state)
SetProject(Box<crate::audio::project::Project>),
/// Install the host's packed-media byte-source factory (for streaming
/// container-packed audio on load). Sent before `SetProject` so bulk
/// activation can open packed sources.
SetBlobSourceFactory(std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>),
/// Duplicate a MIDI clip in the pool, returning the new clip's ID
DuplicateMidiClipSync(MidiClipId),
/// Get whether a track's graph is still the auto-generated default
@ -516,10 +522,10 @@ pub enum QueryResponse {
AudioExported(Result<(), String>),
/// MIDI clip instance added (returns instance ID)
MidiClipInstanceAdded(Result<MidiClipInstanceId, String>),
/// Audio file added to pool (returns pool index)
AudioFileAddedSync(Result<usize, String>),
/// Audio file imported to pool (returns pool index)
AudioImportedSync(Result<usize, String>),
/// Packed-media byte-source factory installed
BlobSourceFactorySet(Result<(), String>),
/// Raw audio samples from pool (samples, sample_rate, channels)
PoolAudioSamples(Result<(Vec<f32>, u32, u32), String>),
/// Project retrieved

89
daw-backend/tests/compressed_source_stream.rs Normal file
View File

@ -0,0 +1,89 @@
//! Integration test for `CompressedReader::open_source` — decoding a streaming
//! audio source from an in-memory byte stream (the packed-in-container path)
//! rather than a filesystem path. Proves the `MediaByteSource` adapter feeds
//! Symphonia correctly (probe + decode + seekable byte length).
use std::io::{Cursor, Read, Seek, SeekFrom};
use daw_backend::audio::disk_reader::{CompressedReader, MediaByteSource};
/// A `MediaByteSource` over an in-memory buffer (stands in for core's BlobReader).
struct VecSource(Cursor<Vec<u8>>);
impl Read for VecSource {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.0.read(buf)
}
}
impl Seek for VecSource {
fn seek(&mut self, pos: SeekFrom) -> std::io::Result<u64> {
self.0.seek(pos)
}
}
impl MediaByteSource for VecSource {
fn byte_len(&self) -> u64 {
self.0.get_ref().len() as u64
}
}
/// Build a minimal PCM16 mono WAV byte buffer holding `frames` samples of a ramp.
fn make_wav(sample_rate: u32, frames: u32) -> Vec<u8> {
let channels: u16 = 1;
let bits: u16 = 16;
let block_align: u16 = channels * bits / 8;
let byte_rate: u32 = sample_rate * block_align as u32;
let data_len: u32 = frames * block_align as u32;
let mut v = Vec::new();
v.extend_from_slice(b"RIFF");
v.extend_from_slice(&(36 + data_len).to_le_bytes());
v.extend_from_slice(b"WAVE");
v.extend_from_slice(b"fmt ");
v.extend_from_slice(&16u32.to_le_bytes());
v.extend_from_slice(&1u16.to_le_bytes()); // PCM
v.extend_from_slice(&channels.to_le_bytes());
v.extend_from_slice(&sample_rate.to_le_bytes());
v.extend_from_slice(&byte_rate.to_le_bytes());
v.extend_from_slice(&block_align.to_le_bytes());
v.extend_from_slice(&bits.to_le_bytes());
v.extend_from_slice(b"data");
v.extend_from_slice(&data_len.to_le_bytes());
for i in 0..frames {
// A ramp from -16000..16000 so values are recognizable.
let s = (((i % 1000) as i32 - 500) * 32) as i16;
v.extend_from_slice(&s.to_le_bytes());
}
v
}
#[test]
fn open_source_decodes_in_memory_wav() {
let sample_rate = 8000;
let frames = 4096;
let bytes = make_wav(sample_rate, frames);
let src = Box::new(VecSource(Cursor::new(bytes)));
let mut reader = CompressedReader::open_source(src, Some("wav"))
.expect("open_source should probe the in-memory WAV");
assert_eq!(reader.sample_rate(), sample_rate);
assert_eq!(reader.channels(), 1);
// Decode the whole stream and count emitted frames.
let mut buf = Vec::new();
let mut decoded = 0usize;
loop {
let n = reader.decode_next(&mut buf).expect("decode_next");
if n == 0 {
break;
}
decoded += n;
}
// Should recover (approximately) all frames — codec frame counts can round.
assert!(
(decoded as i64 - frames as i64).abs() < 64,
"decoded {} vs expected {}",
decoded,
frames
);
}

253
daw-backend/tests/video_audio_stream.rs Normal file
View File

@ -0,0 +1,253 @@
//! Integration tests for `VideoAudioReader` (FFmpeg streaming audio source).
//!
//! These build the daw-backend lib in normal mode, so they're independent of
//! the crate's pre-existing broken `#[cfg(test)]` unit tests (automation.rs).
//! They synthesize a mono 32-bit-float WAV whose sample `i` has value `i/n`, so
//! a decoded sample's value identifies its frame index — letting us assert both
//! in-order decoding and **sample-accurate seeking** (the property video audio
//! needs to stay synced with other clips).
use daw_backend::audio::disk_reader::{
build_waveform_pyramid, CompressedReader, SourceKind, VideoAudioReader,
};
use std::io::Write;
use std::path::Path;
fn write_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
let channels = 1u16;
let bytes_per_sample = 4u32;
let data_size = n * bytes_per_sample;
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
buf.extend_from_slice(b"RIFF");
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
buf.extend_from_slice(b"WAVE");
buf.extend_from_slice(b"fmt ");
buf.extend_from_slice(&16u32.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
buf.extend_from_slice(&channels.to_le_bytes());
buf.extend_from_slice(&sample_rate.to_le_bytes());
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
buf.extend_from_slice(&32u16.to_le_bytes());
buf.extend_from_slice(b"data");
buf.extend_from_slice(&data_size.to_le_bytes());
for i in 0..n {
buf.extend_from_slice(&((i as f32) / (n as f32)).to_le_bytes());
}
let mut f = std::fs::File::create(path).unwrap();
f.write_all(&buf).unwrap();
}
/// Stereo ramp: frame `i` has left = `i/n`, right = `0.5 - i/n` (distinct per
/// channel), interleaved `[L0,R0,L1,R1,…]`. Exercises the channels>1 path.
fn write_stereo_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
let channels = 2u16;
let bytes_per_sample = 4u32;
let data_size = n * channels as u32 * bytes_per_sample;
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
buf.extend_from_slice(b"RIFF");
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
buf.extend_from_slice(b"WAVE");
buf.extend_from_slice(b"fmt ");
buf.extend_from_slice(&16u32.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
buf.extend_from_slice(&channels.to_le_bytes());
buf.extend_from_slice(&sample_rate.to_le_bytes());
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
buf.extend_from_slice(&32u16.to_le_bytes());
buf.extend_from_slice(b"data");
buf.extend_from_slice(&data_size.to_le_bytes());
for i in 0..n {
let l = i as f32 / n as f32;
let r = 0.5 - i as f32 / n as f32;
buf.extend_from_slice(&l.to_le_bytes());
buf.extend_from_slice(&r.to_le_bytes());
}
let mut f = std::fs::File::create(path).unwrap();
f.write_all(&buf).unwrap();
}
fn temp_path(tag: &str) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!("lb_videoaudio_test_{}_{}.wav", std::process::id(), tag));
p
}
#[test]
fn decodes_samples_in_order() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("seq");
write_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
assert_eq!(reader.channels(), 1);
assert_eq!(reader.sample_rate(), sr);
// Probe estimate (used by add_video_audio_sync) should be ~n frames.
let tf = reader.total_frames() as f64;
assert!(
(tf - n as f64).abs() < n as f64 * 0.1,
"total_frames {} not ~{}",
tf, n
);
let mut all = Vec::new();
let mut buf = Vec::new();
loop {
let frames = reader.decode_next(&mut buf).unwrap();
if frames == 0 {
break;
}
all.extend_from_slice(&buf);
}
// Allow a couple of priming/flush samples of slack at the very end.
assert!(all.len() + 4 >= n as usize, "decoded too few samples: {}", all.len());
for (i, &v) in all.iter().enumerate().take(n as usize) {
let expected = i as f32 / n as f32;
assert!((v - expected).abs() < 1e-3, "sample {} = {}, expected {}", i, v, expected);
}
let _ = std::fs::remove_file(&path);
}
/// CompressedReader (symphonia) must seek **sample-accurately** too, so compressed
/// audio stays frame-synced with video audio. Symphonia decodes WAV via the same
/// path; its coarse seek lands on packet boundaries, exercising the decode-discard.
#[test]
fn compressed_reader_seek_is_sample_accurate() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("comp_seek");
write_ramp_wav(&path, n, sr);
let mut reader = CompressedReader::open(&path).unwrap();
assert_eq!(reader.channels(), 1);
assert_eq!(reader.sample_rate(), sr);
for &target in &[2000u64, 137, 3500, 0] {
let actual = reader.seek(target).unwrap();
assert_eq!(actual, target, "seek should report the exact target");
let mut buf = Vec::new();
let mut frames = 0;
for _ in 0..128 {
frames = reader.decode_next(&mut buf).unwrap();
if frames > 0 {
break;
}
}
assert!(frames > 0, "no samples after seek to {}", target);
let expected = target as f32 / n as f32;
assert!(
(buf[0] - expected).abs() < 1e-3,
"compressed seek to {}: first sample = {}, expected {}",
target, buf[0], expected
);
}
let _ = std::fs::remove_file(&path);
}
/// The decode→pyramid bridge should produce an envelope matching the signal,
/// through both reader backends (symphonia + ffmpeg), with bounded memory.
#[test]
fn waveform_pyramid_from_decode_matches_signal() {
let n = 5000u32;
let sr = 8000u32;
let path = temp_path("pyr");
write_ramp_wav(&path, n, sr); // ramp 0 .. (n-1)/n, all positive
for kind in [SourceKind::CompressedAudio, SourceKind::VideoAudio] {
let p = build_waveform_pyramid(&path, kind, 256).unwrap();
assert_eq!(p.channels, 1);
assert_eq!(p.root().len(), 1, "{:?}: root should be one texel", kind);
let root = p.root()[0];
assert!(root.l_min.abs() < 1e-2, "{:?}: root min {} ~ 0", kind, root.l_min);
let expected_max = (n - 1) as f32 / n as f32;
assert!(
(root.l_max - expected_max).abs() < 1e-2,
"{:?}: root max {} ~ {}", kind, root.l_max, expected_max
);
// Frame count is approximate across decoders (priming/resampler overhead);
// the envelope above is the real check. Just confirm it's about right.
assert!((p.total_frames as i64 - n as i64).abs() < 128, "{:?}: frames {}", kind, p.total_frames);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn decodes_stereo_interleaved() {
let n = 2000u32;
let sr = 8000u32;
let path = temp_path("stereo");
write_stereo_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
assert_eq!(reader.channels(), 2);
let mut all = Vec::new();
let mut buf = Vec::new();
loop {
let frames = reader.decode_next(&mut buf).unwrap();
if frames == 0 {
break;
}
// Each decode_next returns whole interleaved frames.
assert_eq!(buf.len() % 2, 0, "stereo decode returned a partial frame");
all.extend_from_slice(&buf);
}
// Interleaved L/R, ~n frames.
assert!(all.len() + 8 >= (n * 2) as usize, "decoded too few samples: {}", all.len());
for i in 0..n as usize {
let l = all[2 * i];
let r = all[2 * i + 1];
assert!((l - i as f32 / n as f32).abs() < 1e-3, "L[{}]={} expected {}", i, l, i as f32 / n as f32);
assert!(
(r - (0.5 - i as f32 / n as f32)).abs() < 1e-3,
"R[{}]={} expected {}", i, r, 0.5 - i as f32 / n as f32
);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn seek_is_sample_accurate() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("seek");
write_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
for &target in &[2000u64, 137, 3500, 0] {
let actual = reader.seek(target).unwrap();
assert_eq!(actual, target);
// Pull the first non-empty decode after the seek.
let mut buf = Vec::new();
let mut frames = 0;
for _ in 0..64 {
frames = reader.decode_next(&mut buf).unwrap();
if frames > 0 {
break;
}
}
assert!(frames > 0, "no samples after seek to {}", target);
let expected = target as f32 / n as f32;
assert!(
(buf[0] - expected).abs() < 1e-3,
"after seek to {}: first sample = {}, expected {}",
target,
buf[0],
expected
);
// And the next few advance in order.
for k in 0..frames.min(8) {
let exp = (target as usize + k) as f32 / n as f32;
assert!((buf[k] - exp).abs() < 1e-3, "seek {}+{}: {} vs {}", target, k, buf[k], exp);
}
}
let _ = std::fs::remove_file(&path);
}

135
daw-backend/tests/waveform_pyramid.rs Normal file
View File

@ -0,0 +1,135 @@
//! Integration tests for the streaming waveform LOD pyramid builder.
//!
//! Convention B: `levels[0]` is the root (coarsest), `levels.last()` the floor
//! (finest). Tests use the `.root()` / `.floor()` accessors so they don't depend
//! on the raw index ordering.
use daw_backend::audio::waveform_pyramid::{Texel, WaveformPyramid, WaveformPyramidBuilder};
fn build_mono(samples: &[f32], floor: u32) -> WaveformPyramid {
let mut b = WaveformPyramidBuilder::new(1, floor);
b.push_interleaved(samples);
b.finish()
}
#[test]
fn floor_level_min_max_per_bucket() {
// 8 samples, floor 4 → two floor texels covering [0..4) and [4..8).
let s: Vec<f32> = (0..8).map(|i| i as f32).collect();
let p = build_mono(&s, 4);
assert_eq!(p.floor().len(), 2);
assert_eq!(p.floor()[0], Texel { l_min: 0.0, l_max: 3.0, r_min: 0.0, r_max: 3.0 });
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 7.0, r_min: 4.0, r_max: 7.0 });
// Root reduces the two floor texels into the envelope [0..8).
assert_eq!(p.root().len(), 1);
assert_eq!(p.root()[0], Texel { l_min: 0.0, l_max: 7.0, r_min: 0.0, r_max: 7.0 });
}
#[test]
fn partial_trailing_bucket_is_flushed() {
// 6 samples, floor 4 → texels [0..4) and a ragged [4..6).
let s: Vec<f32> = (0..6).map(|i| i as f32).collect();
let p = build_mono(&s, 4);
assert_eq!(p.floor().len(), 2);
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 5.0, r_min: 4.0, r_max: 5.0 });
assert_eq!(p.total_frames, 6);
}
#[test]
fn multi_level_envelope_matches_global_min_max() {
let s: Vec<f32> = (0..1000).map(|i| ((i as f32) * 0.01).sin()).collect();
let g_min = s.iter().cloned().fold(f32::INFINITY, f32::min);
let g_max = s.iter().cloned().fold(f32::NEG_INFINITY, f32::max);
let p = build_mono(&s, 16);
assert_eq!(p.root().len(), 1);
assert!((p.root()[0].l_min - g_min).abs() < 1e-6);
assert!((p.root()[0].l_max - g_max).abs() < 1e-6);
// Every level's overall min/max equals the global (extremes are lossless).
for level in &p.levels {
let lmin = level.iter().map(|t| t.l_min).fold(f32::INFINITY, f32::min);
let lmax = level.iter().map(|t| t.l_max).fold(f32::NEG_INFINITY, f32::max);
assert!((lmin - g_min).abs() < 1e-6);
assert!((lmax - g_max).abs() < 1e-6);
}
}
#[test]
fn levels_are_root_first_and_get_finer() {
let s: Vec<f32> = (0..1000).map(|i| i as f32).collect();
let p = build_mono(&s, 16);
// Root first, floor last; strictly finer (more texels) as depth increases.
assert_eq!(p.root().len(), 1);
assert!(p.depth() >= 3);
for w in p.levels.windows(2) {
assert!(w[1].len() >= w[0].len(), "deeper level should be finer");
}
// Floor has ceil(1000/16) = 63 texels.
assert_eq!(p.floor().len(), 63);
}
#[test]
fn stereo_channels_tracked_separately() {
// L ramps up, R ramps down; interleaved.
let n = 64;
let mut s = Vec::new();
for i in 0..n {
s.push(i as f32); // L
s.push(-(i as f32)); // R
}
let mut b = WaveformPyramidBuilder::new(2, 16);
b.push_interleaved(&s);
let p = b.finish();
assert_eq!(p.root().len(), 1);
assert_eq!(p.root()[0].l_min, 0.0);
assert_eq!(p.root()[0].l_max, (n - 1) as f32);
assert_eq!(p.root()[0].r_min, -((n - 1) as f32));
assert_eq!(p.root()[0].r_max, 0.0);
}
#[test]
fn pyramid_size_is_bounded() {
let n = 100_000usize;
let s: Vec<f32> = (0..n).map(|i| (i % 7) as f32).collect();
let floor = 256u32;
let p = build_mono(&s, floor);
let total: usize = p.levels.iter().map(|l| l.len()).sum();
let floor_texels = (n as u32).div_ceil(floor) as usize;
// Geometric bound: < floor_texels * branch/(branch-1) + small per-level slack.
let bound = floor_texels * 4 / 3 + p.depth() + 2;
assert!(total <= bound, "pyramid too big: {} > {}", total, bound);
}
#[test]
fn bytes_round_trip() {
let s: Vec<f32> = (0..3333).map(|i| ((i as f32) * 0.013).sin()).collect();
let p = build_mono(&s, 64);
let bytes = p.to_bytes();
let q = WaveformPyramid::from_bytes(&bytes).unwrap();
assert_eq!(p.floor_samples_per_texel, q.floor_samples_per_texel);
assert_eq!(p.branch, q.branch);
assert_eq!(p.channels, q.channels);
assert_eq!(p.total_frames, q.total_frames);
assert_eq!(p.levels, q.levels);
// Truncated/garbage input is rejected, not panicking.
assert!(WaveformPyramid::from_bytes(&bytes[..bytes.len() - 4]).is_err());
assert!(WaveformPyramid::from_bytes(b"nope").is_err());
}
#[test]
fn pushing_in_arbitrary_chunks_matches() {
// The streaming builder must be agnostic to how samples are chunked.
let s: Vec<f32> = (0..5000).map(|i| ((i * 13) % 97) as f32 - 48.0).collect();
let whole = build_mono(&s, 32);
let mut b = WaveformPyramidBuilder::new(1, 32);
b.reserve_for_frames(5000);
for chunk in s.chunks(37) {
b.push_interleaved(chunk);
}
let chunked = b.finish();
assert_eq!(whole.depth(), chunked.depth());
for (a, c) in whole.levels.iter().zip(chunked.levels.iter()) {
assert_eq!(a, c);
}
}

52
lightningbeam-ui/Cargo.lock generated
View File

@ -2160,6 +2160,18 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
[[package]]
name = "fallible-iterator"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
[[package]]
name = "fallible-streaming-iterator"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]]
name = "fancy-regex"
version = "0.16.2"
@ -2925,6 +2937,9 @@ name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
dependencies = [
"ahash 0.8.12",
]
[[package]]
name = "hashbrown"
@ -2946,6 +2961,15 @@ dependencies = [
"foldhash 0.2.0",
]
[[package]]
name = "hashlink"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
dependencies = [
"hashbrown 0.14.5",
]
[[package]]
name = "heapless"
version = "0.8.0"
@ -3415,6 +3439,17 @@ dependencies = [
"redox_syscall 0.5.18",
]
[[package]]
name = "libsqlite3-sys"
version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f"
dependencies = [
"cc",
"pkg-config",
"vcpkg",
]
[[package]]
name = "libxdo"
version = "0.6.0"
@ -3455,6 +3490,7 @@ dependencies = [
"objc2-foundation 0.3.2",
"pathdiff",
"rstar",
"rusqlite",
"serde",
"serde_json",
"tiny-skia",
@ -3469,7 +3505,7 @@ dependencies = [
[[package]]
name = "lightningbeam-editor"
version = "1.0.3-alpha"
version = "1.0.4-alpha"
dependencies = [
"beamdsp",
"bytemuck",
@ -5478,6 +5514,20 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba"
[[package]]
name = "rusqlite"
version = "0.31.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae"
dependencies = [
"bitflags 2.10.0",
"fallible-iterator",
"fallible-streaming-iterator",
"hashlink",
"libsqlite3-sys",
"smallvec",
]
[[package]]
name = "rustc-hash"
version = "1.1.0"

3
lightningbeam-ui/lightningbeam-core/Cargo.toml
View File

@ -36,6 +36,9 @@ zip = "0.6"
chrono = "0.4"
base64 = "0.21"
pathdiff = "0.2"
# .beam container: SQLite database file. `bundled` compiles SQLite from source
# (no system dependency); `blob` enables incremental blob I/O for streaming.
rusqlite = { version = "0.31", features = ["bundled", "blob"] }
# Audio encoding for embedded files
flacenc = "0.4" # For FLAC encoding (lossless)

21
lightningbeam-ui/lightningbeam-core/src/action.rs
View File

@ -71,6 +71,15 @@ pub trait Action: Send {
/// Get a human-readable description of this action (for UI display)
fn description(&self) -> String;
/// For raster actions that store dirty-rect diffs: the `(layer_id, time)` of the
/// keyframe whose full pixels must be resident before `execute`/`rollback` can
/// apply the diff. The editor faults the frame in (synchronously) before undo/redo
/// so a paged-out clean frame is restored to its container state first. Non-raster
/// actions (and full-buffer ones) return `None`.
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
None
}
/// Execute backend operations after document changes
///
/// Called AFTER execute() succeeds. If this returns an error, execute()
@ -290,6 +299,18 @@ impl ActionExecutor {
self.undo_stack.last().map(|a| a.description())
}
/// `(layer_id, time)` of the raster keyframe the next undo needs resident, if any.
/// The editor faults it in before calling `undo()` so a paged-out clean frame is
/// restored to its container state, giving the diff a correct base to apply onto.
pub fn peek_undo_raster_hint(&self) -> Option<(Uuid, f64)> {
self.undo_stack.last().and_then(|a| a.raster_resident_hint())
}
/// `(layer_id, time)` of the raster keyframe the next redo needs resident, if any.
pub fn peek_redo_raster_hint(&self) -> Option<(Uuid, f64)> {
self.redo_stack.last().and_then(|a| a.raster_resident_hint())
}
/// Get MIDI cache data from the last action on the undo stack (after redo).
/// Returns the notes reflecting execute state.
pub fn last_undo_midi_notes(&self) -> Option<(u32, &[(f64, u8, u8, f64)])> {

20
lightningbeam-ui/lightningbeam-core/src/actions/add_clip_instance.rs
View File

@ -235,11 +235,22 @@ impl Action for AddClipInstanceAction {
}
}
AudioClipType::Sampled { audio_pool_index } => {
// `trim_*` / `clip.duration` are in SECONDS (audio content time),
// while `timeline_*` and the backend's `duration` are in BEATS.
let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
let effective_duration = self.clip_instance.timeline_duration
.unwrap_or(internal_end - internal_start);
let start_time = self.clip_instance.timeline_start;
// `effective_duration` is in BEATS. When `timeline_duration` is set
// it already is; otherwise the clip occupies its natural content
// length, so convert that seconds-span to beats at the clip's start
// (NOT `internal_end - internal_start`, which is seconds — that was
// the seconds-as-beats bug that made clips stop early off 60 BPM).
let effective_duration = self.clip_instance.timeline_duration.unwrap_or_else(|| {
let tempo_map = document.tempo_map();
let content_secs = internal_end - internal_start;
tempo_map.inverse_transform(tempo_map.transform(start_time) + content_secs)
- start_time
});
let instance_id = controller.add_audio_clip(
*backend_track_id,
@ -305,8 +316,11 @@ mod tests {
let layer_id = layer.layer.id;
document.root_mut().add_child(AnyLayer::Vector(layer));
// Create a clip instance (using a fake clip_id since we're just testing the action)
// Register a vector clip so get_clip_duration succeeds
let clip_id = Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 5.0);
document.vector_clips.insert(clip_id, clip);
let clip_instance = ClipInstance::new(clip_id);
let instance_id = clip_instance.id;

52
lightningbeam-ui/lightningbeam-core/src/actions/add_raster_keyframe.rs Normal file
View File

@ -0,0 +1,52 @@
//! Add a blank raster keyframe at the playhead (the explicit "New Keyframe" command
//! for raster layers — mirrors `SetKeyframeAction` for vector, but inserts an empty
//! cel rather than copying the current graph).
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use uuid::Uuid;
pub struct AddRasterKeyframeAction {
layer_id: Uuid,
time: f64,
width: u32,
height: u32,
/// Id of the keyframe created by the last `execute` (so `rollback` can remove
/// exactly that one). `None` if a keyframe already existed at `time` (no-op).
created_id: Option<Uuid>,
}
impl AddRasterKeyframeAction {
pub fn new(layer_id: Uuid, time: f64, width: u32, height: u32) -> Self {
Self { layer_id, time, width, height, created_id: None }
}
}
impl Action for AddRasterKeyframeAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
let rl = match layer {
AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()),
};
// Inserts a blank cel only if one doesn't already exist at this time.
self.created_id = rl.insert_blank_keyframe_at(self.time, self.width, self.height);
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if let Some(id) = self.created_id.take() {
if let Some(AnyLayer::Raster(rl)) = document.get_layer_mut(&self.layer_id) {
rl.remove_keyframe(id);
}
}
Ok(())
}
fn description(&self) -> String {
"New keyframe".to_string()
}
}

60
lightningbeam-ui/lightningbeam-core/src/actions/add_shape.rs
View File

@ -23,6 +23,9 @@ pub struct AddShapeAction {
stroke_color: Option<ShapeColor>,
fill_color: Option<ShapeColor>,
is_closed: bool,
/// When set, the enclosed region is filled with this image asset (instead of a
/// solid colour). The renderer prioritises `image_fill` over colour/gradient.
image_fill: Option<Uuid>,
description_text: String,
/// Snapshot of the graph before insertion (for undo).
graph_before: Option<VectorGraph>,
@ -46,15 +49,53 @@ impl AddShapeAction {
stroke_color,
fill_color,
is_closed,
image_fill: None,
description_text: "Add shape".to_string(),
graph_before: None,
}
}
/// A borderless, axis-aligned rectangle filled with an image asset — the result
/// of importing/dropping an image onto a vector layer.
pub fn image_rect(
layer_id: Uuid,
time: f64,
x: f64,
y: f64,
w: f64,
h: f64,
asset_id: Uuid,
) -> Self {
let mut path = BezPath::new();
path.move_to((x, y));
path.line_to((x + w, y));
path.line_to((x + w, y + h));
path.line_to((x, y + h));
path.close_path();
Self {
layer_id,
time,
path,
stroke_style: None, // invisible edges — just the image
stroke_color: None,
fill_color: None,
is_closed: true,
image_fill: Some(asset_id),
description_text: "Add image".to_string(),
graph_before: None,
}
}
pub fn with_description(mut self, desc: impl Into<String>) -> Self {
self.description_text = desc.into();
self
}
/// Fill the created region with an image asset (image takes render priority).
pub fn with_image_fill(mut self, asset_id: Uuid) -> Self {
self.image_fill = Some(asset_id);
self
}
}
impl Action for AddShapeAction {
@ -87,16 +128,25 @@ impl Action for AddShapeAction {
DEFAULT_SNAP_EPSILON,
);
// Apply fill if this is a closed shape with fill
if self.is_closed {
if let Some(ref fill) = self.fill_color {
// Compute centroid of the path's bounding box and paint-bucket fill
// Apply fill if this is a closed shape with a colour and/or image fill.
if self.is_closed && (self.fill_color.is_some() || self.image_fill.is_some()) {
// Compute centroid of the path's bounding box and paint-bucket fill.
let bbox = self.path.bounding_box();
let centroid = kurbo::Point::new(
(bbox.x0 + bbox.x1) / 2.0,
(bbox.y0 + bbox.y1) / 2.0,
);
graph.paint_bucket(centroid, fill.clone(), FillRule::NonZero, 0.0);
// paint_bucket needs a colour; an image-only fill uses a placeholder
// that the image overrides (cleared below).
let color = self.fill_color.clone().unwrap_or_else(|| ShapeColor::rgba(255, 255, 255, 255));
if let Some(fid) = graph.paint_bucket(centroid, color, FillRule::NonZero, 0.0) {
if let Some(asset_id) = self.image_fill {
let fill = graph.fill_mut(fid);
fill.image_fill = Some(asset_id);
if self.fill_color.is_none() {
fill.color = None; // image-only: don't double-paint a colour
}
}
}
}
}

126
lightningbeam-ui/lightningbeam-core/src/actions/clip_from_geometry.rs Normal file
View File

@ -0,0 +1,126 @@
//! Shared logic for the "Group" and "Convert to Movie Clip" actions: extract the
//! selected DCEL geometry from a vector layer's active keyframe into a new `VectorClip`
//! and drop a `ClipInstance` in its place (so it can then be motion-tweened).
//!
//! A *group* (`is_group = true`) is a static container; a *movie clip* (`is_group =
//! false`) has its own timeline. Both are tweenable via the clip instance's transform.
//!
//! Both the Select tool and the (cut-and-select) RegionSelect tool populate the same
//! `Selection` ID sets, so a single entry point — [`extract_geometry_to_clip`] — handles
//! Group and Convert-to-Movie-Clip from whatever is selected.
use std::collections::HashSet;
use crate::clip::{ClipInstance, VectorClip};
use crate::document::Document;
use crate::layer::{AnyLayer, ShapeKeyframe, VectorLayer};
use crate::object::Transform;
use crate::vector_graph::{EdgeId, FillId, VectorGraph};
use uuid::Uuid;
/// Build a clip holding `sub_graph` and place a `ClipInstance` (with `transform`) on the
/// layer. Shared by both extraction paths. Does **not** touch the source graph — the
/// caller is responsible for having removed the moved geometry first.
fn assemble_clip_from_graph(
document: &mut Document,
layer_id: Uuid,
time: f64,
sub_graph: VectorGraph,
transform: Transform,
clip_id: Uuid,
instance_id: Uuid,
is_group: bool,
clip_name: &str,
) {
let (doc_w, doc_h, doc_dur) = (document.width, document.height, document.duration.max(1.0));
// A vector layer whose single keyframe holds the extracted graph (in the source's
// coordinate space, so an identity/translation placement renders it in place).
let mut inner = VectorLayer::new("Layer 1");
let mut kf = ShapeKeyframe::new(0.0);
kf.graph = sub_graph;
inner.keyframes.push(kf);
let mut clip = VectorClip::with_id(clip_id, clip_name, doc_w, doc_h, doc_dur);
clip.is_group = is_group;
clip.layers.add_root(AnyLayer::Vector(inner));
document.add_vector_clip(clip);
let mut instance = ClipInstance::with_id(instance_id, clip_id);
instance.transform = transform;
if let Some(AnyLayer::Vector(vl)) = document.get_layer_mut(&layer_id) {
// Groups gate visibility by the active keyframe's clip_instance_ids; movie
// clips render unconditionally.
if is_group {
if let Some(kf) = vl.keyframe_at_mut(time) {
kf.clip_instance_ids.push(instance_id);
}
}
vl.clip_instances.push(instance);
}
}
/// Extract the selected geometry into a new clip + place a `ClipInstance`. Returns the
/// pre-extraction graph snapshot for undo. `clip_id`/`instance_id` are caller-provided
/// so undo/redo is stable. The selection sets come straight from the editor selection
/// (`select_fill` already includes each fill's boundary edges); `extract_subgraph`
/// derives which of those edges are shared with non-selected shapes.
pub fn extract_geometry_to_clip(
document: &mut Document,
layer_id: Uuid,
time: f64,
fills: &HashSet<FillId>,
edges: &HashSet<EdgeId>,
clip_id: Uuid,
instance_id: Uuid,
is_group: bool,
clip_name: &str,
) -> Result<VectorGraph, String> {
if fills.is_empty() && edges.is_empty() {
return Err("No geometry selected".to_string());
}
// 1. Extract from the source graph (extract_subgraph removes the moved geometry).
let (graph_before, sub_graph) = {
let layer = document.get_layer_mut(&layer_id).ok_or("Layer not found")?;
let vl = match layer {
AnyLayer::Vector(vl) => vl,
_ => return Err("Not a vector layer".to_string()),
};
let graph = vl.graph_at_time_mut(time).ok_or("No keyframe at time")?;
let before = graph.clone();
// No explicit cut boundary — extract_subgraph derives shared-fill boundaries.
let (sub, _, _) = graph.extract_subgraph(edges, fills, &HashSet::new());
(before, sub)
};
// 2 & 3. Build the clip + place an identity-transform instance (geometry stays put).
assemble_clip_from_graph(
document, layer_id, time, sub_graph, Transform::default(),
clip_id, instance_id, is_group, clip_name,
);
Ok(graph_before)
}
/// Reverse `extract_geometry_to_clip`: remove the clip + instance and restore the graph.
pub fn undo_extract_geometry(
document: &mut Document,
layer_id: Uuid,
time: f64,
clip_id: Uuid,
instance_id: Uuid,
graph_before: &VectorGraph,
) {
document.vector_clips.remove(&clip_id);
document.rebuild_layer_to_clip_map();
if let Some(AnyLayer::Vector(vl)) = document.get_layer_mut(&layer_id) {
vl.clip_instances.retain(|ci| ci.id != instance_id);
if let Some(kf) = vl.keyframe_at_mut(time) {
kf.clip_instance_ids.retain(|id| *id != instance_id);
}
if let Some(graph) = vl.graph_at_time_mut(time) {
*graph = graph_before.clone();
}
}
}

52
lightningbeam-ui/lightningbeam-core/src/actions/convert_to_movie_clip.rs
View File

@ -1,55 +1,57 @@
//! Convert to Movie Clip action — STUB: needs DCEL rewrite
//! Convert to Movie Clip — extract selected geometry into a movie-clip `VectorClip`
//! (its own timeline) + a `ClipInstance` that can be motion-tweened.
use std::collections::HashSet;
use crate::action::Action;
use crate::clip::ClipInstance;
use crate::actions::clip_from_geometry::{extract_geometry_to_clip, undo_extract_geometry};
use crate::document::Document;
use crate::vector_graph::{EdgeId, FillId, VectorGraph};
use uuid::Uuid;
/// Action that converts selected items to a Movie Clip
/// TODO: Rewrite for DCEL
#[allow(dead_code)]
pub struct ConvertToMovieClipAction {
layer_id: Uuid,
time: f64,
shape_ids: Vec<Uuid>,
clip_instance_ids: Vec<Uuid>,
fills: Vec<FillId>,
edges: Vec<EdgeId>,
clip_id: Uuid,
instance_id: Uuid,
created_clip_id: Option<Uuid>,
removed_clip_instances: Vec<ClipInstance>,
graph_before: Option<VectorGraph>,
}
impl ConvertToMovieClipAction {
pub fn new(
layer_id: Uuid,
time: f64,
shape_ids: Vec<Uuid>,
clip_instance_ids: Vec<Uuid>,
fills: Vec<FillId>,
edges: Vec<EdgeId>,
clip_id: Uuid,
instance_id: Uuid,
) -> Self {
Self {
layer_id,
time,
shape_ids,
clip_instance_ids,
instance_id,
created_clip_id: None,
removed_clip_instances: Vec::new(),
}
Self { layer_id, time, fills, edges, clip_id, instance_id, graph_before: None }
}
}
impl Action for ConvertToMovieClipAction {
fn execute(&mut self, _document: &mut Document) -> Result<(), String> {
let _ = (&self.layer_id, self.time, &self.shape_ids, &self.clip_instance_ids, self.instance_id);
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let fills: HashSet<FillId> = self.fills.iter().copied().collect();
let edges: HashSet<EdgeId> = self.edges.iter().copied().collect();
let before = extract_geometry_to_clip(
document, self.layer_id, self.time, &fills, &edges,
self.clip_id, self.instance_id, false, "Movie Clip",
)?;
self.graph_before = Some(before);
Ok(())
}
fn rollback(&mut self, _document: &mut Document) -> Result<(), String> {
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if let Some(before) = &self.graph_before {
undo_extract_geometry(document, self.layer_id, self.time, self.clip_id, self.instance_id, before);
}
Ok(())
}
fn description(&self) -> String {
let count = self.shape_ids.len() + self.clip_instance_ids.len();
format!("Convert {} object(s) to Movie Clip", count)
"Convert to Movie Clip".to_string()
}
}

13
lightningbeam-ui/lightningbeam-core/src/actions/delete_folder.rs
View File

@ -37,6 +37,9 @@ pub struct DeleteFolderAction {
/// Asset IDs that were deleted (for DeleteRecursive strategy)
deleted_asset_ids: Vec<Uuid>,
/// Subfolder IDs that were reparented to the deleted folder's parent (for MoveToParent strategy)
moved_subfolder_ids: Vec<Uuid>,
}
impl DeleteFolderAction {
@ -55,6 +58,7 @@ impl DeleteFolderAction {
removed_folders: Vec::new(),
moved_asset_ids: Vec::new(),
deleted_asset_ids: Vec::new(),
moved_subfolder_ids: Vec::new(),
}
}
}
@ -130,6 +134,7 @@ impl Action for DeleteFolderAction {
for subfolder_id in subfolder_ids {
if let Some(subfolder) = tree.folders.get_mut(&subfolder_id) {
subfolder.parent_id = parent_id;
self.moved_subfolder_ids.push(subfolder_id);
}
}
}
@ -259,6 +264,13 @@ impl Action for DeleteFolderAction {
tree.add_folder(folder.clone());
}
// Restore reparented subfolders back under the deleted folder
for subfolder_id in &self.moved_subfolder_ids {
if let Some(subfolder) = tree.folders.get_mut(subfolder_id) {
subfolder.parent_id = Some(self.folder_id);
}
}
match self.strategy {
DeleteStrategy::MoveToParent => {
// Restore folder_id for moved assets
@ -312,6 +324,7 @@ impl Action for DeleteFolderAction {
self.removed_folders.clear();
self.moved_asset_ids.clear();
self.deleted_asset_ids.clear();
self.moved_subfolder_ids.clear();
Ok(())
}

54
lightningbeam-ui/lightningbeam-core/src/actions/group_shapes.rs
View File

@ -1,56 +1,58 @@
//! Group action — STUB: needs DCEL rewrite
//! Group action — extract selected geometry into a group `VectorClip` + a `ClipInstance`.
use std::collections::HashSet;
use crate::action::Action;
use crate::clip::ClipInstance;
use crate::actions::clip_from_geometry::{extract_geometry_to_clip, undo_extract_geometry};
use crate::document::Document;
use crate::vector_graph::{EdgeId, FillId, VectorGraph};
use uuid::Uuid;
/// Action that groups selected shapes and/or clip instances into a VectorClip
/// TODO: Rewrite for DCEL (group DCEL faces/edges into a sub-clip)
#[allow(dead_code)]
/// Groups the selected DCEL geometry (fills/edges) of a vector layer's active keyframe
/// into a new group clip, placing a clip instance in its place (which can be tweened).
pub struct GroupAction {
layer_id: Uuid,
time: f64,
shape_ids: Vec<Uuid>,
clip_instance_ids: Vec<Uuid>,
fills: Vec<FillId>,
edges: Vec<EdgeId>,
clip_id: Uuid,
instance_id: Uuid,
created_clip_id: Option<Uuid>,
removed_clip_instances: Vec<ClipInstance>,
graph_before: Option<VectorGraph>,
}
impl GroupAction {
pub fn new(
layer_id: Uuid,
time: f64,
shape_ids: Vec<Uuid>,
clip_instance_ids: Vec<Uuid>,
fills: Vec<FillId>,
edges: Vec<EdgeId>,
clip_id: Uuid,
instance_id: Uuid,
) -> Self {
Self {
layer_id,
time,
shape_ids,
clip_instance_ids,
instance_id,
created_clip_id: None,
removed_clip_instances: Vec::new(),
}
Self { layer_id, time, fills, edges, clip_id, instance_id, graph_before: None }
}
}
impl Action for GroupAction {
fn execute(&mut self, _document: &mut Document) -> Result<(), String> {
let _ = (&self.layer_id, self.time, &self.shape_ids, &self.clip_instance_ids, self.instance_id);
// TODO: Implement DCEL-aware grouping
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let fills: HashSet<FillId> = self.fills.iter().copied().collect();
let edges: HashSet<EdgeId> = self.edges.iter().copied().collect();
let before = extract_geometry_to_clip(
document, self.layer_id, self.time, &fills, &edges,
self.clip_id, self.instance_id, true, "Group",
)?;
self.graph_before = Some(before);
Ok(())
}
fn rollback(&mut self, _document: &mut Document) -> Result<(), String> {
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if let Some(before) = &self.graph_before {
undo_extract_geometry(document, self.layer_id, self.time, self.clip_id, self.instance_id, before);
}
Ok(())
}
fn description(&self) -> String {
let count = self.shape_ids.len() + self.clip_instance_ids.len();
format!("Group {} objects", count)
"Group".to_string()
}
}

8
lightningbeam-ui/lightningbeam-core/src/actions/mod.rs
View File

@ -29,15 +29,20 @@ pub mod update_midi_events;
pub mod loop_clip_instances;
pub mod remove_clip_instances;
pub mod set_keyframe;
pub mod set_tween;
pub mod group_shapes;
pub mod convert_to_movie_clip;
pub mod region_split;
pub mod toggle_group_expansion;
pub mod group_layers;
pub mod clip_from_geometry;
pub mod raster_diff;
pub mod raster_stroke;
pub mod raster_fill;
pub mod add_raster_keyframe;
pub mod move_layer;
pub mod set_fill_paint;
pub mod set_image_fill;
pub use add_clip_instance::AddClipInstanceAction;
pub use add_effect::AddEffectAction;
@ -63,6 +68,7 @@ pub use update_midi_events::UpdateMidiEventsAction;
pub use loop_clip_instances::LoopClipInstancesAction;
pub use remove_clip_instances::RemoveClipInstancesAction;
pub use set_keyframe::SetKeyframeAction;
pub use set_tween::SetTweenAction;
pub use group_shapes::GroupAction;
pub use convert_to_movie_clip::ConvertToMovieClipAction;
pub use region_split::RegionSplitAction;
@ -70,7 +76,9 @@ pub use toggle_group_expansion::ToggleGroupExpansionAction;
pub use group_layers::GroupLayersAction;
pub use raster_stroke::RasterStrokeAction;
pub use raster_fill::RasterFillAction;
pub use add_raster_keyframe::AddRasterKeyframeAction;
pub use move_layer::MoveLayerAction;
pub use set_fill_paint::SetFillPaintAction;
pub use set_image_fill::SetImageFillAction;
pub use change_bpm::ChangeBpmAction;
pub use change_fps::ChangeFpsAction;

232
lightningbeam-ui/lightningbeam-core/src/actions/raster_diff.rs Normal file
View File

@ -0,0 +1,232 @@
//! Dirty-rect diff for raster undo/redo.
//!
//! Brush strokes and fills used to store the *entire* before/after RGBA frame in the
//! undo stack (~8 MB each at 1080p → up to ~1.6 GB at the 100-action cap). A
//! [`RasterDiff`] instead stores only the changed bounding box's pixels before and
//! after, which for a typical brush dab is a few tens of KB.
//!
//! Applying a diff overwrites just the bbox of the keyframe's `raw_pixels`, so the
//! buffer **must be resident** (full length `w*h*4`) when `apply_*` runs. The editor
//! guarantees this by faulting the target frame in before undo/redo (a clean evicted
//! frame's container bytes equal its current logical state, so the restored base is
//! correct). If the base is somehow not resident we skip rather than corrupt.
/// Normalize a buffer to full length `n`; an empty/short buffer becomes transparent.
fn normalize(buf: &[u8], n: usize) -> std::borrow::Cow<[u8]> {
if buf.len() == n {
std::borrow::Cow::Borrowed(buf)
} else {
std::borrow::Cow::Owned(vec![0u8; n])
}
}
/// A minimal before/after record of the region a raster edit changed.
#[derive(Clone, Debug)]
pub struct RasterDiff {
full_width: u32,
full_height: u32,
/// Changed bounding box `(x, y, w, h)`; `None` when before == after (no-op).
bbox: Option<(u32, u32, u32, u32)>,
/// bbox-sized RGBA (`w*h*4`) of the region before the edit.
before_region: Vec<u8>,
/// bbox-sized RGBA (`w*h*4`) of the region after the edit.
after_region: Vec<u8>,
/// The pre-edit buffer was blank (empty/unallocated) — i.e. this was the first
/// edit on a fresh keyframe. Lets `apply_after` build from a transparent base
/// (the commit/redo path often starts with empty `raw_pixels`) and `apply_before`
/// restore to blank, instead of requiring a resident base.
before_blank: bool,
}
impl RasterDiff {
/// Build a diff from full before/after buffers. `after` is expected to be the
/// resident post-edit buffer (`width*height*4`); `before` may be empty (a blank
/// keyframe's first stroke), treated as fully transparent.
pub fn compute(before: &[u8], after: &[u8], width: u32, height: u32) -> Self {
let n = width as usize * height as usize * 4;
let before_blank = before.len() != n;
// Normalize both sides to full length; empty/short ⇒ transparent.
let before_full = normalize(before, n);
let after_full = normalize(after, n);
// Find the tight bbox of differing pixels (compare 4-byte texels).
let (w, h) = (width as usize, height as usize);
let (mut min_x, mut min_y, mut max_x, mut max_y) = (usize::MAX, usize::MAX, 0usize, 0usize);
let mut any = false;
for y in 0..h {
let row = y * w * 4;
for x in 0..w {
let i = row + x * 4;
if before_full[i..i + 4] != after_full[i..i + 4] {
any = true;
if x < min_x { min_x = x; }
if x > max_x { max_x = x; }
if y < min_y { min_y = y; }
if y > max_y { max_y = y; }
}
}
}
if !any {
return Self { full_width: width, full_height: height, bbox: None,
before_region: Vec::new(), after_region: Vec::new(), before_blank };
}
let bw = max_x - min_x + 1;
let bh = max_y - min_y + 1;
let crop = |full: &[u8]| -> Vec<u8> {
let mut out = Vec::with_capacity(bw * bh * 4);
for row in 0..bh {
let src = ((min_y + row) * w + min_x) * 4;
out.extend_from_slice(&full[src..src + bw * 4]);
}
out
};
Self {
full_width: width,
full_height: height,
bbox: Some((min_x as u32, min_y as u32, bw as u32, bh as u32)),
before_region: crop(&before_full),
after_region: crop(&after_full),
before_blank,
}
}
/// Approximate retained size in bytes (the two cropped regions).
pub fn byte_size(&self) -> usize {
self.before_region.len() + self.after_region.len()
}
/// Restore the pre-edit pixels into `raw` (undo / first-execute rollback).
pub fn apply_before(&self, raw: &mut Vec<u8>) {
if self.bbox.is_none() {
return; // no change
}
if self.before_blank {
// The frame was blank before this edit (it was the first stroke); undoing
// it returns to blank regardless of the current buffer.
raw.clear();
return;
}
self.stamp_resident(&self.before_region, raw);
}
/// Apply the post-edit pixels into `raw` (commit / redo).
pub fn apply_after(&self, raw: &mut Vec<u8>) {
if self.bbox.is_none() {
return; // no change
}
if self.before_blank {
// Base was blank: build a full transparent buffer then stamp the bbox. The
// commit/redo path frequently starts from empty `raw_pixels` here.
let n = self.full_width as usize * self.full_height as usize * 4;
raw.clear();
raw.resize(n, 0);
}
self.stamp_resident(&self.after_region, raw);
}
/// Stamp a bbox-sized region into `raw`, which must already be full-size. If it
/// isn't (a non-blank base that the editor failed to fault in), skip rather than
/// resize-and-corrupt — the frame will re-page to its container state.
fn stamp_resident(&self, region: &[u8], raw: &mut [u8]) {
let n = self.full_width as usize * self.full_height as usize * 4;
let (x, y, bw, bh) = match self.bbox {
Some(b) => b,
None => return,
};
if raw.len() != n {
eprintln!(
"⚠️ [RASTER_DIFF] base not resident ({} != {}); skipping undo/redo apply",
raw.len(), n
);
return;
}
let (x, y, bw, bh) = (x as usize, y as usize, bw as usize, bh as usize);
let fw = self.full_width as usize;
for row in 0..bh {
let dst = ((y + row) * fw + x) * 4;
let src = row * bw * 4;
raw[dst..dst + bw * 4].copy_from_slice(&region[src..src + bw * 4]);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn solid(w: u32, h: u32, px: [u8; 4]) -> Vec<u8> {
px.iter().copied().cycle().take((w * h * 4) as usize).collect()
}
#[test]
fn roundtrip_reproduces_buffers_exactly() {
let (w, h) = (8, 6);
let before = solid(w, h, [10, 20, 30, 255]);
let mut after = before.clone();
// Change a 2x2 region at (3,2).
for (dy, dx) in [(0, 0), (0, 1), (1, 0), (1, 1)] {
let i = (((2 + dy) * w + (3 + dx)) * 4) as usize;
after[i..i + 4].copy_from_slice(&[200, 100, 50, 255]);
}
let diff = RasterDiff::compute(&before, &after, w, h);
assert_eq!(diff.bbox, Some((3, 2, 2, 2)));
let mut buf = after.clone();
diff.apply_before(&mut buf);
assert_eq!(buf, before, "undo must reproduce the pre-edit buffer exactly");
diff.apply_after(&mut buf);
assert_eq!(buf, after, "redo must reproduce the post-edit buffer exactly");
}
#[test]
fn blank_before_first_stroke() {
let (w, h) = (4, 4);
let n = (w * h * 4) as usize;
let before: Vec<u8> = Vec::new(); // blank keyframe
let mut after = vec![0u8; n];
let i = ((1 * w + 1) * 4) as usize;
after[i..i + 4].copy_from_slice(&[255, 0, 0, 255]);
let diff = RasterDiff::compute(&before, &after, w, h);
assert_eq!(diff.bbox, Some((1, 1, 1, 1)));
// First execute / redo from EMPTY raw_pixels (the real commit path): builds
// the full buffer from transparent + the stroke.
let mut buf: Vec<u8> = Vec::new();
diff.apply_after(&mut buf);
assert_eq!(buf, after, "commit/redo must build the frame from a blank base");
// Undo the first stroke → back to blank (empty).
diff.apply_before(&mut buf);
assert!(buf.is_empty(), "undoing the first stroke restores the blank keyframe");
// Redo again from the now-empty buffer.
diff.apply_after(&mut buf);
assert_eq!(buf, after);
}
#[test]
fn no_change_is_noop() {
let (w, h) = (4, 4);
let buf = solid(w, h, [1, 2, 3, 4]);
let diff = RasterDiff::compute(&buf, &buf, w, h);
assert_eq!(diff.bbox, None);
assert_eq!(diff.byte_size(), 0);
let mut b = buf.clone();
diff.apply_before(&mut b);
assert_eq!(b, buf);
}
#[test]
fn not_resident_base_is_skipped_not_corrupted() {
let (w, h) = (4, 4);
let before = solid(w, h, [9, 9, 9, 255]);
let after = solid(w, h, [1, 2, 3, 255]);
let diff = RasterDiff::compute(&before, &after, w, h);
let mut empty: Vec<u8> = Vec::new();
diff.apply_before(&mut empty); // base not resident
assert!(empty.is_empty(), "must not resize/corrupt a non-resident base");
}
}

35
lightningbeam-ui/lightningbeam-core/src/actions/raster_fill.rs
View File

@ -1,6 +1,7 @@
//! Raster flood-fill action — records and undoes a paint bucket fill on a RasterLayer.
use crate::action::Action;
use crate::actions::raster_diff::RasterDiff;
use crate::document::Document;
use crate::layer::AnyLayer;
use uuid::Uuid;
@ -8,11 +9,13 @@ use uuid::Uuid;
pub struct RasterFillAction {
layer_id: Uuid,
time: f64,
buffer_before: Vec<u8>,
buffer_after: Vec<u8>,
width: u32,
height: u32,
name: String,
diff: RasterDiff,
/// Full post-fill buffer, kept only for the first `execute` (commit); see
/// `RasterStrokeAction::full_after`.
full_after: Option<Vec<u8>>,
}
impl RasterFillAction {
@ -24,7 +27,9 @@ impl RasterFillAction {
width: u32,
height: u32,
) -> Self {
Self { layer_id, time, buffer_before, buffer_after, width, height, name: "Flood fill".to_string() }
let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
Self { layer_id, time, width, height, name: "Flood fill".to_string(),
diff, full_after: Some(buffer_after) }
}
pub fn with_description(mut self, name: &str) -> Self {
@ -41,9 +46,17 @@ impl Action for RasterFillAction {
AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()),
};
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height);
kf.raw_pixels = self.buffer_after.clone();
let _ = (self.width, self.height);
let kf = raster
.keyframe_at_mut(self.time)
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
if let Some(full) = self.full_after.take() {
kf.raw_pixels = full;
} else {
self.diff.apply_after(&mut kf.raw_pixels);
}
kf.texture_dirty = true;
kf.dirty = true;
Ok(())
}
@ -54,13 +67,21 @@ impl Action for RasterFillAction {
AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()),
};
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height);
kf.raw_pixels = self.buffer_before.clone();
let _ = (self.width, self.height);
let kf = raster
.keyframe_at_mut(self.time)
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
self.diff.apply_before(&mut kf.raw_pixels);
kf.texture_dirty = true;
kf.dirty = true;
Ok(())
}
fn description(&self) -> String {
self.name.clone()
}
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
Some((self.layer_id, self.time))
}
}

43
lightningbeam-ui/lightningbeam-core/src/actions/raster_stroke.rs
View File

@ -9,6 +9,7 @@
//! `rollback` → swap in `buffer_before`
use crate::action::Action;
use crate::actions::raster_diff::RasterDiff;
use crate::document::Document;
use crate::layer::AnyLayer;
use uuid::Uuid;
@ -16,16 +17,20 @@ use uuid::Uuid;
/// Action that records a single brush stroke for undo/redo.
///
/// The stroke must already be painted into the document's `raw_pixels` before
/// this action is executed for the first time.
/// this action is executed for the first time. Only the changed bounding box is
/// retained (see [`RasterDiff`]) rather than two full frame buffers.
pub struct RasterStrokeAction {
layer_id: Uuid,
time: f64,
/// Raw RGBA pixels *before* the stroke (for rollback / undo)
buffer_before: Vec<u8>,
/// Raw RGBA pixels *after* the stroke (for execute / redo)
buffer_after: Vec<u8>,
width: u32,
height: u32,
diff: RasterDiff,
/// The full post-stroke buffer, kept ONLY for the first `execute` (the commit),
/// which establishes `raw_pixels` exactly like the old code did — robust no matter
/// what state the working buffer is in (empty new keyframe, GPU-canvas readback,
/// etc.). Taken (dropped) on first execute, so the action sitting in the undo stack
/// retains only the small `diff`; redo then replays via the diff.
full_after: Option<Vec<u8>>,
}
impl RasterStrokeAction {
@ -33,6 +38,8 @@ impl RasterStrokeAction {
///
/// * `buffer_before` – raw RGBA pixels captured just before the stroke began.
/// * `buffer_after` – raw RGBA pixels captured just after the stroke finished.
///
/// The full buffers are diffed down to the changed bbox here and then dropped.
pub fn new(
layer_id: Uuid,
time: f64,
@ -41,28 +48,41 @@ impl RasterStrokeAction {
width: u32,
height: u32,
) -> Self {
Self { layer_id, time, buffer_before, buffer_after, width, height }
let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
Self { layer_id, time, width, height, diff, full_after: Some(buffer_after) }
}
}
impl Action for RasterStrokeAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
kf.raw_pixels = self.buffer_after.clone();
if let Some(full) = self.full_after.take() {
// First execute (commit): assign the full buffer outright.
kf.raw_pixels = full;
} else {
// Redo: replay via the diff onto the (resident) base.
self.diff.apply_after(&mut kf.raw_pixels);
}
kf.texture_dirty = true;
kf.dirty = true;
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
kf.raw_pixels = self.buffer_before.clone();
self.diff.apply_before(&mut kf.raw_pixels);
kf.texture_dirty = true;
kf.dirty = true;
Ok(())
}
fn description(&self) -> String {
"Paint stroke".to_string()
}
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
Some((self.layer_id, self.time))
}
}
fn get_keyframe_mut<'a>(
@ -79,5 +99,10 @@ fn get_keyframe_mut<'a>(
AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()),
};
Ok(raster.ensure_keyframe_at(time, width, height))
let _ = (width, height);
// Edit the ACTIVE keyframe (at-or-before `time`); never create one — keyframes
// are made explicitly via "New Keyframe".
raster
.keyframe_at_mut(time)
.ok_or_else(|| format!("No raster keyframe at/before t={time}"))
}

67
lightningbeam-ui/lightningbeam-core/src/actions/set_image_fill.rs Normal file
View File

@ -0,0 +1,67 @@
//! Action that sets or clears the image fill on one or more VectorGraph fills.
//!
//! `image_fill` is an asset id the renderer maps onto the fill's bounding box; it
//! takes priority over colour/gradient. Setting `None` clears it (the colour/gradient
//! underneath shows again).
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use crate::vector_graph::FillId;
use uuid::Uuid;
pub struct SetImageFillAction {
layer_id: Uuid,
time: f64,
fill_ids: Vec<FillId>,
/// `Some(asset_id)` to set, `None` to clear.
new_image: Option<Uuid>,
/// Per-fill previous `image_fill`, for undo.
old: Vec<(FillId, Option<Uuid>)>,
}
impl SetImageFillAction {
pub fn new(layer_id: Uuid, time: f64, fill_ids: Vec<FillId>, image: Option<Uuid>) -> Self {
Self { layer_id, time, fill_ids, new_image: image, old: Vec::new() }
}
fn get_graph_mut<'a>(
document: &'a mut Document,
layer_id: &Uuid,
time: f64,
) -> Result<&'a mut crate::vector_graph::VectorGraph, String> {
let layer = document
.get_layer_mut(layer_id)
.ok_or_else(|| format!("Layer {} not found", layer_id))?;
match layer {
AnyLayer::Vector(vl) => vl
.graph_at_time_mut(time)
.ok_or_else(|| format!("No keyframe at time {}", time)),
_ => Err("Not a vector layer".to_string()),
}
}
}
impl Action for SetImageFillAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let graph = Self::get_graph_mut(document, &self.layer_id, self.time)?;
self.old.clear();
for &fid in &self.fill_ids {
self.old.push((fid, graph.fill(fid).image_fill));
graph.fill_mut(fid).image_fill = self.new_image;
}
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let graph = Self::get_graph_mut(document, &self.layer_id, self.time)?;
for &(fid, old) in &self.old {
graph.fill_mut(fid).image_fill = old;
}
Ok(())
}
fn description(&self) -> String {
if self.new_image.is_some() { "Set image fill" } else { "Clear image fill" }.to_string()
}
}

133
lightningbeam-ui/lightningbeam-core/src/actions/set_keyframe.rs
View File

@ -8,6 +8,7 @@ use crate::action::Action;
use crate::animation::{AnimationCurve, AnimationTarget, Keyframe, TransformProperty};
use crate::document::Document;
use crate::layer::{AnyLayer, ShapeKeyframe};
use crate::object::Transform;
use uuid::Uuid;
/// Undo info for a clip animation curve
@ -54,11 +55,17 @@ const TRANSFORM_PROPERTIES: &[TransformProperty] = &[
TransformProperty::Opacity,
];
fn transform_default(prop: &TransformProperty) -> f64 {
/// The clip instance's own value for a property (its base transform / opacity).
fn transform_prop_value(t: &Transform, opacity: f64, prop: &TransformProperty) -> f64 {
match prop {
TransformProperty::ScaleX | TransformProperty::ScaleY => 1.0,
TransformProperty::Opacity => 1.0,
_ => 0.0,
TransformProperty::X => t.x,
TransformProperty::Y => t.y,
TransformProperty::Rotation => t.rotation,
TransformProperty::ScaleX => t.scale_x,
TransformProperty::ScaleY => t.scale_y,
TransformProperty::SkewX => t.skew_x,
TransformProperty::SkewY => t.skew_y,
TransformProperty::Opacity => opacity,
}
}
@ -67,6 +74,21 @@ impl Action for SetKeyframeAction {
self.clip_undo_entries.clear();
self.shape_keyframe_created = false;
// Phase 1 (immutable): for each clip instance, gather its base transform and the
// start time of its visibility region, so a brand-new curve can be anchored there.
let mut clip_info: std::collections::HashMap<Uuid, (Transform, f64, f64)> =
std::collections::HashMap::new(); // id -> (base transform, opacity, start time)
if let Some(AnyLayer::Vector(vl)) = document.get_layer(&self.layer_id) {
for clip_id in &self.clip_instance_ids {
if let Some(ci) = vl.clip_instances.iter().find(|c| c.id == *clip_id) {
let start = vl
.group_visibility_start(clip_id, self.time)
.unwrap_or(ci.timeline_start);
clip_info.insert(*clip_id, (ci.transform.clone(), ci.opacity, start));
}
}
}
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
@ -82,23 +104,37 @@ impl Action for SetKeyframeAction {
// Add clip animation keyframes
for clip_id in &self.clip_instance_ids {
let (base_transform, base_opacity, start) = clip_info
.get(clip_id)
.cloned()
.unwrap_or((Transform::new(), 1.0, 0.0));
for prop in TRANSFORM_PROPERTIES {
let target = AnimationTarget::Object {
id: *clip_id,
property: *prop,
};
let default = transform_default(prop);
let value = vl.layer.animation_data.eval(&target, self.time, default);
// Fall back to the clip's OWN value (not a generic default) so a brand-new
// keyframe captures the actual on-stage position, not (0,0)/identity.
let base = transform_prop_value(&base_transform, base_opacity, prop);
let value = vl.layer.animation_data.eval(&target, self.time, base);
let curve_created = vl.layer.animation_data.get_curve(&target).is_none();
if curve_created {
vl.layer
.animation_data
.set_curve(AnimationCurve::new(target.clone(), default));
.set_curve(AnimationCurve::new(target.clone(), base));
}
let curve = vl.layer.animation_data.get_curve_mut(&target).unwrap();
let old_keyframe = curve.get_keyframe_at(self.time, 0.001).cloned();
// When this is the first keyframe of the curve and the clip already existed
// before `time`, anchor a keyframe at its start with the original value.
// Otherwise a single keyframe would Hold-extrapolate backward and move the
// clip on every earlier frame too (the motion-tween first-keyframe bug).
if curve_created && start < self.time - 0.001 {
curve.set_keyframe(Keyframe::linear(start, base));
}
curve.set_keyframe(Keyframe::linear(self.time, value));
self.clip_undo_entries.push(ClipUndoEntry {
@ -145,3 +181,86 @@ impl Action for SetKeyframeAction {
"New keyframe".to_string()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::actions::TransformClipInstancesAction;
use crate::clip::ClipInstance;
use crate::layer::VectorLayer;
use std::collections::HashMap;
fn x_curve_eval(document: &Document, layer_id: Uuid, instance_id: Uuid, time: f64) -> f64 {
let target = AnimationTarget::Object { id: instance_id, property: TransformProperty::X };
match document.get_layer(&layer_id) {
Some(AnyLayer::Vector(vl)) => vl.layer.animation_data.eval(&target, time, f64::NAN),
_ => panic!("no layer"),
}
}
#[test]
fn first_keyframe_then_move_does_not_disturb_earlier_frames() {
// Group created at frame 0 (clip at x=50), keyframe + move at frame 10 → x=200.
// Frame 0 must keep x=50 (the motion-tween first-keyframe bug: it used to become 200).
let mut document = Document::new("Test");
let mut layer = VectorLayer::new("Layer");
let clip_id = Uuid::new_v4();
let instance_id = Uuid::new_v4();
let mut instance = ClipInstance::with_id(instance_id, clip_id);
instance.transform = Transform::with_position(50.0, 50.0);
layer.clip_instances.push(instance);
// The group's visibility starts at a keyframe at time 0 containing the instance.
layer.ensure_keyframe_at(0.0).clip_instance_ids.push(instance_id);
let layer_id = document.root_mut().add_child(AnyLayer::Vector(layer));
// Create a keyframe at frame 10.
SetKeyframeAction::new(layer_id, 10.0, vec![instance_id])
.execute(&mut document)
.unwrap();
// The new curve must be anchored at the start (two keyframes, both at x=50 so far).
assert!((x_curve_eval(&document, layer_id, instance_id, 0.0) - 50.0).abs() < 1e-6);
assert!((x_curve_eval(&document, layer_id, instance_id, 10.0) - 50.0).abs() < 1e-6);
// Move the clip at frame 10 to x=200.
let mut transforms = HashMap::new();
transforms.insert(
instance_id,
(Transform::with_position(50.0, 50.0), Transform::with_position(200.0, 200.0)),
);
TransformClipInstancesAction::new(layer_id, 10.0, transforms)
.execute(&mut document)
.unwrap();
// Frame 0 unchanged; frame 10 moved; midpoint tweens.
assert!((x_curve_eval(&document, layer_id, instance_id, 0.0) - 50.0).abs() < 1e-6, "frame 0 must stay 50");
assert!((x_curve_eval(&document, layer_id, instance_id, 10.0) - 200.0).abs() < 1e-6, "frame 10 must be 200");
assert!((x_curve_eval(&document, layer_id, instance_id, 5.0) - 125.0).abs() < 1e-6, "midpoint tweens");
}
#[test]
fn first_keyframe_at_clip_start_is_not_double_anchored() {
// When the keyframe is created at the clip's own start, there's nothing earlier to
// anchor — a single keyframe is correct.
let mut document = Document::new("Test");
let mut layer = VectorLayer::new("Layer");
let clip_id = Uuid::new_v4();
let instance_id = Uuid::new_v4();
let mut instance = ClipInstance::with_id(instance_id, clip_id);
instance.transform = Transform::with_position(10.0, 0.0);
layer.clip_instances.push(instance);
layer.ensure_keyframe_at(0.0).clip_instance_ids.push(instance_id);
let layer_id = document.root_mut().add_child(AnyLayer::Vector(layer));
SetKeyframeAction::new(layer_id, 0.0, vec![instance_id])
.execute(&mut document)
.unwrap();
let target = AnimationTarget::Object { id: instance_id, property: TransformProperty::X };
if let Some(AnyLayer::Vector(vl)) = document.get_layer(&layer_id) {
let curve = vl.layer.animation_data.get_curve(&target).unwrap();
assert_eq!(curve.keyframes.len(), 1, "keyframe at clip start needs no anchor");
assert!((curve.eval(0.0) - 10.0).abs() < 1e-6);
}
}
}

57
lightningbeam-ui/lightningbeam-core/src/actions/set_tween.rs Normal file
View File

@ -0,0 +1,57 @@
//! Set the tween type on the keyframe at-or-before a time (e.g. "Add Shape Tween").
//!
//! The keyframe's `tween_after` controls how the span between it and the next keyframe is
//! rendered: `None` holds, `Shape` morphs the geometry (when the two keyframes share
//! topology — otherwise rendering falls back to holding).
use crate::action::Action;
use crate::document::Document;
use crate::layer::{AnyLayer, TweenType};
use uuid::Uuid;
pub struct SetTweenAction {
layer_id: Uuid,
time: f64,
new_tween: TweenType,
old_tween: Option<TweenType>,
}
impl SetTweenAction {
pub fn new(layer_id: Uuid, time: f64, new_tween: TweenType) -> Self {
Self { layer_id, time, new_tween, old_tween: None }
}
}
impl Action for SetTweenAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
if let Some(AnyLayer::Vector(vl)) = document.get_layer_mut(&self.layer_id) {
if let Some(kf) = vl.keyframe_at_mut(self.time) {
self.old_tween = Some(kf.tween_after);
kf.tween_after = self.new_tween;
} else {
return Err("No keyframe at-or-before this time".to_string());
}
} else {
return Err("Not a vector layer".to_string());
}
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if let (Some(old), Some(AnyLayer::Vector(vl))) =
(self.old_tween, document.get_layer_mut(&self.layer_id))
{
if let Some(kf) = vl.keyframe_at_mut(self.time) {
kf.tween_after = old;
}
}
Ok(())
}
fn description(&self) -> String {
match self.new_tween {
TweenType::Shape => "Add shape tween".to_string(),
TweenType::None => "Remove tween".to_string(),
}
}
}

4
lightningbeam-ui/lightningbeam-core/src/actions/trim_clip_instances.rs
View File

@ -547,6 +547,8 @@ mod tests {
// Create a clip ID (ClipInstance references clip by ID)
let clip_id = uuid::Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
document.vector_clips.insert(clip_id, clip);
let mut vector_layer = VectorLayer::new("Layer 1");
@ -607,6 +609,8 @@ mod tests {
// Create a clip ID (ClipInstance references clip by ID)
let clip_id = uuid::Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
document.vector_clips.insert(clip_id, clip);
let mut vector_layer = VectorLayer::new("Layer 1");

73
lightningbeam-ui/lightningbeam-core/src/animation.rs
View File

@ -322,6 +322,20 @@ impl AnimationCurve {
}
}
/// True when `time` lies strictly between two keyframes — an in-between frame of a
/// tween (not on a keyframe, not in the pre/post-extrapolation tails).
pub fn is_tween_inbetween(&self, time: f64, tol: f64) -> bool {
if self.keyframes.len() < 2 {
return false;
}
let first = self.keyframes.first().unwrap().time;
let last = self.keyframes.last().unwrap().time;
if time <= first + tol || time >= last - tol {
return false;
}
!self.keyframes.iter().any(|kf| (kf.time - time).abs() <= tol)
}
/// Extrapolate before the first keyframe
fn extrapolate_pre(&self, time: f64, first_kf: &Keyframe) -> f64 {
match self.pre_extrapolation {
@ -516,6 +530,17 @@ impl AnimationData {
self.curves.remove(target)
}
/// True when the object (e.g. a clip instance) is mid motion-tween at `time` — any of
/// its curves has `time` strictly between two keyframes. Used to lock out editing on
/// in-between frames (editing there would silently insert a keyframe and disturb the tween).
pub fn is_object_tweened_at(&self, id: uuid::Uuid, time: f64) -> bool {
const TOL: f64 = 0.001;
self.curves.iter().any(|(target, curve)| {
matches!(target, AnimationTarget::Object { id: oid, .. } if *oid == id)
&& curve.is_tween_inbetween(time, TOL)
})
}
/// Evaluate a property at a given time
pub fn eval(&self, target: &AnimationTarget, time: f64, default: f64) -> f64 {
self.curves
@ -545,3 +570,51 @@ impl AnimationData {
(t, opacity)
}
}
#[cfg(test)]
mod tween_lock_tests {
use super::*;
#[test]
fn curve_in_between_detection() {
let mut c = AnimationCurve::new(
AnimationTarget::Object { id: uuid::Uuid::nil(), property: TransformProperty::X },
0.0,
);
c.set_keyframe(Keyframe::linear(0.0, 0.0));
c.set_keyframe(Keyframe::linear(10.0, 100.0));
assert!(c.is_tween_inbetween(5.0, 0.001), "strictly between keyframes");
assert!(!c.is_tween_inbetween(0.0, 0.001), "on a keyframe");
assert!(!c.is_tween_inbetween(10.0, 0.001), "on a keyframe");
assert!(!c.is_tween_inbetween(15.0, 0.001), "past the last keyframe (extrapolation tail)");
}
#[test]
fn single_keyframe_is_never_in_between() {
let mut c = AnimationCurve::new(
AnimationTarget::Object { id: uuid::Uuid::nil(), property: TransformProperty::X },
0.0,
);
c.set_keyframe(Keyframe::linear(10.0, 100.0));
assert!(!c.is_tween_inbetween(5.0, 0.001));
assert!(!c.is_tween_inbetween(20.0, 0.001));
}
#[test]
fn object_tweened_when_any_curve_is_in_between() {
let id = uuid::Uuid::new_v4();
let mut data = AnimationData::new();
let mut cx = AnimationCurve::new(
AnimationTarget::Object { id, property: TransformProperty::X },
0.0,
);
cx.set_keyframe(Keyframe::linear(0.0, 0.0));
cx.set_keyframe(Keyframe::linear(10.0, 100.0));
data.set_curve(cx);
assert!(data.is_object_tweened_at(id, 5.0));
assert!(!data.is_object_tweened_at(id, 0.0));
assert!(!data.is_object_tweened_at(uuid::Uuid::new_v4(), 5.0), "different object");
}
}

813
lightningbeam-ui/lightningbeam-core/src/beam_archive.rs Normal file
View File

@ -0,0 +1,813 @@
//! SQLite-backed `.beam` project container.
//!
//! The `.beam` format is a single SQLite database file. It replaces the older
//! ZIP-archive format. SQLite gives us, in one file:
//!
//! - **Streaming reads** — packed media is split into chunk rows and read on
//! demand through [`BlobReader`] (`Read + Seek`), so arbitrary-length audio /
//! video never has to be fully decoded into RAM on load.
//! - **In-place, crash-safe mutation** — raster frame write-back and re-save are
//! transactional `UPDATE`s rather than rewriting a whole archive.
//! - **Single-file UX** — behaves like a file on every platform.
//!
//! ## Media storage
//!
//! Each media item is one row in `media` plus, when *packed*, N rows in
//! `media_chunk`:
//!
//! - **Packed** (`MediaStorage::Packed`) — bytes live in the database, split
//! into [`CHUNK_SIZE`]-byte chunks. Chunking keeps each blob well under
//! SQLite's ~2 GB per-blob ceiling and bounds the working set of a streaming
//! reader to a single chunk.
//! - **Referenced** (`MediaStorage::Referenced`) — only an external path is
//! stored; the bytes stay on disk (useful for shared media on a network drive,
//! or media too large/volatile to pack). Callers open the path directly.
//!
//! `project.json` (the serialized `BeamProject`) is stored verbatim in the
//! single-row `project_json` table; only the container and media storage change
//! relative to the legacy format.
use rusqlite::blob::Blob;
use rusqlite::{Connection, DatabaseName, OpenFlags, OptionalExtension};
use std::io::{self, Read, Seek, SeekFrom};
use std::path::Path;
use uuid::Uuid;
/// Default packed-media chunk size: 4 MiB.
///
/// Small enough to bound a streaming reader's per-chunk work and any
/// whole-chunk buffering, large enough to keep row counts modest (a 1 GB file
/// is 256 rows). Comfortably under SQLite's ~2 GB per-blob limit.
pub const CHUNK_SIZE: u64 = 4 * 1024 * 1024;
/// Files at or above this size prompt the user to pick packed vs referenced
/// (and the choice is then persisted as the default). Matches SQLite's
/// practical large-blob threshold.
pub const LARGE_MEDIA_THRESHOLD: u64 = 2 * 1024 * 1024 * 1024;
/// Kind of media stored in the `media` table.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MediaKind {
Audio = 0,
Video = 1,
Raster = 2,
ImageAsset = 3,
/// A precomputed waveform LOD pyramid blob for an audio item (keyed by the
/// same id as the audio it describes). See `daw_backend::audio::waveform_pyramid`.
Waveform = 4,
/// A pack of precomputed video thumbnails for a video clip (keyed by a
/// sentinel-derived id from the clip id). Opaque blob; format owned by the editor.
Thumbnail = 5,
/// A low-res PNG proxy of a raster keyframe (keyed by a sentinel-derived id from
/// the keyframe id). Decoded eagerly on load and shown while the full-res pixels
/// page in, so cold scrubs don't flash blank. See `raster_proxy_media_id`.
RasterProxy = 6,
}
impl MediaKind {
fn from_i64(v: i64) -> Option<Self> {
match v {
0 => Some(Self::Audio),
1 => Some(Self::Video),
2 => Some(Self::Raster),
3 => Some(Self::ImageAsset),
4 => Some(Self::Waveform),
5 => Some(Self::Thumbnail),
6 => Some(Self::RasterProxy),
_ => None,
}
}
}
/// How a media item's bytes are stored.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MediaStorage {
/// Bytes are chunked into `media_chunk` rows inside the database.
Packed = 0,
/// Only an external path is stored; bytes live on disk.
Referenced = 1,
}
impl MediaStorage {
fn from_i64(v: i64) -> Option<Self> {
match v {
0 => Some(Self::Packed),
1 => Some(Self::Referenced),
_ => None,
}
}
}
/// Metadata row for a media item (no bytes).
#[derive(Debug, Clone)]
pub struct MediaInfo {
pub id: Uuid,
pub kind: MediaKind,
/// Original codec / container extension, e.g. `"flac"`, `"mp3"`, `"png"`.
pub codec: String,
pub storage: MediaStorage,
/// Set when `storage == Referenced`.
pub ext_path: Option<String>,
/// Total byte length of the media payload (packed only; 0 for referenced).
pub total_len: u64,
// Kind-specific metadata (nullable; meaning depends on `kind`).
pub channels: Option<u32>,
pub sample_rate: Option<u32>,
pub width: Option<u32>,
pub height: Option<u32>,
}
/// Optional kind-specific metadata supplied when writing a media item.
#[derive(Debug, Clone, Copy, Default)]
pub struct MediaMeta {
pub channels: Option<u32>,
pub sample_rate: Option<u32>,
pub width: Option<u32>,
pub height: Option<u32>,
}
/// A `.beam` project container backed by a SQLite database.
pub struct BeamArchive {
conn: Connection,
chunk_size: u64,
}
impl BeamArchive {
/// Schema version stored in `meta` under `"schema_version"`.
pub const SCHEMA_VERSION: i64 = 1;
/// Create a new (empty) archive at `path`, replacing any existing file.
pub fn create(path: &Path) -> Result<Self, String> {
// Remove any existing file so we start from a clean schema.
if path.exists() {
std::fs::remove_file(path).map_err(|e| format!("Failed to replace {:?}: {}", path, e))?;
}
let conn = Connection::open(path).map_err(map_sql)?;
let mut archive = Self { conn, chunk_size: CHUNK_SIZE };
archive.init_schema()?;
Ok(archive)
}
/// Open an existing archive for read/write.
pub fn open(path: &Path) -> Result<Self, String> {
let conn = Connection::open(path).map_err(map_sql)?;
let archive = Self { conn, chunk_size: CHUNK_SIZE };
archive.verify_schema()?;
Ok(archive)
}
/// Quick check: does `path` look like a SQLite database (vs. a legacy ZIP)?
/// Reads the 16-byte SQLite header magic. Used to route between the SQLite
/// loader and the legacy-ZIP migration path.
pub fn is_sqlite(path: &Path) -> bool {
use std::io::Read as _;
let mut f = match std::fs::File::open(path) {
Ok(f) => f,
Err(_) => return false,
};
let mut magic = [0u8; 16];
if f.read_exact(&mut magic).is_err() {
return false;
}
&magic == b"SQLite format 3\0"
}
fn init_schema(&mut self) -> Result<(), String> {
self.conn
.execute_batch(
"BEGIN;
CREATE TABLE media (
id BLOB PRIMARY KEY, -- 16-byte Uuid
kind INTEGER NOT NULL,
codec TEXT NOT NULL,
storage INTEGER NOT NULL,
ext_path TEXT,
total_len INTEGER NOT NULL DEFAULT 0,
channels INTEGER,
sample_rate INTEGER,
width INTEGER,
height INTEGER
);
CREATE TABLE media_chunk (
id INTEGER PRIMARY KEY, -- rowid, for blob_open
media_id BLOB NOT NULL,
chunk_index INTEGER NOT NULL,
bytes BLOB NOT NULL,
UNIQUE (media_id, chunk_index)
);
CREATE TABLE project_json (
id INTEGER PRIMARY KEY CHECK (id = 0),
data TEXT NOT NULL
);
CREATE TABLE meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
COMMIT;",
)
.map_err(map_sql)?;
self.set_meta("schema_version", &Self::SCHEMA_VERSION.to_string())?;
Ok(())
}
fn verify_schema(&self) -> Result<(), String> {
let v: Option<String> = self.get_meta("schema_version")?;
match v.as_deref().and_then(|s| s.parse::<i64>().ok()) {
Some(n) if n <= Self::SCHEMA_VERSION => Ok(()),
Some(n) => Err(format!(
"Unsupported .beam schema version {} (this build supports up to {})",
n,
Self::SCHEMA_VERSION
)),
None => Err("Not a valid .beam archive (missing schema_version)".to_string()),
}
}
/// Begin a write transaction grouping multiple media/json writes into one
/// atomic, crash-safe commit. Used by saves so unchanged (large) media is
/// never rewritten — only dirty rows are touched, in place.
pub fn transaction(&mut self) -> Result<BeamTxn<'_>, String> {
let tx = self.conn.transaction().map_err(map_sql)?;
Ok(BeamTxn { tx, chunk_size: self.chunk_size })
}
// -- meta key/value --------------------------------------------------
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
set_meta_conn(&self.conn, key, value)
}
pub fn get_meta(&self, key: &str) -> Result<Option<String>, String> {
self.conn
.query_row("SELECT value FROM meta WHERE key = ?1", [key], |r| r.get(0))
.optional()
.map_err(map_sql)
}
// -- project.json ----------------------------------------------------
/// Store the serialized `project.json` (single row).
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
set_project_json_conn(&self.conn, json)
}
/// Read the serialized `project.json`.
pub fn get_project_json(&self) -> Result<String, String> {
self.conn
.query_row("SELECT data FROM project_json WHERE id = 0", [], |r| r.get(0))
.optional()
.map_err(map_sql)?
.ok_or_else(|| "Archive has no project.json".to_string())
}
// -- media write -----------------------------------------------------
/// Write a media item whose bytes are packed (chunked) into the database.
/// Replaces any existing rows for `id`.
pub fn put_media_packed(
&mut self,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
let tx = self.conn.transaction().map_err(map_sql)?;
write_media_packed(&tx, self.chunk_size, id, kind, codec, bytes, meta)?;
tx.commit().map_err(map_sql)?;
Ok(())
}
/// Write a media item that references an external file by path (no bytes
/// stored). Replaces any existing rows for `id`.
pub fn put_media_referenced(
&mut self,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
let tx = self.conn.transaction().map_err(map_sql)?;
write_media_referenced(&tx, id, kind, codec, ext_path, meta)?;
tx.commit().map_err(map_sql)?;
Ok(())
}
// -- media read ------------------------------------------------------
/// Look up a media item's metadata.
pub fn media_info(&self, id: Uuid) -> Result<Option<MediaInfo>, String> {
let id_bytes = id.as_bytes().to_vec();
self.conn
.query_row(
"SELECT kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height
FROM media WHERE id = ?1",
[&id_bytes],
|r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, String>(1)?,
r.get::<_, i64>(2)?,
r.get::<_, Option<String>>(3)?,
r.get::<_, i64>(4)?,
r.get::<_, Option<u32>>(5)?,
r.get::<_, Option<u32>>(6)?,
r.get::<_, Option<u32>>(7)?,
r.get::<_, Option<u32>>(8)?,
))
},
)
.optional()
.map_err(map_sql)?
.map(|(kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)| {
Ok(MediaInfo {
id,
kind: MediaKind::from_i64(kind)
.ok_or_else(|| format!("Unknown media kind {}", kind))?,
codec,
storage: MediaStorage::from_i64(storage)
.ok_or_else(|| format!("Unknown media storage {}", storage))?,
ext_path,
total_len: total_len.max(0) as u64,
channels,
sample_rate,
width,
height,
})
})
.transpose()
}
/// List every media item of a given kind.
pub fn media_ids_of_kind(&self, kind: MediaKind) -> Result<Vec<Uuid>, String> {
let mut stmt = self
.conn
.prepare("SELECT id FROM media WHERE kind = ?1")
.map_err(map_sql)?;
let rows = stmt
.query_map([kind as i64], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::new();
for row in rows {
let bytes = row.map_err(map_sql)?;
out.push(uuid_from_bytes(&bytes)?);
}
Ok(out)
}
/// Read an entire packed media item into memory. Convenience for small
/// media (raster frames, image assets); large media should stream via
/// [`BeamArchive::open_blob_reader`] instead.
pub fn read_media_full(&self, id: Uuid) -> Result<Vec<u8>, String> {
let info = self
.media_info(id)?
.ok_or_else(|| format!("Media {} not found", id))?;
if info.storage != MediaStorage::Packed {
return Err(format!("Media {} is referenced, not packed", id));
}
let id_bytes = id.as_bytes().to_vec();
let mut stmt = self
.conn
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::with_capacity(info.total_len as usize);
for row in rows {
out.extend_from_slice(&row.map_err(map_sql)?);
}
Ok(out)
}
/// Open a streaming reader over a packed media item. The reader owns its own
/// SQLite connection (read-only) so it can live on a separate thread (e.g.
/// the audio disk reader) independent of this archive handle.
pub fn open_blob_reader(&self, db_path: &Path, id: Uuid) -> Result<BlobReader, String> {
let info = self
.media_info(id)?
.ok_or_else(|| format!("Media {} not found", id))?;
if info.storage != MediaStorage::Packed {
return Err(format!("Media {} is referenced, not packed", id));
}
BlobReader::open(db_path, id, info.total_len, self.chunk_size)
}
/// Override the chunk size (testing / tuning). Affects subsequent writes.
#[doc(hidden)]
pub fn set_chunk_size(&mut self, chunk_size: u64) {
assert!(chunk_size > 0);
self.chunk_size = chunk_size;
}
}
/// Streaming reader (`Read + Seek`) over a packed media item's chunk rows.
///
/// Owns a dedicated read-only SQLite connection so it is independent of the
/// writing [`BeamArchive`] handle and can be moved to another thread. Each
/// `read` opens a blob handle on the current chunk's row via `blob_open` (no
/// per-read query — chunk rowids are resolved once up front) and reads up to the
/// chunk boundary; callers that issue many tiny reads should wrap this in a
/// `BufReader`.
pub struct BlobReader {
conn: Connection,
chunk_rowids: Vec<i64>,
chunk_size: u64,
total_len: u64,
pos: u64,
}
impl BlobReader {
fn open(db_path: &Path, id: Uuid, total_len: u64, chunk_size: u64) -> Result<Self, String> {
let conn = Connection::open_with_flags(
db_path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)
.map_err(map_sql)?;
let id_bytes = id.as_bytes().to_vec();
let mut stmt = conn
.prepare("SELECT id FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, i64>(0))
.map_err(map_sql)?;
let mut chunk_rowids = Vec::new();
for row in rows {
chunk_rowids.push(row.map_err(map_sql)?);
}
drop(stmt);
Ok(Self { conn, chunk_rowids, chunk_size, total_len, pos: 0 })
}
/// Total length of the media payload in bytes.
pub fn len(&self) -> u64 {
self.total_len
}
pub fn is_empty(&self) -> bool {
self.total_len == 0
}
fn chunk_blob(&self, chunk_index: usize) -> io::Result<Blob<'_>> {
let rowid = *self
.chunk_rowids
.get(chunk_index)
.ok_or_else(|| io::Error::new(io::ErrorKind::UnexpectedEof, "chunk index out of range"))?;
self.conn
.blob_open(DatabaseName::Main, "media_chunk", "bytes", rowid, true)
.map_err(|e| io::Error::new(io::ErrorKind::Other, e))
}
}
impl Read for BlobReader {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
if self.pos >= self.total_len || buf.is_empty() {
return Ok(0);
}
let chunk_index = (self.pos / self.chunk_size) as usize;
let off_in_chunk = self.pos % self.chunk_size;
// The chunk's length is derivable from total_len/chunk_size, so we don't
// depend on Blob::len(): every chunk but the last is exactly chunk_size.
let chunk_start = chunk_index as u64 * self.chunk_size;
let chunk_len = (self.total_len - chunk_start).min(self.chunk_size);
let avail_in_chunk = (chunk_len - off_in_chunk) as usize;
let avail_total = (self.total_len - self.pos) as usize;
let want = buf.len().min(avail_in_chunk).min(avail_total);
// Scope the blob borrow (it borrows `self.conn`) so it ends before we
// mutate `self.pos`.
let n = {
let mut blob = self.chunk_blob(chunk_index)?;
blob.seek(SeekFrom::Start(off_in_chunk))?;
blob.read(&mut buf[..want])?
};
self.pos += n as u64;
Ok(n)
}
}
impl Seek for BlobReader {
fn seek(&mut self, pos: SeekFrom) -> io::Result<u64> {
let new_pos = match pos {
SeekFrom::Start(n) => n as i64,
SeekFrom::End(n) => self.total_len as i64 + n,
SeekFrom::Current(n) => self.pos as i64 + n,
};
if new_pos < 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"seek before start of media",
));
}
// Allow seeking to/past end (reads then return 0), matching File semantics.
self.pos = new_pos as u64;
Ok(self.pos)
}
}
/// A write transaction over a [`BeamArchive`]. All writes are buffered until
/// [`BeamTxn::commit`]; dropping without committing rolls back. Lets a save
/// touch only the rows that changed, in place, without rewriting unchanged media.
pub struct BeamTxn<'a> {
tx: rusqlite::Transaction<'a>,
chunk_size: u64,
}
impl BeamTxn<'_> {
pub fn put_media_packed(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
write_media_packed(&self.tx, self.chunk_size, id, kind, codec, bytes, meta)
}
pub fn put_media_referenced(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
write_media_referenced(&self.tx, id, kind, codec, ext_path, meta)
}
/// Like [`BeamTxn::put_media_packed`] but streams the bytes from a file on
/// disk chunk-by-chunk, so an arbitrarily large file is never fully loaded
/// into memory. `total_len` is taken from the bytes actually read.
pub fn put_media_packed_from_path(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
path: &Path,
meta: MediaMeta,
) -> Result<(), String> {
write_media_packed_from_path(&self.tx, self.chunk_size, id, kind, codec, path, meta)
}
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
set_project_json_conn(&self.tx, json)
}
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
set_meta_conn(&self.tx, key, value)
}
/// Does a media row with this id already exist?
pub fn media_exists(&self, id: Uuid) -> Result<bool, String> {
let id_bytes = id.as_bytes().to_vec();
let n: i64 = self
.tx
.query_row("SELECT COUNT(*) FROM media WHERE id = ?1", [&id_bytes], |r| r.get(0))
.map_err(map_sql)?;
Ok(n > 0)
}
/// Every media id currently in the archive.
pub fn all_media_ids(&self) -> Result<Vec<Uuid>, String> {
let mut stmt = self.tx.prepare("SELECT id FROM media").map_err(map_sql)?;
let rows = stmt.query_map([], |r| r.get::<_, Vec<u8>>(0)).map_err(map_sql)?;
let mut out = Vec::new();
for row in rows {
out.push(uuid_from_bytes(&row.map_err(map_sql)?)?);
}
Ok(out)
}
/// Delete a media row (and its chunks).
pub fn delete_media(&self, id: Uuid) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
self.tx
.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
self.tx
.execute("DELETE FROM media WHERE id = ?1", [&id_bytes])
.map_err(map_sql)?;
Ok(())
}
/// Delete every media row whose id is not in `keep` (orphan cleanup).
pub fn retain_media(&self, keep: &std::collections::HashSet<Uuid>) -> Result<usize, String> {
let mut removed = 0;
for id in self.all_media_ids()? {
if !keep.contains(&id) {
self.delete_media(id)?;
removed += 1;
}
}
Ok(removed)
}
pub fn commit(self) -> Result<(), String> {
self.tx.commit().map_err(map_sql)
}
}
// -- shared write helpers (used by both BeamArchive and BeamTxn) --------------
fn write_media_packed(
conn: &Connection,
chunk_size: u64,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Packed as i64,
bytes.len() as i64,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
for (chunk_index, chunk) in bytes.chunks(chunk_size as usize).enumerate() {
conn.execute(
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
rusqlite::params![id_bytes, chunk_index as i64, chunk],
)
.map_err(map_sql)?;
}
Ok(())
}
fn write_media_packed_from_path(
conn: &Connection,
chunk_size: u64,
id: Uuid,
kind: MediaKind,
codec: &str,
path: &Path,
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
let file = std::fs::File::open(path).map_err(|e| format!("Failed to open {:?}: {}", path, e))?;
let mut reader = std::io::BufReader::new(file);
let mut buf = vec![0u8; chunk_size as usize];
let mut chunk_index: i64 = 0;
let mut total_len: u64 = 0;
loop {
// Fill `buf` up to chunk_size, tolerating short reads.
let mut filled = 0usize;
while filled < buf.len() {
let n = reader
.read(&mut buf[filled..])
.map_err(|e| format!("Failed to read {:?}: {}", path, e))?;
if n == 0 {
break;
}
filled += n;
}
if filled == 0 {
break;
}
conn.execute(
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
rusqlite::params![id_bytes, chunk_index, &buf[..filled]],
)
.map_err(map_sql)?;
chunk_index += 1;
total_len += filled as u64;
if filled < buf.len() {
break; // reached EOF
}
}
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Packed as i64,
total_len as i64,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
Ok(())
}
fn write_media_referenced(
conn: &Connection,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, ?5, 0, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Referenced as i64,
ext_path,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
Ok(())
}
fn set_project_json_conn(conn: &Connection, json: &str) -> Result<(), String> {
conn.execute(
"INSERT INTO project_json (id, data) VALUES (0, ?1)
ON CONFLICT(id) DO UPDATE SET data = excluded.data",
[json],
)
.map_err(map_sql)?;
Ok(())
}
fn set_meta_conn(conn: &Connection, key: &str, value: &str) -> Result<(), String> {
conn.execute(
"INSERT INTO meta (key, value) VALUES (?1, ?2)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
rusqlite::params![key, value],
)
.map_err(map_sql)?;
Ok(())
}
/// Read a packed media item's full bytes via a fresh **read-only** connection,
/// without an open [`BeamArchive`] handle. Returns `Ok(None)` if the item has no
/// chunk rows. Used for on-demand raster fault-in: a read-only connection can't
/// conflict with an in-place save and needs no long-lived handle.
pub fn read_packed_media_readonly(db_path: &Path, id: Uuid) -> Result<Option<Vec<u8>>, String> {
let conn = Connection::open_with_flags(
db_path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)
.map_err(map_sql)?;
let id_bytes = id.as_bytes().to_vec();
let mut stmt = conn
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::new();
let mut any = false;
for row in rows {
out.extend_from_slice(&row.map_err(map_sql)?);
any = true;
}
Ok(if any { Some(out) } else { None })
}
fn map_sql(e: rusqlite::Error) -> String {
format!("SQLite error: {}", e)
}
fn uuid_from_bytes(bytes: &[u8]) -> Result<Uuid, String> {
let arr: [u8; 16] = bytes
.try_into()
.map_err(|_| format!("Invalid uuid blob length {}", bytes.len()))?;
Ok(Uuid::from_bytes(arr))
}
// Tests live in `tests/beam_archive.rs` (integration tests), so they compile the
// library in non-test mode and don't depend on the crate's other `#[cfg(test)]`
// modules.

21
lightningbeam-ui/lightningbeam-core/src/brush_engine.rs
View File

@ -575,6 +575,27 @@ pub fn encode_png(img: &RgbaImage) -> Result<Vec<u8>, String> {
Ok(buf.into_inner())
}
/// Long-edge cap (pixels) for raster keyframe proxies — low-res page-in placeholders.
pub const RASTER_PROXY_MAX_EDGE: u32 = 192;
/// Downscale a full RGBA buffer to a low-res proxy PNG (long edge ≤
/// `RASTER_PROXY_MAX_EDGE`). Returns `None` on invalid dims/length or encode failure.
pub fn encode_raster_proxy_png(raw: &[u8], width: u32, height: u32) -> Option<Vec<u8>> {
if width == 0 || height == 0 || raw.len() != (width as usize * height as usize * 4) {
return None;
}
let long = width.max(height);
let (pw, ph) = if long <= RASTER_PROXY_MAX_EDGE {
(width, height)
} else {
let s = RASTER_PROXY_MAX_EDGE as f32 / long as f32;
(((width as f32 * s).round() as u32).max(1), ((height as f32 * s).round() as u32).max(1))
};
let img = RgbaImage::from_raw(width, height, raw.to_vec())?;
let resized = image::imageops::resize(&img, pw, ph, image::imageops::FilterType::Triangle);
encode_png(&resized).ok()
}
/// Decode PNG bytes into an `RgbaImage`
pub fn decode_png(data: &[u8]) -> Result<RgbaImage, String> {
image::load_from_memory(data)

19
lightningbeam-ui/lightningbeam-core/src/clip.rs
View File

@ -263,9 +263,11 @@ pub struct ImageAsset {
/// Image height in pixels
pub height: u32,
/// Embedded image data (for project portability)
/// If None, the image will be loaded from path when needed
#[serde(skip_serializing_if = "Option::is_none")]
/// Raw image file bytes. NOT serialized to project JSON — persisted as a
/// `MediaKind::ImageAsset` row in the `.beam` container (chunked, pageable) and
/// read back on load. `default` so new projects (bytes in the container, not JSON)
/// deserialize; old projects with base64-embedded `data` still load via deserialize.
#[serde(default, skip_serializing)]
pub data: Option<Vec<u8>>,
/// Folder this asset belongs to (None = root of category)
@ -902,7 +904,7 @@ mod tests {
#[test]
fn test_audio_clip_midi() {
let clip = AudioClip::new_midi("Piano Melody", 1, 60.0);
let clip = AudioClip::new_midi("Piano Melody", 1, daw_backend::Beats(60.0));
assert_eq!(clip.name, "Piano Melody");
assert_eq!(clip.duration, 60.0);
match &clip.clip_type {
@ -952,7 +954,10 @@ mod tests {
assert_eq!(instance.trim_start, 2.0);
assert_eq!(instance.trim_end, Some(8.0));
assert_eq!(instance.effective_duration(10.0), 6.0);
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
// beats-domain effective duration equals the seconds content window.
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(10.0, &tempo_map), 6.0);
}
#[test]
@ -963,7 +968,9 @@ mod tests {
assert_eq!(instance.trim_start, 2.0);
assert_eq!(instance.trim_end, None);
assert_eq!(instance.effective_duration(10.0), 8.0);
// At 60 BPM the tempo map is identity (1 beat == 1 second).
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(10.0, &tempo_map), 8.0);
}
#[test]

23
lightningbeam-ui/lightningbeam-core/src/document.rs
View File

@ -650,6 +650,29 @@ impl Document {
layers
}
/// Get mutable references to all layers across the entire document
/// (root + nested groups + inside all vector clips). Mirrors [`all_layers`].
pub fn all_layers_mut(&mut self) -> Vec<&mut AnyLayer> {
let mut layers: Vec<&mut AnyLayer> = Vec::new();
// Iterative walk with an explicit stack of child slices. Group layers are
// descended into but not themselves collected (they hold no keyframes).
let mut stack: Vec<&mut [AnyLayer]> = vec![&mut self.root.children];
while let Some(list) = stack.pop() {
for layer in list {
match layer {
AnyLayer::Group(g) => {
stack.push(&mut g.children);
}
other => layers.push(other),
}
}
}
for clip in self.vector_clips.values_mut() {
layers.extend(clip.layers.root_data_mut());
}
layers
}
// === CLIP LIBRARY METHODS ===
/// Add a vector clip to the library

10
lightningbeam-ui/lightningbeam-core/src/effect_layer.rs
View File

@ -240,14 +240,18 @@ mod tests {
let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end
layer.add_clip_instance(effect2);
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
// query time in seconds equals the effect's beats-domain extents.
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
// At time 2: only effect1 active
assert_eq!(layer.active_clip_instances_at(2.0, 60.0).len(), 1);
assert_eq!(layer.active_clip_instances_at(2.0, &tempo_map).len(), 1);
// At time 4: both effects active
assert_eq!(layer.active_clip_instances_at(4.0, 60.0).len(), 2);
assert_eq!(layer.active_clip_instances_at(4.0, &tempo_map).len(), 2);
// At time 7: only effect2 active
assert_eq!(layer.active_clip_instances_at(7.0, 60.0).len(), 1);
assert_eq!(layer.active_clip_instances_at(7.0, &tempo_map).len(), 1);
}
#[test]

6
lightningbeam-ui/lightningbeam-core/src/effect_registry.rs
View File

@ -75,7 +75,7 @@ impl EffectRegistry {
INVERT_ID,
"Invert",
EffectCategory::Color,
include_str!("shaders/effect_invert.wgsl"),
format!("{}\n{}", crate::gpu::COLOR_WGSL, include_str!("shaders/effect_invert.wgsl")),
vec![
EffectParameterDef::float_range("amount", "Amount", 1.0, 0.0, 1.0),
],
@ -88,7 +88,7 @@ impl EffectRegistry {
BRIGHTNESS_CONTRAST_ID,
"Brightness/Contrast",
EffectCategory::Color,
include_str!("shaders/effect_brightness_contrast.wgsl"),
format!("{}\n{}", crate::gpu::COLOR_WGSL, include_str!("shaders/effect_brightness_contrast.wgsl")),
vec![
EffectParameterDef::float_range("brightness", "Brightness", 0.0, -1.0, 1.0),
EffectParameterDef::float_range("contrast", "Contrast", 1.0, 0.0, 3.0),
@ -102,7 +102,7 @@ impl EffectRegistry {
HUE_SATURATION_ID,
"Hue/Saturation",
EffectCategory::Color,
include_str!("shaders/effect_hue_saturation.wgsl"),
format!("{}\n{}", crate::gpu::COLOR_WGSL, include_str!("shaders/effect_hue_saturation.wgsl")),
vec![
EffectParameterDef::angle("hue", "Hue Shift", 0.0),
EffectParameterDef::float_range("saturation", "Saturation", 1.0, 0.0, 3.0),

820
lightningbeam-ui/lightningbeam-core/src/file_io.rs
View File

@ -1,20 +1,25 @@
//! File I/O for .beam project files
//!
//! This module handles saving and loading Lightningbeam projects in the .beam format,
//! which is a ZIP archive containing:
//! - project.json (compressed) - Project metadata and structure
//! - media/ directory (uncompressed) - Embedded media files (FLAC for audio)
//! The `.beam` format is a single **SQLite database** (see [`crate::beam_archive`]):
//! - `project_json` table — serialized project metadata and structure
//! - `media` / `media_chunk` tables — audio and raster media (packed as chunked
//! blobs, or referenced by external path)
//!
//! Older `.beam` files are ZIP archives; [`load_beam`] detects and reads those
//! too (via [`load_beam_zip_legacy`]). Saving always writes the SQLite form, so
//! opening a legacy file and saving migrates it.
use crate::beam_archive::{BeamArchive, MediaKind, MediaMeta, LARGE_MEDIA_THRESHOLD};
use crate::document::Document;
use daw_backend::audio::pool::AudioPoolEntry;
use daw_backend::audio::project::Project as AudioProject;
use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::fs::File;
use std::io::{Read, Write};
use std::io::Read;
use std::path::{Path, PathBuf};
use zip::write::FileOptions;
use zip::{CompressionMethod, ZipArchive, ZipWriter};
use flacenc::error::Verify;
use uuid::Uuid;
use zip::ZipArchive;
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
/// File format version
@ -51,9 +56,7 @@ pub struct SerializedAudioBackend {
/// Audio project (tracks, MIDI clips, etc.)
pub project: AudioProject,
/// Audio pool entries (metadata and paths for audio files)
/// Note: embedded_data field from daw-backend is ignored; embedded files
/// are stored as FLAC in the ZIP's media/audio/ directory instead
/// Audio pool entries (metadata and media references for audio files)
pub audio_pool_entries: Vec<AudioPoolEntry>,
/// Mapping from UI layer UUIDs to backend TrackIds
@ -63,6 +66,25 @@ pub struct SerializedAudioBackend {
}
/// How to store a media file at or above [`LARGE_MEDIA_THRESHOLD`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum LargeMediaMode {
/// Not yet decided — prompt the user the first time a large file is imported.
/// Treated as [`LargeMediaMode::Reference`] at save time. Resetting the
/// preference to `Ask` re-triggers the prompt (useful for testing).
Ask,
/// Pack the bytes into the `.beam` container (chunked, streamed from disk).
Pack,
/// Keep the file external and store only a path reference.
Reference,
}
impl Default for LargeMediaMode {
fn default() -> Self {
LargeMediaMode::Ask
}
}
/// Settings for saving a project
#[derive(Debug, Clone)]
pub struct SaveSettings {
@ -74,6 +96,10 @@ pub struct SaveSettings {
/// Force linking all media files (don't embed any)
pub force_link_all: bool,
/// How to store files at/above [`LARGE_MEDIA_THRESHOLD`] (pack vs reference).
/// `Ask` behaves as `Reference` here (safe default: don't bloat the DB).
pub large_media_mode: LargeMediaMode,
}
impl Default for SaveSettings {
@ -82,6 +108,7 @@ impl Default for SaveSettings {
auto_embed_threshold_bytes: 10_000_000, // 10 MB
force_embed_all: false,
force_link_all: false,
large_media_mode: LargeMediaMode::Ask,
}
}
}
@ -100,6 +127,11 @@ pub struct LoadedProject {
/// Loaded audio pool entries
pub audio_pool_entries: Vec<AudioPoolEntry>,
/// Persisted video-thumbnail packs by clip id (opaque LBTN blobs; decoded and
/// inserted into the VideoManager by the editor). Clips present here don't need
/// their thumbnails regenerated on load.
pub thumbnail_blobs: std::collections::HashMap<uuid::Uuid, Vec<u8>>,
/// List of files that couldn't be found
pub missing_files: Vec<MissingFileInfo>,
}
@ -125,282 +157,345 @@ pub enum MediaFileType {
Image,
}
/// Save a project to a .beam file
/// Save a project to a `.beam` file (SQLite container).
///
/// This function:
/// 1. Prepares audio project for save (saves AudioGraph presets)
/// 2. Serializes project data to JSON
/// 3. Creates ZIP archive with compressed project.json
/// 4. Embeds media files as FLAC (for audio) in media/ directory
/// Re-saving an existing SQLite `.beam` updates it **in place** inside a single
/// transaction: unchanged (large) media is never rewritten, only changed rows
/// are touched, and the commit is atomic/crash-safe. A brand-new file or a
/// legacy-ZIP migration is written to a temp file and atomically renamed (there
/// is no large existing container to copy in that case).
///
/// # Arguments
/// * `path` - Path to save the .beam file
/// * `document` - UI document state
/// * `audio_project` - Audio backend project
/// * `audio_pool_entries` - Serialized audio pool entries
/// * `settings` - Save settings (embedding preferences)
///
/// # Returns
/// Ok(()) on success, or error message
/// Audio and raster media become rows in the `media` table — packed as chunked
/// blobs, or referenced by external path for files at/above
/// [`LARGE_MEDIA_THRESHOLD`]. `project.json` goes in the `project_json` table.
/// Whether a stored media codec is an audio format the disk reader (Symphonia)
/// can stream directly from a packed blob. Video-container audio tracks and any
/// unknown formats fall back to the legacy reconstitution-and-decode path.
fn is_streamable_audio_codec(codec: &str) -> bool {
matches!(
codec.to_lowercase().as_str(),
"mp3" | "flac" | "ogg" | "oga" | "wav" | "wave" | "aiff" | "aif"
| "aac" | "m4a" | "opus" | "alac" | "caf"
)
}
/// A `Sync` wrapper over core's `BlobReader` so it satisfies Symphonia's
/// `MediaSource: Send + Sync`. `BlobReader` holds a rusqlite `Connection`
/// (`Send` but `!Sync`); the disk reader uses it single-threaded, so the
/// hot Read/Seek path goes through `Mutex::get_mut` (no runtime locking).
struct SyncBlobReader {
inner: std::sync::Mutex<crate::beam_archive::BlobReader>,
len: u64,
}
impl std::io::Read for SyncBlobReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.inner.get_mut().unwrap().read(buf)
}
}
impl std::io::Seek for SyncBlobReader {
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
self.inner.get_mut().unwrap().seek(pos)
}
}
impl daw_backend::audio::MediaByteSource for SyncBlobReader {
fn byte_len(&self) -> u64 {
self.len
}
}
/// The host's packed-media byte-source factory: opens an independent streaming
/// reader over a `.beam` container's packed audio by media id. Installed into the
/// engine on load so container-packed audio streams without a full decode.
#[derive(Debug)]
struct BeamBlobFactory {
db_path: PathBuf,
}
impl daw_backend::audio::AudioBlobSourceFactory for BeamBlobFactory {
fn open(
&self,
media_id: &str,
) -> Result<Box<dyn daw_backend::audio::MediaByteSource>, String> {
let id = Uuid::parse_str(media_id).map_err(|e| format!("bad media id {}: {}", media_id, e))?;
let archive = BeamArchive::open(&self.db_path)?;
let reader = archive.open_blob_reader(&self.db_path, id)?;
let len = reader.len();
Ok(Box::new(SyncBlobReader { inner: std::sync::Mutex::new(reader), len }))
}
}
/// Build a packed-media byte-source factory for a `.beam` file, to install into
/// the engine (`EngineController::set_blob_source_factory`) before loading so its
/// packed audio can be streamed.
pub fn blob_source_factory(
beam_path: &Path,
) -> std::sync::Arc<dyn daw_backend::audio::AudioBlobSourceFactory> {
std::sync::Arc::new(BeamBlobFactory { db_path: beam_path.to_path_buf() })
}
/// Deterministic id for the waveform-pyramid media row of audio pool entry
/// `pool_index`, within a single project container. Stable across saves (so an
/// in-place re-save reuses the row instead of orphaning/rewriting it) and
/// independent of how the audio bytes are stored. The top 32 bits are a fixed
/// "LBWF" sentinel so it can't collide with the random v4 ids used elsewhere.
fn waveform_media_id(pool_index: usize) -> Uuid {
const SENTINEL: u128 = 0x4C42_5746u128 << 96; // "LBWF" in the high 32 bits
Uuid::from_u128(SENTINEL | (pool_index as u128))
}
/// Deterministic id for the thumbnail-pack media row of a video clip. Derived from
/// the clip id by XOR with a fixed constant — bijective and stable across saves, so
/// it reuses the row in place, and it can't (in practice) collide with the random
/// v4 ids used for other media of a different kind.
fn thumbnail_media_id(clip_id: Uuid) -> Uuid {
// "LBTN" repeated — moves clip ids into a distinct region of the id space.
const SENTINEL: u128 = 0x4C42_544E_4C42_544E_4C42_544E_4C42_544E;
Uuid::from_u128(clip_id.as_u128() ^ SENTINEL)
}
/// Derived id for a raster keyframe's low-res proxy row (distinct from the keyframe's
/// own full-res `Raster` row, which is keyed by the raw keyframe id).
fn raster_proxy_media_id(kf_id: Uuid) -> Uuid {
// "LBPX" repeated — distinct id region from the full raster + thumbnail rows.
const SENTINEL: u128 = 0x4C42_5058_4C42_5058_4C42_5058_4C42_5058;
Uuid::from_u128(kf_id.as_u128() ^ SENTINEL)
}
pub fn save_beam(
path: &Path,
document: &Document,
audio_project: &mut AudioProject,
audio_pool_entries: Vec<AudioPoolEntry>,
layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>,
thumbnail_blobs: &std::collections::HashMap<uuid::Uuid, Vec<u8>>,
_settings: &SaveSettings,
) -> Result<(), String> {
let fn_start = std::time::Instant::now();
eprintln!("📊 [SAVE_BEAM] Starting save_beam()...");
eprintln!("📊 [SAVE_BEAM] Starting save_beam() (SQLite container)...");
// 1. Create backup if file exists and open it for reading old audio files
let step1_start = std::time::Instant::now();
let mut old_zip = if path.exists() {
let backup_path = path.with_extension("beam.backup");
std::fs::copy(path, &backup_path)
.map_err(|e| format!("Failed to create backup: {}", e))?;
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
let in_place = path.exists() && BeamArchive::is_sqlite(path);
// Open the backup as a ZIP archive for reading
match File::open(&backup_path) {
Ok(file) => match ZipArchive::new(file) {
Ok(archive) => {
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup and open for reading took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
Some(archive)
}
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup as ZIP: {}, will not copy old audio files", e);
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
None
}
},
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup: {}, will not copy old audio files", e);
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
None
}
}
// In-place for an existing SQLite container (don't rewrite unchanged media);
// temp + atomic rename for new files / legacy-ZIP migration.
let tmp_path = path.with_extension("beam.tmp");
let mut archive = if in_place {
BeamArchive::open(path)?
} else {
eprintln!("📊 [SAVE_BEAM] Step 1: No backup needed (new file)");
None
BeamArchive::create(&tmp_path)?
};
// 2. Graph presets are already populated by the engine thread (in GetProject handler)
// before cloning. Do NOT call prepare_for_save() here — the cloned project has
// default empty graphs (AudioTrack::clone() doesn't copy the graph), so calling
// prepare_for_save() would overwrite the good presets with empty ones.
let step2_start = std::time::Instant::now();
eprintln!("📊 [SAVE_BEAM] Step 2: (graph presets already prepared) took {:.2}ms", step2_start.elapsed().as_secs_f64() * 1000.0);
let now = chrono::Utc::now().to_rfc3339();
let created = if in_place {
archive.get_meta("created").ok().flatten().unwrap_or_else(|| now.clone())
} else {
now.clone()
};
// 3. Create ZIP writer
let step3_start = std::time::Instant::now();
let file = File::create(path)
.map_err(|e| format!("Failed to create file: {}", e))?;
let mut zip = ZipWriter::new(file);
eprintln!("📊 [SAVE_BEAM] Step 3: Create ZIP writer took {:.2}ms", step3_start.elapsed().as_secs_f64() * 1000.0);
let txn = archive.transaction()?;
// 4. Process audio pool entries and write embedded audio files to ZIP
// Priority: old ZIP file > external file > encode PCM as FLAC
let step4_start = std::time::Instant::now();
let mut modified_entries = Vec::new();
let mut flac_encode_time = 0.0;
let mut zip_write_time = 0.0;
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
// --- audio pool entries -> media rows (packed) or external references ---
let mut modified_entries = Vec::with_capacity(audio_pool_entries.len());
let mut live_media: HashSet<Uuid> = HashSet::new();
for entry in &audio_pool_entries {
let mut modified_entry = entry.clone();
let mut e = entry.clone();
let existing_id = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok());
// Try to get audio data from various sources (in priority order)
let audio_source: Option<(Vec<u8>, String)> = if let Some(ref rel_path) = entry.relative_path {
// Priority 1: Check if file is in the old ZIP
if rel_path.starts_with("media/audio/") {
if let Some(ref mut old_zip_archive) = old_zip {
match old_zip_archive.by_name(rel_path) {
Ok(mut file) => {
let mut bytes = Vec::new();
if file.read_to_end(&mut bytes).is_ok() {
let extension = rel_path.split('.').last().unwrap_or("bin").to_string();
eprintln!("📊 [SAVE_BEAM] Copying from old ZIP: {}", rel_path);
Some((bytes, extension))
} else {
eprintln!("⚠️ [SAVE_BEAM] Failed to read {} from old ZIP", rel_path);
None
// Already packed in this archive (in-place re-save): leave the bytes
// untouched, just keep the reference.
if let Some(id) = existing_id {
if txn.media_exists(id)? {
live_media.insert(id);
e.media_id = Some(id.to_string());
e.relative_path = None;
e.embedded_data = None;
modified_entries.push(e);
continue;
}
}
Err(_) => {
eprintln!("⚠️ [SAVE_BEAM] File {} not found in old ZIP", rel_path);
None
}
}
} else {
None
}
}
// Priority 2: Check external filesystem
else {
let full_path = project_dir.join(rel_path);
if full_path.exists() {
match std::fs::read(&full_path) {
Ok(bytes) => {
let extension = full_path.extension()
.and_then(|e| e.to_str())
.unwrap_or("bin")
.to_string();
eprintln!("📊 [SAVE_BEAM] Using external file: {:?}", full_path);
Some((bytes, extension))
}
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to read {:?}: {}", full_path, e);
None
}
}
} else {
eprintln!("⚠️ [SAVE_BEAM] External file not found: {:?}", full_path);
None
}
}
} else {
None
// Otherwise resolve the source: external file (Priority 2, streamed from
// disk so a huge file is never fully loaded), or embedded data (Priority 3).
let meta = MediaMeta {
channels: Some(entry.channels),
sample_rate: Some(entry.sample_rate),
..Default::default()
};
let mut wrote_packed: Option<Uuid> = None;
let mut referenced: Option<String> = None;
if let Some((audio_bytes, extension)) = audio_source {
// We have the original file - copy it directly
let zip_filename = format!("media/audio/{}.{}", entry.pool_index, extension);
let file_options = FileOptions::default()
.compression_method(CompressionMethod::Stored);
zip.start_file(&zip_filename, file_options)
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?;
let write_start = std::time::Instant::now();
zip.write_all(&audio_bytes)
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?;
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0;
// Update entry to point to ZIP file
modified_entry.embedded_data = None;
modified_entry.relative_path = Some(zip_filename);
} else if let Some(ref embedded_data) = entry.embedded_data {
// Priority 3: No original file - encode PCM as FLAC
eprintln!("📊 [SAVE_BEAM] Encoding PCM to FLAC for pool {} (no original file)", entry.pool_index);
// Embedded data is always PCM - encode as FLAC
let audio_bytes = BASE64_STANDARD.decode(&embedded_data.data_base64)
.map_err(|e| format!("Failed to decode base64 audio data for pool index {}: {}", entry.pool_index, e))?;
let zip_filename = format!("media/audio/{}.flac", entry.pool_index);
let file_options = FileOptions::default()
.compression_method(CompressionMethod::Stored);
zip.start_file(&zip_filename, file_options)
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?;
// Encode PCM samples to FLAC
let flac_start = std::time::Instant::now();
// The audio_bytes are raw PCM samples (interleaved f32 little-endian)
let samples: Vec<f32> = audio_bytes
.chunks_exact(4)
.map(|chunk| f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]))
.collect();
// Convert f32 samples to i32 for FLAC encoding
let samples_i32: Vec<i32> = samples
.iter()
.map(|&s| {
let clamped = s.clamp(-1.0, 1.0);
(clamped * 8388607.0) as i32
})
.collect();
// Configure FLAC encoder
let config = flacenc::config::Encoder::default()
.into_verified()
.map_err(|(_, e)| format!("FLAC encoder config error: {:?}", e))?;
let source = flacenc::source::MemSource::from_samples(
&samples_i32,
entry.channels as usize,
24,
entry.sample_rate as usize,
);
// Encode to FLAC
let flac_stream = flacenc::encode_with_fixed_block_size(
&config,
source,
config.block_size,
).map_err(|e| format!("FLAC encoding failed: {:?}", e))?;
// Convert stream to bytes
use flacenc::component::BitRepr;
let mut sink = flacenc::bitsink::ByteSink::new();
flac_stream.write(&mut sink)
.map_err(|e| format!("Failed to write FLAC stream: {:?}", e))?;
let flac_bytes = sink.as_slice();
flac_encode_time += flac_start.elapsed().as_secs_f64() * 1000.0;
let write_start = std::time::Instant::now();
zip.write_all(flac_bytes)
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?;
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0;
// Update entry to point to ZIP file instead of embedding data
modified_entry.embedded_data = None;
modified_entry.relative_path = Some(zip_filename);
if let Some(rel) = entry.relative_path.as_ref() {
let full = if Path::new(rel).is_absolute() {
PathBuf::from(rel)
} else {
project_dir.join(rel)
};
// Require an actual file: an empty/blank `relative_path` resolves to the
// project directory itself (`join("")` == dir), which `exists()` accepts
// but can't be read as media. `is_file()` skips dirs + missing paths, so
// such an entry correctly falls through to embedded data below.
if full.is_file() {
let size = std::fs::metadata(&full).map(|m| m.len()).unwrap_or(0);
let codec = full
.extension()
.and_then(|x| x.to_str())
.unwrap_or("bin")
.to_lowercase();
// Video-audio entries are always referenced (the video is already
// referenced by its VideoClip; reloaded by re-probing via FFmpeg).
// Otherwise large files honor the user's pack-vs-reference choice
// (`Ask` == reference); smaller files are always packed.
let reference_it = entry.is_video_audio
|| (size >= LARGE_MEDIA_THRESHOLD
&& _settings.large_media_mode != LargeMediaMode::Pack);
if reference_it {
referenced = Some(rel.clone());
} else {
let id = existing_id.unwrap_or_else(Uuid::new_v4);
txn.put_media_packed_from_path(id, MediaKind::Audio, &codec, &full, meta)?;
wrote_packed = Some(id);
}
}
}
modified_entries.push(modified_entry);
if wrote_packed.is_none() && referenced.is_none() {
if let Some(ed) = entry.embedded_data.as_ref() {
if let Ok(bytes) = BASE64_STANDARD.decode(&ed.data_base64) {
let id = existing_id.unwrap_or_else(Uuid::new_v4);
txn.put_media_packed(id, MediaKind::Audio, &ed.format.to_lowercase(), &bytes, meta)?;
wrote_packed = Some(id);
}
eprintln!("📊 [SAVE_BEAM] Step 4: Process audio pool ({} entries) took {:.2}ms",
audio_pool_entries.len(), step4_start.elapsed().as_secs_f64() * 1000.0);
if flac_encode_time > 0.0 {
eprintln!("📊 [SAVE_BEAM] - FLAC encoding: {:.2}ms", flac_encode_time);
}
if zip_write_time > 0.0 {
eprintln!("📊 [SAVE_BEAM] - ZIP writing: {:.2}ms", zip_write_time);
}
// 4b. Write raster layer PNG buffers to ZIP (media/raster/<keyframe-uuid>.png)
let step4b_start = std::time::Instant::now();
let raster_file_options = FileOptions::default()
.compression_method(CompressionMethod::Stored); // PNG is already compressed
if let Some(id) = wrote_packed {
live_media.insert(id);
e.media_id = Some(id.to_string());
e.relative_path = None;
e.embedded_data = None;
} else if let Some(rel) = referenced {
e.media_id = None;
e.relative_path = Some(rel);
e.embedded_data = None;
} // else: nothing available — keep original references (reported missing on load)
// Persist this entry's waveform pyramid (keyed by pool index, independent
// of the audio storage above). Reuse the row in place on re-save.
let wf_id = waveform_media_id(entry.pool_index);
if let Some(blob) = entry.waveform_blob.as_ref() {
txn.put_media_packed(wf_id, MediaKind::Waveform, "lbwf", blob, MediaMeta::default())?;
live_media.insert(wf_id);
} else if txn.media_exists(wf_id)? {
// Unchanged this save — keep the stored waveform row.
live_media.insert(wf_id);
}
modified_entries.push(e);
}
// --- raster keyframes -> media rows (PNG), keyed by keyframe id ---
// (Phase 0 writes all resident frames each save; a disk-dirty flag to skip
// unchanged frames in place is deferred to Phase 3.)
// Walk ALL layers (incl. nested in groups/clips) so nested raster keyframes
// are persisted too, and so `live_media` covers them — matching the load path,
// which arms `needs_fault_in` recursively. Top-level-only projects are unaffected.
let mut raster_count = 0usize;
for layer in &document.root.children {
for layer in document.all_layers() {
if let crate::layer::AnyLayer::Raster(rl) = layer {
for kf in &rl.keyframes {
if !kf.raw_pixels.is_empty() {
// Encode raw RGBA to PNG for storage
let img = crate::brush_engine::image_from_raw(
kf.raw_pixels.clone(), kf.width, kf.height,
);
let img =
crate::brush_engine::image_from_raw(kf.raw_pixels.clone(), kf.width, kf.height);
match crate::brush_engine::encode_png(&img) {
Ok(png_bytes) => {
let zip_path = kf.media_path.clone();
zip.start_file(&zip_path, raster_file_options)
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_path, e))?;
zip.write_all(&png_bytes)
.map_err(|e| format!("Failed to write {}: {}", zip_path, e))?;
txn.put_media_packed(
kf.id,
MediaKind::Raster,
"png",
&png_bytes,
MediaMeta {
width: Some(kf.width),
height: Some(kf.height),
..Default::default()
},
)?;
live_media.insert(kf.id);
raster_count += 1;
}
Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster PNG {}: {}", kf.media_path, e),
Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster {}: {}", kf.id, e),
}
// Low-res proxy alongside the full PNG (shown while the full pages
// in on a later load). Regenerated from the resident pixels.
let proxy_id = raster_proxy_media_id(kf.id);
if let Some(proxy_png) =
crate::brush_engine::encode_raster_proxy_png(&kf.raw_pixels, kf.width, kf.height)
{
txn.put_media_packed(
proxy_id, MediaKind::RasterProxy, "png", &proxy_png, MediaMeta::default(),
)?;
live_media.insert(proxy_id);
}
} else if txn.media_exists(kf.id)? {
// Pixels not resident but already stored — keep both rows.
live_media.insert(kf.id);
let proxy_id = raster_proxy_media_id(kf.id);
if txn.media_exists(proxy_id)? {
live_media.insert(proxy_id);
}
}
}
}
}
eprintln!("📊 [SAVE_BEAM] Step 4b: Write {} raster PNG buffers took {:.2}ms",
raster_count, step4b_start.elapsed().as_secs_f64() * 1000.0);
// 5. Build BeamProject structure with modified entries
let step5_start = std::time::Instant::now();
let now = chrono::Utc::now().to_rfc3339();
// --- video thumbnail packs -> media rows (opaque LBTN blob), keyed by a
// sentinel-derived id from the video clip id ---
for clip_id in document.video_clips.keys() {
let tn_id = thumbnail_media_id(*clip_id);
if let Some(blob) = thumbnail_blobs.get(clip_id) {
txn.put_media_packed(tn_id, MediaKind::Thumbnail, "lbtn", blob, MediaMeta::default())?;
live_media.insert(tn_id);
} else if txn.media_exists(tn_id)? {
// Not regenerated this session — keep the stored pack.
live_media.insert(tn_id);
}
}
// --- image assets -> media rows (original file bytes), keyed by asset id ---
let mut image_count = 0usize;
for (id, asset) in &document.image_assets {
if let Some(ref data) = asset.data {
let ext = asset
.path
.extension()
.and_then(|e| e.to_str())
.unwrap_or("img")
.to_lowercase();
txn.put_media_packed(
*id,
MediaKind::ImageAsset,
&ext,
data,
MediaMeta { width: Some(asset.width), height: Some(asset.height), ..Default::default() },
)?;
live_media.insert(*id);
image_count += 1;
} else if txn.media_exists(*id)? {
// Bytes not resident (paged out) but already stored — keep the row.
live_media.insert(*id);
}
}
let _ = image_count;
// --- orphan cleanup: drop media for removed clips/keyframes ---
let removed = txn.retain_media(&live_media)?;
// --- project.json + meta ---
let beam_project = BeamProject {
version: BEAM_VERSION.to_string(),
created: now.clone(),
modified: now,
created: created.clone(),
modified: now.clone(),
ui_state: document.clone(),
audio_backend: SerializedAudioBackend {
sample_rate: 48000, // TODO: Get from audio engine
@ -409,52 +504,222 @@ pub fn save_beam(
layer_to_track_map: layer_to_track_map.clone(),
},
};
eprintln!("📊 [SAVE_BEAM] Step 5: Build BeamProject structure took {:.2}ms", step5_start.elapsed().as_secs_f64() * 1000.0);
// 6. Write project.json (compressed with DEFLATE)
let step6_start = std::time::Instant::now();
let json_options = FileOptions::default()
.compression_method(CompressionMethod::Deflated)
.compression_level(Some(6));
zip.start_file("project.json", json_options)
.map_err(|e| format!("Failed to create project.json in ZIP: {}", e))?;
let json = serde_json::to_string_pretty(&beam_project)
let json = serde_json::to_string(&beam_project)
.map_err(|e| format!("JSON serialization failed: {}", e))?;
txn.set_project_json(&json)?;
txn.set_meta("version", BEAM_VERSION)?;
txn.set_meta("created", &created)?;
txn.set_meta("modified", &now)?;
txn.commit()?;
zip.write_all(json.as_bytes())
.map_err(|e| format!("Failed to write project.json: {}", e))?;
eprintln!("📊 [SAVE_BEAM] Step 6: Write project.json ({} bytes) took {:.2}ms", json.len(), step6_start.elapsed().as_secs_f64() * 1000.0);
// 7. Finalize ZIP
let step7_start = std::time::Instant::now();
zip.finish()
.map_err(|e| format!("Failed to finalize ZIP: {}", e))?;
eprintln!("📊 [SAVE_BEAM] Step 7: Finalize ZIP took {:.2}ms", step7_start.elapsed().as_secs_f64() * 1000.0);
eprintln!("📊 [SAVE_BEAM] ✅ Total save_beam() time: {:.2}ms", fn_start.elapsed().as_secs_f64() * 1000.0);
// Close the connection before renaming (required on Windows; harmless elsewhere).
drop(archive);
if !in_place {
std::fs::rename(&tmp_path, path)
.map_err(|e| format!("Failed to finalize {:?}: {}", path, e))?;
}
eprintln!(
"📊 [SAVE_BEAM] ✅ Saved {} audio + {} raster media, {} orphans removed, in {:.2}ms",
audio_pool_entries.len(),
raster_count,
removed,
fn_start.elapsed().as_secs_f64() * 1000.0
);
Ok(())
}
/// Load a project from a .beam file
/// Load a project from a `.beam` file.
///
/// This function:
/// 1. Opens ZIP archive and reads project.json
/// 2. Deserializes project data
/// 3. Loads embedded media files from archive
/// 4. Attempts to load external media files
/// 5. Rebuilds AudioGraphs from presets with correct sample_rate
///
/// # Arguments
/// * `path` - Path to the .beam file
///
/// # Returns
/// LoadedProject on success (with missing_files list), or error message
/// Detects the container format: SQLite (current) or legacy ZIP, and dispatches
/// accordingly. Both produce an identical [`LoadedProject`].
pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
if BeamArchive::is_sqlite(path) {
load_beam_sqlite(path)
} else {
load_beam_zip_legacy(path)
}
}
/// Load a project from a SQLite `.beam` container.
///
/// Phase 0 reconstitutes packed audio into each entry's `embedded_data` so the
/// existing (full-decode) audio pool loader keeps working unchanged; Phase 1b
/// replaces this with streaming reads via `BlobReader`.
fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
let fn_start = std::time::Instant::now();
eprintln!("📊 [LOAD_BEAM] Starting load_beam()...");
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (SQLite container)...");
let archive = BeamArchive::open(path)?;
let json = archive.get_project_json()?;
let beam_project: BeamProject = serde_json::from_str(&json)
.map_err(|e| format!("Failed to deserialize project.json: {}", e))?;
if beam_project.version != BEAM_VERSION {
return Err(format!(
"Unsupported file version: {} (expected {})",
beam_project.version, BEAM_VERSION
));
}
let mut document = beam_project.ui_state;
document.tempo_map_mut().rebuild_seconds();
let mut audio_project = beam_project.audio_backend.project;
audio_project
.rebuild_audio_graphs(DEFAULT_BUFFER_SIZE)
.map_err(|e| format!("Failed to rebuild audio graphs: {}", e))?;
let layer_to_track_map = beam_project.audio_backend.layer_to_track_map;
// For each packed audio item: stream it (leave `embedded_data` empty so the
// pool builds a Compressed placeholder backed by the blob factory) when it's a
// recognized audio codec; otherwise fall back to the legacy reconstitution
// (whole bytes → base64 → decode), which still covers video-container audio
// tracks symphonia can't stream and any unknown formats.
let mut restored_entries = Vec::with_capacity(beam_project.audio_backend.audio_pool_entries.len());
for entry in &beam_project.audio_backend.audio_pool_entries {
let mut e = entry.clone();
if let Some(id) = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok()) {
match archive.media_info(id) {
Ok(Some(info)) => {
if is_streamable_audio_codec(&info.codec) {
// Stream: keep media_id, no embedded bytes. The engine opens
// the packed blob via the factory at activation time.
e.embedded_data = None;
e.relative_path = None;
} else {
match archive.read_media_full(id) {
Ok(bytes) => {
e.embedded_data = Some(daw_backend::audio::pool::EmbeddedAudioData {
data_base64: BASE64_STANDARD.encode(&bytes),
format: info.codec,
});
e.relative_path = None;
}
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read audio media {}: {}", id, err),
}
}
}
Ok(None) => eprintln!("⚠️ [LOAD_BEAM] Audio media {} missing from archive", id),
Err(err) => eprintln!("⚠️ [LOAD_BEAM] media_info({}) failed: {}", id, err),
}
}
// Restore this entry's persisted waveform pyramid, if present — avoids
// re-decoding the source media just to redraw the overview.
let wf_id = waveform_media_id(entry.pool_index);
if let Ok(Some(_)) = archive.media_info(wf_id) {
match archive.read_media_full(wf_id) {
Ok(bytes) => e.waveform_blob = Some(bytes),
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read waveform {}: {}", wf_id, err),
}
}
restored_entries.push(e);
}
// Raster keyframes are NOT eagerly decoded (Phase 3 paging): `raw_pixels` stays
// empty and is faulted in on demand from the container's `Raster` rows via the
// editor's `RasterStore` (keyed by `kf.id`). Loading a big paint project is now
// instant and only the resident window lives in RAM. Mark every keyframe
// `needs_fault_in` (recursively, incl. nested layers) so the renderer requests a
// page-in; a freshly-created keyframe stays `false` (blank-resident, nothing to load).
// Proxies (low-res, ~tens of KB each) ARE decoded eagerly so a cold scrub onto a
// not-yet-paged frame shows the proxy instantly instead of flashing blank.
let mut raster_load_count = 0usize;
let mut proxy_load_count = 0usize;
for layer in document.all_layers_mut() {
if let crate::layer::AnyLayer::Raster(rl) = layer {
for kf in &mut rl.keyframes {
kf.needs_fault_in = true;
raster_load_count += 1;
let proxy_id = raster_proxy_media_id(kf.id);
if let Ok(Some(_)) = archive.media_info(proxy_id) {
if let Ok(bytes) = archive.read_media_full(proxy_id) {
if let Ok(img) = crate::brush_engine::decode_png(&bytes) {
let (w, h) = (img.width(), img.height());
kf.proxy = Some(crate::raster_layer::RasterProxy {
width: w,
height: h,
pixels: img.into_raw(),
});
proxy_load_count += 1;
}
}
}
}
}
}
let _ = proxy_load_count;
// Image-asset bytes are NOT eagerly read (Phase 4 Tier 1 paging): `ImageAsset.data`
// stays empty and the renderer's ImageCache pages bytes from the container on a
// decode miss (keyed by asset id). Old base64 projects keep their deserialized
// `data` (no container row). Loading is instant; only rendered images touch disk.
// Missing external files (referenced entries whose file no longer exists).
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
let missing_files: Vec<MissingFileInfo> = restored_entries
.iter()
.enumerate()
.filter_map(|(idx, entry)| {
if entry.embedded_data.is_none() && entry.media_id.is_none() {
if let Some(rel) = entry.relative_path.as_ref() {
let full = if Path::new(rel).is_absolute() {
PathBuf::from(rel)
} else {
project_dir.join(rel)
};
if !full.exists() {
return Some(MissingFileInfo {
pool_index: idx,
original_path: full,
file_type: MediaFileType::Audio,
});
}
}
}
None
})
.collect();
// Persisted video thumbnail packs (opaque LBTN blobs), keyed by clip id. The
// editor decodes + inserts them and skips regeneration for these clips.
let mut thumbnail_blobs = std::collections::HashMap::new();
for clip_id in document.video_clips.keys() {
let tn_id = thumbnail_media_id(*clip_id);
if let Ok(Some(info)) = archive.media_info(tn_id) {
if info.kind == MediaKind::Thumbnail {
match archive.read_media_full(tn_id) {
Ok(bytes) => { thumbnail_blobs.insert(*clip_id, bytes); }
Err(e) => eprintln!("⚠️ [LOAD_BEAM] Failed to read thumbnails for {}: {}", clip_id, e),
}
}
}
}
eprintln!(
"📊 [LOAD_BEAM] ✅ Loaded {} audio entries, {} raster frames, {} thumbnail packs in {:.2}ms",
restored_entries.len(),
raster_load_count,
thumbnail_blobs.len(),
fn_start.elapsed().as_secs_f64() * 1000.0
);
Ok(LoadedProject {
document,
audio_project,
layer_to_track_map,
audio_pool_entries: restored_entries,
thumbnail_blobs,
missing_files,
})
}
/// Load a project from a legacy ZIP `.beam` archive (pre-SQLite format).
/// Retained for backward compatibility; saving converts to SQLite.
fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
let fn_start = std::time::Instant::now();
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (legacy ZIP)...");
// 1. Open ZIP archive
let step1_start = std::time::Instant::now();
@ -673,6 +938,7 @@ pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
audio_project,
layer_to_track_map,
audio_pool_entries: restored_entries,
thumbnail_blobs: std::collections::HashMap::new(), // legacy ZIP has no thumbnail packs
missing_files,
})
}

35
lightningbeam-ui/lightningbeam-core/src/gpu/color_convert.rs
View File

@ -6,6 +6,41 @@
use super::HDR_FORMAT;
/// Shared WGSL sRGB transfer functions — the single source of the sRGB OETF/EOTF
/// used by every gamma-aware shader. Prepend it to a shader's source (it defines
/// the functions before the body, so call order doesn't matter):
/// `srgb_to_linear_channel` / `linear_to_srgb_channel` (scalar) and
/// `srgb_to_linear` / `linear_to_srgb` (vec3). `linear_to_srgb_channel` clamps to
/// [0,1] (its outputs target 8-bit / SDR display surfaces).
pub const COLOR_WGSL: &str = r#"
fn srgb_to_linear_channel(c: f32) -> f32 {
return select(pow((c + 0.055) / 1.055, 2.4), c / 12.92, c <= 0.04045);
}
fn linear_to_srgb_channel(c: f32) -> f32 {
let x = clamp(c, 0.0, 1.0);
return select(1.055 * pow(x, 1.0 / 2.4) - 0.055, x * 12.92, x <= 0.0031308);
}
fn srgb_to_linear(c: vec3<f32>) -> vec3<f32> {
return vec3<f32>(srgb_to_linear_channel(c.r), srgb_to_linear_channel(c.g), srgb_to_linear_channel(c.b));
}
fn linear_to_srgb(c: vec3<f32>) -> vec3<f32> {
return vec3<f32>(linear_to_srgb_channel(c.r), linear_to_srgb_channel(c.g), linear_to_srgb_channel(c.b));
}
"#;
/// sRGB → linear for one channel in `[0, 1]` (CPU twin of the WGSL
/// `srgb_to_linear_channel`). The single source of the EOTF for CPU code.
pub fn srgb_to_linear(c: f32) -> f32 {
if c <= 0.04045 { c / 12.92 } else { ((c + 0.055) / 1.055).powf(2.4) }
}
/// linear → sRGB for one channel, clamped to `[0, 1]` (CPU twin of the WGSL
/// `linear_to_srgb_channel`). The single source of the OETF for CPU code.
pub fn linear_to_srgb(c: f32) -> f32 {
let c = c.clamp(0.0, 1.0);
if c <= 0.0031308 { c * 12.92 } else { 1.055 * c.powf(1.0 / 2.4) - 0.055 }
}
/// GPU pipeline for sRGB to linear color space conversion
///
/// Converts Rgba8Srgb textures to Rgba16Float linear textures.

23
lightningbeam-ui/lightningbeam-core/src/gpu/compositor.rs
View File

@ -105,6 +105,9 @@ pub struct CompositorLayer {
pub opacity: f32,
/// Blend mode for this layer
pub blend_mode: BlendMode,
/// Screen-blend RGB tint; `[0,0,0,0]` (the default) is a no-op. Used by
/// onion-skin ghosts (warm = past, cool = future).
pub tint: [f32; 4],
}
impl CompositorLayer {
@ -113,12 +116,19 @@ impl CompositorLayer {
buffer,
opacity: opacity.clamp(0.0, 1.0),
blend_mode,
tint: [0.0; 4],
}
}
pub fn normal(buffer: BufferHandle, opacity: f32) -> Self {
Self::new(buffer, opacity, BlendMode::Normal)
}
/// Screen-blend the layer toward an RGB tint (for onion-skin ghosts).
pub fn with_tint(mut self, r: f32, g: f32, b: f32) -> Self {
self.tint = [r, g, b, 0.0];
self
}
}
/// Uniform data for the composite shader
@ -129,8 +139,10 @@ pub struct CompositeUniforms {
pub opacity: f32,
/// Blend mode index
pub blend_mode: u32,
/// Padding for alignment
/// Padding to 16 bytes before the vec4 tint
pub _padding: [u32; 2],
/// Screen-blend tint ((0,0,0) = none); `.w` unused.
pub tint: [f32; 4],
}
/// Compositor for blending layers
@ -323,6 +335,7 @@ impl Compositor {
opacity: layer.opacity,
blend_mode: layer.blend_mode.to_index(),
_padding: [0, 0],
tint: layer.tint,
};
queue.write_buffer(&uniforms_buffer, 0, bytemuck::bytes_of(&uniforms));
@ -382,6 +395,8 @@ struct Uniforms {
opacity: f32,
blend_mode: u32,
_padding: vec2<u32>,
// Screen-blend tint ((0,0,0) = no tint). Used by onion-skin ghosts.
tint: vec4<f32>,
}
@group(0) @binding(0) var source_tex: texture_2d<f32>;
@ -526,7 +541,11 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
// Apply opacity
let src_alpha = src.a * uniforms.opacity;
// Screen-blend tint (recolors blacks toward the tint; no-op at tint=0).
let t = uniforms.tint.rgb;
let tinted = src.rgb + t - src.rgb * t;
// Output premultiplied alpha in linear color space
return vec4<f32>(src.rgb * src_alpha, src_alpha);
return vec4<f32>(tinted * src_alpha, src_alpha);
}
"#;

2
lightningbeam-ui/lightningbeam-core/src/gpu/mod.rs
View File

@ -14,7 +14,7 @@ pub mod yuv_converter;
// Re-export commonly used types
pub use buffer_pool::{BufferHandle, BufferPool, BufferSpec, BufferFormat};
pub use color_convert::SrgbToLinearConverter;
pub use color_convert::{SrgbToLinearConverter, COLOR_WGSL, srgb_to_linear, linear_to_srgb};
pub use compositor::{Compositor, CompositorLayer, BlendMode};
pub use effect_processor::{EffectProcessor, EffectUniforms};
pub use yuv_converter::YuvConverter;

170
lightningbeam-ui/lightningbeam-core/src/layer.rs
View File

@ -382,6 +382,29 @@ impl VectorLayer {
self.keyframe_at(time).map(|kf| &kf.graph)
}
/// The VectorGraph to *render* at `time`. When the keyframe at-or-before `time` has
/// `tween_after == Shape` and the next keyframe shares its topology, returns an owned
/// graph morphed between them; otherwise borrows the held keyframe's graph. Editing
/// should keep using `graph_at_time`/`graph_at_time_mut` (the held keyframe).
pub fn tweened_graph_at(&self, time: f64) -> Option<std::borrow::Cow<'_, VectorGraph>> {
use std::borrow::Cow;
let idx = self.keyframes.partition_point(|kf| kf.time <= time);
if idx == 0 {
return None;
}
let a = &self.keyframes[idx - 1];
if a.tween_after == TweenType::Shape && idx < self.keyframes.len() {
let b = &self.keyframes[idx];
if b.time > a.time {
let t = ((time - a.time) / (b.time - a.time)).clamp(0.0, 1.0);
if let Some(g) = a.graph.interpolated(&b.graph, t) {
return Some(Cow::Owned(g));
}
}
}
Some(Cow::Borrowed(&a.graph))
}
/// Get a mutable VectorGraph at a given time
pub fn graph_at_time_mut(&mut self, time: f64) -> Option<&mut VectorGraph> {
self.keyframe_at_mut(time).map(|kf| &mut kf.graph)
@ -433,6 +456,25 @@ impl VectorLayer {
time + frame_duration
}
/// Start time of the group clip instance's visibility region that contains `time`:
/// the time of the earliest keyframe reachable by walking back through consecutive
/// keyframes that all contain the clip instance. Returns `None` if the clip instance
/// isn't a keyframe-gated group visible at `time` (e.g. a movie clip).
pub fn group_visibility_start(&self, clip_instance_id: &Uuid, time: f64) -> Option<f64> {
let after = self.keyframes.partition_point(|kf| kf.time <= time);
if after == 0 {
return None;
}
let mut idx = after - 1; // keyframe at-or-before `time`
if !self.keyframes[idx].clip_instance_ids.contains(clip_instance_id) {
return None;
}
while idx > 0 && self.keyframes[idx - 1].clip_instance_ids.contains(clip_instance_id) {
idx -= 1;
}
Some(self.keyframes[idx].time)
}
// Shape-based methods removed — use DCEL methods instead.
// - shapes_at_time_mut → graph_at_time_mut
// - get_shape_in_keyframe → use DCEL vertex/edge/face accessors
@ -451,30 +493,53 @@ impl VectorLayer {
&mut self.keyframes[insert_idx]
}
/// Insert a new keyframe at time by cloning the DCEL from the active keyframe.
/// Insert a new keyframe at time, taking the geometry the layer shows there.
/// If a keyframe already exists at the exact time, does nothing and returns it.
///
/// Inside a shape-tween span this captures the *interpolated* geometry at `time` (not
/// the left keyframe's), and inherits the span's `tween_after` so the new keyframe keeps
/// tweening toward the next one — i.e. splitting a tween in two leaves the motion intact.
pub fn insert_keyframe_from_current(&mut self, time: f64) -> &mut ShapeKeyframe {
let tolerance = 0.001;
if let Some(idx) = self.keyframe_index_at_exact(time, tolerance) {
return &mut self.keyframes[idx];
}
// Clone graph and clip instance IDs from the active keyframe
let (cloned_graph, cloned_clip_ids) = self
// Geometry shown at `time` (interpolated if mid-tween, else the held keyframe).
let cloned_graph = self
.tweened_graph_at(time)
.map(|g| g.into_owned())
.unwrap_or_else(VectorGraph::new);
// Inherit tween + clip instances from the active (left) keyframe.
let (tween_after, cloned_clip_ids) = self
.keyframe_at(time)
.map(|kf| {
(kf.graph.clone(), kf.clip_instance_ids.clone())
})
.unwrap_or_else(|| (VectorGraph::new(), Vec::new()));
.map(|kf| (kf.tween_after, kf.clip_instance_ids.clone()))
.unwrap_or((TweenType::None, Vec::new()));
let insert_idx = self.keyframes.partition_point(|kf| kf.time < time);
let mut kf = ShapeKeyframe::new(time);
kf.graph = cloned_graph;
kf.tween_after = tween_after;
kf.clip_instance_ids = cloned_clip_ids;
self.keyframes.insert(insert_idx, kf);
&mut self.keyframes[insert_idx]
}
/// True when `time` falls strictly inside a shape-tween span — i.e. an in-between frame
/// (the left keyframe has `tween_after == Shape` and `time` is not on either keyframe).
/// Editing such a frame would silently modify the left keyframe, so the editor blocks it.
pub fn is_tween_inbetween(&self, time: f64) -> bool {
let tol = 0.001;
let idx = self.keyframes.partition_point(|kf| kf.time <= time);
if idx == 0 || idx >= self.keyframes.len() {
return false;
}
let (a, b) = (&self.keyframes[idx - 1], &self.keyframes[idx]);
a.tween_after == TweenType::Shape
&& (time - a.time).abs() > tol
&& (b.time - time).abs() > tol
}
/// Remove a keyframe at the exact time (within tolerance).
/// Returns the removed keyframe if found.
pub(crate) fn remove_keyframe_at(&mut self, time: f64, tolerance: f64) -> Option<ShapeKeyframe> {
@ -1050,6 +1115,97 @@ impl AnyLayer {
mod tests {
use super::*;
#[test]
fn tweened_graph_at_morphs_between_shape_keyframes() {
use crate::vector_graph::{Direction, FillRule, ShapeColor};
use kurbo::{CubicBez, Point};
// Build a single-vertex-ish graph at a given x via one degenerate fill is overkill;
// use one vertex + one edge (a loop) is also odd. Use two vertices + one edge and
// just check the vertex lerp through the layer's tween path.
let mk = |x: f64| {
let mut g = VectorGraph::new();
let v0 = g.alloc_vertex(Point::new(x, 0.0));
let v1 = g.alloc_vertex(Point::new(x + 10.0, 0.0));
let c = CubicBez::new(
Point::new(x, 0.0),
Point::new(x + 3.0, 0.0),
Point::new(x + 7.0, 0.0),
Point::new(x + 10.0, 0.0),
);
g.alloc_edge(c, v0, v1, None, Some(ShapeColor::rgb(0, 0, 0)));
g.alloc_fill(vec![(crate::vector_graph::EdgeId(0), Direction::Forward)],
ShapeColor::rgb(255, 0, 0), FillRule::NonZero);
g
};
let mut layer = VectorLayer::new("L");
layer.keyframes.clear();
let mut kf0 = ShapeKeyframe::new(0.0);
kf0.graph = mk(0.0);
kf0.tween_after = TweenType::Shape;
let mut kf10 = ShapeKeyframe::new(10.0);
kf10.graph = mk(100.0);
layer.keyframes.push(kf0);
layer.keyframes.push(kf10);
// Midway through the tween, vertex 0 is halfway (x=50).
let g = layer.tweened_graph_at(5.0).unwrap();
assert!((g.vertices[0].position.x - 50.0).abs() < 1e-6);
// Without the tween flag, it holds the left keyframe (x=0).
layer.keyframes[0].tween_after = TweenType::None;
let g = layer.tweened_graph_at(5.0).unwrap();
assert!((g.vertices[0].position.x - 0.0).abs() < 1e-6);
}
#[test]
fn inserting_keyframe_mid_tween_captures_interpolated_geometry_and_inherits_tween() {
use crate::vector_graph::{Direction, FillRule, ShapeColor};
use kurbo::{CubicBez, Point};
let mk = |x: f64| {
let mut g = VectorGraph::new();
let v0 = g.alloc_vertex(Point::new(x, 0.0));
let v1 = g.alloc_vertex(Point::new(x + 10.0, 0.0));
let c = CubicBez::new(
Point::new(x, 0.0),
Point::new(x + 3.0, 0.0),
Point::new(x + 7.0, 0.0),
Point::new(x + 10.0, 0.0),
);
g.alloc_edge(c, v0, v1, None, Some(ShapeColor::rgb(0, 0, 0)));
g.alloc_fill(vec![(crate::vector_graph::EdgeId(0), Direction::Forward)],
ShapeColor::rgb(255, 0, 0), FillRule::NonZero);
g
};
let mut layer = VectorLayer::new("L");
layer.keyframes.clear();
let mut kf_a = ShapeKeyframe::new(0.0);
kf_a.graph = mk(0.0);
kf_a.tween_after = TweenType::Shape;
let mut kf_b = ShapeKeyframe::new(10.0);
kf_b.graph = mk(100.0);
layer.keyframes.push(kf_a);
layer.keyframes.push(kf_b);
// Insert keyframe C at the midpoint of the A→B shape tween.
layer.insert_keyframe_from_current(5.0);
let c = layer.keyframe_at(5.0).expect("keyframe C exists");
// C took the interpolated geometry (x=50), not A's geometry (x=0).
assert!((c.graph.vertices[0].position.x - 50.0).abs() < 1e-6,
"C should hold interpolated geometry, got {}", c.graph.vertices[0].position.x);
// C inherits the shape tween so it still morphs toward B.
assert_eq!(c.tween_after, TweenType::Shape, "C should inherit the shape tween");
// The frame between C and B is still an in-between (tween continues past C).
assert!(layer.is_tween_inbetween(7.0));
// C itself is a keyframe, not an in-between.
assert!(!layer.is_tween_inbetween(5.0));
}
#[test]
fn test_layer_creation() {
let layer = Layer::new(LayerType::Vector, "Test Layer");

2
lightningbeam-ui/lightningbeam-core/src/lib.rs
View File

@ -42,6 +42,7 @@ pub mod segment_builder;
pub mod planar_graph;
pub mod file_types;
pub mod file_io;
pub mod beam_archive;
pub mod export;
pub mod clipboard;
pub(crate) mod clipboard_platform;
@ -53,6 +54,7 @@ pub mod svg_export;
pub mod snap;
pub mod webcam;
pub mod raster_layer;
pub mod raster_store;
pub mod brush_settings;
pub mod brush_engine;
pub mod raster_draw;

60
lightningbeam-ui/lightningbeam-core/src/raster_layer.rs
View File

@ -92,6 +92,16 @@ impl Default for TweenType {
}
}
/// A low-res decoded RGBA proxy of a keyframe's pixels, shown while the full-res
/// buffer pages in from the container so cold scrubs don't flash blank.
#[derive(Clone, Debug)]
pub struct RasterProxy {
pub width: u32,
pub height: u32,
/// RGBA, `width * height * 4` bytes.
pub pixels: Vec<u8>,
}
/// A single keyframe of a raster layer
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct RasterKeyframe {
@ -117,6 +127,24 @@ pub struct RasterKeyframe {
/// Always `true` after load; cleared by the renderer after uploading.
#[serde(skip, default = "default_true")]
pub texture_dirty: bool,
/// Phase 3 paging: the keyframe's pixels live in the container and must be
/// faulted in (`raw_pixels` empty *and* this true ⇒ page in from the store).
/// A *new* keyframe is `false` (intentionally blank/resident, nothing to load);
/// set true on load and again when evicted. Never serialized.
#[serde(skip)]
pub needs_fault_in: bool,
/// Phase 3a eviction: set `true` whenever user editing mutates `raw_pixels`
/// (brush, fill, paint-bucket, floating-selection commit/lift, undo/redo of
/// those). A dirty keyframe's current pixels are NOT yet persisted in the
/// container, so it must NEVER be evicted (doing so would silently lose the
/// unsaved edit). Cleared on a successful save. Never serialized.
#[serde(skip)]
pub dirty: bool,
/// Phase 3a-3: low-res proxy decoded from the container on load, rendered while
/// the full pixels page in (removes the cold-scrub blank flash). `None` if the
/// keyframe has no persisted proxy yet (new/unsaved, or pre-proxy project).
#[serde(skip)]
pub proxy: Option<RasterProxy>,
}
fn default_true() -> bool { true }
@ -140,6 +168,9 @@ impl RasterKeyframe {
tween_after: TweenType::Hold,
raw_pixels: Vec::new(),
texture_dirty: true,
needs_fault_in: false,
dirty: false,
proxy: None,
}
}
}
@ -204,6 +235,35 @@ impl RasterLayer {
&mut self.keyframes[insert_idx]
}
/// Insert a blank keyframe at `time` if none exists there (within tolerance).
/// Returns the new keyframe's id if one was created, `None` if a keyframe already
/// existed. Used by the explicit "New Keyframe" command (blank cel).
pub fn insert_blank_keyframe_at(&mut self, time: f64, width: u32, height: u32) -> Option<Uuid> {
if self.keyframe_index_at_exact(time, 0.001).is_some() {
return None;
}
let (w, h) = if width == 0 || height == 0 {
self.keyframe_at(time)
.map(|kf| (kf.width, kf.height))
.unwrap_or((1920, 1080))
} else {
(width, height)
};
let insert_idx = self.keyframes.partition_point(|kf| kf.time < time);
let kf = RasterKeyframe::new(time, w, h);
let id = kf.id;
self.keyframes.insert(insert_idx, kf);
Some(id)
}
/// Remove the keyframe with the given id, returning it if found.
pub fn remove_keyframe(&mut self, id: Uuid) -> Option<RasterKeyframe> {
self.keyframes
.iter()
.position(|kf| kf.id == id)
.map(|pos| self.keyframes.remove(pos))
}
/// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`.
pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> {
self.keyframe_at(time).map(|kf| kf.media_path.as_str())

56
lightningbeam-ui/lightningbeam-core/src/raster_store.rs Normal file
View File

@ -0,0 +1,56 @@
//! On-demand loader for raster keyframe pixels backed by the project `.beam`
//! container (Phase 3 paging).
//!
//! Raster keyframes are no longer eagerly decoded at load; `raw_pixels` stays
//! empty until something needs the frame, then it is faulted in from the
//! container's `Raster` media row (keyed by the keyframe id). The store holds only
//! the container path and reads through a fresh **read-only** connection per call,
//! so it never conflicts with an in-place save and keeps no long-lived handle.
//! `None` path = an unsaved document (nothing to fault in).
use std::path::PathBuf;
use uuid::Uuid;
/// Faults in raster keyframe pixels from the project container on demand.
#[derive(Default, Clone)]
pub struct RasterStore {
path: Option<PathBuf>,
}
impl RasterStore {
pub fn new(path: Option<PathBuf>) -> Self {
Self { path }
}
/// Point the store at a (possibly new) container path, or `None` for an
/// unsaved document. Call on load and on save-as.
pub fn set_path(&mut self, path: Option<PathBuf>) {
self.path = path;
}
pub fn has_path(&self) -> bool {
self.path.is_some()
}
/// Decode the keyframe's full RGBA pixels from the container, or `None` if the
/// container has no row for it (or decoding fails). The returned buffer is the
/// working `raw_pixels` representation (`width*height*4` sRGB-premultiplied RGBA).
pub fn load_pixels(&self, kf_id: Uuid) -> Option<Vec<u8>> {
let path = self.path.as_ref()?;
let png = match crate::beam_archive::read_packed_media_readonly(path, kf_id) {
Ok(Some(bytes)) => bytes,
Ok(None) => return None,
Err(e) => {
eprintln!("[RasterStore] read {} failed: {}", kf_id, e);
return None;
}
};
match crate::brush_engine::decode_png(&png) {
Ok(img) => Some(img.into_raw()),
Err(e) => {
eprintln!("[RasterStore] decode {} failed: {}", kf_id, e);
None
}
}
}
}

209
lightningbeam-ui/lightningbeam-core/src/renderer.rs
View File

@ -22,43 +22,111 @@ use vello::peniko::{Blob, Fill, ImageAlphaType, ImageBrush, ImageData, ImageForm
use vello::Scene;
/// Cache for decoded image data to avoid re-decoding every frame
/// Decoded-image cache, bounded by a byte budget with usage-LRU eviction (Phase 4
/// asset paging). The decoded RGBA (~`w·h·4` per image) is the heavy, evictable cost;
/// a miss re-decodes from `asset.data`. Recency is bumped on every access, so images
/// actually rendered each frame stay resident and unused ones age out under pressure.
pub struct ImageCache {
cache: HashMap<Uuid, Arc<ImageBrush>>,
/// CPU path: tiny-skia pixmaps decoded from the same assets (premultiplied RGBA8)
cpu_cache: HashMap<Uuid, Arc<tiny_skia::Pixmap>>,
/// Recency order (least-recent first) of resident asset ids.
lru: Vec<Uuid>,
/// Decoded bytes per resident asset (counted once; GPU/CPU are ~equal and a render
/// session uses one path) and the running total.
sizes: HashMap<Uuid, usize>,
bytes: usize,
/// `.beam` container path for lazily loading compressed `ImageAsset` bytes on a
/// decode miss (Tier 1 paging) when `asset.data` isn't resident.
container_path: Option<std::path::PathBuf>,
}
impl ImageCache {
/// Max decoded-image bytes kept resident before LRU eviction.
const BUDGET: usize = 256 * 1024 * 1024;
/// Create a new empty image cache
pub fn new() -> Self {
Self {
cache: HashMap::new(),
cpu_cache: HashMap::new(),
lru: Vec::new(),
sizes: HashMap::new(),
bytes: 0,
container_path: None,
}
}
/// Set the `.beam` container path used to lazily load image bytes that aren't
/// resident in `asset.data` (Tier 1 paging). Cheap to call each frame.
pub fn set_container_path(&mut self, path: Option<std::path::PathBuf>) {
self.container_path = path;
}
/// Resolve an asset's compressed bytes: prefer the resident `asset.data` (imported
/// this session, or an old base64 project), else page from the container.
fn resolve_bytes<'a>(&self, asset: &'a ImageAsset) -> Option<std::borrow::Cow<'a, [u8]>> {
if let Some(d) = &asset.data {
return Some(std::borrow::Cow::Borrowed(d.as_slice()));
}
let path = self.container_path.as_ref()?;
crate::beam_archive::read_packed_media_readonly(path, asset.id)
.ok()
.flatten()
.map(std::borrow::Cow::Owned)
}
/// Mark `id` (size `size` bytes) as most-recently-used; evict LRU entries over budget.
fn touch(&mut self, id: Uuid, size: usize) {
if !self.sizes.contains_key(&id) {
self.sizes.insert(id, size);
self.bytes += size;
}
if let Some(pos) = self.lru.iter().position(|x| *x == id) {
self.lru.remove(pos);
}
self.lru.push(id);
// Keep at least the just-touched entry resident.
while self.bytes > Self::BUDGET && self.lru.len() > 1 {
let old = self.lru.remove(0);
self.cache.remove(&old);
self.cpu_cache.remove(&old);
if let Some(sz) = self.sizes.remove(&old) {
self.bytes -= sz;
}
}
}
/// Get or decode an image, caching the result
pub fn get_or_decode(&mut self, asset: &ImageAsset) -> Option<Arc<ImageBrush>> {
if let Some(cached) = self.cache.get(&asset.id) {
return Some(Arc::clone(cached));
let size = (asset.width as usize) * (asset.height as usize) * 4;
if let Some(cached) = self.cache.get(&asset.id).map(Arc::clone) {
self.touch(asset.id, size);
return Some(cached);
}
// Decode and cache
let image = decode_image_asset(asset)?;
// Decode and cache (bytes from asset.data or paged from the container).
let bytes = self.resolve_bytes(asset)?;
let image = decode_image_brush(&bytes)?;
let arc_image = Arc::new(image);
self.cache.insert(asset.id, Arc::clone(&arc_image));
self.touch(asset.id, size);
Some(arc_image)
}
/// Get or decode an image as a premultiplied tiny-skia Pixmap (CPU render path).
pub fn get_or_decode_cpu(&mut self, asset: &ImageAsset) -> Option<Arc<tiny_skia::Pixmap>> {
if let Some(cached) = self.cpu_cache.get(&asset.id) {
return Some(Arc::clone(cached));
let size = (asset.width as usize) * (asset.height as usize) * 4;
if let Some(cached) = self.cpu_cache.get(&asset.id).map(Arc::clone) {
self.touch(asset.id, size);
return Some(cached);
}
let pixmap = decode_image_to_pixmap(asset)?;
let bytes = self.resolve_bytes(asset)?;
let pixmap = decode_image_to_pixmap(&bytes)?;
let arc = Arc::new(pixmap);
self.cpu_cache.insert(asset.id, Arc::clone(&arc));
self.touch(asset.id, size);
Some(arc)
}
@ -66,12 +134,21 @@ impl ImageCache {
pub fn invalidate(&mut self, id: &Uuid) {
self.cache.remove(id);
self.cpu_cache.remove(id);
if let Some(pos) = self.lru.iter().position(|x| x == id) {
self.lru.remove(pos);
}
if let Some(sz) = self.sizes.remove(id) {
self.bytes -= sz;
}
}
/// Clear all cached images
pub fn clear(&mut self) {
self.cache.clear();
self.cpu_cache.clear();
self.lru.clear();
self.sizes.clear();
self.bytes = 0;
}
}
@ -81,12 +158,34 @@ impl Default for ImageCache {
}
}
/// Decode an image asset to a premultiplied tiny-skia Pixmap (CPU render path).
fn decode_image_to_pixmap(asset: &ImageAsset) -> Option<tiny_skia::Pixmap> {
let data = asset.data.as_ref()?;
/// Image asset ids referenced by the visible vector layers' active keyframes at `time`
/// (top-level + group children). Used to prefetch/decode images ahead during playback.
/// (Recursing into nested clip instances is a refinement.)
pub fn assets_needed_at(document: &Document, time: f64) -> Vec<Uuid> {
let mut ids = Vec::new();
for layer in document.all_layers() {
if let crate::layer::AnyLayer::Vector(vl) = layer {
if !vl.layer.visible {
continue;
}
if let Some(kf) = vl.keyframe_at(time) {
for fill in &kf.graph.fills {
if let Some(id) = fill.image_fill {
ids.push(id);
}
}
}
}
}
ids
}
/// Decode image bytes to a premultiplied tiny-skia Pixmap (CPU render path).
fn decode_image_to_pixmap(data: &[u8]) -> Option<tiny_skia::Pixmap> {
let img = image::load_from_memory(data).ok()?;
let rgba = img.to_rgba8();
let mut pixmap = tiny_skia::Pixmap::new(asset.width, asset.height)?;
let (iw, ih) = rgba.dimensions();
let mut pixmap = tiny_skia::Pixmap::new(iw, ih)?;
for (dst, src) in pixmap.pixels_mut().iter_mut().zip(rgba.pixels()) {
let [r, g, b, a] = src.0;
// Convert straight alpha (image crate output) to premultiplied (tiny-skia internal format)
@ -100,21 +199,17 @@ fn decode_image_to_pixmap(asset: &ImageAsset) -> Option<tiny_skia::Pixmap> {
Some(pixmap)
}
/// Decode an image asset to peniko ImageBrush
fn decode_image_asset(asset: &ImageAsset) -> Option<ImageBrush> {
// Get the raw file data
let data = asset.data.as_ref()?;
// Decode using the image crate
/// Decode image bytes to a peniko ImageBrush (GPU render path).
fn decode_image_brush(data: &[u8]) -> Option<ImageBrush> {
let img = image::load_from_memory(data).ok()?;
let rgba = img.to_rgba8();
let (iw, ih) = rgba.dimensions();
// Create peniko ImageData then ImageBrush
let image_data = ImageData {
data: Blob::from(rgba.into_raw()),
format: ImageFormat::Rgba8,
width: asset.width,
height: asset.height,
width: iw,
height: ih,
alpha_type: ImageAlphaType::Alpha,
};
Some(ImageBrush::new(image_data))
@ -423,11 +518,25 @@ pub fn render_layer_isolated(
* Affine::scale_non_uniform(scale_x, scale_y)
* skew_transform;
// The decoded frame is scaled down to fit the document (decoder caps
// at the canvas size), so its pixel size is smaller than the clip's
// native dimensions. The instance is blitted treating the texture as
// `frame.width × frame.height`, while `clip_transform` is expressed in
// the clip's native space — so scale frame-px → clip-native-px first,
// else the frame renders small in a corner with its edges streaked.
let frame_to_clip = if frame.width > 0 && frame.height > 0 {
Affine::scale_non_uniform(
video_clip.width / frame.width as f64,
video_clip.height / frame.height as f64,
)
} else {
Affine::IDENTITY
};
instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(),
width: frame.width,
height: frame.height,
transform: base_transform * clip_transform,
transform: base_transform * clip_transform * frame_to_clip,
opacity: (layer_opacity * inst_opacity) as f32,
});
}
@ -1028,12 +1137,26 @@ fn render_video_layer(
// Create rectangle path for the video frame
let video_rect = Rect::new(0.0, 0.0, video_clip.width, video_clip.height);
// The decoded frame is scaled down to fit the document (the decoder caps at
// the canvas size to bound memory), so its pixel dimensions are smaller than
// the clip's native display size. Scale the image brush from frame-pixel
// space to the clip rect; without this the image is drawn 1:1 in a corner
// and its edge pixels pad the rest (small frame with "stretched corners").
let brush_transform = if frame.width > 0 && frame.height > 0 {
Affine::scale_non_uniform(
video_clip.width / frame.width as f64,
video_clip.height / frame.height as f64,
)
} else {
Affine::IDENTITY
};
// Render video frame as image fill
scene.fill(
Fill::NonZero,
instance_transform,
&image_with_alpha,
None,
Some(brush_transform),
&video_rect,
);
clip_rendered = true;
@ -1134,7 +1257,15 @@ pub fn render_vector_graph(
if let Some(image_asset) = document.get_image_asset(&image_asset_id) {
if let Some(image) = image_cache.get_or_decode(image_asset) {
let image_with_alpha = (*image).clone().with_alpha(opacity_f32);
scene.fill(fill_rule, base_transform, &image_with_alpha, None, &path);
// Map the image (native pixel space, origin 0,0) onto the fill's
// bounding box, so it sits where the shape is and scales to fit
// (1:1 for an image-sized rectangle).
let bbox = vello::kurbo::Shape::bounding_box(&path);
let iw = (image_asset.width.max(1)) as f64;
let ih = (image_asset.height.max(1)) as f64;
let brush_transform = Affine::translate((bbox.x0, bbox.y0))
* Affine::scale_non_uniform(bbox.width() / iw, bbox.height() / ih);
scene.fill(fill_rule, base_transform, &image_with_alpha, Some(brush_transform), &path);
filled = true;
}
}
@ -1225,7 +1356,12 @@ fn render_vector_layer(
// Cascade opacity: parent_opacity × layer.opacity
let layer_opacity = parent_opacity * layer.layer.opacity;
// Render clip instances first (they appear under shape instances)
// Render the layer's own VectorGraph (loose shapes) first, then clip instances
// (groups / movie clips) on top. Shape tweens are applied here.
if let Some(graph) = layer.tweened_graph_at(time) {
render_vector_graph(&graph, scene, base_transform, layer_opacity, document, image_cache);
}
for clip_instance in &layer.clip_instances {
// For groups, compute the visibility end from keyframe data
let group_end_time = document.vector_clips.get(&clip_instance.clip_id)
@ -1236,11 +1372,6 @@ fn render_vector_layer(
});
render_clip_instance(document, time, clip_instance, layer_opacity, scene, base_transform, &layer.layer.animation_data, image_cache, video_manager, group_end_time);
}
// Render VectorGraph from active keyframe
if let Some(graph) = layer.graph_at_time(time) {
render_vector_graph(graph, scene, base_transform, layer_opacity, document, image_cache);
}
}
// ============================================================================
@ -1458,12 +1589,21 @@ fn render_vector_graph_cpu(
if let Some(image_asset_id) = fill.image_fill {
if let Some(asset) = document.get_image_asset(&image_asset_id) {
if let Some(img_pixmap) = image_cache.get_or_decode_cpu(asset) {
// Map the image's native pixel space onto the fill's bounding box.
let bbox: kurbo::Rect = vello::kurbo::Shape::bounding_box(&path);
let iw = (asset.width.max(1)) as f32;
let ih = (asset.height.max(1)) as f32;
let sx = (bbox.width() as f32) / iw;
let sy = (bbox.height() as f32) / ih;
let pat_tf = tiny_skia::Transform::from_row(
sx, 0.0, 0.0, sy, bbox.x0 as f32, bbox.y0 as f32,
);
let pattern = tiny_skia::Pattern::new(
tiny_skia::Pixmap::as_ref(&img_pixmap),
tiny_skia::SpreadMode::Pad,
tiny_skia::FilterQuality::Bilinear,
opacity,
tiny_skia::Transform::identity(),
pat_tf,
);
let mut paint = tiny_skia::Paint::default();
paint.shader = pattern;
@ -1527,6 +1667,11 @@ fn render_vector_layer_cpu(
) {
let layer_opacity = parent_opacity * layer.layer.opacity;
// Loose shapes first, then clip instances (groups / movie clips) on top.
if let Some(graph) = layer.tweened_graph_at(time) {
render_vector_graph_cpu(&graph, pixmap, affine_to_ts(base_transform), layer_opacity as f32, document, image_cache);
}
for clip_instance in &layer.clip_instances {
let group_end_time = document.vector_clips.get(&clip_instance.clip_id)
.filter(|vc| vc.is_group)
@ -1539,10 +1684,6 @@ fn render_vector_layer_cpu(
&layer.layer.animation_data, image_cache, group_end_time,
);
}
if let Some(graph) = layer.graph_at_time(time) {
render_vector_graph_cpu(graph, pixmap, affine_to_ts(base_transform), layer_opacity as f32, document, image_cache);
}
}
/// Render a clip instance (and its nested layers) to a CPU pixmap.

39
lightningbeam-ui/lightningbeam-core/src/selection.rs
View File

@ -4,9 +4,8 @@
use crate::vector_graph::{VectorGraph, EdgeId, FillId, VertexId};
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::collections::HashSet;
use uuid::Uuid;
use vello::kurbo::{Affine, BezPath};
/// Shape of a raster pixel selection, in canvas pixel coordinates.
#[derive(Clone, Debug)]
@ -440,42 +439,6 @@ impl Selection {
}
}
/// Represents a temporary region-based selection.
///
/// When a region select is active, the region boundary is inserted into the
/// DCEL as invisible edges, splitting existing geometry. Faces inside the
/// region are added to the normal `Selection`. If the user performs an
/// operation, the selection is committed; if they deselect, the DCEL is
/// restored from the snapshot.
#[derive(Clone, Debug)]
pub struct RegionSelection {
/// The clipping region as a closed BezPath (polygon or rect)
pub region_path: BezPath,
/// Layer containing the affected elements
pub layer_id: Uuid,
/// Keyframe time
pub time: f64,
/// Snapshot of the graph before region boundary insertion, for revert
pub graph_snapshot: VectorGraph,
/// The extracted graph containing geometry inside the region
pub selected_graph: VectorGraph,
/// Transform applied to the selected graph (e.g. from dragging)
pub transform: Affine,
/// Whether the selection has been committed (via an operation on the selection)
pub committed: bool,
/// IDs of the invisible edges inserted for the region boundary stroke.
/// These exist in the main graph (remainder side). Deleted during merge-back.
pub region_edge_ids: Vec<EdgeId>,
/// Action epoch recorded when this selection was created.
/// Compared against `ActionExecutor::epoch()` on deselect to decide
/// whether merge-back is needed or a clean snapshot restore suffices.
pub action_epoch_at_selection: u64,
/// selected_graph VID → main graph VID for boundary vertices (shared between both graphs).
pub boundary_vertex_map: HashMap<VertexId, VertexId>,
/// selected_graph boundary EID → main graph boundary EID (duplicated edges to skip on merge).
pub boundary_edge_map: HashMap<EdgeId, EdgeId>,
}
#[cfg(test)]
mod tests {
use super::*;

15
lightningbeam-ui/lightningbeam-core/src/shaders/effect_brightness_contrast.wgsl
View File

@ -31,14 +31,23 @@ fn vs_main(@builtin(vertex_index) vertex_index: u32) -> VertexOutput {
return out;
}
// The HDR pipeline feeds these shaders LINEAR light, but brightness/contrast
// (additive brightness, contrast pivoting around 0.5 perceptual mid-gray) are
// defined in gamma/display space. Convert to sRGB, adjust there, then convert
// back to linear so the controls behave like standard editors.
// sRGB helpers (linear_to_srgb / srgb_to_linear) come from the prepended
// COLOR_WGSL prelude.
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let src = textureSample(source_tex, source_sampler, in.uv);
let brightness = uniforms.params0.x; // -1 to 1
let contrast = uniforms.params0.y; // 0 to 3
let src_srgb = linear_to_srgb(src.rgb);
// Apply brightness (additive)
var color = src.rgb + vec3<f32>(brightness);
var color = src_srgb + vec3<f32>(brightness);
// Apply contrast (multiply around midpoint 0.5)
color = (color - vec3<f32>(0.5)) * contrast + vec3<f32>(0.5);
@ -46,6 +55,6 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
// Clamp to valid range
color = clamp(color, vec3<f32>(0.0), vec3<f32>(1.0));
let result = mix(src.rgb, color, uniforms.mix);
return vec4<f32>(result, src.a);
let result_srgb = mix(src_srgb, color, uniforms.mix);
return vec4<f32>(srgb_to_linear(result_srgb), src.a);
}

14
lightningbeam-ui/lightningbeam-core/src/shaders/effect_hue_saturation.wgsl
View File

@ -84,6 +84,12 @@ fn hsl_to_rgb(hsl: vec3<f32>) -> vec3<f32> {
);
}
// The HDR pipeline feeds this shader LINEAR light, but the HSL model (and the
// lightness/saturation axes users expect) is defined on gamma-encoded sRGB.
// Convert to sRGB, run the HSL adjustment there, then convert back to linear.
// sRGB helpers (linear_to_srgb / srgb_to_linear) come from the prepended
// COLOR_WGSL prelude.
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let src = textureSample(source_tex, source_sampler, in.uv);
@ -91,8 +97,10 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let saturation = uniforms.params0.y; // Multiplier (1.0 = no change)
let lightness = uniforms.params0.z; // Additive (-1 to 1)
let src_srgb = linear_to_srgb(src.rgb);
// Convert to HSL
var hsl = rgb_to_hsl(src.rgb);
var hsl = rgb_to_hsl(src_srgb);
// Apply adjustments
hsl.x = fract(hsl.x + hue_shift); // Shift hue (wrapping)
@ -102,6 +110,6 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
// Convert back to RGB
let adjusted = hsl_to_rgb(hsl);
let result = mix(src.rgb, adjusted, uniforms.mix);
return vec4<f32>(result, src.a);
let result_srgb = mix(src_srgb, adjusted, uniforms.mix);
return vec4<f32>(srgb_to_linear(result_srgb), src.a);
}

12
lightningbeam-ui/lightningbeam-core/src/shaders/effect_invert.wgsl
View File

@ -33,13 +33,19 @@ fn vs_main(@builtin(vertex_index) vertex_index: u32) -> VertexOutput {
return out;
}
// The HDR pipeline feeds these shaders LINEAR light, but "invert" is a
// perceptual operation defined in gamma/display space (Photoshop, GIMP, etc.).
// Convert to sRGB, invert there, then convert back to linear. The sRGB helpers
// (linear_to_srgb / srgb_to_linear) come from the prepended COLOR_WGSL prelude.
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let src = textureSample(source_tex, source_sampler, in.uv);
let amount = uniforms.params0.x; // params[0]
let inverted = vec3<f32>(1.0) - src.rgb;
let result = mix(src.rgb, inverted, amount * uniforms.mix);
let src_srgb = linear_to_srgb(src.rgb);
let inverted = vec3<f32>(1.0) - src_srgb;
let result_srgb = mix(src_srgb, inverted, amount * uniforms.mix);
return vec4<f32>(result, src.a);
return vec4<f32>(srgb_to_linear(result_srgb), src.a);
}

2
lightningbeam-ui/lightningbeam-core/src/shape.rs
View File

@ -395,7 +395,7 @@ mod tests {
let shape = Shape::new(path);
assert_eq!(shape.versions.len(), 1);
assert!(shape.fill_color.is_some());
assert!(shape.fill_color.is_none());
}
#[test]

821
lightningbeam-ui/lightningbeam-core/src/vector_graph/mod.rs
View File

@ -417,6 +417,152 @@ impl VectorGraph {
self.boundary_to_bezpath(&fill.boundary)
}
/// Interpolate toward `other` by `t` ∈ [0,1] for a same-topology shape tween.
///
/// Returns `None` if the two graphs don't share identical topology — same vertex,
/// edge and fill structure (counts, deleted flags, edge endpoints, fill boundaries).
/// In that case the caller should hold the source keyframe instead of morphing.
/// Vertex positions, edge curves, stroke widths and stroke/fill colours are lerped.
pub fn interpolated(&self, other: &VectorGraph, t: f64) -> Option<VectorGraph> {
if self.vertices.len() != other.vertices.len()
|| self.edges.len() != other.edges.len()
|| self.fills.len() != other.fills.len()
{
return None;
}
for (a, b) in self.vertices.iter().zip(&other.vertices) {
if a.deleted != b.deleted {
return None;
}
}
for (a, b) in self.edges.iter().zip(&other.edges) {
if a.deleted != b.deleted || a.vertices != b.vertices {
return None;
}
}
for (a, b) in self.fills.iter().zip(&other.fills) {
if a.deleted != b.deleted || a.boundary != b.boundary {
return None;
}
}
let lf = |x: f64, y: f64| x + (y - x) * t;
let lp = |p: Point, q: Point| Point::new(lf(p.x, q.x), lf(p.y, q.y));
let lc = |a: Option<ShapeColor>, b: Option<ShapeColor>| match (a, b) {
(Some(a), Some(b)) => {
let c = |x: u8, y: u8| (lf(x as f64, y as f64)).round().clamp(0.0, 255.0) as u8;
Some(ShapeColor::new(c(a.r, b.r), c(a.g, b.g), c(a.b, b.b), c(a.a, b.a)))
}
(a, _) => a,
};
let mut g = self.clone();
for (i, v) in g.vertices.iter_mut().enumerate() {
v.position = lp(self.vertices[i].position, other.vertices[i].position);
}
for (i, e) in g.edges.iter_mut().enumerate() {
let (a, b) = (self.edges[i].curve, other.edges[i].curve);
e.curve = CubicBez::new(lp(a.p0, b.p0), lp(a.p1, b.p1), lp(a.p2, b.p2), lp(a.p3, b.p3));
if let (Some(s), Some(sa), Some(sb)) = (
e.stroke_style.as_mut(),
self.edges[i].stroke_style.as_ref(),
other.edges[i].stroke_style.as_ref(),
) {
s.width = lf(sa.width, sb.width);
}
e.stroke_color = lc(self.edges[i].stroke_color, other.edges[i].stroke_color);
}
for (i, f) in g.fills.iter_mut().enumerate() {
f.color = lc(self.fills[i].color, other.fills[i].color);
}
Some(g)
}
/// A point guaranteed to lie inside the fill — for point-in-region classification
/// (e.g. deciding whether a fill is inside a lasso). Prefers the polygon area-centroid,
/// but for a non-convex fill (e.g. an L-shape, where the area-centroid can fall in the
/// concavity *outside* the shape) it steps just inward from a boundary edge instead.
/// The naive average of boundary-edge midpoints is NOT reliable here — it can land
/// outside a non-convex fill and misclassify it.
pub fn fill_interior_point(&self, fill_id: FillId) -> Point {
let boundary = self.fills[fill_id.idx()].boundary.clone();
self.boundary_interior_point(&boundary)
}
/// A point guaranteed to lie inside the region enclosed by a `(edge, direction)`
/// boundary loop. See [`fill_interior_point`].
pub fn boundary_interior_point(&self, boundary: &[(EdgeId, Direction)]) -> Point {
use kurbo::{ParamCurve, Shape, Vec2};
let path = self.boundary_to_bezpath(boundary);
// Ordered polygon corners: the directed start point of each boundary edge.
let mut pts: Vec<Point> = Vec::new();
for &(eid, dir) in boundary {
if eid.is_none() {
continue;
}
let c = self.edges[eid.idx()].curve;
pts.push(match dir {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
});
}
if pts.len() < 3 {
if pts.is_empty() {
return Point::ZERO;
}
let (sx, sy) = pts.iter().fold((0.0, 0.0), |(x, y), p| (x + p.x, y + p.y));
return Point::new(sx / pts.len() as f64, sy / pts.len() as f64);
}
// Shoelace area-centroid.
let (mut a2, mut cx, mut cy) = (0.0, 0.0, 0.0);
for i in 0..pts.len() {
let p0 = pts[i];
let p1 = pts[(i + 1) % pts.len()];
let cross = p0.x * p1.y - p1.x * p0.y;
a2 += cross;
cx += (p0.x + p1.x) * cross;
cy += (p0.y + p1.y) * cross;
}
if a2.abs() > 1e-9 {
let c = Point::new(cx / (3.0 * a2), cy / (3.0 * a2));
if path.winding(c) != 0 {
return c;
}
}
// Fallback: step a small distance inward from a boundary edge midpoint.
let (mut minx, mut miny, mut maxx, mut maxy) = (f64::MAX, f64::MAX, f64::MIN, f64::MIN);
for p in &pts {
minx = minx.min(p.x);
miny = miny.min(p.y);
maxx = maxx.max(p.x);
maxy = maxy.max(p.y);
}
let eps = ((maxx - minx).min(maxy - miny) * 1e-3).max(1e-4);
for &(eid, _) in boundary {
if eid.is_none() {
continue;
}
let c = self.edges[eid.idx()].curve;
let mid = c.eval(0.5);
let tangent = c.eval(0.5001) - c.eval(0.4999);
let len = tangent.hypot();
if len < 1e-12 {
continue;
}
let n = Vec2::new(-tangent.y / len, tangent.x / len);
for s in [1.0_f64, -1.0] {
let cand = mid + n * (s * eps);
if path.winding(cand) != 0 {
return cand;
}
}
}
pts[0]
}
// -------------------------------------------------------------------
// Vertex editing
// -------------------------------------------------------------------
@ -653,6 +799,10 @@ impl VectorGraph {
}
let mut all_new_edges = Vec::new();
// Sub-edges produced when a stroke segment splits an existing edge (incl. an earlier
// segment of this same stroke). Tracked separately so they reach the fill re-tracer
// without polluting the returned edge list.
let mut split_products: Vec<EdgeId> = Vec::new();
let mut prev_end_vertex: Option<VertexId> = None;
for (seg_idx, seg) in expanded_segments.iter().enumerate() {
@ -706,7 +856,14 @@ impl VectorGraph {
let remapped_t = original_edge_t / head_end;
let remapped_t = remapped_t.clamp(ENDPOINT_T_MARGIN, 1.0 - ENDPOINT_T_MARGIN);
let (mid_v, _sub_a, _sub_b) = self.split_edge(eid, remapped_t);
let (mid_v, sub_a, sub_b) = self.split_edge(eid, remapped_t);
// Track the split products so the fill re-tracer sees them: when a later
// stroke segment crosses an earlier one, these sub-edges are part of the
// stroke's arrangement but would otherwise be invisible to the re-trace.
// (Kept out of `all_new_edges` so the returned edge list stays the stroke's
// own edges only.)
split_products.push(sub_a);
split_products.push(sub_b);
// Snap vertex to intersection point
self.vertices[mid_v.idx()].position = point;
// Merge with nearby existing vertex if within snap distance
@ -791,127 +948,511 @@ impl VectorGraph {
prev_end_vertex = Some(end_v);
}
// Fill splitting pass: for each new edge, check if both endpoints
// lie on any fill's boundary — if so, split that fill.
let edges_to_check = all_new_edges.clone();
for &eid in &edges_to_check {
let v0 = self.edges[eid.idx()].vertices[0];
let v1 = self.edges[eid.idx()].vertices[1];
// Weld dangling stroke endpoints onto a near-coincident existing vertex. A
// self-intersecting freehand stroke can create an intersection vertex a fraction of
// a pixel away from a segment endpoint that should be the same point; if they don't
// merge, the stroke's loop is broken by a degree-1 stub and the cut is lost.
self.weld_dangling_endpoints(&all_new_edges, snap_epsilon.max(1.0));
// Find fills where both v0 and v1 appear as boundary vertices
let fill_ids: Vec<FillId> = self.fills
.iter()
.enumerate()
.filter(|(_, f)| !f.deleted)
.filter(|(_, f)| {
let has_v0 = f.boundary.iter().any(|&(be, _)| {
let e = &self.edges[be.idx()];
e.vertices[0] == v0 || e.vertices[1] == v0
});
let has_v1 = f.boundary.iter().any(|&(be, _)| {
let e = &self.edges[be.idx()];
e.vertices[0] == v1 || e.vertices[1] == v1
});
has_v0 && has_v1
})
.map(|(i, _)| FillId(i as u32))
.collect();
// Coincident-edge cleanup: a new edge that lands exactly on an existing edge
// between the same two vertices (e.g. drawing a shape whose edge snaps onto an
// existing one) must not be duplicated — duplicates produce zero-area "sliver"
// fills. Merge such duplicates before splitting/filling.
self.dedupe_coincident_new_edges(&all_new_edges);
for fid in fill_ids {
self.split_fill_by_edge(fid, eid);
}
}
// Re-derive the fills touched by the new edges. Rather than incrementally splitting
// along single cut edges (which can't handle a lasso whose path is interrupted by a
// hole/notch in a non-convex fill), we re-trace the planar faces of the affected
// sub-arrangement and rebuild the fills from them. This is robust for arbitrary
// holed/concave fills (e.g. cutting across geometry left behind by a prior group).
// Include split products so a self-crossing stroke's full arrangement is seen.
let mut retrace_edges = all_new_edges.clone();
retrace_edges.extend(split_products);
self.retrace_fills_after_cut(&retrace_edges);
// Drop any zero-area fills (e.g. slivers left between coincident edges).
self.remove_degenerate_fills();
all_new_edges
}
/// When a new edge splits a fill (both endpoints on the fill's boundary),
/// split the fill into two fills.
pub fn split_fill_by_edge(
&mut self,
fill_id: FillId,
splitting_edge: EdgeId,
) -> Option<(FillId, FillId)> {
let fill = &self.fills[fill_id.idx()];
if fill.deleted {
return None;
/// Merge edges from the latest stroke that are geometrically coincident with an
/// existing edge between the same two vertices (drawing a shape whose edge lands exactly
/// on an existing edge). Keeps the existing edge, redirects fill references, and frees
/// the duplicate — preventing zero-area "sliver" fills between the two copies.
/// Weld degree-1 endpoints of freshly inserted edges onto a near-coincident existing
/// vertex. A self-intersecting freehand stroke can create an intersection vertex a
/// fraction of a pixel from a segment endpoint that should be the same point; if they
/// don't merge, the stroke's loop is broken by a degree-1 stub and the cut is lost.
fn weld_dangling_endpoints(&mut self, new_edges: &[EdgeId], eps: f64) {
// Repeatedly weld the closest dangling new endpoint to its nearest neighbour.
loop {
let mut to_merge: Option<(VertexId, VertexId)> = None; // (keep, merge)
'scan: for &e in new_edges {
if self.edges[e.idx()].deleted {
continue;
}
for &v in &self.edges[e.idx()].vertices {
if self.vertices[v.idx()].deleted || self.edges_at_vertex(v).len() != 1 {
continue;
}
let pv = self.vertices[v.idx()].position;
let mut best: Option<(f64, VertexId)> = None;
for ui in 0..self.vertices.len() {
if ui == v.idx() || self.vertices[ui].deleted {
continue;
}
let pu = self.vertices[ui].position;
let d = (pu.x - pv.x).hypot(pu.y - pv.y);
if d < eps && best.map_or(true, |(bd, _)| d < bd) {
best = Some((d, VertexId(ui as u32)));
}
}
if let Some((_, keep)) = best {
to_merge = Some((keep, v));
break 'scan;
}
}
}
match to_merge {
Some((keep, merge)) => self.merge_vertices(keep, merge),
None => break,
}
}
// Drop any edges that collapsed to zero length (both endpoints welded together).
let degenerate: Vec<EdgeId> = self
.edges
.iter()
.enumerate()
.filter(|(_, e)| !e.deleted && e.vertices[0] == e.vertices[1])
.map(|(i, _)| EdgeId(i as u32))
.collect();
for e in degenerate {
for fill in &mut self.fills {
if !fill.deleted {
fill.boundary.retain(|&(fe, _)| fe != e);
}
}
self.free_edge(e);
}
}
let split_v0 = self.edges[splitting_edge.idx()].vertices[0];
let split_v1 = self.edges[splitting_edge.idx()].vertices[1];
fn dedupe_coincident_new_edges(&mut self, new_edges: &[EdgeId]) {
for &ne in new_edges {
if self.edges[ne.idx()].deleted {
continue;
}
let va = self.edges[ne.idx()].vertices[0];
let vb = self.edges[ne.idx()].vertices[1];
let candidates: Vec<EdgeId> = self
.edges_at_vertex(va)
.into_iter()
.filter(|&e| e != ne && !self.edges[e.idx()].deleted)
.filter(|&e| {
let v = self.edges[e.idx()].vertices;
(v[0] == va && v[1] == vb) || (v[0] == vb && v[1] == va)
})
.collect();
for c in candidates {
if self.edges[c.idx()].deleted {
continue;
}
if self.curves_coincident(ne, c) {
self.redirect_edge_in_fills(ne, c);
self.free_edge(ne);
break;
}
}
}
}
// Find the positions in the boundary where the splitting edge's
// endpoint vertices appear as the "arrival" vertex of a directed edge.
let boundary = fill.boundary.clone();
/// Whether two edges (already known to share both endpoints) trace the same path.
/// Coincident duplicates in practice are straight collinear segments (a shape edge
/// snapping onto an existing edge), so we treat "both are straight lines between the
/// same endpoints" as coincident. Comparing curve `eval(t)` directly is unreliable —
/// split sub-edges are non-uniformly parameterised, so equal `t` ≠ equal point.
fn curves_coincident(&self, a: EdgeId, b: EdgeId) -> bool {
let is_straight = |c: kurbo::CubicBez| {
let chord = c.p3 - c.p0;
let len = chord.hypot();
if len < 1e-9 {
return true; // zero-length chord — degenerate, treat as coincident
}
// Perpendicular distance of each control point from the p0→p3 chord.
let dist = |p: Point| ((p - c.p0).cross(chord)).abs() / len;
dist(c.p1) < 1e-2 && dist(c.p2) < 1e-2
};
is_straight(self.edges[a.idx()].curve) && is_straight(self.edges[b.idx()].curve)
}
// Helper: get the "end" vertex of a directed boundary edge
let end_vertex = |eid: EdgeId, dir: Direction| -> VertexId {
/// Replace every `from` boundary reference with `to`, preserving traversal direction.
fn redirect_edge_in_fills(&mut self, from: EdgeId, to: EdgeId) {
let f = self.edges[from.idx()].vertices;
let t = self.edges[to.idx()].vertices;
let same_dir = t[0] == f[0] && t[1] == f[1];
for fill in &mut self.fills {
if fill.deleted {
continue;
}
for entry in &mut fill.boundary {
if entry.0 == from {
entry.1 = match (entry.1, same_dir) {
(Direction::Forward, true) | (Direction::Backward, false) => {
Direction::Forward
}
(Direction::Backward, true) | (Direction::Forward, false) => {
Direction::Backward
}
};
entry.0 = to;
}
}
}
}
/// Drop fills that enclose ~zero area (degenerate slivers from coincident edges).
fn remove_degenerate_fills(&mut self) {
for i in 0..self.fills.len() {
if self.fills[i].deleted {
continue;
}
let mut pts: Vec<Point> = Vec::new();
for &(eid, dir) in &self.fills[i].boundary {
if eid.is_none() {
continue;
}
let c = self.edges[eid.idx()].curve;
pts.push(match dir {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
});
}
let area = if pts.len() < 3 {
0.0
} else {
let mut a2 = 0.0;
for k in 0..pts.len() {
let p0 = pts[k];
let p1 = pts[(k + 1) % pts.len()];
a2 += p0.x * p1.y - p1.x * p0.y;
}
(a2 * 0.5).abs()
};
if area < 1e-6 {
self.fills[i].deleted = true;
self.free_fills.push(i as u32);
}
}
}
/// Directed end vertex of a `(edge, direction)` boundary entry.
#[inline]
fn entry_end_vertex(&self, eid: EdgeId, dir: Direction) -> VertexId {
match dir {
Direction::Forward => self.edges[eid.idx()].vertices[1],
Direction::Backward => self.edges[eid.idx()].vertices[0],
}
}
/// Re-derive the fills touched by a freshly inserted stroke by re-tracing the planar
/// faces of the affected sub-arrangement. This replaces incremental "split a fill by a
/// cut edge" logic, which can't handle a cut whose path is interrupted by a hole/notch
/// in a non-convex fill. Each affected fill is deleted and rebuilt from the traced
/// faces that lie inside it (inheriting its colour/rule); faces outside it — or in a
/// hole — are dropped.
fn retrace_fills_after_cut(&mut self, new_edges: &[EdgeId]) {
use kurbo::{ParamCurve, Shape};
let new_set: HashSet<EdgeId> = new_edges
.iter()
.filter(|&&e| !e.is_none() && !self.edges[e.idx()].deleted)
.copied()
.collect();
if new_set.is_empty() {
return;
}
let new_verts: HashSet<VertexId> =
new_set.iter().flat_map(|&e| self.edges[e.idx()].vertices).collect();
// Affected fills: any non-deleted fill that shares a vertex with a new edge.
let affected: Vec<FillId> = (0..self.fills.len())
.filter(|&i| !self.fills[i].deleted)
.filter(|&i| {
self.fills[i].boundary.iter().any(|&(e, _)| {
!e.is_none()
&& self.edges[e.idx()].vertices.iter().any(|v| new_verts.contains(v))
})
})
.map(|i| FillId(i as u32))
.collect();
if affected.is_empty() {
return;
}
// Snapshot each affected fill's path + attributes before we delete them.
let originals: Vec<(kurbo::BezPath, Option<ShapeColor>, FillRule)> = affected
.iter()
.map(|&f| {
(
self.fill_to_bezpath(f),
self.fills[f.idx()].color,
self.fills[f.idx()].fill_rule,
)
})
.collect();
// Edge set for the local arrangement: every affected fill's boundary edges plus the
// ENTIRE inserted stroke. We include the whole stroke (not just the segments whose
// midpoint is inside a fill) because a wiggly freehand lasso has segments that dip
// just outside the fill; excluding them would break the inside-arc chain into
// dangling fragments that then get pruned away, losing the cut entirely. The stroke
// forms closed loops, so it contributes no dangling edges; faces that end up outside
// every affected fill are discarded by the classification below.
let mut edge_set: HashSet<EdgeId> = HashSet::new();
for &f in &affected {
for &(e, _) in &self.fills[f.idx()].boundary {
if !e.is_none() && !self.edges[e.idx()].deleted {
edge_set.insert(e);
}
}
}
edge_set.extend(new_set.iter().copied());
// Expand to the induced subgraph on the covered vertices. A self-intersecting
// freehand stroke splits its own edges via `split_edge`, whose sub-edges aren't in
// `new_edges`; without them the local arrangement has gaps and the stroke's loop
// looks like dangling fragments. Adding every edge whose endpoints are both already
// covered closes those gaps (the sub-edges connect already-covered stroke vertices).
loop {
let verts: HashSet<VertexId> = edge_set
.iter()
.flat_map(|&e| self.edges[e.idx()].vertices)
.collect();
let added: Vec<EdgeId> = (0..self.edges.len())
.map(|i| EdgeId(i as u32))
.filter(|&e| !self.edges[e.idx()].deleted && !edge_set.contains(&e))
.filter(|&e| {
let [a, b] = self.edges[e.idx()].vertices;
verts.contains(&a) && verts.contains(&b)
})
.collect();
if added.is_empty() {
break;
}
edge_set.extend(added);
}
// Prune dangling edges (a vertex with degree < 2 in the local arrangement). They
// form spikes, never real face boundaries — a freehand lasso that wiggles or nearly
// self-touches leaves such stubs, and tracing them produces a face that runs out and
// back along the same edge (a degenerate self-touching boundary). Iterate, since
// removing one stub can expose another.
loop {
let mut degree: HashMap<VertexId, usize> = HashMap::new();
for &e in &edge_set {
for &v in &self.edges[e.idx()].vertices {
*degree.entry(v).or_default() += 1;
}
}
let dangling: Vec<EdgeId> = edge_set
.iter()
.copied()
.filter(|&e| {
let [a, b] = self.edges[e.idx()].vertices;
a == b || degree[&a] < 2 || degree[&b] < 2
})
.collect();
if dangling.is_empty() {
break;
}
for e in dangling {
edge_set.remove(&e);
}
}
let faces = self.trace_faces(&edge_set);
// Replace the affected fills with the re-traced bounded faces that fall inside them.
for &f in &affected {
self.fills[f.idx()].deleted = true;
self.free_fills.push(f.0);
}
for mut face in faces {
// Collapse degenerate "spikes" — a sequence that runs out to a point and back
// (e.g. across near-coincident duplicate tiny edges from a dense freehand path).
self.collapse_boundary_spikes(&mut face);
if face.len() < 3 {
continue;
}
// Only bounded (counter-clockwise, positive-area) faces are real regions; the
// outer face is clockwise/negative.
if self.face_signed_area(&face) <= 1e-6 {
continue;
}
let sample = self.boundary_interior_point(&face);
if let Some((_, color, rule)) =
originals.iter().find(|(p, _, _)| p.winding(sample) != 0)
{
self.alloc_fill(face, *color, *rule);
}
}
}
/// Remove out-and-back "spikes" from a face boundary: consecutive entries where the
/// second exactly reverses the first (the boundary returns to where it started, e.g.
/// bouncing across near-coincident duplicate edges). These are zero-area and would make
/// `boundary_to_bezpath` render a stray hair; collapsing them yields a simple loop.
fn collapse_boundary_spikes(&self, face: &mut Vec<(EdgeId, Direction)>) {
let dstart = |entry: &(EdgeId, Direction)| -> Point {
let c = self.edges[entry.0.idx()].curve;
match entry.1 {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
}
};
// Find positions where boundary edges arrive at split_v0 and split_v1
let mut pos_v0: Option<usize> = None;
let mut pos_v1: Option<usize> = None;
for (i, &(eid, dir)) in boundary.iter().enumerate() {
let ev = end_vertex(eid, dir);
if ev == split_v0 && pos_v0.is_none() {
pos_v0 = Some(i);
let dend = |entry: &(EdgeId, Direction)| -> Point {
let c = self.edges[entry.0.idx()].curve;
match entry.1 {
Direction::Forward => c.p3,
Direction::Backward => c.p0,
}
if ev == split_v1 && pos_v1.is_none() {
pos_v1 = Some(i);
}
}
let pos_v0 = pos_v0?;
let pos_v1 = pos_v1?;
// Ensure we have two distinct positions
if pos_v0 == pos_v1 {
return None;
}
// Walk boundary in two halves:
// Half A: from pos_v0+1 to pos_v1 (inclusive), then splitting_edge Forward
// Half B: from pos_v1+1 to pos_v0 (wrapping), then splitting_edge Backward
let n = boundary.len();
let color = fill.color;
let fill_rule = fill.fill_rule;
let mut half_a = Vec::new();
let mut idx = (pos_v0 + 1) % n;
};
const EPS: f64 = 0.5;
loop {
half_a.push(boundary[idx]);
if idx == pos_v1 {
let n = face.len();
if n < 2 {
break;
}
idx = (idx + 1) % n;
}
half_a.push((splitting_edge, Direction::Forward));
let mut half_b = Vec::new();
idx = (pos_v1 + 1) % n;
loop {
half_b.push(boundary[idx]);
if idx == pos_v0 {
let mut collapsed = false;
for i in 0..n {
let j = (i + 1) % n;
// entries i and j cancel when j ends back at i's start.
let si = dstart(&face[i]);
let ej = dend(&face[j]);
if (si.x - ej.x).hypot(si.y - ej.y) < EPS {
let (hi, lo) = if i > j { (i, j) } else { (j, i) };
face.remove(hi);
face.remove(lo);
collapsed = true;
break;
}
idx = (idx + 1) % n;
}
half_b.push((splitting_edge, Direction::Backward));
if !collapsed {
break;
}
}
}
// Delete the original fill
self.fills[fill_id.idx()].deleted = true;
self.free_fills.push(fill_id.0);
/// Trace all faces of the planar arrangement formed by `edge_set`, using the standard
/// angular next-edge rule (turn to the clockwise-adjacent dart of the twin at each
/// vertex). Returns each face as an ordered `(edge, direction)` loop. Bounded faces
/// come out counter-clockwise (positive signed area); the outer face clockwise.
fn trace_faces(&self, edge_set: &HashSet<EdgeId>) -> Vec<Vec<(EdgeId, Direction)>> {
// Outgoing darts per vertex, sorted by outgoing angle (CCW).
let mut out: HashMap<VertexId, Vec<(f64, (EdgeId, Direction))>> = HashMap::new();
for &e in edge_set {
if self.edges[e.idx()].deleted {
continue;
}
let [a, b] = self.edges[e.idx()].vertices;
out.entry(a)
.or_default()
.push((self.dart_angle(e, Direction::Forward), (e, Direction::Forward)));
out.entry(b)
.or_default()
.push((self.dart_angle(e, Direction::Backward), (e, Direction::Backward)));
}
for darts in out.values_mut() {
darts.sort_by(|x, y| x.0.partial_cmp(&y.0).unwrap_or(std::cmp::Ordering::Equal));
}
// Create two new fills
let fill_a = self.alloc_fill(half_a, color, fill_rule);
let fill_b = self.alloc_fill(half_b, color, fill_rule);
let mut visited: HashSet<(EdgeId, Direction)> = HashSet::new();
let mut faces: Vec<Vec<(EdgeId, Direction)>> = Vec::new();
let cap = edge_set.len() * 2 + 4;
for &e in edge_set {
if self.edges[e.idx()].deleted {
continue;
}
for dir in [Direction::Forward, Direction::Backward] {
let start = (e, dir);
if visited.contains(&start) {
continue;
}
let mut face: Vec<(EdgeId, Direction)> = Vec::new();
let mut d = start;
loop {
visited.insert(d);
face.push(d);
// Next dart: at the end vertex, the dart clockwise-adjacent to the twin.
let end_v = self.entry_end_vertex(d.0, d.1);
let twin = (
d.0,
match d.1 {
Direction::Forward => Direction::Backward,
Direction::Backward => Direction::Forward,
},
);
let darts = match out.get(&end_v) {
Some(d) => d,
None => break,
};
let Some(idx) = darts.iter().position(|&(_, dd)| dd == twin) else {
break;
};
let next = darts[(idx + darts.len() - 1) % darts.len()].1;
if next == start {
break;
}
if visited.contains(&next) || face.len() > cap {
break;
}
d = next;
}
if face.len() >= 3 {
faces.push(face);
}
}
}
faces
}
Some((fill_a, fill_b))
/// Outgoing direction angle of a dart at its start vertex.
fn dart_angle(&self, e: EdgeId, dir: Direction) -> f64 {
let c = self.edges[e.idx()].curve;
let (base, cands) = match dir {
Direction::Forward => (c.p0, [c.p1, c.p2, c.p3]),
Direction::Backward => (c.p3, [c.p2, c.p1, c.p0]),
};
for cand in cands {
let d = cand - base;
if d.hypot() > 1e-9 {
return d.y.atan2(d.x);
}
}
0.0
}
/// Signed area of a face given as an ordered `(edge, direction)` loop (CCW positive).
fn face_signed_area(&self, face: &[(EdgeId, Direction)]) -> f64 {
let pts: Vec<Point> = face
.iter()
.map(|&(e, dir)| {
let c = self.edges[e.idx()].curve;
match dir {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
}
})
.collect();
if pts.len() < 3 {
return 0.0;
}
let mut a2 = 0.0;
for i in 0..pts.len() {
let p0 = pts[i];
let p1 = pts[(i + 1) % pts.len()];
a2 += p0.x * p1.y - p1.x * p0.y;
}
a2 * 0.5
}
/// Merge two fills that share a boundary edge (e.g., after edge deletion).
@ -1458,11 +1999,14 @@ impl VectorGraph {
// ── Region selection: extract / merge subgraph ──────────────────────
/// Extract a subgraph containing `inside_edges` and `inside_fills`.
/// Extract a subgraph containing `inside_edges` and `inside_fills` (typically a
/// geometry selection — `select_fill` already includes each fill's boundary edges).
///
/// Boundary edges (`boundary_edge_ids`) are **duplicated** — they exist in
/// both the returned graph and `self`, so both sides have closed fill
/// boundaries when the selection is moved.
/// **Boundary edges** are *duplicated* (copied into the returned graph but kept in
/// `self`, so remaining shapes keep closed boundaries). They are `explicit_boundary`
/// (a cut the caller knows about, e.g. a lasso region — pass an empty set if none)
/// UNION any inside edge still shared with a non-extracted fill (derived here, so a
/// plain geometry selection needs no boundary analysis from the caller).
///
/// Returns `(new_graph, vertex_map, edge_map)` where the maps go from
/// old (self) IDs to new (returned graph) IDs.
@ -1470,17 +2014,58 @@ impl VectorGraph {
&mut self,
inside_edges: &HashSet<EdgeId>,
inside_fills: &HashSet<FillId>,
boundary_edge_ids: &HashSet<EdgeId>,
explicit_boundary: &HashSet<EdgeId>,
) -> (VectorGraph, HashMap<VertexId, VertexId>, HashMap<EdgeId, EdgeId>) {
let mut new_graph = VectorGraph::new();
let mut vtx_map: HashMap<VertexId, VertexId> = HashMap::new();
let mut edge_map: HashMap<EdgeId, EdgeId> = HashMap::new();
// Collect all edge IDs we need to copy into the new graph
let edges_to_copy: HashSet<EdgeId> = inside_edges
.union(boundary_edge_ids)
.copied()
.collect();
// Augment the inside set with every boundary edge of an extracted fill. A
// selection might not enumerate them all (e.g. lasso/region selection populates
// edges differently than `select_fill`); without this an extracted fill would be
// copied with `EdgeId::NONE` standing in for a missing edge, and that NONE later
// panics any code that indexes `fill.boundary` (e.g. `insert_stroke`).
let inside_edges: HashSet<EdgeId> = {
let mut s = inside_edges.clone();
for &fid in inside_fills {
if fid.is_none() {
continue;
}
if let Some(fill) = self.fills.get(fid.idx()) {
if fill.deleted {
continue;
}
for &(eid, _) in &fill.boundary {
if !eid.is_none() {
s.insert(eid);
}
}
}
}
s
};
let inside_edges = &inside_edges;
// Boundary = `explicit_boundary` (e.g. a region/lasso cut the caller knows about)
// UNION any inside edge still referenced by a fill we're NOT extracting (a shared
// DCEL edge — must be duplicated, not moved, or that fill dangles). Deriving the
// latter here means a plain geometry selection needs no boundary analysis.
let mut boundary_edge_ids: HashSet<EdgeId> = explicit_boundary.clone();
for (i, fill) in self.fills.iter().enumerate() {
if fill.deleted || inside_fills.contains(&FillId(i as u32)) {
continue;
}
for &(eid, _) in &fill.boundary {
if !eid.is_none() && inside_edges.contains(&eid) {
boundary_edge_ids.insert(eid);
}
}
}
let boundary_edge_ids = &boundary_edge_ids;
// Copy all inside edges + any boundary edges (the explicit ones may not be in
// inside_edges); boundary edges are kept in self below.
let edges_to_copy: HashSet<EdgeId> = inside_edges.union(boundary_edge_ids).copied().collect();
// Collect all vertices referenced by edges we're copying
let mut referenced_vids: HashSet<VertexId> = HashSet::new();
@ -1493,16 +2078,21 @@ impl VectorGraph {
}
}
// Determine which vertices are interior (exclusively owned by the
// extracted subgraph) vs boundary (shared with remaining geometry).
// A vertex is interior if ALL of its incident edges are either in
// inside_edges or boundary_edge_ids.
// Determine which vertices are safe to free from `self`. A vertex can only be
// freed if EVERY one of its incident edges is actually being removed from `self`,
// i.e. is an inside edge that is NOT a boundary edge. Boundary edges are kept
// (duplicated) in `self`, so a vertex touching one is still referenced and must
// remain — otherwise it becomes a freed-but-referenced vertex whose slot a later
// `alloc_vertex` reuses, corrupting the remaining fill.
let mut interior_vertices: HashSet<VertexId> = HashSet::new();
let mut boundary_vertices: HashSet<VertexId> = HashSet::new();
for &vid in &referenced_vids {
let incident = self.edges_at_vertex(vid);
let all_inside = incident.iter().all(|&eid| edges_to_copy.contains(&eid));
if all_inside {
let all_removed = !incident.is_empty()
&& incident
.iter()
.all(|&eid| inside_edges.contains(&eid) && !boundary_edge_ids.contains(&eid));
if all_removed {
interior_vertices.insert(vid);
} else {
boundary_vertices.insert(vid);
@ -1566,9 +2156,10 @@ impl VectorGraph {
new_graph.fills[new_fid.idx()].image_fill = fill.image_fill;
}
// Remove inside_edges from self (but NOT boundary edges — those are duplicated)
// Remove inside_edges from self, EXCEPT boundary edges (those are duplicated —
// a non-extracted fill still needs them).
for &eid in inside_edges {
if !eid.is_none() && !self.edges[eid.idx()].deleted {
if !eid.is_none() && !boundary_edge_ids.contains(&eid) && !self.edges[eid.idx()].deleted {
self.free_edge(eid);
}
}

4
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/mod.rs
View File

@ -10,3 +10,7 @@ mod editing;
mod gap_close;
#[cfg(test)]
mod region;
#[cfg(test)]
mod region_cut_select;
#[cfg(test)]
mod tween;

660
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_cut_select.rs Normal file
View File

@ -0,0 +1,660 @@
//! Reproduces the unified region-select behaviour at the graph level: cutting a shape
//! along a lasso outline (`insert_stroke`) and classifying which resulting fills/edges
//! fall inside the lasso — the same logic the editor's `execute_region_select` runs.
//!
//! The shape is built exactly as the editor builds a rectangle: `insert_stroke` of a
//! closed rect loop, then `paint_bucket` to create the fill (NOT a hand-rolled
//! `alloc_fill`), because the bug is in how that traced fill's boundary survives being
//! split by the lasso cut.
use super::super::*;
use kurbo::{BezPath, CubicBez, ParamCurve, Point, Shape};
/// Straight-line cubic from a to b.
fn line(a: Point, b: Point) -> CubicBez {
CubicBez::new(
a,
Point::new(a.x + (b.x - a.x) / 3.0, a.y + (b.y - a.y) / 3.0),
Point::new(a.x + 2.0 * (b.x - a.x) / 3.0, a.y + 2.0 * (b.y - a.y) / 3.0),
b,
)
}
fn closed_rect_path(min_x: f64, min_y: f64, max_x: f64, max_y: f64) -> BezPath {
let mut p = BezPath::new();
p.move_to((min_x, min_y));
p.line_to((max_x, min_y));
p.line_to((max_x, max_y));
p.line_to((min_x, max_y));
p.close_path();
p
}
/// Draw a filled rectangle into an existing graph the way the editor's rectangle tool does.
fn draw_filled_rect(g: &mut VectorGraph, min_x: f64, min_y: f64, max_x: f64, max_y: f64) {
let style = StrokeStyle { width: 1.0, ..Default::default() };
let color = ShapeColor::rgb(0, 0, 0);
for segs in bezpath_to_cubic_segments(&closed_rect_path(min_x, min_y, max_x, max_y)) {
g.insert_stroke(&segs, Some(style.clone()), Some(color), 0.5);
}
let centroid = Point::new((min_x + max_x) / 2.0, (min_y + max_y) / 2.0);
g.paint_bucket(centroid, ShapeColor::rgb(255, 0, 0), FillRule::NonZero, 0.0)
.expect("paint_bucket should create a fill");
}
/// Build a filled rectangle the way the editor's rectangle tool does.
fn editor_filled_rect(min_x: f64, min_y: f64, max_x: f64, max_y: f64) -> VectorGraph {
let mut g = VectorGraph::new();
draw_filled_rect(&mut g, min_x, min_y, max_x, max_y);
g
}
/// Closed rectangular lasso: cubic segments (for insert_stroke) + BezPath (for winding).
fn rect_lasso(min_x: f64, min_y: f64, max_x: f64, max_y: f64) -> (Vec<CubicBez>, BezPath) {
let p = [
Point::new(min_x, min_y),
Point::new(max_x, min_y),
Point::new(max_x, max_y),
Point::new(min_x, max_y),
];
let segs = vec![line(p[0], p[1]), line(p[1], p[2]), line(p[2], p[3]), line(p[3], p[0])];
let mut path = BezPath::new();
path.move_to(p[0]);
for pt in &p[1..] {
path.line_to(*pt);
}
path.close_path();
(segs, path)
}
// Classify-against-lasso uses the same guaranteed-interior probe point the editor uses.
fn fill_centroid(g: &VectorGraph, fid: FillId) -> Point {
g.fill_interior_point(fid)
}
/// Directed start/end points of a boundary entry.
fn dir_start(g: &VectorGraph, eid: EdgeId, dir: Direction) -> Point {
let c = g.edge(eid).curve;
match dir {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
}
}
fn dir_end(g: &VectorGraph, eid: EdgeId, dir: Direction) -> Point {
let c = g.edge(eid).curve;
match dir {
Direction::Forward => c.p3,
Direction::Backward => c.p0,
}
}
/// Describe a fill's boundary as an ordered list of (start → end) segments for diagnostics.
fn describe_boundary(g: &VectorGraph, fid: FillId) -> Vec<(EdgeId, Point, Point)> {
g.fill(fid)
.boundary
.iter()
.filter(|(e, _)| !e.is_none())
.map(|&(e, d)| (e, dir_start(g, e, d), dir_end(g, e, d)))
.collect()
}
/// Assert no fill boundary uses the same edge twice (the signature of a degenerate
/// "spike" where the trace runs out and back along a dangling edge).
fn assert_no_spikes(g: &VectorGraph) {
for (fi, f) in g.fills.iter().enumerate() {
if f.deleted { continue; }
let mut seen = std::collections::HashSet::new();
for &(e, _) in &f.boundary {
if e.is_none() { continue; }
assert!(seen.insert(e.0), "fill {fi} uses edge {} twice (spike)", e.0);
}
}
}
/// Assert every non-deleted fill's boundary is a connected closed loop.
fn assert_all_fills_connected(g: &VectorGraph) {
for (i, f) in g.fills.iter().enumerate() {
if f.deleted {
continue;
}
let fid = FillId(i as u32);
let b = describe_boundary(g, fid);
if b.is_empty() {
continue;
}
let n = b.len();
for k in 0..n {
let (_, _, end) = b[k];
let (_, next_start, _) = b[(k + 1) % n];
// Tolerance well below any real gap (pixels) but above accumulated float drift.
assert!(
(end.x - next_start.x).abs() < 1e-2 && (end.y - next_start.y).abs() < 1e-2,
"fill {i} boundary disconnected between edge {k} (ends {end:?}) and \
edge {} (starts {next_start:?}); boundary = {b:#?}",
(k + 1) % n
);
}
}
}
#[test]
fn single_vertical_cut_produces_two_connected_fills() {
// Rectangle (0,0)-(200,100), one vertical cut at x=100.
let mut g = editor_filled_rect(0.0, 0.0, 200.0, 100.0);
let cut = vec![line(Point::new(100.0, -10.0), Point::new(100.0, 110.0))];
g.insert_stroke(&cut, None, None, 1.0);
let live = g.fills.iter().filter(|f| !f.deleted).count();
assert_eq!(live, 2, "one cut should split the rect into 2 fills, got {live}");
assert_all_fills_connected(&g);
}
#[test]
fn lasso_over_second_of_two_rects_splits_that_rect() {
// Two separate rectangles; lasso a vertical strip over the SECOND one.
let mut g = editor_filled_rect(0.0, 0.0, 200.0, 100.0);
draw_filled_rect(&mut g, 0.0, 150.0, 200.0, 250.0);
assert_eq!(g.fills.iter().filter(|f| !f.deleted).count(), 2, "two rects → two fills");
let (segs, lasso_path) = rect_lasso(50.0, 120.0, 150.0, 300.0); // crosses rect B only
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
// Rect B should now be split into 3 fills (left/middle/right strips); rect A is untouched.
// So 1 (rect A) + 3 (rect B pieces) = 4 fills total.
let live = g.fills.iter().filter(|f| !f.deleted).count();
let inside_fills: Vec<FillId> = g
.fills
.iter()
.enumerate()
.filter(|(_, f)| !f.deleted)
.map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso_path.winding(fill_centroid(&g, fid)) != 0)
.collect();
let dump = || {
g.fills
.iter()
.enumerate()
.filter(|(_, f)| !f.deleted)
.map(|(i, _)| (i, fill_centroid(&g, FillId(i as u32))))
.collect::<Vec<_>>()
};
assert_eq!(live, 4, "rect B should split into 3 (total 4 fills); got {live}: {:#?}", dump());
assert_eq!(
inside_fills.len(),
1,
"exactly the center strip of rect B should be inside; got {:#?}",
dump()
);
assert_all_fills_connected(&g);
}
#[test]
fn lasso_strip_through_side_by_side_second_rect() {
// Two rectangles side by side; lasso a vertical strip through the SECOND (right) one.
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
draw_filled_rect(&mut g, 150.0, 0.0, 250.0, 100.0);
let (segs, lasso_path) = rect_lasso(180.0, -50.0, 220.0, 150.0);
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
assert_all_fills_connected(&g);
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso_path.winding(fill_centroid(&g, fid)) != 0).collect();
let dump: Vec<_> = g.fills.iter().enumerate().filter(|(_, f)| !f.deleted)
.map(|(i, _)| (i, fill_centroid(&g, FillId(i as u32)), describe_boundary(&g, FillId(i as u32)).len())).collect();
assert_eq!(inside.len(), 1, "one strip of the right rect should be inside; got {dump:#?}");
}
/// No live fill may reference a freed edge or vertex (whose slot a later alloc reuses).
fn assert_no_freed_but_referenced(g: &VectorGraph) {
let freed_v: std::collections::HashSet<u32> = g.free_vertices.iter().copied().collect();
let freed_e: std::collections::HashSet<u32> = g.free_edges.iter().copied().collect();
for (fi, f) in g.fills.iter().enumerate() {
if f.deleted { continue; }
for &(e, _) in &f.boundary {
if e.is_none() { continue; }
assert!(!freed_e.contains(&e.0), "live fill {fi} references freed edge {}", e.0);
for &v in &g.edge(e).vertices {
assert!(!freed_v.contains(&v.0), "live fill {fi} references freed vertex {}", v.0);
}
}
}
}
#[test]
fn extract_after_cut_keeps_remaining_fill_intact() {
// Cut a rectangle's corner with a lasso, then extract (Group) the clipped corner. The
// extraction must not free vertices/edges still referenced by the L-shaped remainder —
// previously it freed the shared cut vertices, and a later `alloc_vertex` reused those
// slots and corrupted the remainder (the "second lasso deletes all faces" bug).
let mut g = editor_filled_rect(0.0, 0.0, 200.0, 100.0);
let (segs, lasso) = rect_lasso(100.0, -50.0, 300.0, 50.0); // clips the top-right corner
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
// Pick the fill inside the lasso (the clipped corner) and extract it.
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso.winding(g.fill_interior_point(fid)) != 0).collect();
assert_eq!(inside.len(), 1, "one corner fill should be inside the lasso");
let inside_fills: HashSet<FillId> = inside.iter().copied().collect();
let inside_edges: HashSet<EdgeId> = g.fill(inside[0]).boundary.iter()
.filter_map(|&(e, _)| (!e.is_none()).then_some(e)).collect();
let _ = g.extract_subgraph(&inside_edges, &inside_fills, &HashSet::new());
assert_no_freed_but_referenced(&g);
assert_all_fills_connected(&g);
// Simulate the next lasso allocating vertices, then re-validate: a corrupted (freed but
// referenced) vertex would now be at a bogus position.
let (segs2, _) = rect_lasso(20.0, 20.0, 60.0, 80.0);
g.insert_stroke(&segs2, None, None, 1.0);
g.gc_invisible_edges();
assert_all_fills_connected(&g);
}
/// Captured region-select cases (`LIGHTNINGBEAM_DUMP_REGION=1`), embedded so the regression
/// survives `/tmp` being cleared. Each is `{ "graph": <VectorGraph>, "segments": [[[x,y]*4]*] }`.
/// They span: two separate rects, side-by-side, overlapping, a notched post-group fill, and
/// dense self-intersecting freehand lassos (dump 3 is the boundary-spike repro).
const REGION_DUMPS: &[&str] = &[
include_str!("region_dumps/dump0.json"),
include_str!("region_dumps/dump1.json"),
include_str!("region_dumps/dump2.json"),
include_str!("region_dumps/dump3.json"),
include_str!("region_dumps/dump4.json"),
];
#[test]
fn dumped_region_selects_are_valid() {
// Replays each captured region-select cut and asserts it yields only valid, non-corrupt
// geometry (no freed-but-referenced refs, no spikes, every fill a connected loop).
for json in REGION_DUMPS {
let v: serde_json::Value = serde_json::from_str(json).unwrap();
let mut g: VectorGraph = serde_json::from_value(v["graph"].clone()).unwrap();
let segs: Vec<CubicBez> = v["segments"].as_array().unwrap().iter().map(|s| {
let p = |i: usize| { let a = s[i].as_array().unwrap(); Point::new(a[0].as_f64().unwrap(), a[1].as_f64().unwrap()) };
CubicBez::new(p(0), p(1), p(2), p(3))
}).collect();
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
assert_no_freed_but_referenced(&g);
assert_no_spikes(&g);
assert_all_fills_connected(&g);
}
}
#[test]
fn near_coincident_needle_does_not_spike() {
// Smoke test: a stroke poking into a fill and returning along a near-coincident path
// must still yield valid, non-corrupt geometry. (The dense-freehand accordion that the
// boundary-spike collapse specifically fixes is only reliably reproduced by the captured
// region dumps; this is a lightweight portable guard for the same family of inputs.)
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
g.insert_stroke(
&[
line(Point::new(50.0, -10.0), Point::new(50.0, 50.0)),
line(Point::new(50.0, 50.0), Point::new(50.0003, -10.0)),
],
None, None, 1.0,
);
g.gc_invisible_edges();
assert_no_spikes(&g);
assert_all_fills_connected(&g);
}
#[test]
fn second_stroke_crossing_first_splits_into_quadrants() {
// A later stroke that crosses an earlier stroke's edge inside a fill triggers an
// edge-domain `split_edge`, whose sub-edges aren't tracked in the second stroke's new
// edges. The retrace must still see them (induced-subgraph expansion) or the cut breaks
// and unravels. Two crossing cuts through a rectangle must yield four connected quadrants.
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
g.insert_stroke(&[line(Point::new(-10.0, 50.0), Point::new(110.0, 50.0))], None, None, 1.0); // horizontal
assert_eq!(g.fills.iter().filter(|f| !f.deleted).count(), 2, "horizontal cut → 2");
g.insert_stroke(&[line(Point::new(50.0, -10.0), Point::new(50.0, 110.0))], None, None, 1.0); // vertical, crosses it
g.gc_invisible_edges();
let live: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32)).collect();
assert_eq!(live.len(), 4, "two crossing cuts → four quadrants; got {}", live.len());
assert_all_fills_connected(&g);
assert_no_spikes(&g);
for &fid in &live {
assert!((polygon_area(&g, fid) - 2500.0).abs() < 1.0, "each quadrant is 50x50, got {}", polygon_area(&g, fid));
}
}
#[test]
fn stroke_dead_ending_inside_fill_does_not_spike() {
// A stroke (e.g. a freehand lasso that wiggles or nearly self-touches) can leave a
// dangling stub: an edge whose inner endpoint has degree 1. Re-tracing must not run out
// and back along it (a self-touching "spike" boundary); the fill stays a clean loop.
let mut g = editor_filled_rect(0.0, 0.0, 200.0, 100.0);
// Open stroke entering the top edge and dead-ending inside at (100,50).
let stub = vec![line(Point::new(50.0, -20.0), Point::new(100.0, 50.0))];
g.insert_stroke(&stub, None, None, 1.0);
g.gc_invisible_edges();
let live: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32)).collect();
assert_eq!(live.len(), 1, "a dead-end stub must not split the fill; got {}", live.len());
assert_all_fills_connected(&g);
assert_no_spikes(&g);
// The fill is still the full rectangle.
assert!((polygon_area(&g, live[0]) - 20000.0).abs() < 1.0,
"fill should remain the 200x100 rectangle, area {}", polygon_area(&g, live[0]));
}
#[test]
fn groupfail_two_notched_fills_second_lasso() {
// Faithful reconstruction of the captured post-group state (dump_1): two adjacent
// fills sharing the x=337 column, each with an invisible notch (the hole left by the
// first lasso+group). A second lasso (398,249)-(495,327) clips fill 3's corner.
use std::collections::HashMap;
let mut g = VectorGraph::new();
let vp = [
(130.0, 232.0), (337.0, 232.0), (337.0, 362.0), (130.0, 362.0), (337.0, 297.0),
(535.0, 297.0), (535.0, 417.0), (337.0, 417.0), (0.0, 0.0), (337.0, 315.0),
(220.0, 315.0), (456.0, 315.0), (456.0, 345.0), (337.0, 345.0), (220.0, 345.0),
];
let vs: Vec<_> = vp.iter().map(|&(x, y)| g.alloc_vertex(Point::new(x, y))).collect();
let style = StrokeStyle { width: 3.0, ..Default::default() };
let color = ShapeColor::rgb(0, 0, 0);
// (name, v_start, v_end, visible)
let edge_defs: &[(&str, usize, usize, bool)] = &[
("e0", 0, 1, true), ("e1", 4, 5, true), ("e2", 2, 3, true), ("e3", 3, 0, true),
("e4", 1, 4, true), ("e5", 7, 2, true), ("e6", 5, 6, true), ("e7", 6, 7, true),
("e8", 10, 9, false), ("e9", 12, 13, false), ("e11", 4, 9, true), ("e12", 9, 11, false),
("e13", 11, 12, false), ("e15", 13, 2, true), ("e16", 13, 14, false), ("e17", 14, 10, false),
];
let mut em: HashMap<&str, EdgeId> = HashMap::new();
for &(name, a, b, vis) in edge_defs {
let (ss, sc) = if vis { (Some(style.clone()), Some(color)) } else { (None, None) };
let e = g.alloc_edge(line(vp[a].into(), vp[b].into()), vs[a], vs[b], ss, sc);
em.insert(name, e);
}
let mk = |spec: &[(&str, Direction)]| -> Vec<(EdgeId, Direction)> {
spec.iter().map(|&(n, d)| (em[n], d)).collect()
};
use Direction::{Backward as B, Forward as F};
g.alloc_fill(mk(&[("e11", B), ("e4", B), ("e0", B), ("e3", B), ("e2", B), ("e15", B), ("e16", F), ("e17", F), ("e8", F)]),
ShapeColor::rgb(100, 100, 255), FillRule::NonZero);
g.alloc_fill(mk(&[("e15", F), ("e5", B), ("e7", B), ("e6", B), ("e1", B), ("e11", F), ("e12", F), ("e13", F), ("e9", F)]),
ShapeColor::rgb(100, 100, 255), FillRule::NonZero);
assert_eq!(g.fills.iter().filter(|f| !f.deleted).count(), 2, "starts with 2 fills");
let (segs, lasso) = rect_lasso(398.0, 249.0, 495.0, 327.0);
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
let live = g.fills.iter().filter(|f| !f.deleted).count();
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso.winding(g.fill_interior_point(fid)) != 0).collect();
eprintln!("groupfail: {live} live fills, {} inside lasso", inside.len());
for (i, f) in g.fills.iter().enumerate() {
if f.deleted { continue; }
let fid = FillId(i as u32);
eprintln!(" fill {i}: bbox {:?} area {:.0}", fill_bbox(&g, fid), polygon_area(&g, fid));
}
assert_all_fills_connected(&g);
// The two original shapes must survive (no faces wrongly deleted); the lasso adds a cut.
assert!(live >= 2, "must keep both shapes plus the cut; got {live}");
assert_eq!(inside.len(), 1, "exactly the clipped corner of fill 3 should be selected");
}
#[test]
fn lasso_across_notched_fill_never_corrupts_boundaries() {
// A rectangle with a rectangular notch bitten out of its top edge — the shape a fill
// is left as after grouping a lasso selection (the notch is the extracted region's
// hole). A second lasso crossing the notch must never produce disconnected/corrupt
// fill boundaries (it previously incorporated the lasso's out-of-shape edges, drawing
// stray diagonals). It may under-select on such complex geometry, but must stay valid.
let mut g = VectorGraph::new();
let pts = [
Point::new(0.0, 0.0), Point::new(80.0, 0.0), Point::new(80.0, 40.0),
Point::new(120.0, 40.0), Point::new(120.0, 0.0), Point::new(200.0, 0.0),
Point::new(200.0, 100.0), Point::new(0.0, 100.0),
];
let style = StrokeStyle { width: 1.0, ..Default::default() };
let color = ShapeColor::rgb(0, 0, 0);
let v: Vec<_> = pts.iter().map(|&p| g.alloc_vertex(p)).collect();
let mut boundary = Vec::new();
for i in 0..pts.len() {
let e = g.alloc_edge(line(pts[i], pts[(i + 1) % pts.len()]), v[i], v[(i + 1) % pts.len()],
Some(style.clone()), Some(color));
boundary.push((e, Direction::Forward));
}
g.alloc_fill(boundary, ShapeColor::rgb(255, 0, 0), FillRule::NonZero);
// Lasso a rectangle that straddles the notch and the shape body. Inside the fill it
// covers (60..140, 0..60) minus the notch (80..120, 0..40) — an arch shape — which the
// cut must isolate as one connected fill, with the remainder outside.
let (segs, lasso) = rect_lasso(60.0, -20.0, 140.0, 60.0);
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
// Every resulting fill is a valid connected loop (never corrupt).
assert_all_fills_connected(&g);
// Exactly one fill lies inside the lasso, and it is the arch (area = 80*60 - 40*40).
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso.winding(g.fill_interior_point(fid)) != 0).collect();
assert_eq!(inside.len(), 1, "the arch (lasso ∩ notched fill) should be one selected fill");
let area = polygon_area(&g, inside[0]);
assert!((area - 3200.0).abs() < 1.0, "arch area should be 3200, got {area}");
}
/// Absolute polygon area of a fill from its boundary corner points.
fn polygon_area(g: &VectorGraph, fid: FillId) -> f64 {
let pts: Vec<Point> = g.fill(fid).boundary.iter()
.filter(|(e, _)| !e.is_none())
.map(|&(e, d)| dir_start(g, e, d))
.collect();
if pts.len() < 3 { return 0.0; }
let mut a2 = 0.0;
for i in 0..pts.len() {
let p0 = pts[i]; let p1 = pts[(i + 1) % pts.len()];
a2 += p0.x * p1.y - p1.x * p0.y;
}
(a2 * 0.5).abs()
}
#[test]
fn two_edge_adjacent_rects_make_two_clean_fills() {
// Two rectangles sharing the x=100 edge. The second's left edge lands exactly on the
// first's right edge; without coincident-edge cleanup this produced duplicate edges and
// zero-area "sliver" fills (4 fills instead of 2) before any lasso.
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
draw_filled_rect(&mut g, 100.0, 0.0, 200.0, 100.0);
let live: Vec<_> = g.fills.iter().enumerate().filter(|(_, f)| !f.deleted)
.map(|(i, _)| (i, fill_bbox(&g, FillId(i as u32)))).collect();
assert_eq!(live.len(), 2, "two adjacent rects should make exactly two fills; got {live:?}");
assert_all_fills_connected(&g);
}
#[test]
fn lasso_strip_through_adjacent_second_rect() {
// Two rectangles sharing an edge (snapped adjacent), lasso a strip through the second.
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
draw_filled_rect(&mut g, 100.0, 0.0, 200.0, 100.0); // shares the x=100 edge with the first
let (segs, lasso_path) = rect_lasso(130.0, -50.0, 170.0, 150.0); // strip through 2nd
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
assert_all_fills_connected(&g);
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso_path.winding(fill_centroid(&g, fid)) != 0).collect();
let dump: Vec<_> = g.fills.iter().enumerate().filter(|(_, f)| !f.deleted)
.map(|(i, _)| (i, fill_centroid(&g, FillId(i as u32)), describe_boundary(&g, FillId(i as u32)).len())).collect();
assert_eq!(inside.len(), 1, "one strip of the 2nd rect should be inside; got {dump:#?}");
}
#[test]
fn lasso_strip_through_overlapping_second_rect() {
// Second rectangle overlaps the first; lasso a strip through the second's free part.
let mut g = editor_filled_rect(0.0, 0.0, 100.0, 100.0);
draw_filled_rect(&mut g, 60.0, 60.0, 200.0, 160.0);
let (segs, _lasso) = rect_lasso(120.0, 40.0, 160.0, 180.0);
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
assert_all_fills_connected(&g);
}
/// Bounding box of a fill's boundary edge endpoints.
fn fill_bbox(g: &VectorGraph, fid: FillId) -> (f64, f64, f64, f64) {
let (mut minx, mut miny, mut maxx, mut maxy) = (f64::MAX, f64::MAX, f64::MIN, f64::MIN);
for &(eid, _) in &g.fill(fid).boundary {
if eid.is_none() {
continue;
}
for &vid in &g.edge(eid).vertices {
let p = g.vertex(vid).position;
minx = minx.min(p.x);
miny = miny.min(p.y);
maxx = maxx.max(p.x);
maxy = maxy.max(p.y);
}
}
(minx, miny, maxx, maxy)
}
#[test]
fn lasso_corner_clip_of_second_rect_splits_off_corner() {
// Captured repro (/tmp dump): two separate rects; the lasso clips the top-left CORNER
// of the second rect, so the cut path turns a corner at a vertex INTERIOR to the rect.
let mut g = editor_filled_rect(128.37, 255.97, 290.91, 336.55); // rect 1
draw_filled_rect(&mut g, 417.39, 311.62, 620.25, 442.55); // rect 2
let (segs, lasso) = rect_lasso(360.12, 277.92, 537.32, 397.14); // clips rect-2 corner
g.insert_stroke(&segs, None, None, 1.0);
g.gc_invisible_edges();
assert_all_fills_connected(&g);
let inside: Vec<FillId> = g.fills.iter().enumerate()
.filter(|(_, f)| !f.deleted).map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso.winding(fill_centroid(&g, fid)) != 0).collect();
let dump: Vec<_> = g.fills.iter().enumerate().filter(|(_, f)| !f.deleted)
.map(|(i, _)| (i, fill_centroid(&g, FillId(i as u32)), fill_bbox(&g, FillId(i as u32)), describe_boundary(&g, FillId(i as u32)).len())).collect();
assert_eq!(inside.len(), 1, "exactly the clipped corner should be inside; fills = {dump:#?}");
// The selected fill must be the clipped CORNER, not the whole rect 2.
let (minx, miny, maxx, maxy) = fill_bbox(&g, inside[0]);
let eps = 1.0;
assert!(
(minx - 417.39).abs() < eps && (miny - 311.62).abs() < eps
&& (maxx - 537.32).abs() < eps && (maxy - 397.14).abs() < eps,
"expected clipped-corner bbox (417.4,311.6)-(537.3,397.1), got ({minx:.1},{miny:.1})-({maxx:.1},{maxy:.1}) \
— whole rect2 is (417,311)-(620,442); the fill wasn't split"
);
}
#[test]
fn lasso_over_middle_of_rect_selects_clean_center_strip() {
// Rectangle (0,0)-(200,100); lasso a vertical strip (50,-50)-(150,150).
let mut g = editor_filled_rect(0.0, 0.0, 200.0, 100.0);
let (segs, lasso_path) = rect_lasso(50.0, -50.0, 150.0, 150.0);
g.insert_stroke(&segs, None, None, 1.0);
// The editor runs this right after the cut: the lasso's top/bottom edges and the
// out-of-shape extensions of its sides are invisible and unreferenced — they must be
// collected so they can't be selected/edited later.
g.gc_invisible_edges();
for (i, e) in g.edges.iter().enumerate() {
if e.deleted || e.stroke_style.is_some() || e.stroke_color.is_some() {
continue;
}
let eid = EdgeId(i as u32);
let in_a_fill = g
.fills
.iter()
.any(|f| !f.deleted && f.boundary.iter().any(|&(fe, _)| fe == eid));
assert!(
in_a_fill,
"stray invisible edge {i} ({:?}) survived gc — not part of any fill",
e.curve
);
}
// Classify fills by centroid winding (the editor's logic).
let inside_fills: Vec<FillId> = g
.fills
.iter()
.enumerate()
.filter(|(_, f)| !f.deleted)
.map(|(i, _)| FillId(i as u32))
.filter(|&fid| lasso_path.winding(fill_centroid(&g, fid)) != 0)
.collect();
let dump = || {
g.fills
.iter()
.enumerate()
.filter(|(_, f)| !f.deleted)
.map(|(i, _)| {
let fid = FillId(i as u32);
(i, fill_centroid(&g, fid), describe_boundary(&g, fid))
})
.collect::<Vec<_>>()
};
assert_eq!(
inside_fills.len(),
1,
"expected exactly one inside fill (the center strip); live fills = {:#?}",
dump()
);
let fid = inside_fills[0];
let boundary = describe_boundary(&g, fid);
// The center strip is a rectangle: it must have 4 boundary edges that form a
// *connected closed loop* (each edge's directed end == the next edge's directed start).
// Before the fix, the split produced a 2-edge boundary (left cut + top segment) whose
// bbox is still (50,0)-(150,100) but which `fill_to_bezpath` renders as left+top edges
// plus a diagonal close — the artifact the user reported.
assert_eq!(
boundary.len(),
4,
"center strip should have 4 boundary edges, got {}: {:#?}",
boundary.len(),
boundary
);
let eps = 1e-6;
let n = boundary.len();
for i in 0..n {
let (_, _, end) = boundary[i];
let (_, next_start, _) = boundary[(i + 1) % n];
assert!(
(end.x - next_start.x).abs() < eps && (end.y - next_start.y).abs() < eps,
"boundary is disconnected between edge {i} (ends {end:?}) and edge {} \
(starts {next_start:?}); full boundary = {boundary:#?}",
(i + 1) % n
);
}
// And the loop should visit the four expected corners.
let corners = [
Point::new(50.0, 0.0),
Point::new(150.0, 0.0),
Point::new(150.0, 100.0),
Point::new(50.0, 100.0),
];
for c in corners {
let hit = boundary
.iter()
.any(|&(_, s, _)| (s.x - c.x).abs() < 0.5 && (s.y - c.y).abs() < 0.5);
assert!(hit, "center strip boundary should pass through corner {c:?}: {boundary:#?}");
}
}

1
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_dumps/dump0.json Normal file
View File

File diff suppressed because one or more lines are too long

1
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_dumps/dump1.json Normal file
View File

File diff suppressed because one or more lines are too long

1
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_dumps/dump2.json Normal file
View File

File diff suppressed because one or more lines are too long

1
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_dumps/dump3.json Normal file
View File

File diff suppressed because one or more lines are too long

1
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/region_dumps/dump4.json Normal file
View File

File diff suppressed because one or more lines are too long

78
lightningbeam-ui/lightningbeam-core/src/vector_graph/tests/tween.rs Normal file
View File

@ -0,0 +1,78 @@
//! Tests for same-topology shape-tween interpolation (`VectorGraph::interpolated`).
use super::super::*;
use kurbo::{CubicBez, Point};
fn line(a: Point, b: Point) -> CubicBez {
CubicBez::new(
a,
Point::new(a.x + (b.x - a.x) / 3.0, a.y + (b.y - a.y) / 3.0),
Point::new(a.x + 2.0 * (b.x - a.x) / 3.0, a.y + 2.0 * (b.y - a.y) / 3.0),
b,
)
}
/// Triangle (3 verts, 3 edges, 1 fill) offset by (ox, oy).
fn triangle(ox: f64, oy: f64) -> VectorGraph {
let mut g = VectorGraph::new();
let p = [
Point::new(ox, oy),
Point::new(ox + 100.0, oy),
Point::new(ox + 50.0, oy + 100.0),
];
let v: Vec<_> = p.iter().map(|&pt| g.alloc_vertex(pt)).collect();
let style = StrokeStyle { width: 1.0, ..Default::default() };
let mut boundary = Vec::new();
for i in 0..3 {
let e = g.alloc_edge(
line(p[i], p[(i + 1) % 3]),
v[i],
v[(i + 1) % 3],
Some(style.clone()),
Some(ShapeColor::rgb(0, 0, 0)),
);
boundary.push((e, Direction::Forward));
}
g.alloc_fill(boundary, ShapeColor::rgb(255, 0, 0), FillRule::NonZero);
g
}
#[test]
fn interpolate_same_topology_lerps_positions() {
let a = triangle(0.0, 0.0);
let b = triangle(100.0, 50.0);
let mid = a.interpolated(&b, 0.5).expect("same topology should interpolate");
// Vertex 0: (0,0) and (100,50) → (50,25). Curve endpoints follow.
assert!((mid.vertices[0].position.x - 50.0).abs() < 1e-6);
assert!((mid.vertices[0].position.y - 25.0).abs() < 1e-6);
assert!((mid.edges[0].curve.p0.x - 50.0).abs() < 1e-6);
// Endpoints: t=0 is `a`, t=1 is `b`.
assert!((a.interpolated(&b, 0.0).unwrap().vertices[0].position.x - 0.0).abs() < 1e-6);
assert!((a.interpolated(&b, 1.0).unwrap().vertices[0].position.x - 100.0).abs() < 1e-6);
}
#[test]
fn interpolate_lerps_fill_color() {
let mut a = triangle(0.0, 0.0);
let mut b = triangle(0.0, 0.0);
a.fills[0].color = Some(ShapeColor::rgb(0, 0, 0));
b.fills[0].color = Some(ShapeColor::rgb(100, 200, 40));
let mid = a.interpolated(&b, 0.5).unwrap();
let c = mid.fills[0].color.unwrap();
assert_eq!((c.r, c.g, c.b), (50, 100, 20));
}
#[test]
fn interpolate_topology_mismatch_returns_none() {
let a = triangle(0.0, 0.0);
let mut more_verts = triangle(0.0, 0.0);
more_verts.alloc_vertex(Point::new(999.0, 999.0));
assert!(a.interpolated(&more_verts, 0.5).is_none(), "different vertex count");
// Same counts but a moved edge endpoint (different vertices) is still a mismatch.
let mut rewired = triangle(0.0, 0.0);
rewired.edges[0].vertices = [VertexId(2), VertexId(1)];
assert!(a.interpolated(&rewired, 0.5).is_none(), "different edge endpoints");
}

484
lightningbeam-ui/lightningbeam-core/src/video.rs
View File

@ -97,7 +97,7 @@ impl VideoDecoder {
// Optionally build keyframe index for fast seeking
let keyframe_positions = if build_keyframes {
eprintln!("[Video Decoder] Building keyframe index for {}", path);
let positions = Self::build_keyframe_index(&path, stream_index)?;
let positions = Self::scan_keyframes(&path, stream_index)?;
eprintln!("[Video Decoder] Found {} keyframes", positions.len());
positions
} else {
@ -125,14 +125,19 @@ impl VideoDecoder {
})
}
/// Build keyframe index for this decoder
/// This can be called asynchronously after decoder creation
fn build_and_set_keyframe_index(&mut self) -> Result<(), String> {
eprintln!("[Video Decoder] Building keyframe index for {}", self.path);
let positions = Self::build_keyframe_index(&self.path, self.stream_index)?;
eprintln!("[Video Decoder] Found {} keyframes", positions.len());
/// Source file path this decoder reads from.
pub fn path(&self) -> &str {
&self.path
}
/// Parameters needed to scan keyframes off-thread (path + video stream index).
pub fn keyframe_scan_params(&self) -> (String, usize) {
(self.path.clone(), self.stream_index)
}
/// Replace the keyframe index (built off-thread via [`VideoDecoder::scan_keyframes`]).
pub fn set_keyframe_index(&mut self, positions: Vec<i64>) {
self.keyframe_positions = positions;
Ok(())
}
/// Get the output width (scaled dimensions)
@ -150,9 +155,10 @@ impl VideoDecoder {
self.get_frame(timestamp)
}
/// Build an index of all keyframe positions in the video
/// This enables fast seeking by knowing exactly where keyframes are
fn build_keyframe_index(path: &str, stream_index: usize) -> Result<Vec<i64>, String> {
/// Build an index of all keyframe positions in the video by scanning packets
/// from a fresh input. Does not touch `self` — call it off-thread (it is slow
/// on long videos) and hand the result to [`VideoDecoder::set_keyframe_index`].
pub fn scan_keyframes(path: &str, stream_index: usize) -> Result<Vec<i64>, String> {
let mut input = ffmpeg::format::input(path)
.map_err(|e| format!("Failed to open video for indexing: {}", e))?;
@ -340,6 +346,58 @@ impl VideoDecoder {
}
}
/// Generate timeline thumbnails for a video using a **dedicated** decoder that
/// is independent of any shared playback decoder — so thumbnail work never holds
/// a lock the UI/playback needs.
///
/// Thumbnails are sampled at keyframes ~`interval_secs` apart. Decoding at a
/// keyframe is cheap (≈one frame) versus decoding forward to an arbitrary
/// timestamp (the whole GOP). Frames are decoded directly at `thumb_width` (so
/// `get_thumbnail_at`'s 128-wide assumption holds) and tightly packed RGBA is
/// handed to `on_thumb` as `(timestamp_secs, data)`.
pub fn generate_keyframe_thumbnails(
path: &str,
interval_secs: f64,
thumb_width: u32,
mut should_skip: impl FnMut(f64) -> bool,
mut on_thumb: impl FnMut(f64, Arc<Vec<u8>>),
) -> Result<(), String> {
// Own decoder at thumbnail resolution; builds its own keyframe index. The
// large max-height lets width be the constraining dimension, so output width
// is exactly `thumb_width`.
let mut decoder = VideoDecoder::new(
path.to_string(),
4,
Some(thumb_width),
Some(100_000),
true, // build keyframe index (needed to sample at keyframes)
)?;
let keyframe_secs: Vec<f64> = decoder
.keyframe_positions
.iter()
.map(|&ts| ts as f64 * decoder.time_base)
.collect();
let mut last_emitted = f64::NEG_INFINITY;
for ks in keyframe_secs {
if ks - last_emitted < interval_secs {
continue;
}
// This keyframe is a target slot; advance regardless of skip so the chosen
// slots are deterministic (lets a resumed pass target the same timestamps).
last_emitted = ks;
// Skip slots already covered (resume after a partial save / dedup).
if should_skip(ks) {
continue;
}
if let Ok(rgba) = decoder.get_frame(ks) {
on_thumb(ks, Arc::new(rgba));
}
}
Ok(())
}
/// Probe video file for metadata without creating a full decoder
pub fn probe_video(path: &str) -> Result<VideoMetadata, String> {
ffmpeg::init().map_err(|e| e.to_string())?;
@ -407,18 +465,33 @@ pub struct VideoManager {
/// Pool of video decoders, one per clip
decoders: HashMap<Uuid, Arc<Mutex<VideoDecoder>>>,
/// Frame cache: (clip_id, timestamp_ms) -> frame
/// Stores raw RGBA data for zero-copy rendering
frame_cache: HashMap<(Uuid, i64), Arc<VideoFrame>>,
/// Frame cache: (clip_id, timestamp_ms) -> frame. Stores decoded RGBA for
/// zero-copy rendering. Bounded by a **byte budget** (not a frame count, which
/// would be unsafe across resolutions — a 4K frame is ~33MB vs ~2MB at 800x600)
/// so playback of arbitrarily long video never grows unbounded.
frame_cache: LruCache<(Uuid, i64), Arc<VideoFrame>>,
/// Running total of bytes held in `frame_cache` (sum of each frame's RGBA len),
/// kept in sync on insert/evict/remove so eviction is O(1) per frame.
frame_cache_bytes: usize,
/// Thumbnail cache: clip_id -> Vec of (timestamp, rgba_data)
/// Low-resolution (64px width) thumbnails for scrubbing
thumbnail_cache: HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>>,
/// Clips whose thumbnail generation finished. Only complete sets are worth
/// persisting — a partial set (saved mid-generation) is dropped so the load
/// regenerates it fully rather than leaving it permanently incomplete.
thumbnails_complete: std::collections::HashSet<Uuid>,
/// Maximum number of frames to cache per decoder
cache_size: usize,
}
/// Byte budget for [`VideoManager::frame_cache`] (decoded full-resolution frames).
/// At ~2MB/frame (800x600) this holds ~128 frames; at ~33MB/frame (4K) ~8 — in
/// both cases enough for the current frame plus a scrub window, while bounding RAM.
const FRAME_CACHE_BYTE_BUDGET: usize = 256 * 1024 * 1024;
impl VideoManager {
/// Create a new video manager with default cache size
pub fn new() -> Self {
@ -429,8 +502,10 @@ impl VideoManager {
pub fn with_cache_size(cache_size: usize) -> Self {
Self {
decoders: HashMap::new(),
frame_cache: HashMap::new(),
frame_cache: LruCache::unbounded(),
frame_cache_bytes: 0,
thumbnail_cache: HashMap::new(),
thumbnails_complete: std::collections::HashSet::new(),
cache_size,
}
}
@ -440,8 +515,9 @@ impl VideoManager {
/// `target_width` and `target_height` specify the maximum dimensions
/// for decoded frames. Video will be scaled down if larger.
///
/// The keyframe index is NOT built during this call - use `build_keyframe_index_async`
/// in a background thread to build it asynchronously.
/// The keyframe index is NOT built during this call — scan it off-thread via
/// [`VideoDecoder::scan_keyframes`] and store it with
/// [`VideoDecoder::set_keyframe_index`] so the slow scan never blocks playback.
pub fn load_video(
&mut self,
clip_id: Uuid,
@ -467,20 +543,6 @@ impl VideoManager {
Ok(metadata)
}
/// Build keyframe index for a loaded video asynchronously
///
/// This should be called from a background thread after load_video()
/// to avoid blocking the UI during import.
pub fn build_keyframe_index(&self, clip_id: &Uuid) -> Result<(), String> {
let decoder_arc = self.decoders.get(clip_id)
.ok_or_else(|| format!("Video clip {} not found", clip_id))?;
let mut decoder = decoder_arc.lock()
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
decoder.build_and_set_keyframe_index()
}
/// Get a decoded frame for a specific clip at a specific timestamp
///
/// Returns None if the clip is not loaded or decoding fails.
@ -495,14 +557,16 @@ impl VideoManager {
return Some(Arc::clone(cached_frame));
}
// Get decoder for this clip
let decoder_arc = self.decoders.get(clip_id)?;
// Get decoder for this clip. Clone the Arc so we don't hold a borrow of
// `self.decoders` across the `&mut self` cache insert below.
let decoder_arc = Arc::clone(self.decoders.get(clip_id)?);
let mut decoder = decoder_arc.lock().ok()?;
// Decode the frame
let rgba_data = decoder.get_frame(timestamp).ok()?;
let width = decoder.output_width;
let height = decoder.output_height;
drop(decoder); // release the lock before touching `self`
// Create VideoFrame and cache it
let frame = Arc::new(VideoFrame {
@ -512,65 +576,27 @@ impl VideoManager {
timestamp,
});
self.frame_cache.insert(cache_key, Arc::clone(&frame));
self.cache_frame(cache_key, Arc::clone(&frame));
Some(frame)
}
/// Generate thumbnails for a video clip (single batch version - use generate_thumbnails_progressive instead)
///
/// Thumbnails are generated every 5 seconds at 128px width.
/// This should be called in a background thread to avoid blocking.
/// Thumbnails are inserted into the cache progressively as they're generated,
/// allowing the UI to display them immediately.
///
/// DEPRECATED: Use generate_thumbnails_progressive which releases the lock between thumbnails.
pub fn generate_thumbnails(&mut self, clip_id: &Uuid, duration: f64) -> Result<(), String> {
let decoder_arc = self.decoders.get(clip_id)
.ok_or("Clip not loaded")?
.clone();
let mut decoder = decoder_arc.lock()
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
// Initialize thumbnail cache entry with empty vec
self.thumbnail_cache.insert(*clip_id, Vec::new());
let interval = 5.0; // Generate thumbnail every 5 seconds
let mut t = 0.0;
while t < duration {
// Decode frame at this timestamp
if let Ok(rgba_data) = decoder.get_frame(t) {
// Decode already scaled to output dimensions, but we want 128px width for thumbnails
// We need to scale down further
let current_width = decoder.output_width;
let current_height = decoder.output_height;
// Calculate thumbnail dimensions (128px width, maintain aspect ratio)
let thumb_width = 128u32;
let aspect_ratio = current_height as f32 / current_width as f32;
let thumb_height = (thumb_width as f32 * aspect_ratio) as u32;
// Simple nearest-neighbor downsampling for thumbnails
let thumb_data = downsample_rgba(
&rgba_data,
current_width,
current_height,
thumb_width,
thumb_height,
);
// Insert thumbnail into cache immediately so UI can display it
if let Some(thumbnails) = self.thumbnail_cache.get_mut(clip_id) {
thumbnails.push((t, Arc::new(thumb_data)));
/// Insert a frame into the byte-budgeted cache, evicting least-recently-used
/// frames until the total is within [`FRAME_CACHE_BYTE_BUDGET`].
fn cache_frame(&mut self, key: (Uuid, i64), frame: Arc<VideoFrame>) {
let bytes = frame.rgba_data.len();
if let Some(old) = self.frame_cache.put(key, frame) {
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(old.rgba_data.len());
}
self.frame_cache_bytes += bytes;
// Keep at least one frame resident even if it alone exceeds the budget.
while self.frame_cache_bytes > FRAME_CACHE_BYTE_BUDGET && self.frame_cache.len() > 1 {
if let Some((_, evicted)) = self.frame_cache.pop_lru() {
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(evicted.rgba_data.len());
} else {
break;
}
}
t += interval;
}
Ok(())
}
/// Get the decoder Arc for a clip (for external thumbnail generation)
@ -579,18 +605,57 @@ impl VideoManager {
self.decoders.get(clip_id).cloned()
}
/// Insert a thumbnail into the cache (for external thumbnail generation)
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) {
self.thumbnail_cache
.entry(*clip_id)
.or_insert_with(Vec::new)
.push((timestamp, data));
/// Snapshot all cached thumbnails for persistence (clip id -> sorted
/// (timestamp, rgba) pairs). Cheap: clones the `Arc`s, not the pixel data.
/// Partial sets are persisted too — pair with [`complete_thumbnail_clips`] so
/// the load knows which clips still need generation resumed.
pub fn snapshot_all_thumbnails(&self) -> HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>> {
self.thumbnail_cache.clone()
}
/// Get the thumbnail closest to the specified timestamp
/// The set of clips whose thumbnail generation has finished (a full keyframe
/// pass). A persisted set flagged incomplete is resumed on load.
pub fn complete_thumbnail_clips(&self) -> std::collections::HashSet<Uuid> {
self.thumbnails_complete.clone()
}
/// Mark a clip's thumbnail generation as complete (called when the background
/// generator finishes the full keyframe pass).
pub fn mark_thumbnails_complete(&mut self, clip_id: &Uuid) {
self.thumbnails_complete.insert(*clip_id);
}
/// Whether the clip already has a thumbnail within `tol` seconds of `ts`.
/// Lets the generator skip keyframes already covered (resume / dedup).
pub fn has_thumbnail_near(&self, clip_id: &Uuid, ts: f64, tol: f64) -> bool {
self.thumbnail_cache
.get(clip_id)
.map_or(false, |v| v.iter().any(|(t, _)| (t - ts).abs() < tol))
}
/// Insert a thumbnail into the cache, keeping it **sorted by timestamp** and
/// **deduped** (an existing entry at the same timestamp is replaced). Sorted
/// order is required by `get_thumbnail_at`'s binary search, and dedup makes
/// concurrent restore + resumed generation idempotent (no double inserts).
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) {
let vec = self.thumbnail_cache.entry(*clip_id).or_default();
match vec.binary_search_by(|(t, _)| {
t.partial_cmp(&timestamp).unwrap_or(std::cmp::Ordering::Equal)
}) {
Ok(i) => vec[i] = (timestamp, data),
Err(i) => vec.insert(i, (timestamp, data)),
}
}
/// Get the thumbnail closest to the specified timestamp.
///
/// Returns `(actual_timestamp, width, height, data)` — `actual_timestamp` is
/// the time of the thumbnail actually chosen (which may differ from the
/// requested `timestamp`, and changes as closer thumbnails finish generating).
/// Callers key their GPU texture cache on it so the on-clip strip refreshes as
/// better thumbnails load instead of freezing on the first one.
/// Returns None if no thumbnails have been generated for this clip.
pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(u32, u32, Arc<Vec<u8>>)> {
pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(f64, u32, u32, Arc<Vec<u8>>)> {
let thumbnails = self.thumbnail_cache.get(clip_id)?;
if thumbnails.is_empty() {
@ -618,30 +683,43 @@ impl VideoManager {
}
});
let (_, rgba_data) = &thumbnails[idx];
let (actual_ts, rgba_data) = &thumbnails[idx];
// Return (width, height, data)
// Return (actual_timestamp, width, height, data)
// Thumbnails are always 128px width
let thumb_width = 128;
let thumb_height = (rgba_data.len() / (thumb_width * 4)) as u32;
Some((thumb_width as u32, thumb_height, Arc::clone(rgba_data)))
Some((*actual_ts, thumb_width as u32, thumb_height, Arc::clone(rgba_data)))
}
/// Remove a video clip and its cached data
pub fn unload_video(&mut self, clip_id: &Uuid) {
self.decoders.remove(clip_id);
// Remove all cached frames for this clip
self.frame_cache.retain(|(id, _), _| id != clip_id);
// Remove all cached frames for this clip (LruCache has no retain; collect
// matching keys, then pop each, keeping the byte total in sync).
let keys: Vec<(Uuid, i64)> = self
.frame_cache
.iter()
.filter(|((id, _), _)| id == clip_id)
.map(|(k, _)| *k)
.collect();
for key in keys {
if let Some(frame) = self.frame_cache.pop(&key) {
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(frame.rgba_data.len());
}
}
// Remove thumbnails
self.thumbnail_cache.remove(clip_id);
self.thumbnails_complete.remove(clip_id);
}
/// Clear all frame caches (useful for memory management)
pub fn clear_frame_cache(&mut self) {
self.frame_cache.clear();
self.frame_cache_bytes = 0;
}
}
@ -650,211 +728,3 @@ impl Default for VideoManager {
Self::new()
}
}
/// Simple nearest-neighbor downsampling for RGBA images
pub fn downsample_rgba_public(
src: &[u8],
src_width: u32,
src_height: u32,
dst_width: u32,
dst_height: u32,
) -> Vec<u8> {
downsample_rgba(src, src_width, src_height, dst_width, dst_height)
}
/// Simple nearest-neighbor downsampling for RGBA images (internal)
fn downsample_rgba(
src: &[u8],
src_width: u32,
src_height: u32,
dst_width: u32,
dst_height: u32,
) -> Vec<u8> {
let mut dst = Vec::with_capacity((dst_width * dst_height * 4) as usize);
let x_ratio = src_width as f32 / dst_width as f32;
let y_ratio = src_height as f32 / dst_height as f32;
for y in 0..dst_height {
for x in 0..dst_width {
let src_x = (x as f32 * x_ratio) as u32;
let src_y = (y as f32 * y_ratio) as u32;
let src_idx = ((src_y * src_width + src_x) * 4) as usize;
// Copy RGBA bytes
dst.push(src[src_idx]); // R
dst.push(src[src_idx + 1]); // G
dst.push(src[src_idx + 2]); // B
dst.push(src[src_idx + 3]); // A
}
}
dst
}
/// Extracted audio data from a video file
#[derive(Debug, Clone)]
pub struct ExtractedAudio {
pub samples: Vec<f32>,
pub channels: u32,
pub sample_rate: u32,
pub duration: f64,
}
/// Extract audio from a video file
///
/// This function performs the slow FFmpeg decoding without holding any locks.
/// The caller can then quickly add the audio to the DAW backend in a background thread.
///
/// Returns None if the video has no audio stream.
pub fn extract_audio_from_video(path: &str) -> Result<Option<ExtractedAudio>, String> {
ffmpeg::init().map_err(|e| e.to_string())?;
// Open video file
let mut input = ffmpeg::format::input(path)
.map_err(|e| format!("Failed to open video: {}", e))?;
// Find audio stream
let audio_stream_opt = input.streams()
.best(ffmpeg::media::Type::Audio);
// Return None if no audio stream
if audio_stream_opt.is_none() {
return Ok(None);
}
let audio_stream = audio_stream_opt.unwrap();
let audio_index = audio_stream.index();
// Get audio properties
let context_decoder = ffmpeg::codec::context::Context::from_parameters(
audio_stream.parameters()
).map_err(|e| e.to_string())?;
let mut audio_decoder = context_decoder.decoder().audio()
.map_err(|e| e.to_string())?;
let sample_rate = audio_decoder.rate();
let channels = audio_decoder.channels() as u32;
// Decode all audio frames
let mut audio_samples: Vec<f32> = Vec::new();
for (stream, packet) in input.packets() {
if stream.index() == audio_index {
audio_decoder.send_packet(&packet)
.map_err(|e| e.to_string())?;
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
// Convert audio to f32 packed format
let format = audio_frame.format();
let frame_channels = audio_frame.channels() as usize;
// Create resampler to convert to f32 packed
let mut resampler = ffmpeg::software::resampling::context::Context::get(
format,
audio_frame.channel_layout(),
sample_rate,
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
audio_frame.channel_layout(),
sample_rate,
).map_err(|e| e.to_string())?;
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
resampler.run(&audio_frame, &mut resampled_frame)
.map_err(|e| e.to_string())?;
// Extract f32 samples (interleaved format)
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
let total_samples = resampled_frame.samples() * frame_channels;
// Safety checks before creating slice from FFmpeg data
// 1. Verify f32 alignment (required: 4 bytes)
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
}
// 2. Verify the frame actually has enough data
let byte_size = resampled_frame.data(0).len();
let expected_bytes = total_samples * std::mem::size_of::<f32>();
if byte_size < expected_bytes {
return Err(format!(
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
byte_size, expected_bytes
));
}
// SAFETY: We verified alignment and bounds above.
// The slice lifetime is tied to resampled_frame which lives until
// after extend_from_slice completes.
let samples_slice = unsafe {
std::slice::from_raw_parts(data_ptr, total_samples)
};
audio_samples.extend_from_slice(samples_slice);
}
}
}
// Flush audio decoder
audio_decoder.send_eof().map_err(|e| e.to_string())?;
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
let format = audio_frame.format();
let frame_channels = audio_frame.channels() as usize;
let mut resampler = ffmpeg::software::resampling::context::Context::get(
format,
audio_frame.channel_layout(),
sample_rate,
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
audio_frame.channel_layout(),
sample_rate,
).map_err(|e| e.to_string())?;
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
resampler.run(&audio_frame, &mut resampled_frame)
.map_err(|e| e.to_string())?;
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
let total_samples = resampled_frame.samples() * frame_channels;
// Safety checks before creating slice from FFmpeg data
// 1. Verify f32 alignment (required: 4 bytes)
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
}
// 2. Verify the frame actually has enough data
let byte_size = resampled_frame.data(0).len();
let expected_bytes = total_samples * std::mem::size_of::<f32>();
if byte_size < expected_bytes {
return Err(format!(
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
byte_size, expected_bytes
));
}
// SAFETY: We verified alignment and bounds above.
// The slice lifetime is tied to resampled_frame which lives until
// after extend_from_slice completes.
let samples_slice = unsafe {
std::slice::from_raw_parts(data_ptr, total_samples)
};
audio_samples.extend_from_slice(samples_slice);
}
// Calculate duration
let total_samples_per_channel = audio_samples.len() / channels as usize;
let duration = total_samples_per_channel as f64 / sample_rate as f64;
Ok(Some(ExtractedAudio {
samples: audio_samples,
channels,
sample_rate,
duration,
}))
}

207
lightningbeam-ui/lightningbeam-core/tests/beam_archive.rs Normal file
View File

@ -0,0 +1,207 @@
//! Integration tests for the SQLite-backed `.beam` container.
//!
//! These are integration tests (not `#[cfg(test)]` unit tests) so they build the
//! library in normal mode and exercise only the public API — independent of any
//! pre-existing breakage in the crate's internal test modules.
use lightningbeam_core::beam_archive::{BeamArchive, MediaKind, MediaMeta, MediaStorage};
use std::io::{Read, Seek, SeekFrom};
use std::sync::atomic::{AtomicU64, Ordering};
use uuid::Uuid;
fn temp_db_path(tag: &str) -> std::path::PathBuf {
static N: AtomicU64 = AtomicU64::new(0);
let n = N.fetch_add(1, Ordering::Relaxed);
let mut p = std::env::temp_dir();
p.push(format!("beam_archive_it_{}_{}_{}.beam", std::process::id(), tag, n));
let _ = std::fs::remove_file(&p);
p
}
#[test]
fn project_json_roundtrip() {
let path = temp_db_path("json");
let archive = BeamArchive::create(&path).unwrap();
archive.set_project_json("{\"hello\":\"world\"}").unwrap();
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
drop(archive);
let archive = BeamArchive::open(&path).unwrap();
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
assert!(BeamArchive::is_sqlite(&path));
let _ = std::fs::remove_file(&path);
}
#[test]
fn packed_media_roundtrip_full() {
let path = temp_db_path("full");
let mut archive = BeamArchive::create(&path).unwrap();
let id = Uuid::from_u128(0x1234);
archive.set_chunk_size(1000);
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
archive
.put_media_packed(
id,
MediaKind::Audio,
"flac",
&data,
MediaMeta { channels: Some(2), sample_rate: Some(44100), ..Default::default() },
)
.unwrap();
let info = archive.media_info(id).unwrap().unwrap();
assert_eq!(info.kind, MediaKind::Audio);
assert_eq!(info.codec, "flac");
assert_eq!(info.storage, MediaStorage::Packed);
assert_eq!(info.total_len, 3500);
assert_eq!(info.channels, Some(2));
assert_eq!(info.sample_rate, Some(44100));
assert_eq!(archive.read_media_full(id).unwrap(), data);
assert_eq!(archive.media_ids_of_kind(MediaKind::Audio).unwrap(), vec![id]);
let _ = std::fs::remove_file(&path);
}
#[test]
fn blob_reader_streams_and_seeks() {
let path = temp_db_path("stream");
let mut archive = BeamArchive::create(&path).unwrap();
archive.set_chunk_size(1000);
let id = Uuid::from_u128(0xBEEF);
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
archive
.put_media_packed(id, MediaKind::Audio, "mp3", &data, MediaMeta::default())
.unwrap();
let mut reader = archive.open_blob_reader(&path, id).unwrap();
assert_eq!(reader.len(), 3500);
// Sequential read in odd-sized buffers crosses chunk boundaries.
let mut got = Vec::new();
let mut buf = [0u8; 333];
loop {
let n = reader.read(&mut buf).unwrap();
if n == 0 {
break;
}
got.extend_from_slice(&buf[..n]);
}
assert_eq!(got, data);
// Seek to a mid-chunk position and read across a boundary.
reader.seek(SeekFrom::Start(990)).unwrap();
let mut window = [0u8; 20];
let mut filled = 0;
while filled < window.len() {
let n = reader.read(&mut window[filled..]).unwrap();
assert!(n > 0);
filled += n;
}
assert_eq!(&window[..], &data[990..1010]);
// Seek from end and read the tail.
reader.seek(SeekFrom::End(-10)).unwrap();
let mut tail = Vec::new();
reader.read_to_end(&mut tail).unwrap();
assert_eq!(tail, &data[3490..]);
let _ = std::fs::remove_file(&path);
}
#[test]
fn referenced_media_records_path() {
let path = temp_db_path("ref");
let mut archive = BeamArchive::create(&path).unwrap();
let id = Uuid::from_u128(0xCAFE);
archive
.put_media_referenced(
id,
MediaKind::Video,
"mp4",
"/mnt/share/big.mp4",
MediaMeta { width: Some(3840), height: Some(2160), ..Default::default() },
)
.unwrap();
let info = archive.media_info(id).unwrap().unwrap();
assert_eq!(info.storage, MediaStorage::Referenced);
assert_eq!(info.ext_path.as_deref(), Some("/mnt/share/big.mp4"));
assert_eq!(info.width, Some(3840));
// Streaming a referenced item is an error (caller opens the path directly).
assert!(archive.open_blob_reader(&path, id).is_err());
let _ = std::fs::remove_file(&path);
}
#[test]
fn transaction_groups_writes_and_orphan_cleanup() {
let path = temp_db_path("txn");
let keep = Uuid::from_u128(1);
let orphan = Uuid::from_u128(2);
// First save: two media items committed in one transaction.
{
let mut archive = BeamArchive::create(&path).unwrap();
let txn = archive.transaction().unwrap();
txn.put_media_packed(keep, MediaKind::Audio, "flac", &vec![9u8; 10], MediaMeta::default())
.unwrap();
txn.put_media_packed(orphan, MediaKind::Audio, "mp3", &vec![8u8; 10], MediaMeta::default())
.unwrap();
txn.set_project_json("{}").unwrap();
txn.commit().unwrap();
}
{
let archive = BeamArchive::open(&path).unwrap();
assert!(archive.media_info(keep).unwrap().is_some());
assert!(archive.media_info(orphan).unwrap().is_some());
}
// Second save (in place): keep only `keep`; `orphan` should be retained-out.
{
let mut archive = BeamArchive::open(&path).unwrap();
let txn = archive.transaction().unwrap();
// `keep` already present → in-place save leaves it untouched.
assert!(txn.media_exists(keep).unwrap());
let mut live = std::collections::HashSet::new();
live.insert(keep);
let removed = txn.retain_media(&live).unwrap();
assert_eq!(removed, 1);
txn.commit().unwrap();
}
{
let archive = BeamArchive::open(&path).unwrap();
assert!(archive.media_info(keep).unwrap().is_some());
assert!(archive.media_info(orphan).unwrap().is_none());
// `keep`'s bytes survived untouched.
assert_eq!(archive.read_media_full(keep).unwrap(), vec![9u8; 10]);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn rolled_back_transaction_writes_nothing() {
let path = temp_db_path("rollback");
let id = Uuid::from_u128(42);
let mut archive = BeamArchive::create(&path).unwrap();
{
let txn = archive.transaction().unwrap();
txn.put_media_packed(id, MediaKind::Audio, "flac", &vec![1u8; 5], MediaMeta::default())
.unwrap();
// Drop without commit → rollback.
}
assert!(archive.media_info(id).unwrap().is_none());
let _ = std::fs::remove_file(&path);
}
#[test]
fn overwrite_media_replaces_chunks() {
let path = temp_db_path("overwrite");
let mut archive = BeamArchive::create(&path).unwrap();
archive.set_chunk_size(100);
let id = Uuid::from_u128(7);
archive
.put_media_packed(id, MediaKind::Raster, "png", &vec![1u8; 250], MediaMeta::default())
.unwrap();
// Overwrite with shorter data — stale chunks must be gone.
archive
.put_media_packed(id, MediaKind::Raster, "png", &vec![2u8; 50], MediaMeta::default())
.unwrap();
assert_eq!(archive.read_media_full(id).unwrap(), vec![2u8; 50]);
let _ = std::fs::remove_file(&path);
}

2
lightningbeam-ui/lightningbeam-editor/Cargo.toml
View File

@ -1,6 +1,6 @@
[package]
name = "lightningbeam-editor"
version = "1.0.4-alpha"
version = "1.0.5-alpha"
edition = "2021"
description = "Multimedia editor for audio, video and 2D animation"
license = "GPL-3.0-or-later"

16
lightningbeam-ui/lightningbeam-editor/src/config.rs
View File

@ -1,6 +1,7 @@
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use crate::keymap::KeybindingConfig;
use lightningbeam_core::file_io::LargeMediaMode;
/// Application configuration (persistent)
#[derive(Debug, Clone, Serialize, Deserialize)]
@ -57,6 +58,18 @@ pub struct AppConfig {
/// Custom keyboard shortcut overrides (sparse — only non-default bindings stored)
#[serde(default)]
pub keybindings: KeybindingConfig,
/// How to store media files at/above the large-media threshold (~2GB).
/// `Ask` (default) prompts the first time such a file is imported, then the
/// chosen mode is persisted here. Reset to `Ask` to be prompted again.
#[serde(default)]
pub large_media_default: LargeMediaMode,
/// Finest-level resolution of the waveform LOD pyramid: source frames per
/// floor texel (`B`). Smaller = larger on-disk pyramid but zoom-in re-decodes
/// sooner; larger = smaller pyramid, wider re-decode span. Default 256.
#[serde(default = "defaults::waveform_floor_samples_per_texel")]
pub waveform_floor_samples_per_texel: u32,
}
impl Default for AppConfig {
@ -75,6 +88,8 @@ impl Default for AppConfig {
waveform_stereo: defaults::waveform_stereo(),
theme_mode: defaults::theme_mode(),
keybindings: KeybindingConfig::default(),
large_media_default: LargeMediaMode::default(),
waveform_floor_samples_per_texel: defaults::waveform_floor_samples_per_texel(),
}
}
}
@ -276,4 +291,5 @@ mod defaults {
pub fn debug() -> bool { false }
pub fn waveform_stereo() -> bool { false }
pub fn theme_mode() -> String { "system".to_string() }
pub fn waveform_floor_samples_per_texel() -> u32 { 256 }
}

34
lightningbeam-ui/lightningbeam-editor/src/debug_overlay.rs
View File

@ -39,6 +39,33 @@ pub fn update_prepare_timing(
t.composite_ms = composite_ms;
}
}
/// GPU memory the editor tracks itself (wgpu has no allocator query). Currently the
/// raster-layer texture cache — the only unbounded-by-default VRAM consumer.
#[derive(Debug, Clone, Default)]
pub struct GpuMemoryStats {
pub raster_cache_entries: usize,
pub raster_cache_bytes: usize,
}
static GPU_MEMORY: OnceLock<Mutex<GpuMemoryStats>> = OnceLock::new();
/// Called by the GPU brush whenever the raster-layer cache changes.
pub fn update_gpu_memory(raster_cache_entries: usize, raster_cache_bytes: usize) {
let cell = GPU_MEMORY.get_or_init(|| Mutex::new(GpuMemoryStats::default()));
if let Ok(mut s) = cell.lock() {
s.raster_cache_entries = raster_cache_entries;
s.raster_cache_bytes = raster_cache_bytes;
}
}
fn get_gpu_memory() -> GpuMemoryStats {
GPU_MEMORY
.get_or_init(|| Mutex::new(GpuMemoryStats::default()))
.lock()
.map(|s| s.clone())
.unwrap_or_default()
}
const DEVICE_REFRESH_INTERVAL: Duration = Duration::from_secs(2); // Refresh devices every 2 seconds
const MEMORY_REFRESH_INTERVAL: Duration = Duration::from_millis(500); // Refresh memory every 500ms
@ -52,6 +79,7 @@ pub struct DebugStats {
pub frame_time_ms: f32, // Current frame time in milliseconds
pub memory_physical_mb: usize,
pub memory_virtual_mb: usize,
pub gpu_memory: GpuMemoryStats,
pub gpu_name: String,
pub gpu_backend: String,
pub gpu_driver: String,
@ -218,6 +246,7 @@ impl DebugStatsCollector {
frame_time_ms,
memory_physical_mb,
memory_virtual_mb,
gpu_memory: get_gpu_memory(),
gpu_name,
gpu_backend,
gpu_driver,
@ -286,6 +315,11 @@ pub fn render_debug_overlay(ctx: &egui::Context, stats: &DebugStats) {
ui.colored_label(egui::Color32::YELLOW, format!("Memory: ({}µs)", stats.timing_memory_us));
ui.label(format!("Physical: {} MB", stats.memory_physical_mb));
ui.label(format!("Virtual: {} MB", stats.memory_virtual_mb));
ui.label(format!(
"VRAM (raster cache): {:.1} MB ({} frames)",
stats.gpu_memory.raster_cache_bytes as f64 / (1024.0 * 1024.0),
stats.gpu_memory.raster_cache_entries,
));
ui.add_space(8.0);

77
lightningbeam-ui/lightningbeam-editor/src/effect_thumbnails.rs
View File

@ -11,6 +11,36 @@ use uuid::Uuid;
/// Size of effect thumbnails in pixels
pub const EFFECT_THUMBNAIL_SIZE: u32 = 64;
use lightningbeam_core::gpu::{srgb_to_linear, linear_to_srgb};
/// sRGB-u8 RGBA → linear-`f16` RGBA bytes (little-endian). Feeds the effect
/// shaders linear light at float precision, matching the live HDR pipeline (an
/// 8-bit linear intermediate would band in shadows). RGB go through the sRGB
/// EOTF; alpha is linear.
fn srgb_image_to_linear_f16(rgba: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(rgba.len() * 2);
for px in rgba.chunks_exact(4) {
for &c in &px[..3] {
out.extend_from_slice(&half::f16::from_f32(srgb_to_linear(c as f32 / 255.0)).to_le_bytes());
}
out.extend_from_slice(&half::f16::from_f32(px[3] as f32 / 255.0).to_le_bytes());
}
out
}
/// linear-`f16` RGBA bytes → sRGB-u8 RGBA. Inverse of [`srgb_image_to_linear_f16`].
fn linear_f16_to_srgb_image(f16_rgba: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(f16_rgba.len() / 2);
for texel in f16_rgba.chunks_exact(8) {
let ch = |i: usize| half::f16::from_le_bytes([texel[i], texel[i + 1]]).to_f32();
out.push((linear_to_srgb(ch(0)) * 255.0 + 0.5) as u8);
out.push((linear_to_srgb(ch(2)) * 255.0 + 0.5) as u8);
out.push((linear_to_srgb(ch(4)) * 255.0 + 0.5) as u8);
out.push((ch(6).clamp(0.0, 1.0) * 255.0 + 0.5) as u8);
}
out
}
/// Embedded still-life image for effect preview thumbnails
const EFFECT_PREVIEW_IMAGE_BYTES: &[u8] = include_bytes!("../../../src/assets/still-life.jpg");
@ -39,10 +69,18 @@ impl EffectThumbnailGenerator {
/// Create a new effect thumbnail generator
pub fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Self {
// Load and decode the source image
let source_rgba = Self::load_source_image();
// The effect shaders operate in LINEAR light (matching the live HDR
// pipeline, which feeds them a linear Rgba16Float texture). The preview
// image is sRGB-encoded, so linearize it before upload and re-encode the
// result after readback. This keeps thumbnails consistent with the live
// render for every effect, including the gamma-space perceptual ones.
// Linearize to f16 (float precision — an 8-bit linear intermediate would
// band in shadows, the reason the live canvas is Rgba16Float).
let source_f16 = srgb_image_to_linear_f16(&Self::load_source_image());
// Create effect processor (using Rgba8Unorm for thumbnail output)
let effect_processor = EffectProcessor::new(device, wgpu::TextureFormat::Rgba8Unorm);
// Effect processor + textures use Rgba16Float linear, matching the live
// pipeline so thumbnails render identically to the on-canvas effect.
let effect_processor = EffectProcessor::new(device, wgpu::TextureFormat::Rgba16Float);
// Create source texture
let source_texture = device.create_texture(&wgpu::TextureDescriptor {
@ -55,12 +93,12 @@ impl EffectThumbnailGenerator {
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
// Upload source image data
// Upload source image data (Rgba16Float = 8 bytes/texel).
queue.write_texture(
wgpu::TexelCopyTextureInfo {
texture: &source_texture,
@ -68,10 +106,10 @@ impl EffectThumbnailGenerator {
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
&source_rgba,
&source_f16,
wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(EFFECT_THUMBNAIL_SIZE * 4),
bytes_per_row: Some(EFFECT_THUMBNAIL_SIZE * 8),
rows_per_image: Some(EFFECT_THUMBNAIL_SIZE),
},
wgpu::Extent3d {
@ -94,17 +132,15 @@ impl EffectThumbnailGenerator {
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::COPY_SRC,
view_formats: &[],
});
let dest_view = dest_texture.create_view(&wgpu::TextureViewDescriptor::default());
// Create readback buffer
let _buffer_size = (EFFECT_THUMBNAIL_SIZE * EFFECT_THUMBNAIL_SIZE * 4) as u64;
// Align to 256 bytes for wgpu requirements
let aligned_bytes_per_row = ((EFFECT_THUMBNAIL_SIZE * 4 + 255) / 256) * 256;
// Create readback buffer (Rgba16Float = 8 bytes/texel, rows 256-aligned).
let aligned_bytes_per_row = ((EFFECT_THUMBNAIL_SIZE * 8 + 255) / 256) * 256;
let readback_buffer = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("effect_thumbnail_readback"),
size: (aligned_bytes_per_row * EFFECT_THUMBNAIL_SIZE) as u64,
@ -248,8 +284,8 @@ impl EffectThumbnailGenerator {
return None;
}
// Copy result to readback buffer
let aligned_bytes_per_row = ((EFFECT_THUMBNAIL_SIZE * 4 + 255) / 256) * 256;
// Copy result to readback buffer (Rgba16Float = 8 bytes/texel).
let aligned_bytes_per_row = ((EFFECT_THUMBNAIL_SIZE * 8 + 255) / 256) * 256;
encoder.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo {
texture: &self.dest_texture,
@ -291,20 +327,21 @@ impl EffectThumbnailGenerator {
return None;
}
// Copy data from mapped buffer (handling row alignment)
// De-stride the linear-f16 result (drop the 256-byte row padding).
let data = buffer_slice.get_mapped_range();
let mut rgba = Vec::with_capacity((EFFECT_THUMBNAIL_SIZE * EFFECT_THUMBNAIL_SIZE * 4) as usize);
let row_tight = (EFFECT_THUMBNAIL_SIZE * 8) as usize;
let mut f16_rgba = Vec::with_capacity(row_tight * EFFECT_THUMBNAIL_SIZE as usize);
for row in 0..EFFECT_THUMBNAIL_SIZE {
let row_start = (row * aligned_bytes_per_row) as usize;
let row_end = row_start + (EFFECT_THUMBNAIL_SIZE * 4) as usize;
rgba.extend_from_slice(&data[row_start..row_end]);
f16_rgba.extend_from_slice(&data[row_start..row_start + row_tight]);
}
drop(data);
self.readback_buffer.unmap();
Some(rgba)
// Result is linear f16 (the effect ran in linear light); re-encode to
// sRGB-u8 for display, mirroring the live pipeline's linear→sRGB output.
Some(linear_f16_to_srgb_image(&f16_rgba))
}
/// Get all effect IDs that have pending thumbnail requests

78
lightningbeam-ui/lightningbeam-editor/src/export/cpu_yuv_converter.rs
View File

@ -13,6 +13,11 @@ use ffmpeg_next as ffmpeg;
pub struct CpuYuvConverter {
width: u32,
height: u32,
/// swscale context + reusable source/dest frames, built once and reused every frame
/// (creating them per call was a measurable per-output-frame export cost).
scaler: ffmpeg::software::scaling::Context,
rgba_frame: ffmpeg::frame::Video,
yuv_frame: ffmpeg::frame::Video,
}
impl CpuYuvConverter {
@ -22,7 +27,20 @@ impl CpuYuvConverter {
/// * `width` - Frame width in pixels
/// * `height` - Frame height in pixels
pub fn new(width: u32, height: u32) -> Result<Self, String> {
Ok(Self { width, height })
// BT.709 (HD) RGBA→YUV420p context, created once.
let scaler = ffmpeg::software::scaling::Context::get(
ffmpeg::format::Pixel::RGBA,
width,
height,
ffmpeg::format::Pixel::YUV420P,
width,
height,
ffmpeg::software::scaling::Flags::BILINEAR,
)
.map_err(|e| format!("Failed to create swscale context: {}", e))?;
let rgba_frame = ffmpeg::frame::Video::new(ffmpeg::format::Pixel::RGBA, width, height);
let yuv_frame = ffmpeg::frame::Video::new(ffmpeg::format::Pixel::YUV420P, width, height);
Ok(Self { width, height, scaler, rgba_frame, yuv_frame })
}
/// Convert RGBA data to YUV420p planes
@ -40,7 +58,7 @@ impl CpuYuvConverter {
///
/// # Panics
/// Panics if rgba_data length doesn't match width * height * 4
pub fn convert(&self, rgba_data: &[u8]) -> Result<(Vec<u8>, Vec<u8>, Vec<u8>), String> {
pub fn convert(&mut self, rgba_data: &[u8]) -> Result<(Vec<u8>, Vec<u8>, Vec<u8>), String> {
let expected_size = (self.width * self.height * 4) as usize;
assert_eq!(
rgba_data.len(),
@ -50,51 +68,17 @@ impl CpuYuvConverter {
rgba_data.len()
);
// Create source RGBA frame
let mut rgba_frame = ffmpeg::frame::Video::new(
ffmpeg::format::Pixel::RGBA,
self.width,
self.height,
);
// Copy RGBA data into source frame
// ffmpeg-next provides mutable access to the frame data
let frame_data = rgba_frame.data_mut(0);
frame_data.copy_from_slice(rgba_data);
// Create destination YUV420p frame
let mut yuv_frame = ffmpeg::frame::Video::new(
ffmpeg::format::Pixel::YUV420P,
self.width,
self.height,
);
// Create swscale context for RGBA→YUV420p conversion
// Uses BT.709 color matrix (HD standard)
let mut scaler = ffmpeg::software::scaling::Context::get(
ffmpeg::format::Pixel::RGBA,
self.width,
self.height,
ffmpeg::format::Pixel::YUV420P,
self.width,
self.height,
ffmpeg::software::scaling::Flags::BILINEAR,
)
.map_err(|e| format!("Failed to create swscale context: {}", e))?;
// Perform the conversion (SIMD-optimized)
scaler
.run(&rgba_frame, &mut yuv_frame)
// Copy RGBA into the reused source frame, run the reused scaler into the reused
// dest frame (SIMD-optimized), then extract planes.
self.rgba_frame.data_mut(0).copy_from_slice(rgba_data);
self.scaler
.run(&self.rgba_frame, &mut self.yuv_frame)
.map_err(|e| format!("swscale conversion failed: {}", e))?;
// Extract planar YUV data
// YUV420p has 3 planes:
// - Y: full resolution (width × height)
// - U: quarter resolution (width/2 × height/2)
// - V: quarter resolution (width/2 × height/2)
let y_plane = yuv_frame.data(0).to_vec();
let u_plane = yuv_frame.data(1).to_vec();
let v_plane = yuv_frame.data(2).to_vec();
// YUV420p planes: Y full-res, U/V quarter-res (2×2 subsampled).
let y_plane = self.yuv_frame.data(0).to_vec();
let u_plane = self.yuv_frame.data(1).to_vec();
let v_plane = self.yuv_frame.data(2).to_vec();
Ok((y_plane, u_plane, v_plane))
}
@ -112,7 +96,7 @@ mod tests {
#[test]
fn test_conversion_output_sizes() {
let converter = CpuYuvConverter::new(1920, 1080).unwrap();
let mut converter = CpuYuvConverter::new(1920, 1080).unwrap();
// Create dummy RGBA data (all black)
let rgba_data = vec![0u8; 1920 * 1080 * 4];
@ -133,7 +117,7 @@ mod tests {
#[test]
#[should_panic(expected = "RGBA data size mismatch")]
fn test_wrong_input_size_panics() {
let converter = CpuYuvConverter::new(1920, 1080).unwrap();
let mut converter = CpuYuvConverter::new(1920, 1080).unwrap();
// Wrong size input
let rgba_data = vec![0u8; 1000];

216
lightningbeam-ui/lightningbeam-editor/src/export/mod.rs
View File

@ -113,10 +113,10 @@ struct ParallelExportState {
video_progress_rx: Receiver<ExportProgress>,
/// Audio progress channel
audio_progress_rx: Receiver<ExportProgress>,
/// Video encoder thread handle
video_thread: std::thread::JoinHandle<()>,
/// Audio export thread handle
audio_thread: std::thread::JoinHandle<()>,
/// Video encoder thread handle (taken when the mux thread is spawned).
video_thread: Option<std::thread::JoinHandle<()>>,
/// Audio export thread handle (taken when the mux thread is spawned).
audio_thread: Option<std::thread::JoinHandle<()>>,
/// Temporary video file path
temp_video_path: PathBuf,
/// Temporary audio file path
@ -127,6 +127,9 @@ struct ParallelExportState {
video_progress: Option<ExportProgress>,
/// Latest audio progress
audio_progress: Option<ExportProgress>,
/// Result channel for the background mux. `Some` once muxing has started; the
/// mux runs off the UI thread so the app stays responsive during finalization.
mux_rx: Option<Receiver<Result<(), String>>>,
}
impl ExportOrchestrator {
@ -220,47 +223,33 @@ impl ExportOrchestrator {
parallel.audio_progress = Some(progress);
}
// Check if both are complete
let video_complete = matches!(parallel.video_progress, Some(ExportProgress::Complete { .. }));
let audio_complete = matches!(parallel.audio_progress, Some(ExportProgress::Complete { .. }));
if video_complete && audio_complete {
println!("🎬🎵 [PARALLEL] Both video and audio complete, starting mux");
// Take parallel state to extract file paths
let parallel_state = self.parallel_export.take().unwrap();
// Wait for threads to finish
parallel_state.video_thread.join().ok();
parallel_state.audio_thread.join().ok();
// Start muxing
match Self::mux_video_and_audio(
&parallel_state.temp_video_path,
&parallel_state.temp_audio_path,
&parallel_state.final_output_path,
) {
Ok(()) => {
// If a background mux is already running, poll it without blocking the UI.
if parallel.mux_rx.is_some() {
match parallel.mux_rx.as_ref().unwrap().try_recv() {
Ok(Ok(())) => {
println!("✅ [MUX] Muxing complete, cleaning up temp files");
// Clean up temp files
std::fs::remove_file(&parallel_state.temp_video_path).ok();
std::fs::remove_file(&parallel_state.temp_audio_path).ok();
return Some(ExportProgress::Complete {
output_path: parallel_state.final_output_path,
});
let state = self.parallel_export.take().unwrap();
std::fs::remove_file(&state.temp_video_path).ok();
std::fs::remove_file(&state.temp_audio_path).ok();
return Some(ExportProgress::Complete { output_path: state.final_output_path });
}
Err(err) => {
Ok(Err(err)) => {
println!("❌ [MUX] Muxing failed: {}", err);
return Some(ExportProgress::Error {
message: format!("Muxing failed: {}", err),
});
self.parallel_export = None;
return Some(ExportProgress::Error { message: format!("Muxing failed: {}", err) });
}
Err(std::sync::mpsc::TryRecvError::Empty) => {
// Still muxing — keep the UI responsive and show finalizing state.
return Some(ExportProgress::Finalizing);
}
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
self.parallel_export = None;
return Some(ExportProgress::Error { message: "Mux thread terminated unexpectedly".to_string() });
}
}
}
// Check for errors
// Check for errors before completion.
if let Some(ExportProgress::Error { ref message }) = parallel.video_progress {
return Some(ExportProgress::Error { message: format!("Video: {}", message) });
}
@ -268,6 +257,30 @@ impl ExportOrchestrator {
return Some(ExportProgress::Error { message: format!("Audio: {}", message) });
}
// Both streams done → spawn the mux on a background thread (the previous
// implementation muxed synchronously here on the UI thread, which froze the
// app for the whole re-mux pass after progress already hit 100%).
let video_complete = matches!(parallel.video_progress, Some(ExportProgress::Complete { .. }));
let audio_complete = matches!(parallel.audio_progress, Some(ExportProgress::Complete { .. }));
if video_complete && audio_complete {
println!("🎬🎵 [PARALLEL] Both video and audio complete, starting background mux");
let video_thread = parallel.video_thread.take();
let audio_thread = parallel.audio_thread.take();
let video_path = parallel.temp_video_path.clone();
let audio_path = parallel.temp_audio_path.clone();
let output_path = parallel.final_output_path.clone();
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
// The export threads have signalled Complete; join is near-instant.
if let Some(t) = video_thread { t.join().ok(); }
if let Some(t) = audio_thread { t.join().ok(); }
let result = Self::mux_video_and_audio(&video_path, &audio_path, &output_path);
tx.send(result).ok();
});
parallel.mux_rx = Some(rx);
return Some(ExportProgress::Finalizing);
}
// Return combined progress
match (&parallel.video_progress, &parallel.audio_progress) {
(Some(ExportProgress::FrameRendered { frame, total }), _) => {
@ -371,90 +384,90 @@ impl ExportOrchestrator {
println!("🎵 [MUX] Audio stream - Input TB: {}/{}, Output TB: {}/{}",
audio_input_tb.0, audio_input_tb.1, audio_output_tb.0, audio_output_tb.1);
// Collect all packets with their stream info and timestamps
let mut video_packets = Vec::new();
for (stream, packet) in video_input.packets() {
if stream.index() == video_stream_index {
video_packets.push(packet);
// Stream-merge the two inputs by PTS, writing each packet as it's read —
// O(1) memory (one pending packet per stream) instead of collecting every
// packet first, so muxing a long export never grows unbounded.
let video_idx = video_stream_index;
let audio_idx = audio_stream_index;
let mut v_iter = video_input.packets();
let mut a_iter = audio_input.packets();
// Pull the next packet belonging to the desired stream from each input.
let mut next_video = move || -> Option<ffmpeg::Packet> {
loop {
match v_iter.next() {
Some((stream, packet)) => {
if stream.index() == video_idx {
return Some(packet);
}
}
let mut audio_packets = Vec::new();
for (stream, packet) in audio_input.packets() {
if stream.index() == audio_stream_index {
audio_packets.push(packet);
None => return None,
}
}
println!("🎬 [MUX] Collected {} video packets, {} audio packets",
video_packets.len(), audio_packets.len());
// Report first and last timestamps
if !video_packets.is_empty() {
println!("🎬 [MUX] Video PTS range: {} to {}",
video_packets[0].pts().unwrap_or(0),
video_packets[video_packets.len()-1].pts().unwrap_or(0));
};
let mut next_audio = move || -> Option<ffmpeg::Packet> {
loop {
match a_iter.next() {
Some((stream, packet)) => {
if stream.index() == audio_idx {
return Some(packet);
}
if !audio_packets.is_empty() {
println!("🎵 [MUX] Audio PTS range: {} to {}",
audio_packets[0].pts().unwrap_or(0),
audio_packets[audio_packets.len()-1].pts().unwrap_or(0));
}
None => return None,
}
}
};
// Interleave packets by comparing timestamps in a common time base (use microseconds)
let mut v_idx = 0;
let mut a_idx = 0;
let mut interleave_log_count = 0;
let mut pending_v = next_video();
let mut pending_a = next_audio();
let mut v_count = 0usize;
let mut a_count = 0usize;
let mut log_count = 0;
while v_idx < video_packets.len() || a_idx < audio_packets.len() {
let write_video = if v_idx >= video_packets.len() {
false // No more video
} else if a_idx >= audio_packets.len() {
true // No more audio, write video
} else {
// Compare timestamps - convert both to microseconds
let v_pts = video_packets[v_idx].pts().unwrap_or(0);
let a_pts = audio_packets[a_idx].pts().unwrap_or(0);
// Convert to microseconds: pts * 1000000 * tb.num / tb.den
let v_us = v_pts * 1_000_000 * video_input_tb.0 as i64 / video_input_tb.1 as i64;
let a_us = a_pts * 1_000_000 * audio_input_tb.0 as i64 / audio_input_tb.1 as i64;
v_us <= a_us // Write video if it comes before or at same time as audio
loop {
// Write whichever pending packet has the earlier PTS (in a common
// microsecond base); when one stream is exhausted, drain the other.
let write_video = match (&pending_v, &pending_a) {
(None, None) => break,
(Some(_), None) => true,
(None, Some(_)) => false,
(Some(v), Some(a)) => {
let v_us = v.pts().unwrap_or(0) * 1_000_000 * video_input_tb.0 as i64
/ video_input_tb.1 as i64;
let a_us = a.pts().unwrap_or(0) * 1_000_000 * audio_input_tb.0 as i64
/ audio_input_tb.1 as i64;
v_us <= a_us
}
};
if write_video {
let mut packet = video_packets[v_idx].clone();
let mut packet = pending_v.take().unwrap();
packet.set_stream(0);
packet.rescale_ts(video_input_tb, video_output_tb);
if interleave_log_count < 10 {
println!("🎬 [MUX] Writing V packet {} - PTS={:?}, DTS={:?}, Duration={:?}",
v_idx, packet.pts(), packet.dts(), packet.duration());
interleave_log_count += 1;
if log_count < 10 {
println!("🎬 [MUX] Writing V packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
log_count += 1;
}
packet.write_interleaved(&mut output)
.map_err(|e| format!("Failed to write video packet: {}", e))?;
v_idx += 1;
v_count += 1;
pending_v = next_video();
} else {
let mut packet = audio_packets[a_idx].clone();
let mut packet = pending_a.take().unwrap();
packet.set_stream(1);
packet.rescale_ts(audio_input_tb, audio_output_tb);
if interleave_log_count < 10 {
println!("🎵 [MUX] Writing A packet {} - PTS={:?}, DTS={:?}, Duration={:?}",
a_idx, packet.pts(), packet.dts(), packet.duration());
interleave_log_count += 1;
if log_count < 10 {
println!("🎵 [MUX] Writing A packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
log_count += 1;
}
packet.write_interleaved(&mut output)
.map_err(|e| format!("Failed to write audio packet: {}", e))?;
a_idx += 1;
a_count += 1;
pending_a = next_audio();
}
}
println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_idx, a_idx);
println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_count, a_count);
// Write trailer
output.write_trailer().map_err(|e| format!("Failed to write trailer: {}", e))?;
@ -515,6 +528,7 @@ impl ExportOrchestrator {
image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<bool, String> {
if self.cancel_flag.load(Ordering::Relaxed) {
self.image_state = None;
@ -562,6 +576,7 @@ impl ExportOrchestrator {
output_view,
floating_selection,
state.settings.allow_transparency,
raster_store,
)?;
queue.submit(Some(encoder.finish()));
@ -979,13 +994,14 @@ impl ExportOrchestrator {
self.parallel_export = Some(ParallelExportState {
video_progress_rx,
audio_progress_rx,
video_thread,
audio_thread,
video_thread: Some(video_thread),
audio_thread: Some(audio_thread),
temp_video_path,
temp_audio_path,
final_output_path: output_path,
video_progress: None,
audio_progress: None,
mux_rx: None,
});
println!("🎬🎵 [PARALLEL EXPORT] Both threads spawned, ready for frames");
@ -1015,6 +1031,7 @@ impl ExportOrchestrator {
renderer: &mut vello::Renderer,
image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<bool, String> {
use std::time::Instant;
@ -1112,6 +1129,7 @@ impl ExportOrchestrator {
gpu_resources, &acquired.rgba_texture_view,
None, // No floating selection during video export
false, // Video export is never transparent
raster_store,
)?;
let render_end = Instant::now();

123
lightningbeam-ui/lightningbeam-editor/src/export/video_exporter.rs
View File

@ -191,7 +191,9 @@ impl ExportGpuResources {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("linear_to_srgb_shader"),
source: wgpu::ShaderSource::Wgsl(LINEAR_TO_SRGB_SHADER.into()),
source: wgpu::ShaderSource::Wgsl(
format!("{}\n{}", lightningbeam_core::gpu::COLOR_WGSL, LINEAR_TO_SRGB_SHADER).into(),
),
});
let linear_to_srgb_pipeline = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
@ -310,32 +312,24 @@ fn vs_main(@builtin(vertex_index) vertex_index: u32) -> VertexOutput {
return out;
}
// Linear to sRGB color space conversion (per channel)
fn linear_to_srgb_channel(c: f32) -> f32 {
return select(
1.055 * pow(c, 1.0 / 2.4) - 0.055,
c * 12.92,
c <= 0.0031308
);
}
fn linear_to_srgb(color: vec3<f32>) -> vec3<f32> {
return vec3<f32>(
linear_to_srgb_channel(color.r),
linear_to_srgb_channel(color.g),
linear_to_srgb_channel(color.b)
);
}
// linear_to_srgb / linear_to_srgb_channel are provided by the prepended
// COLOR_WGSL prelude (see the create_shader_module call site).
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let src = textureSample(source_tex, source_sampler, in.uv);
// Convert linear HDR to sRGB
let srgb = linear_to_srgb(src.rgb);
// The compositor accumulates PREMULTIPLIED linear color. Unpremultiply
// before the sRGB OETF (srgb(rgb*a) != srgb(rgb)*a) and emit STRAIGHT
// alpha, which is what PNG export / the readback path expect. For opaque
// pixels (a == 1, the normal video case) this is an exact identity.
let a = src.a;
let straight = select(src.rgb / a, vec3<f32>(0.0), a <= 0.0);
// Alpha stays unchanged
return vec4<f32>(srgb, src.a);
// Convert linear HDR to sRGB
let srgb = linear_to_srgb(straight);
return vec4<f32>(srgb, a);
}
"#;
@ -522,12 +516,27 @@ pub fn setup_video_encoder(
encoder.set_bit_rate((bitrate_kbps * 1000) as usize);
encoder.set_gop(framerate as u32); // 1 second GOP
// Tag the color metadata so players interpret the YUV correctly. Our
// RGB→YUV conversion uses the BT.709 matrix with FULL-range (0–255) luma
// and no transfer applied to the already-sRGB-encoded RGB. Tagging this
// as full-range BT.709 (matrix/primaries/transfer) prevents the level/
// hue shift that occurs when a player assumes limited-range or BT.601.
// colorspace (matrix) and range have safe setters; primaries and trc are
// generic AVCodecContext options set via the open dictionary below.
encoder.set_colorspace(ffmpeg::color::Space::BT709);
encoder.set_color_range(ffmpeg::color::Range::JPEG); // full range
println!("📐 Video dimensions: {}×{} (aligned to {}×{} for H.264)",
width, height, aligned_width, aligned_height);
// Open encoder with codec (like working MP3 export)
// Open encoder with codec (like working MP3 export). color_primaries and
// color_trc have no typed setter on the encoder, so pass them as generic
// AVCodecContext options (BT.709) through the open dictionary.
let mut color_opts = ffmpeg::Dictionary::new();
color_opts.set("color_primaries", "bt709");
color_opts.set("color_trc", "bt709");
let encoder = encoder
.open_as(codec)
.open_as_with(codec, color_opts)
.map_err(|e| format!("Failed to open video encoder: {}", e))?;
Ok((encoder, codec))
@ -973,8 +982,18 @@ pub fn render_frame_to_rgba_hdr(
// Set document time to the frame timestamp
document.current_time = timestamp;
// Use identity transform for export (document coordinates = pixel coordinates)
let base_transform = Affine::IDENTITY;
// Scale the document to the export resolution. The core renderer bakes this
// base transform into every layer (vector scenes, raster and video layer
// transforms), so the whole stage scales up/down to fill the output. When the
// export size matches the document this is the identity.
let base_transform = if document.width > 0.0 && document.height > 0.0 {
Affine::scale_non_uniform(
width as f64 / document.width,
height as f64 / document.height,
)
} else {
Affine::IDENTITY
};
// Render document for compositing (returns per-layer scenes)
let composite_result = render_document_for_compositing(
@ -1160,6 +1179,39 @@ pub fn render_frame_to_rgba_hdr(
///
/// # Returns
/// Command encoder ready for submission (caller submits via ReadbackPipeline)
/// Fault in raster keyframe pixels needed to composite the document at its current
/// time, decoding them from the project `.beam` container via `raster_store`.
///
/// Mutates the document in place: for every raster layer's active keyframe whose
/// `raw_pixels` are empty, loads + sets them (and marks `texture_dirty`). A no-op
/// when `raster_store` is `None`/unsaved or everything is already resident.
fn fault_in_raster_for_frame(
document: &mut Document,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) {
let store = match raster_store {
Some(s) if s.has_path() => s,
_ => return,
};
let now = document.current_time;
for layer in document.all_layers_mut() {
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
// Resolve the active keyframe id at the current time, then fault it in.
let kf_id = match rl.keyframe_at(now) {
Some(kf) if kf.raw_pixels.is_empty() && kf.needs_fault_in => kf.id,
_ => continue,
};
if let Some(kf) = rl.keyframes.iter_mut().find(|kf| kf.id == kf_id) {
if let Some(pixels) = store.load_pixels(kf_id) {
kf.raw_pixels = pixels;
kf.texture_dirty = true;
}
kf.needs_fault_in = false;
}
}
}
}
pub fn render_frame_to_gpu_rgba(
document: &mut Document,
timestamp: f64,
@ -1174,14 +1226,31 @@ pub fn render_frame_to_gpu_rgba(
rgba_texture_view: &wgpu::TextureView,
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
allow_transparency: bool,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<wgpu::CommandEncoder, String> {
use vello::kurbo::Affine;
// Set document time to the frame timestamp
document.current_time = timestamp;
// Use identity transform for export (document coordinates = pixel coordinates)
let base_transform = Affine::IDENTITY;
// Fault in raster keyframe pixels for this frame (Phase 3 paging). Offline
// export renders synchronously with no "next frame", so unlike the live canvas
// we must page the pixels in here, before compositing. Cheap no-op when every
// keyframe is already resident or when the document is unsaved (no store path).
fault_in_raster_for_frame(document, raster_store);
// Scale the document to the export resolution. The core renderer bakes this
// base transform into every layer (vector scenes, raster and video layer
// transforms), so the whole stage scales up/down to fill the output. When the
// export size matches the document this is the identity.
let base_transform = if document.width > 0.0 && document.height > 0.0 {
Affine::scale_non_uniform(
width as f64 / document.width,
height as f64 / document.height,
)
} else {
Affine::IDENTITY
};
// Render document for compositing (returns per-layer scenes)
let composite_result = render_document_for_compositing(

659
lightningbeam-ui/lightningbeam-editor/src/gpu_brush.rs
View File

@ -24,24 +24,94 @@ use lightningbeam_core::brush_engine::GpuDab;
// Colour-space helpers
// ---------------------------------------------------------------------------
/// Decode one sRGB-encoded byte to linear float [0, 1].
fn srgb_to_linear(c: f32) -> f32 {
if c <= 0.04045 {
c / 12.92
} else {
((c + 0.055) / 1.055).powf(2.4)
use lightningbeam_core::gpu::srgb_to_linear;
// Lookup tables that keep the per-pixel `powf`/f16 conversions out of the canvas
// upload/readback loops. Doing the sRGB transfer per pixel was ~110ms for an
// 800x600 readback in a debug build; precomputing it once turns each channel
// into a table index.
/// Upload encode: sRGB byte → linear f16 little-endian bytes (for RGB channels).
fn srgb_to_linear_f16_lut() -> &'static [[u8; 2]; 256] {
static LUT: std::sync::OnceLock<[[u8; 2]; 256]> = std::sync::OnceLock::new();
LUT.get_or_init(|| {
let mut lut = [[0u8; 2]; 256];
for (i, out) in lut.iter_mut().enumerate() {
*out = half::f16::from_f32(srgb_to_linear(i as f32 / 255.0)).to_le_bytes();
}
lut
})
}
/// Upload encode: byte → linear f16 little-endian bytes (for the alpha channel,
/// which is not gamma-encoded).
fn linear_f16_lut() -> &'static [[u8; 2]; 256] {
static LUT: std::sync::OnceLock<[[u8; 2]; 256]> = std::sync::OnceLock::new();
LUT.get_or_init(|| {
let mut lut = [[0u8; 2]; 256];
for (i, out) in lut.iter_mut().enumerate() {
*out = half::f16::from_f32(i as f32 / 255.0).to_le_bytes();
}
lut
})
}
// ---------------------------------------------------------------------------
// Incremental ping-pong sync
// ---------------------------------------------------------------------------
/// Tile size (px) for incremental canvas sync copies between the ping-pong
/// textures. The brush keeps both textures identical; each frame only the tiles
/// touched by that frame's dabs are copied, so this bounds both the wasted bytes
/// per touched tile and the number of copy regions.
const SYNC_TILE: u32 = 128;
/// Coalesced copy rectangles (x, y, w, h) covering the tiles touched by `dabs`,
/// clamped to the canvas. Adjacent tiles in a row are merged into one rectangle
/// to keep the number of `copy_texture_to_texture` calls small.
fn dirty_tile_rects(dabs: &[GpuDab], canvas_w: u32, canvas_h: u32) -> Vec<(u32, u32, u32, u32)> {
if canvas_w == 0 || canvas_h == 0 || dabs.is_empty() {
return Vec::new();
}
let tiles_x = canvas_w.div_ceil(SYNC_TILE);
let tiles_y = canvas_h.div_ceil(SYNC_TILE);
let mut mask = vec![false; (tiles_x * tiles_y) as usize];
for d in dabs {
let r = d.radius + 1.0;
// Dab pixel bbox clamped to the canvas, then mapped to tile indices.
let px0 = (d.x - r).floor().clamp(0.0, (canvas_w - 1) as f32) as u32;
let py0 = (d.y - r).floor().clamp(0.0, (canvas_h - 1) as f32) as u32;
let px1 = (d.x + r).ceil().clamp(0.0, (canvas_w - 1) as f32) as u32;
let py1 = (d.y + r).ceil().clamp(0.0, (canvas_h - 1) as f32) as u32;
for ty in (py0 / SYNC_TILE)..=(py1 / SYNC_TILE) {
for tx in (px0 / SYNC_TILE)..=(px1 / SYNC_TILE) {
mask[(ty * tiles_x + tx) as usize] = true;
}
}
}
/// Encode one linear float [0, 1] to an sRGB-encoded byte.
fn linear_to_srgb_byte(c: u8) -> u8 {
let f = c as f32 / 255.0;
let encoded = if f <= 0.0031308 {
f * 12.92
// Merge horizontal runs of set tiles in each tile-row into one rectangle.
let mut rects = Vec::new();
for ty in 0..tiles_y {
let mut tx = 0;
while tx < tiles_x {
if mask[(ty * tiles_x + tx) as usize] {
let run_start = tx;
while tx < tiles_x && mask[(ty * tiles_x + tx) as usize] {
tx += 1;
}
let x = run_start * SYNC_TILE;
let y = ty * SYNC_TILE;
let w = (tx * SYNC_TILE).min(canvas_w) - x;
let h = ((ty + 1) * SYNC_TILE).min(canvas_h) - y;
rects.push((x, y, w, h));
} else {
1.055 * f.powf(1.0 / 2.4) - 0.055
};
(encoded * 255.0 + 0.5) as u8
tx += 1;
}
}
}
rects
}
// ---------------------------------------------------------------------------
@ -68,7 +138,7 @@ impl CanvasPair {
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
usage: wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::STORAGE_BINDING
| wgpu::TextureUsages::COPY_SRC
@ -93,18 +163,24 @@ impl CanvasPair {
/// `pixels` is expected to be **sRGB-encoded premultiplied** (the format stored
/// in `raw_pixels` / PNG files). The values are decoded to linear premultiplied
/// before being written to the canvas, which operates entirely in linear space.
/// The canvas is `Rgba16Float`, so linear values are stored at 16-bit float
/// precision — storing linear light in 8 bits would band badly in shadows.
pub fn upload(&self, queue: &wgpu::Queue, pixels: &[u8]) {
// Decode sRGB-premultiplied → linear premultiplied for the GPU canvas.
// Decode sRGB-premultiplied → linear premultiplied f16 for the GPU canvas.
// LUT-driven so there is no per-pixel powf / float conversion.
let rgb_lut = srgb_to_linear_f16_lut();
let a_lut = linear_f16_lut();
let linear: Vec<u8> = pixels.chunks_exact(4).flat_map(|p| {
let r = (srgb_to_linear(p[0] as f32 / 255.0) * 255.0 + 0.5) as u8;
let g = (srgb_to_linear(p[1] as f32 / 255.0) * 255.0 + 0.5) as u8;
let b = (srgb_to_linear(p[2] as f32 / 255.0) * 255.0 + 0.5) as u8;
[r, g, b, p[3]]
let r = rgb_lut[p[0] as usize];
let g = rgb_lut[p[1] as usize];
let b = rgb_lut[p[2] as usize];
let a = a_lut[p[3] as usize];
[r[0], r[1], g[0], g[1], b[0], b[1], a[0], a[1]]
}).collect();
let layout = wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(self.width * 4),
bytes_per_row: Some(self.width * 8), // Rgba16Float = 8 bytes/texel
rows_per_image: Some(self.height),
};
let extent = wgpu::Extent3d {
@ -204,7 +280,7 @@ impl RasterTransformPipeline {
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
@ -384,7 +460,7 @@ impl WarpApplyPipeline {
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
@ -596,7 +672,7 @@ impl LiquifyBrushPipeline {
// Gradient-fill pipeline
// ---------------------------------------------------------------------------
/// One gradient stop on the GPU side. Colors are linear straight-alpha [0..1].
/// One gradient stop on the GPU side. Colors are sRGB straight-alpha [0..1].
/// Must be 32 bytes (8 × f32) to match `GradientStop` in `gradient_fill.wgsl`.
#[repr(C)]
#[derive(Clone, Copy, bytemuck::Pod, bytemuck::Zeroable)]
@ -611,13 +687,18 @@ pub struct GpuGradientStop {
impl GpuGradientStop {
/// Construct from sRGB u8 bytes (as stored in `ShapeColor`).
/// RGB is converted to linear; alpha is kept linear (not gamma-encoded).
///
/// Stops are kept in sRGB (gamma) space so the shader interpolates between
/// them in gamma space — matching the CPU raster path (`Gradient::eval`) and
/// the vector/peniko path, and the gamma-space gradients users expect from
/// tools like Photoshop/Flash. The shader converts the interpolated color to
/// linear before compositing into the linear canvas.
pub fn from_srgb_u8(position: f32, r: u8, g: u8, b: u8, a: u8) -> Self {
Self {
position,
r: srgb_to_linear(r as f32 / 255.0),
g: srgb_to_linear(g as f32 / 255.0),
b: srgb_to_linear(b as f32 / 255.0),
r: r as f32 / 255.0,
g: g as f32 / 255.0,
b: b as f32 / 255.0,
a: a as f32 / 255.0,
_pad: [0.0; 3],
}
@ -654,7 +735,7 @@ impl GradientFillPipeline {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("gradient_fill_shader"),
source: wgpu::ShaderSource::Wgsl(
include_str!("panes/shaders/gradient_fill.wgsl").into(),
color_wgsl(include_str!("panes/shaders/gradient_fill.wgsl")).into(),
),
});
@ -701,7 +782,7 @@ impl GradientFillPipeline {
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
@ -780,23 +861,18 @@ impl GradientFillPipeline {
/// Compute pipeline: composites the scratch buffer C over the source A → output B.
///
/// Binding layout (see `alpha_composite.wgsl`):
/// 0 = tex_a (texture_2d<f32>, Rgba8Unorm, sampled, not filterable)
/// 1 = tex_c (texture_2d<f32>, Rgba8Unorm, sampled, not filterable)
/// 0 = tex_a (texture_2d<f32>, Rgba16Float, sampled, not filterable)
/// 1 = tex_c (texture_2d<f32>, Rgba16Float, sampled, not filterable)
/// 2 = tex_b (texture_storage_2d<rgba8unorm, write>)
struct AlphaCompositePipeline {
pipeline: wgpu::ComputePipeline,
bg_layout: wgpu::BindGroupLayout,
/// Prepend the shared WGSL sRGB color functions ([`COLOR_WGSL`]) to a shader
/// source so the OETF/EOTF live in exactly one place.
fn color_wgsl(shader_src: &str) -> String {
format!("{}\n{}", lightningbeam_core::gpu::COLOR_WGSL, shader_src)
}
impl AlphaCompositePipeline {
fn new(device: &wgpu::Device) -> Self {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("alpha_composite_shader"),
source: wgpu::ShaderSource::Wgsl(
include_str!("panes/shaders/alpha_composite.wgsl").into(),
),
});
let sampled_entry = |binding: u32| wgpu::BindGroupLayoutEntry {
/// A compute `texture_2d<f32>` sampled binding (non-filterable).
fn sampled_tex_entry(binding: u32) -> wgpu::BindGroupLayoutEntry {
wgpu::BindGroupLayoutEntry {
binding,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Texture {
@ -805,41 +881,135 @@ impl AlphaCompositePipeline {
multisampled: false,
},
count: None,
};
let bg_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("alpha_composite_bgl"),
entries: &[
sampled_entry(0), // tex_a
sampled_entry(1), // tex_c
}
}
/// A compute write-only storage-texture binding of the given format.
fn storage_tex_entry(binding: u32, format: wgpu::TextureFormat) -> wgpu::BindGroupLayoutEntry {
wgpu::BindGroupLayoutEntry {
binding: 2,
binding,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: wgpu::TextureFormat::Rgba8Unorm,
format,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
},
],
}
}
/// Build a compute pipeline + matching bind-group layout from WGSL source and
/// layout entries (entry point `main`). Collapses the otherwise-identical
/// pipeline/layout construction boilerplate shared by the compute pipelines.
fn build_compute_pipeline(
device: &wgpu::Device,
label: &str,
shader_src: &str,
entries: &[wgpu::BindGroupLayoutEntry],
) -> (wgpu::ComputePipeline, wgpu::BindGroupLayout) {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some(label),
source: wgpu::ShaderSource::Wgsl(shader_src.into()),
});
let bg_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some(label),
entries,
});
let layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("alpha_composite_layout"),
label: Some(label),
bind_group_layouts: &[&bg_layout],
push_constant_ranges: &[],
});
let pipeline = device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor {
label: Some("alpha_composite_pipeline"),
label: Some(label),
layout: Some(&layout),
module: &shader,
entry_point: Some("main"),
compilation_options: Default::default(),
cache: None,
});
(pipeline, bg_layout)
}
struct AlphaCompositePipeline {
pipeline: wgpu::ComputePipeline,
bg_layout: wgpu::BindGroupLayout,
}
impl AlphaCompositePipeline {
fn new(device: &wgpu::Device) -> Self {
let (pipeline, bg_layout) = build_compute_pipeline(
device,
"alpha_composite",
include_str!("panes/shaders/alpha_composite.wgsl"),
&[
sampled_tex_entry(0), // tex_a
sampled_tex_entry(1), // tex_c
storage_tex_entry(2, wgpu::TextureFormat::Rgba16Float), // tex_b (out)
],
);
Self { pipeline, bg_layout }
}
}
/// Compute pipeline that converts the linear-premultiplied `Rgba16Float` canvas
/// into an `Rgba8Unorm` sRGB-premultiplied texture for fast readback.
struct ReadbackSrgbPipeline {
pipeline: wgpu::ComputePipeline,
bg_layout: wgpu::BindGroupLayout,
}
impl ReadbackSrgbPipeline {
fn new(device: &wgpu::Device) -> Self {
let (pipeline, bg_layout) = build_compute_pipeline(
device,
"canvas_readback_srgb",
&color_wgsl(include_str!("panes/shaders/canvas_readback_srgb.wgsl")),
&[
sampled_tex_entry(0), // src (linear)
storage_tex_entry(1, wgpu::TextureFormat::Rgba8Unorm), // dst (sRGB)
],
);
Self { pipeline, bg_layout }
}
}
/// Reusable scratch for `readback_canvas`: the Rgba8Unorm conversion target plus
/// its MAP_READ staging buffer, kept across calls and rebuilt only on size change.
struct ReadbackScratch {
width: u32,
height: u32,
view: wgpu::TextureView,
tex: wgpu::Texture,
staging: wgpu::Buffer,
/// 256-aligned bytes-per-row of the staging buffer (Rgba8 = 4 B/texel).
bytes_per_row_aligned: u32,
}
impl ReadbackScratch {
fn new(device: &wgpu::Device, width: u32, height: u32) -> Self {
let tex = device.create_texture(&wgpu::TextureDescriptor {
label: Some("canvas_readback_srgb"),
size: wgpu::Extent3d { width, height, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::STORAGE_BINDING | wgpu::TextureUsages::COPY_SRC,
view_formats: &[],
});
let view = tex.create_view(&wgpu::TextureViewDescriptor::default());
let bytes_per_row_aligned = ((width * 4 + 255) / 256) * 256;
let staging = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("canvas_readback_buf"),
size: (bytes_per_row_aligned * height) as u64,
usage: wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
});
Self { width, height, view, tex, staging, bytes_per_row_aligned }
}
}
// GpuBrushEngine
// ---------------------------------------------------------------------------
@ -860,18 +1030,36 @@ pub struct GpuBrushEngine {
/// Lazily created on first unified-tool composite dispatch.
composite_pipeline: Option<AlphaCompositePipeline>,
/// Lazily-created pipeline converting the canvas to sRGB for fast readback.
readback_srgb_pipeline: Option<ReadbackSrgbPipeline>,
/// Reused scratch (texture + staging buffer) for `readback_canvas`, recreated
/// only when the canvas size changes, to avoid per-stroke GPU allocations.
readback_scratch: Option<ReadbackScratch>,
/// Canvas texture pairs keyed by keyframe UUID.
pub canvases: HashMap<Uuid, CanvasPair>,
/// Displacement map buffers keyed by a caller-supplied UUID.
pub displacement_bufs: HashMap<Uuid, DisplacementBuffer>,
/// Persistent `Rgba8Unorm` textures for idle raster layers.
/// Persistent `Rgba16Float` textures for idle raster layers.
///
/// Keyed by keyframe UUID (same ID space as `canvases`). Entries are uploaded
/// once when `RasterKeyframe::texture_dirty` is set, then reused every frame.
/// Separate from `canvases` so tool teardown never accidentally removes them.
pub raster_layer_cache: HashMap<Uuid, CanvasPair>,
/// Recency order for `raster_layer_cache` (least-recent first), bumped on every
/// `ensure_layer_texture` so the visible frames stay at the back. Scrubbing a long
/// timeline would otherwise grow this map without bound (~w·h·16 bytes of VRAM per
/// entry); we evict the oldest once it exceeds `RASTER_LAYER_CACHE_MAX`.
raster_layer_lru: Vec<Uuid>,
/// Low-res proxy textures (one small `CanvasPair` per keyframe), shown while the
/// full-res pixels page in. Keyed by keyframe id; separate from
/// `raster_layer_cache` so a proxy and its full frame never collide. Bounded by
/// its own (generous, since each is tiny) recency LRU.
proxy_layer_cache: HashMap<Uuid, CanvasPair>,
proxy_layer_lru: Vec<Uuid>,
}
/// CPU-side parameters uniform for the compute shader.
@ -889,8 +1077,18 @@ struct DabParams {
}
impl GpuBrushEngine {
/// Max number of idle raster-layer textures kept resident in VRAM. Scrubbing a
/// long timeline would otherwise cache one ~`w·h·16`-byte pair per visited frame
/// without bound; the least-recently-used are evicted past this (re-uploaded on
/// revisit from the faulted-in pixels).
const RASTER_LAYER_CACHE_MAX: usize = 12;
/// Proxies are ~1/100th the VRAM of a full frame, so we can keep many resident
/// for instant cold-scrub display before evicting the least-recently-used.
const RASTER_PROXY_CACHE_MAX: usize = 64;
/// Create the pipeline. Returns `Err` if the device lacks the required
/// storage-texture capability for `Rgba8Unorm`.
/// storage-texture capability for `Rgba16Float`.
pub fn new(device: &wgpu::Device) -> Self {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("brush_dab_shader"),
@ -942,7 +1140,7 @@ impl GpuBrushEngine {
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: wgpu::TextureFormat::Rgba8Unorm,
format: wgpu::TextureFormat::Rgba16Float,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
@ -978,9 +1176,14 @@ impl GpuBrushEngine {
liquify_brush_pipeline: None,
gradient_fill_pipeline: None,
composite_pipeline: None,
readback_srgb_pipeline: None,
readback_scratch: None,
canvases: HashMap::new(),
displacement_bufs: HashMap::new(),
raster_layer_cache: HashMap::new(),
raster_layer_lru: Vec::new(),
proxy_layer_cache: HashMap::new(),
proxy_layer_lru: Vec::new(),
}
}
@ -1022,12 +1225,14 @@ impl GpuBrushEngine {
) {
if dabs.is_empty() { return; }
// Smudge dabs must be applied one at a time so each dab reads the canvas
// state written by the previous dab. Use bbox-only copies (union of current
// and previous dab) to avoid an expensive full-canvas copy per dab.
// render_dabs_batch keeps both ping-pong textures identical after every
// call, so smudge — whose each dab must read the previous dab's output —
// simply dispatches one dab at a time: each per-dab call leaves src fully
// authoritative for the next. Paint/erase dabs are independent within a
// frame (the shader accumulates them over the shared source), so they
// dispatch together as one batch.
let is_smudge = dabs.first().map(|d| d.blend_mode == 2).unwrap_or(false);
if is_smudge {
let mut prev_bbox: Option<(i32, i32, i32, i32)> = None;
for dab in dabs {
let r = dab.radius + 1.0;
let cur_bbox = (
@ -1036,31 +1241,24 @@ impl GpuBrushEngine {
(dab.x + r).ceil() as i32,
(dab.y + r).ceil() as i32,
);
// Expand copy region to include the previous dab's bbox so the
// pixels it wrote are visible as the source for this dab's smudge.
let copy_bbox = match prev_bbox {
Some(pb) => (cur_bbox.0.min(pb.0), cur_bbox.1.min(pb.1),
cur_bbox.2.max(pb.2), cur_bbox.3.max(pb.3)),
None => cur_bbox,
};
self.render_dabs_batch(device, queue, keyframe_id,
std::slice::from_ref(dab), cur_bbox, Some(copy_bbox), canvas_w, canvas_h);
prev_bbox = Some(cur_bbox);
std::slice::from_ref(dab), cur_bbox, canvas_w, canvas_h);
}
} else {
self.render_dabs_batch(device, queue, keyframe_id, dabs, bbox, None, canvas_w, canvas_h);
self.render_dabs_batch(device, queue, keyframe_id, dabs, bbox, canvas_w, canvas_h);
}
}
/// Inner batch dispatch.
/// Dispatch one batch of dabs and keep the ping-pong textures identical.
///
/// `dispatch_bbox` — region dispatched to the compute shader (usually the union of all dab bboxes).
/// `copy_bbox` — region to copy src→dst before dispatch:
/// - `None` → copy the full canvas (required for paint/erase batches so
/// dabs outside the current frame's region are preserved).
/// - `Some(r)` → copy only region `r` (sufficient for sequential smudge dabs
/// because both textures hold identical data outside previously
/// touched regions, so no full copy is needed).
/// Reads `src` and writes the result over `dispatch_bbox` into `dst`, then
/// copies only the tiles the dabs touched back from `dst` to `src` so both
/// textures stay authoritative — no full-canvas copy. The textures start equal
/// (seeded by `upload` at stroke start) and this preserves that invariant, so a
/// single dab per call is enough for smudge's read-after-write dependency.
///
/// `dispatch_bbox` is the region dispatched to the compute shader (the union of
/// the batch's dab bboxes).
fn render_dabs_batch(
&mut self,
device: &wgpu::Device,
@ -1068,7 +1266,6 @@ impl GpuBrushEngine {
keyframe_id: Uuid,
dabs: &[GpuDab],
dispatch_bbox: (i32, i32, i32, i32),
copy_bbox: Option<(i32, i32, i32, i32)>,
canvas_w: u32,
canvas_h: u32,
) {
@ -1088,61 +1285,7 @@ impl GpuBrushEngine {
let bbox_w = x1 - x0;
let bbox_h = y1 - y0;
// Step 1: Copy src→dst.
// For paint/erase batches (copy_bbox = None): copy the ENTIRE canvas so dst
// starts with all previous dabs — a bbox-only copy would lose dabs outside
// this frame's region after swap.
// For smudge (copy_bbox = Some(r)): copy only the union of the current and
// previous dab bboxes. Outside that region both textures hold identical
// data so no full copy is needed.
let mut copy_enc = device.create_command_encoder(
&wgpu::CommandEncoderDescriptor { label: Some("canvas_copy_encoder") },
);
match copy_bbox {
None => {
copy_enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo {
texture: canvas.src(),
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyTextureInfo {
texture: canvas.dst(),
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
wgpu::Extent3d { width: canvas_w, height: canvas_h, depth_or_array_layers: 1 },
);
}
Some(cb) => {
let cx0 = cb.0.max(0) as u32;
let cy0 = cb.1.max(0) as u32;
let cx1 = (cb.2 as u32).min(canvas_w);
let cy1 = (cb.3 as u32).min(canvas_h);
if cx1 > cx0 && cy1 > cy0 {
copy_enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo {
texture: canvas.src(),
mip_level: 0,
origin: wgpu::Origin3d { x: cx0, y: cy0, z: 0 },
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyTextureInfo {
texture: canvas.dst(),
mip_level: 0,
origin: wgpu::Origin3d { x: cx0, y: cy0, z: 0 },
aspect: wgpu::TextureAspect::All,
},
wgpu::Extent3d { width: cx1 - cx0, height: cy1 - cy0, depth_or_array_layers: 1 },
);
}
}
}
queue.submit(Some(copy_enc.finish()));
// Step 2: Upload all dabs as a single storage buffer.
// Step 1: Upload all dabs as a single storage buffer.
let dab_bytes = bytemuck::cast_slice(dabs);
let dab_buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("dab_storage_buf"),
@ -1181,7 +1324,7 @@ impl GpuBrushEngine {
],
});
// Step 3: Single dispatch over the union bounding box.
// Step 2: Single dispatch over the union bounding box.
let mut compute_enc = device.create_command_encoder(
&wgpu::CommandEncoderDescriptor { label: Some("brush_dab_encoder") },
);
@ -1193,52 +1336,98 @@ impl GpuBrushEngine {
pass.set_bind_group(0, &bg, &[]);
pass.dispatch_workgroups(bbox_w.div_ceil(8), bbox_h.div_ceil(8), 1);
}
// Step 3: Sync only the tiles these dabs touched from dst back to src, so
// both ping-pong textures stay identical and authoritative — only the bytes
// that actually changed are moved (no full-canvas copy). The copies share
// the compute encoder, so wgpu orders them after the dispatch writes.
for (rx, ry, rw, rh) in dirty_tile_rects(dabs, canvas_w, canvas_h) {
compute_enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo {
texture: canvas.dst(),
mip_level: 0,
origin: wgpu::Origin3d { x: rx, y: ry, z: 0 },
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyTextureInfo {
texture: canvas.src(),
mip_level: 0,
origin: wgpu::Origin3d { x: rx, y: ry, z: 0 },
aspect: wgpu::TextureAspect::All,
},
wgpu::Extent3d { width: rw, height: rh, depth_or_array_layers: 1 },
);
}
queue.submit(Some(compute_enc.finish()));
// Step 4: Swap once — dst (with all dabs applied) becomes the new src.
canvas.swap();
}
/// Read the current canvas back to a CPU `Vec<u8>` (raw RGBA, row-major).
/// Read the current canvas back to a CPU `Vec<u8>` (sRGB-premultiplied RGBA,
/// row-major).
///
/// **Blocks** until the GPU work is complete (`Maintain::Wait`).
/// Should only be called at stroke end, not every frame.
/// The linear→sRGB conversion runs on the GPU into an `Rgba8Unorm` scratch
/// texture, so the CPU side is just a per-row `memcpy` (no per-pixel decode).
/// **Blocks** until the GPU work is complete. Call at stroke end, not per frame.
///
/// Returns `None` if no canvas exists for `keyframe_id`.
pub fn readback_canvas(
&self,
&mut self,
device: &wgpu::Device,
queue: &wgpu::Queue,
keyframe_id: Uuid,
) -> Option<Vec<u8>> {
// Lazily build the conversion pipeline and (re)create the cached scratch if
// the canvas size changed — these mutable borrows end before the shared
// borrows below.
if self.readback_srgb_pipeline.is_none() {
self.readback_srgb_pipeline = Some(ReadbackSrgbPipeline::new(device));
}
let (width, height) = {
let canvas = self.canvases.get(&keyframe_id)?;
let width = canvas.width;
let height = canvas.height;
(canvas.width, canvas.height)
};
if self.readback_scratch.as_ref().map_or(true, |s| s.width != width || s.height != height) {
self.readback_scratch = Some(ReadbackScratch::new(device, width, height));
}
// wgpu requires bytes_per_row to be a multiple of 256
let bytes_per_row_aligned =
((width * 4 + 255) / 256) * 256;
let total_bytes = (bytes_per_row_aligned * height) as u64;
let pipeline = self.readback_srgb_pipeline.as_ref().unwrap();
let scratch = self.readback_scratch.as_ref().unwrap();
let canvas = self.canvases.get(&keyframe_id)?;
let staging = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("canvas_readback_buf"),
size: total_bytes,
usage: wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
// GPU pass: linear-premultiplied Rgba16Float → sRGB-premultiplied Rgba8Unorm.
// Reading back 8-bit sRGB lets the CPU just memcpy each row.
let bg = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("canvas_readback_srgb_bg"),
layout: &pipeline.bg_layout,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: wgpu::BindingResource::TextureView(canvas.src_view()) },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&scratch.view) },
],
});
let bytes_per_row_aligned = scratch.bytes_per_row_aligned;
let mut encoder = device.create_command_encoder(
&wgpu::CommandEncoderDescriptor { label: Some("canvas_readback_encoder") },
);
{
let mut pass = encoder.begin_compute_pass(
&wgpu::ComputePassDescriptor { label: Some("canvas_readback_srgb_pass"), timestamp_writes: None },
);
pass.set_pipeline(&pipeline.pipeline);
pass.set_bind_group(0, &bg, &[]);
pass.dispatch_workgroups(width.div_ceil(8), height.div_ceil(8), 1);
}
encoder.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo {
texture: canvas.src(),
texture: &scratch.tex,
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyBufferInfo {
buffer: &staging,
buffer: &scratch.staging,
layout: wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(bytes_per_row_aligned),
@ -1249,8 +1438,8 @@ impl GpuBrushEngine {
);
queue.submit(Some(encoder.finish()));
// Block until complete
let slice = staging.slice(..);
// Block until complete.
let slice = scratch.staging.slice(..);
let (tx, rx) = std::sync::mpsc::channel();
slice.map_async(wgpu::MapMode::Read, move |r| { let _ = tx.send(r); });
let _ = device.poll(wgpu::PollType::wait_indefinitely());
@ -1258,27 +1447,19 @@ impl GpuBrushEngine {
let mapped = slice.get_mapped_range();
// De-stride: copy only `width * 4` bytes per row (drop alignment padding)
let bytes_per_row_tight = (width * 4) as usize;
let bytes_per_row_src = bytes_per_row_aligned as usize;
// De-stride with a per-row memcpy (dropping the 256-byte row padding). The
// bytes are already sRGB-premultiplied RGBA8 from the GPU pass, which is
// what Vello expects (ImageAlphaType::Premultiplied with sRGB channels).
let row_tight = (width * 4) as usize;
let row_src = bytes_per_row_aligned as usize;
let mut pixels = vec![0u8; (width * height * 4) as usize];
for row in 0..height as usize {
let src = &mapped[row * bytes_per_row_src .. row * bytes_per_row_src + bytes_per_row_tight];
let dst = &mut pixels[row * bytes_per_row_tight .. (row + 1) * bytes_per_row_tight];
dst.copy_from_slice(src);
let src = &mapped[row * row_src .. row * row_src + row_tight];
pixels[row * row_tight .. (row + 1) * row_tight].copy_from_slice(src);
}
drop(mapped);
staging.unmap();
// Encode linear premultiplied → sRGB-encoded premultiplied so the returned
// bytes match what Vello expects (ImageAlphaType::Premultiplied with sRGB
// channels). Alpha is left unchanged.
for pixel in pixels.chunks_exact_mut(4) {
pixel[0] = linear_to_srgb_byte(pixel[0]);
pixel[1] = linear_to_srgb_byte(pixel[1]);
pixel[2] = linear_to_srgb_byte(pixel[2]);
}
scratch.staging.unmap();
Some(pixels)
}
@ -1376,6 +1557,33 @@ impl GpuBrushEngine {
}
self.raster_layer_cache.insert(kf_id, canvas);
}
// Bump recency (the frame was used this render) and evict the least-recently
// used textures over budget. The current frame is pushed to the back, so it
// (and every other frame rendered this pass) is never the eviction victim.
if let Some(pos) = self.raster_layer_lru.iter().position(|id| *id == kf_id) {
self.raster_layer_lru.remove(pos);
}
self.raster_layer_lru.push(kf_id);
let mut evicted = false;
while self.raster_layer_lru.len() > Self::RASTER_LAYER_CACHE_MAX {
let old = self.raster_layer_lru.remove(0);
self.raster_layer_cache.remove(&old);
evicted = true;
}
if needs_new || evicted {
self.report_raster_cache_vram();
}
}
/// Estimated VRAM footprint of `raster_layer_cache` (two `Rgba16Float` textures =
/// `w·h·16` bytes per entry), published to the F3 debug overlay.
fn report_raster_cache_vram(&self) {
let bytes = |cache: &HashMap<Uuid, CanvasPair>| -> usize {
cache.values().map(|c| (c.width as usize) * (c.height as usize) * 16).sum()
};
let total = bytes(&self.raster_layer_cache) + bytes(&self.proxy_layer_cache);
let count = self.raster_layer_cache.len() + self.proxy_layer_cache.len();
crate::debug_overlay::update_gpu_memory(count, total);
}
/// Get the cached display texture for a raster layer keyframe.
@ -1383,9 +1591,55 @@ impl GpuBrushEngine {
self.raster_layer_cache.get(kf_id)
}
/// Ensure a low-res proxy texture exists for `kf_id` (uploaded once; proxies are
/// immutable). Bumps recency and evicts the least-recently-used past the budget.
/// `pixels` is sRGB-premultiplied RGBA of length `w * h * 4`.
pub fn ensure_proxy_texture(
&mut self,
device: &wgpu::Device,
queue: &wgpu::Queue,
kf_id: Uuid,
pixels: &[u8],
w: u32,
h: u32,
) {
if pixels.len() != (w * h * 4) as usize {
return; // malformed proxy — skip rather than panic in the render loop
}
let mut changed = false;
if !self.proxy_layer_cache.contains_key(&kf_id) {
let canvas = CanvasPair::new(device, w, h);
canvas.upload(queue, pixels);
self.proxy_layer_cache.insert(kf_id, canvas);
changed = true;
}
if let Some(pos) = self.proxy_layer_lru.iter().position(|id| *id == kf_id) {
self.proxy_layer_lru.remove(pos);
}
self.proxy_layer_lru.push(kf_id);
while self.proxy_layer_lru.len() > Self::RASTER_PROXY_CACHE_MAX {
let old = self.proxy_layer_lru.remove(0);
self.proxy_layer_cache.remove(&old);
changed = true;
}
if changed {
self.report_raster_cache_vram();
}
}
/// Get the cached low-res proxy texture for a raster keyframe.
pub fn get_proxy_texture(&self, kf_id: &Uuid) -> Option<&CanvasPair> {
self.proxy_layer_cache.get(kf_id)
}
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted).
pub fn remove_layer_texture(&mut self, kf_id: &Uuid) {
self.raster_layer_cache.remove(kf_id);
if self.raster_layer_cache.remove(kf_id).is_some() {
if let Some(pos) = self.raster_layer_lru.iter().position(|id| id == kf_id) {
self.raster_layer_lru.remove(pos);
}
self.report_raster_cache_vram();
}
}
/// Composite the accumulated-dab scratch buffer C over the source A, writing the
@ -1716,6 +1970,9 @@ pub struct CanvasBlitPipeline {
pub pipeline: wgpu::RenderPipeline,
pub bg_layout: wgpu::BindGroupLayout,
pub sampler: wgpu::Sampler,
/// Bilinear sampler for smooth upscaling (used by `blit_smooth`, e.g. low-res
/// proxies). The default `sampler` stays nearest to keep the real canvas crisp.
pub linear_sampler: wgpu::Sampler,
/// Nearest-neighbour sampler used for the selection mask texture.
pub mask_sampler: wgpu::Sampler,
}
@ -1772,12 +2029,24 @@ impl BlitTransform {
// y' = b*x + d*y + f
// Column-major 3×3: col0=(a,b,0), col1=(c,d,0), col2=(e,f,1)
let [a, b, c, d, e, f] = combined.as_coeffs();
// The .w slots are unused by the 3×3 matrix; the shader reads them as an RGB
// screen-blend tint ((0,0,0) = no tint — the default for all normal blits).
Self {
col0: [a as f32, b as f32, 0.0, 0.0],
col1: [c as f32, d as f32, 0.0, 0.0],
col2: [e as f32, f as f32, 1.0, 0.0],
}
}
/// Screen-blend the sampled color toward an RGB tint (for onion-skin ghosts), so
/// blacks/outlines pick up the tint. Packed into the matrix uniform's `.w` slots,
/// so no extra binding is needed.
pub fn with_tint(mut self, r: f32, g: f32, b: f32) -> Self {
self.col0[3] = r;
self.col1[3] = g;
self.col2[3] = b;
self
}
}
impl CanvasBlitPipeline {
@ -1891,6 +2160,17 @@ impl CanvasBlitPipeline {
..Default::default()
});
let linear_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("canvas_blit_linear_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
address_mode_v: wgpu::AddressMode::ClampToEdge,
address_mode_w: wgpu::AddressMode::ClampToEdge,
mag_filter: wgpu::FilterMode::Linear,
min_filter: wgpu::FilterMode::Linear,
mipmap_filter: wgpu::FilterMode::Linear,
..Default::default()
});
let mask_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("canvas_mask_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
@ -1902,7 +2182,7 @@ impl CanvasBlitPipeline {
..Default::default()
});
Self { pipeline, bg_layout, sampler, mask_sampler }
Self { pipeline, bg_layout, sampler, linear_sampler, mask_sampler }
}
/// Render the canvas texture into `target_view` (Rgba16Float) with the given camera.
@ -1910,6 +2190,7 @@ impl CanvasBlitPipeline {
/// `target_view` is cleared to transparent before writing.
/// `mask_view` is an R8Unorm texture in canvas-pixel space: 255 = keep, 0 = discard.
/// Pass `None` to use the built-in 1×1 all-white default (no masking).
/// Blit with the default nearest-neighbour sampler (crisp; for real canvas pixels).
pub fn blit(
&self,
device: &wgpu::Device,
@ -1918,6 +2199,32 @@ impl CanvasBlitPipeline {
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.sampler);
}
/// Blit with a bilinear sampler — smooth upscaling for low-res sources (proxies).
pub fn blit_smooth(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
canvas_view: &wgpu::TextureView,
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler);
}
fn blit_with(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
canvas_view: &wgpu::TextureView,
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
canvas_sampler: &wgpu::Sampler,
) {
// When no mask is provided, create a temporary 1×1 all-white texture.
// (queue is already available here, unlike in new())
@ -1970,7 +2277,7 @@ impl CanvasBlitPipeline {
},
wgpu::BindGroupEntry {
binding: 1,
resource: wgpu::BindingResource::Sampler(&self.sampler),
resource: wgpu::BindingResource::Sampler(canvas_sampler),
},
wgpu::BindGroupEntry {
binding: 2,

9
lightningbeam-ui/lightningbeam-editor/src/keymap.rs
View File

@ -103,6 +103,7 @@ pub enum AppAction {
TogglePlayPause,
CancelAction,
ToggleDebugOverlay,
ToggleOnionSkin,
#[cfg(debug_assertions)]
ToggleTestMode,
@ -150,7 +151,7 @@ impl AppAction {
Self::ToolErase | Self::ToolSmudge | Self::ToolSelectLasso | Self::ToolSplit => "Tools",
Self::TogglePlayPause | Self::CancelAction |
Self::ToggleDebugOverlay => "Global",
Self::ToggleDebugOverlay | Self::ToggleOnionSkin => "Global",
#[cfg(debug_assertions)]
Self::ToggleTestMode => "Global",
@ -246,6 +247,7 @@ impl AppAction {
Self::TogglePlayPause => "Toggle Play/Pause",
Self::CancelAction => "Cancel / Escape",
Self::ToggleDebugOverlay => "Toggle Debug Overlay",
Self::ToggleOnionSkin => "Toggle Onion Skinning",
#[cfg(debug_assertions)]
Self::ToggleTestMode => "Toggle Test Mode",
Self::PianoRollDelete => "Piano Roll: Delete",
@ -281,7 +283,7 @@ impl AppAction {
Self::ToolEyedropper, Self::ToolLine, Self::ToolPolygon,
Self::ToolBezierEdit, Self::ToolText, Self::ToolRegionSelect,
Self::ToolErase, Self::ToolSmudge, Self::ToolSelectLasso, Self::ToolSplit,
Self::TogglePlayPause, Self::CancelAction, Self::ToggleDebugOverlay,
Self::TogglePlayPause, Self::CancelAction, Self::ToggleDebugOverlay, Self::ToggleOnionSkin,
#[cfg(debug_assertions)]
Self::ToggleTestMode,
Self::PianoRollDelete, Self::StageDelete,
@ -330,6 +332,7 @@ impl From<MenuAction> for AppAction {
MenuAction::AddTestClip => Self::AddTestClip,
MenuAction::DeleteLayer => Self::DeleteLayer,
MenuAction::ToggleLayerVisibility => Self::ToggleLayerVisibility,
MenuAction::ToggleOnionSkin => Self::ToggleOnionSkin,
MenuAction::ShowMasterTrack => Self::ToggleLayerVisibility, // not directly mappable
MenuAction::NewKeyframe => Self::NewKeyframe,
MenuAction::NewBlankKeyframe => Self::NewBlankKeyframe,
@ -392,6 +395,7 @@ impl TryFrom<AppAction> for MenuAction {
AppAction::AddTestClip => MenuAction::AddTestClip,
AppAction::DeleteLayer => MenuAction::DeleteLayer,
AppAction::ToggleLayerVisibility => MenuAction::ToggleLayerVisibility,
AppAction::ToggleOnionSkin => MenuAction::ToggleOnionSkin,
AppAction::NewKeyframe => MenuAction::NewKeyframe,
AppAction::NewBlankKeyframe => MenuAction::NewBlankKeyframe,
AppAction::DeleteFrame => MenuAction::DeleteFrame,
@ -507,6 +511,7 @@ pub fn all_defaults() -> HashMap<AppAction, Option<Shortcut>> {
defaults.insert(AppAction::TogglePlayPause, Some(Shortcut::new(ShortcutKey::Space, nc, ns, na)));
defaults.insert(AppAction::CancelAction, Some(Shortcut::new(ShortcutKey::Escape, nc, ns, na)));
defaults.insert(AppAction::ToggleDebugOverlay, Some(Shortcut::new(ShortcutKey::F3, nc, ns, na)));
defaults.insert(AppAction::ToggleOnionSkin, Some(Shortcut::new(ShortcutKey::O, nc, ns, na)));
#[cfg(debug_assertions)]
defaults.insert(AppAction::ToggleTestMode, Some(Shortcut::new(ShortcutKey::F5, nc, ns, na)));

1165
lightningbeam-ui/lightningbeam-editor/src/main.rs
View File

File diff suppressed because it is too large Load Diff

4
lightningbeam-ui/lightningbeam-editor/src/menu.rs
View File

@ -332,6 +332,7 @@ pub enum MenuAction {
ZoomOut,
ActualSize,
RecenterView,
ToggleOnionSkin,
NextLayout,
PreviousLayout,
#[allow(dead_code)] // Handler exists in main.rs, menu item not yet wired
@ -432,6 +433,7 @@ impl MenuItemDef {
const ZOOM_OUT: Self = Self { label: "Zoom Out", action: MenuAction::ZoomOut, shortcut: Some(Shortcut::new(ShortcutKey::Minus, CTRL, NO_SHIFT, NO_ALT)) };
const ACTUAL_SIZE: Self = Self { label: "Actual Size", action: MenuAction::ActualSize, shortcut: Some(Shortcut::new(ShortcutKey::Num0, CTRL, NO_SHIFT, NO_ALT)) };
const RECENTER_VIEW: Self = Self { label: "Recenter View", action: MenuAction::RecenterView, shortcut: None };
const TOGGLE_ONION_SKIN: Self = Self { label: "Onion Skinning", action: MenuAction::ToggleOnionSkin, shortcut: Some(Shortcut::new(ShortcutKey::O, NO_CTRL, NO_SHIFT, NO_ALT)) };
const NEXT_LAYOUT: Self = Self { label: "Next Layout", action: MenuAction::NextLayout, shortcut: Some(Shortcut::new(ShortcutKey::BracketRight, CTRL, NO_SHIFT, NO_ALT)) };
const PREVIOUS_LAYOUT: Self = Self { label: "Previous Layout", action: MenuAction::PreviousLayout, shortcut: Some(Shortcut::new(ShortcutKey::BracketLeft, CTRL, NO_SHIFT, NO_ALT)) };
@ -455,6 +457,7 @@ impl MenuItemDef {
&Self::DELETE, &Self::SELECT_ALL, &Self::SELECT_NONE,
&Self::GROUP, &Self::ADD_LAYER, &Self::NEW_KEYFRAME,
&Self::ZOOM_IN, &Self::ZOOM_OUT, &Self::ACTUAL_SIZE,
&Self::TOGGLE_ONION_SKIN,
&Self::NEXT_LAYOUT, &Self::PREVIOUS_LAYOUT,
&Self::SETTINGS, &Self::CLOSE_WINDOW,
]
@ -566,6 +569,7 @@ impl MenuItemDef {
MenuDef::Item(&Self::ZOOM_OUT),
MenuDef::Item(&Self::ACTUAL_SIZE),
MenuDef::Item(&Self::RECENTER_VIEW),
MenuDef::Item(&Self::TOGGLE_ONION_SKIN),
MenuDef::Separator,
MenuDef::Submenu {
label: "Layout",

56
lightningbeam-ui/lightningbeam-editor/src/notifications.rs
View File

@ -3,52 +3,52 @@
use notify_rust::Notification;
use std::path::Path;
/// Send a desktop notification for successful export
///
/// # Arguments
/// * `output_path` - Path to the exported file
///
/// # Returns
/// Ok(()) if notification sent successfully, Err with log message if failed
pub fn notify_export_complete(output_path: &Path) -> Result<(), String> {
// `notify_rust`'s `Notification::show()` is a SYNCHRONOUS D-Bus call. When no
// notification daemon is running it can block for the service-activation timeout
// (~25s) before failing, so these are fire-and-forget on a detached thread — the
// UI must never wait on them (e.g. they're triggered from the export-complete
// handler on the UI thread).
/// Show a desktop notification for a successful export (fire-and-forget).
pub fn notify_export_complete(output_path: &Path) {
let filename = output_path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("unknown");
.unwrap_or("unknown")
.to_string();
Notification::new()
std::thread::spawn(move || {
if let Err(e) = Notification::new()
.summary("Export Complete")
.body(&format!("Successfully exported: {}", filename))
.icon("dialog-information") // Standard icon name (freedesktop.org)
.timeout(5000) // 5 seconds
.show()
.map_err(|e| format!("Failed to show notification: {}", e))?;
Ok(())
{
eprintln!("⚠️ Could not send desktop notification: {}", e);
}
});
}
/// Send a desktop notification for export error
///
/// # Arguments
/// * `error_message` - The error message to display
///
/// # Returns
/// Ok(()) if notification sent successfully, Err with log message if failed
pub fn notify_export_error(error_message: &str) -> Result<(), String> {
// Truncate very long error messages
let truncated = if error_message.len() > 100 {
format!("{}...", &error_message[..97])
/// Show a desktop notification for an export error (fire-and-forget).
pub fn notify_export_error(error_message: &str) {
// Truncate very long error messages (on a char boundary).
let truncated = if error_message.chars().count() > 100 {
let prefix: String = error_message.chars().take(97).collect();
format!("{}...", prefix)
} else {
error_message.to_string()
};
Notification::new()
std::thread::spawn(move || {
if let Err(e) = Notification::new()
.summary("Export Failed")
.body(&truncated)
.icon("dialog-error") // Standard error icon
.timeout(10000) // 10 seconds for errors (longer to read)
.show()
.map_err(|e| format!("Failed to show notification: {}", e))?;
Ok(())
{
eprintln!("⚠️ Could not send desktop notification: {}", e);
}
});
}

137
lightningbeam-ui/lightningbeam-editor/src/panes/infopanel.rs
View File

@ -12,7 +12,7 @@
use eframe::egui::{self, DragValue, Ui};
use lightningbeam_core::brush_settings::{bundled_brushes, BrushSettings};
use lightningbeam_core::actions::{SetDocumentPropertiesAction, SetShapePropertiesAction, SetFillPaintAction};
use lightningbeam_core::actions::{SetDocumentPropertiesAction, SetShapePropertiesAction, SetFillPaintAction, SetImageFillAction};
use lightningbeam_core::gradient::ShapeGradient;
use lightningbeam_core::layer::{AnyLayer, LayerTrait};
use lightningbeam_core::selection::FocusSelection;
@ -785,6 +785,26 @@ impl InfopanelPane {
let edge_ids: Vec<lightningbeam_core::vector_graph::EdgeId> = shared.selection.selected_edges()
.iter().map(|eid| lightningbeam_core::vector_graph::EdgeId(eid.0)).collect();
// Image-fill state for the selected fills + the document's image assets, for
// the picker below. Gathered now (immutable read) before `shared` is borrowed mut.
let image_assets: Vec<(Uuid, String)> = {
let doc = shared.action_executor.document();
let mut v: Vec<(Uuid, String)> = doc.image_assets.iter()
.map(|(id, a)| (*id, a.name.clone())).collect();
v.sort_by(|a, b| a.1.cmp(&b.1));
v
};
let current_image_fill: Option<Uuid> = {
let doc = shared.action_executor.document();
match doc.get_layer(&layer_id) {
Some(lightningbeam_core::layer::AnyLayer::Vector(vl)) => vl
.graph_at_time(time)
.and_then(|g| face_ids.first().map(|&fid| g.fill(fid).image_fill))
.flatten(),
_ => None,
}
};
egui::CollapsingHeader::new("Shape")
.id_salt(("shape", path))
.default_open(self.shape_section_open)
@ -792,47 +812,67 @@ impl InfopanelPane {
self.shape_section_open = true;
ui.add_space(4.0);
// Fill — determine current fill type
// Fill — determine current fill type. `image_fill` takes render priority,
// so when set it's the active type (overriding colour/gradient underneath);
// switching to None/Solid/Gradient clears it.
let has_image = current_image_fill.is_some();
let has_gradient = matches!(&info.fill_gradient, Some(Some(_)));
let has_solid = matches!(&info.fill_color, Some(Some(_)));
let fill_is_none = matches!(&info.fill_color, Some(None))
&& matches!(&info.fill_gradient, Some(None));
let fill_mixed = info.fill_color.is_none() && info.fill_gradient.is_none();
// Fill type toggle row
// Fill type toggle row: None | Solid | Gradient | Image
ui.horizontal(|ui| {
ui.label("Fill:");
if fill_mixed {
ui.label("--");
} else {
if ui.selectable_label(fill_is_none, "None").clicked() && !fill_is_none {
let action = SetFillPaintAction::solid(
layer_id, time, face_ids.clone(), None,
);
shared.pending_actions.push(Box::new(action));
if ui.selectable_label(fill_is_none && !has_image, "None").clicked() {
if has_image {
shared.pending_actions.push(Box::new(
SetImageFillAction::new(layer_id, time, face_ids.clone(), None)));
}
if ui.selectable_label(has_solid || (!has_gradient && !fill_is_none), "Solid").clicked() && !has_solid {
// Switch to solid: use existing color or default to black
shared.pending_actions.push(Box::new(
SetFillPaintAction::solid(layer_id, time, face_ids.clone(), None)));
}
if ui.selectable_label(has_solid && !has_image, "Solid").clicked() {
if has_image {
shared.pending_actions.push(Box::new(
SetImageFillAction::new(layer_id, time, face_ids.clone(), None)));
}
if !has_solid {
let color = info.fill_color.flatten()
.unwrap_or(ShapeColor::rgba(0, 0, 0, 255));
let action = SetFillPaintAction::solid(
layer_id, time, face_ids.clone(), Some(color),
);
shared.pending_actions.push(Box::new(action));
shared.pending_actions.push(Box::new(
SetFillPaintAction::solid(layer_id, time, face_ids.clone(), Some(color))));
}
}
if ui.selectable_label(has_gradient && !has_image, "Gradient").clicked() {
if has_image {
shared.pending_actions.push(Box::new(
SetImageFillAction::new(layer_id, time, face_ids.clone(), None)));
}
if !has_gradient {
let grad = info.fill_gradient.clone().flatten().unwrap_or_default();
shared.pending_actions.push(Box::new(
SetFillPaintAction::gradient(layer_id, time, face_ids.clone(), Some(grad))));
}
}
// Image tab — only offered if there are imported image assets.
if !image_assets.is_empty() || has_image {
if ui.selectable_label(has_image, "Image").clicked() && !has_image {
if let Some((aid, _)) = image_assets.first() {
shared.pending_actions.push(Box::new(
SetImageFillAction::new(layer_id, time, face_ids.clone(), Some(*aid))));
}
}
if ui.selectable_label(has_gradient, "Gradient").clicked() && !has_gradient {
let grad = info.fill_gradient.clone().flatten()
.unwrap_or_default();
let action = SetFillPaintAction::gradient(
layer_id, time, face_ids.clone(), Some(grad),
);
shared.pending_actions.push(Box::new(action));
}
}
});
// Solid fill color editor
if !fill_mixed && has_solid {
if !fill_mixed && has_solid && !has_image {
if let Some(Some(color)) = info.fill_color {
ui.horizontal(|ui| {
let mut rgba = [color.r, color.g, color.b, color.a];
@ -848,7 +888,7 @@ impl InfopanelPane {
}
// Gradient fill editor
if !fill_mixed && has_gradient {
if !fill_mixed && has_gradient && !has_image {
if let Some(Some(mut grad)) = info.fill_gradient.clone() {
if gradient_stop_editor(ui, &mut grad, &mut self.selected_shape_gradient_stop) {
let action = SetFillPaintAction::gradient(
@ -859,6 +899,28 @@ impl InfopanelPane {
}
}
// Image fill editor — pick which asset (active Image type).
if has_image {
ui.horizontal(|ui| {
ui.label("Image:");
let selected_text = current_image_fill
.and_then(|id| image_assets.iter().find(|(aid, _)| *aid == id))
.map(|(_, n)| n.clone())
.unwrap_or_else(|| "(missing)".to_string());
egui::ComboBox::from_id_salt(("image_fill", path))
.selected_text(selected_text)
.show_ui(ui, |ui| {
for (aid, name) in &image_assets {
if ui.selectable_label(current_image_fill == Some(*aid), name).clicked() {
shared.pending_actions.push(Box::new(
SetImageFillAction::new(layer_id, time, face_ids.clone(), Some(*aid)),
));
}
}
});
});
}
// Stroke color
ui.horizontal(|ui| {
ui.label("Stroke:");
@ -1035,6 +1097,29 @@ impl InfopanelPane {
});
}
/// Render the onion-skinning view settings (global; not tied to selection).
fn render_onion_section(&mut self, ui: &mut Ui, path: &NodePath, shared: &mut SharedPaneState) {
egui::CollapsingHeader::new("Onion Skin")
.id_salt(("onion", path))
.default_open(shared.onion_skin.enabled)
.show(ui, |ui| {
ui.add_space(4.0);
ui.checkbox(&mut shared.onion_skin.enabled, "Enabled");
ui.add_enabled_ui(shared.onion_skin.enabled, |ui| {
ui.horizontal(|ui| {
ui.label("Frames before:");
ui.add(DragValue::new(&mut shared.onion_skin.frames_before).range(0..=5));
});
ui.horizontal(|ui| {
ui.label("Frames after:");
ui.add(DragValue::new(&mut shared.onion_skin.frames_after).range(0..=5));
});
ui.add(egui::Slider::new(&mut shared.onion_skin.opacity, 0.0..=1.0).text("Opacity"));
});
ui.add_space(4.0);
});
}
/// Render layer info section
fn render_layer_section(&self, ui: &mut Ui, path: &NodePath, shared: &SharedPaneState, layer_ids: &[Uuid]) {
let document = shared.action_executor.document();
@ -1478,6 +1563,12 @@ impl PaneRenderer for InfopanelPane {
}
}
}
// Onion-skinning view settings — always available, regardless of selection.
ui.add_space(8.0);
ui.separator();
ui.add_space(4.0);
self.render_onion_section(ui, path, shared);
});
}

54
lightningbeam-ui/lightningbeam-editor/src/panes/mod.rs
View File

@ -143,7 +143,45 @@ pub fn find_sampled_audio_track(document: &lightningbeam_core::document::Documen
}
/// Shared state that all panes can access
/// Onion-skinning view settings (editor-only; not saved with the document). Ghosts the
/// active layer's neighbouring keyframes, warm-tinted for past and cool for future.
#[derive(Clone, Copy, Debug)]
pub struct OnionSkinSettings {
pub enabled: bool,
pub frames_before: usize,
pub frames_after: usize,
/// Opacity of the nearest ghost; further ghosts fall off linearly.
pub opacity: f32,
}
impl Default for OnionSkinSettings {
fn default() -> Self {
Self { enabled: false, frames_before: 2, frames_after: 2, opacity: 0.35 }
}
}
impl OnionSkinSettings {
/// RGB tint multipliers for past (warm) and future (cool) ghosts.
pub const PAST_TINT: [f32; 3] = [1.0, 0.45, 0.45];
pub const FUTURE_TINT: [f32; 3] = [0.45, 0.6, 1.0];
/// Opacity for the `n`-th ghost away from the current frame (n = 1 is nearest),
/// linearly falling off so the furthest ghost is faintest.
pub fn ghost_opacity(&self, n: usize, total: usize) -> f32 {
if total == 0 { return self.opacity; }
let falloff = 1.0 - (n.saturating_sub(1) as f32) / (total as f32);
self.opacity * falloff.clamp(0.15, 1.0)
}
}
pub struct SharedPaneState<'a> {
/// Current `.beam` container path (for lazily paging image-asset bytes in the
/// renderer's ImageCache). `None` before the project is first saved/loaded.
pub container_path: Option<std::path::PathBuf>,
/// Effective onion-skin settings (already gated to off during playback by main.rs).
pub onion: OnionSkinSettings,
/// The raw onion-skin settings, mutable — edited by the Info Panel's controls.
pub onion_skin: &'a mut OnionSkinSettings,
pub tool_icon_cache: &'a mut crate::ToolIconCache,
#[allow(dead_code)] // Used by pane chrome rendering in main.rs
pub icon_cache: &'a mut crate::IconCache,
@ -242,6 +280,16 @@ pub struct SharedPaneState<'a> {
pub raw_audio_cache: &'a std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
/// Pool indices needing GPU waveform texture upload
pub waveform_gpu_dirty: &'a mut std::collections::HashSet<usize>,
/// Pools whose `raw_audio_cache` entry is a packed min/max floor rather than
/// raw samples (pool_index -> `B`, floor frames-per-texel). Drives the GPU
/// min/max upload path and the floor's effective rate `sr/B` in the renderer.
pub waveform_minmax_pools: &'a std::collections::HashMap<usize, u32>,
/// Miss-sink for on-demand raster keyframe pixel faulting (Phase 3 paging).
/// The canvas inserts the id of any raster keyframe it wants to upload whose
/// `raw_pixels` aren't resident; the App drains this at the top of the next
/// `update()` and faults the pixels in from the project container.
pub raster_fault_requests:
&'a std::sync::Arc<std::sync::Mutex<std::collections::HashSet<uuid::Uuid>>>,
/// Effect ID to load into shader editor (set by asset library, consumed by shader editor)
pub effect_to_load: &'a mut Option<Uuid>,
/// Queue for effect thumbnail requests (effect IDs to generate thumbnails for)
@ -277,8 +325,10 @@ pub struct SharedPaneState<'a> {
pub script_to_edit: &'a mut Option<Uuid>,
/// Script ID that was just saved (triggers auto-recompile of nodes using it)
pub script_saved: &'a mut Option<Uuid>,
/// Active region selection (temporary split state)
pub region_selection: &'a mut Option<lightningbeam_core::selection::RegionSelection>,
/// Undo-stack depth recorded before the first region-select cut of the current
/// selection session. `Some` while a region selection is live and unchanged; lets a
/// later deselect heal (undo) the cut if nothing was edited. See `resolve_pending_region_cut`.
pub pending_region_cut_base: &'a mut Option<usize>,
/// Region select mode (Rectangle or Lasso)
pub region_select_mode: &'a mut lightningbeam_core::tool::RegionSelectMode,
/// Lasso select sub-mode (Freehand / Polygonal / Magnetic)

4
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/alpha_composite.wgsl
View File

@ -5,12 +5,12 @@
//
// B[px] = C[px] + A[px] * (1 − C[px].a) (Porter-Duff src-over, C over A)
//
// All textures are Rgba8Unorm, linear premultiplied RGBA.
// All textures are Rgba16Float, linear premultiplied RGBA.
// Dispatch: ceil(w/8) × ceil(h/8) × 1.
@group(0) @binding(0) var tex_a: texture_2d<f32>; // source (A)
@group(0) @binding(1) var tex_c: texture_2d<f32>; // accumulated dabs (C)
@group(0) @binding(2) var tex_b: texture_storage_2d<rgba8unorm, write>; // output (B)
@group(0) @binding(2) var tex_b: texture_storage_2d<rgba16float, write>; // output (B)
@compute @workgroup_size(8, 8)
fn main(@builtin(global_invocation_id) gid: vec3<u32>) {

2
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/brush_dab.wgsl
View File

@ -37,7 +37,7 @@ struct Params {
@group(0) @binding(0) var<storage, read> dabs: array<GpuDab>;
@group(0) @binding(1) var<uniform> params: Params;
@group(0) @binding(2) var canvas_src: texture_2d<f32>;
@group(0) @binding(3) var canvas_dst: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(3) var canvas_dst: texture_storage_2d<rgba16float, write>;
// ---------------------------------------------------------------------------
// Manual bilinear sample from canvas_src at sub-pixel coordinates (px, py).

8
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/canvas_blit.wgsl
View File

@ -67,5 +67,11 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let mask = textureSample(mask_tex, mask_sampler, canvas_uv).r;
let masked_a = c.a * mask;
let inv_a = select(0.0, 1.0 / c.a, c.a > 1e-6);
return vec4<f32>(c.r * inv_a, c.g * inv_a, c.b * inv_a, masked_a);
// RGB tint packed into the matrix uniform's unused .w slots ((0,0,0) = no tint).
// Screen-blended so it recolors blacks too (outlines → warm/cool ghosts) and
// leaves white alone: out = base + tint - base*tint. Used by onion-skin ghosts.
let tint = vec3<f32>(transform.col0.w, transform.col1.w, transform.col2.w);
let base = vec3<f32>(c.r * inv_a, c.g * inv_a, c.b * inv_a);
let tinted = base + tint - base * tint;
return vec4<f32>(tinted, masked_a);
}

27
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/canvas_readback_srgb.wgsl Normal file
View File

@ -0,0 +1,27 @@
// Canvas readback color conversion.
//
// Converts the Rgba16Float linear-PREMULTIPLIED canvas into an Rgba8Unorm
// sRGB-encoded premultiplied texture so the CPU readback is a pure row memcpy
// instead of a per-pixel sRGB decode. Matches the bytes the previous CPU decode
// produced: the sRGB OETF is applied per channel to the premultiplied RGB, and
// alpha (which is not gamma-encoded) is passed through.
// linear_to_srgb_channel is provided by the prepended COLOR_WGSL prelude.
@group(0) @binding(0) var src: texture_2d<f32>; // linear premultiplied
@group(0) @binding(1) var dst: texture_storage_2d<rgba8unorm, write>; // sRGB premultiplied
@compute @workgroup_size(8, 8)
fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
let dim = textureDimensions(src);
if gid.x >= dim.x || gid.y >= dim.y {
return;
}
let p = vec2<i32>(i32(gid.x), i32(gid.y));
let c = textureLoad(src, p, 0);
let srgb = vec3<f32>(
linear_to_srgb_channel(c.r),
linear_to_srgb_channel(c.g),
linear_to_srgb_channel(c.b),
);
textureStore(dst, p, vec4<f32>(srgb, c.a));
}

23
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/gradient_fill.wgsl
View File

@ -2,8 +2,10 @@
//
// Reads the anchor canvas (before_pixels), composites a gradient over it, and
// writes the result to the display canvas. All color values in the canvas are
// linear premultiplied RGBA. The stop colors passed via `stops` are linear
// straight-alpha [0..1] (sRGB→linear conversion is done on the CPU).
// linear premultiplied RGBA. The stop colors passed via `stops` are sRGB
// straight-alpha [0..1]; the gradient is interpolated in sRGB (gamma) space to
// match the CPU raster and vector gradient paths, then the interpolated color is
// converted to linear before compositing.
//
// Dispatch: ceil(canvas_w / 8) × ceil(canvas_h / 8) × 1
@ -25,7 +27,7 @@ struct Params {
// 32 bytes per stop (8 × f32), matching `GpuGradientStop` on the Rust side.
struct GradientStop {
position: f32,
r: f32, // linear [0..1], straight-alpha
r: f32, // sRGB [0..1], straight-alpha
g: f32,
b: f32,
a: f32,
@ -34,10 +36,11 @@ struct GradientStop {
_pad2: f32,
}
// srgb_to_linear_channel is provided by the prepended COLOR_WGSL prelude.
@group(0) @binding(0) var<uniform> params: Params;
@group(0) @binding(1) var src: texture_2d<f32>;
@group(0) @binding(2) var<storage, read> stops: array<GradientStop>;
@group(0) @binding(3) var dst: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(3) var dst: texture_storage_2d<rgba16float, write>;
fn apply_extend(t: f32) -> f32 {
if params.extend_mode == 0u {
@ -122,7 +125,15 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
}
let t = apply_extend(t_raw);
let grad = eval_gradient(t); // straight-alpha linear RGBA
let grad = eval_gradient(t); // straight-alpha sRGB RGBA (interpolated in gamma space)
// Convert the interpolated sRGB color to linear for compositing. Alpha is
// not gamma-encoded, so it passes through unchanged.
let grad_rgb_lin = vec3<f32>(
srgb_to_linear_channel(grad.r),
srgb_to_linear_channel(grad.g),
srgb_to_linear_channel(grad.b),
);
// Effective alpha: gradient alpha × tool opacity.
let a = grad.a * params.opacity;
@ -131,7 +142,7 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
// src_px.rgb is premultiplied (= straight_rgb * src_a).
// Output is also premultiplied.
let out_a = a + src_px.a * (1.0 - a);
let out_rgb = grad.rgb * a + src_px.rgb * (1.0 - a);
let out_rgb = grad_rgb_lin * a + src_px.rgb * (1.0 - a);
textureStore(dst, vec2<i32>(i32(gid.x), i32(gid.y)), vec4<f32>(out_rgb, out_a));
}

28
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/linear_to_srgb.wgsl
View File

@ -30,27 +30,23 @@ fn vs_main(@builtin(vertex_index) vertex_index: u32) -> VertexOutput {
return out;
}
// Linear to sRGB conversion for a single channel
// Formula: c <= 0.0031308 ? c*12.92 : 1.055*pow(c, 1/2.4) - 0.055
fn linear_to_srgb_channel(c: f32) -> f32 {
let clamped = clamp(c, 0.0, 1.0);
return select(
1.055 * pow(clamped, 1.0 / 2.4) - 0.055,
clamped * 12.92,
clamped <= 0.0031308
);
}
// linear_to_srgb_channel is provided by the prepended COLOR_WGSL prelude.
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
// Sample linear HDR texture
// Sample linear HDR texture. The compositor accumulates PREMULTIPLIED
// linear color, so unpremultiply before applying the sRGB OETF:
// srgb(rgb*a) != srgb(rgb)*a, so encoding premultiplied color directly
// corrupts antialiased edges and transparent pixels. We output straight
// alpha to match the straight-alpha display blit and PNG export.
let linear = textureSample(input_tex, input_sampler, in.uv);
let a = linear.a;
let straight = select(linear.rgb / a, vec3<f32>(0.0), a <= 0.0);
// Convert from linear to sRGB for display (alpha stays linear)
return vec4<f32>(
linear_to_srgb_channel(linear.r),
linear_to_srgb_channel(linear.g),
linear_to_srgb_channel(linear.b),
linear.a
linear_to_srgb_channel(straight.r),
linear_to_srgb_channel(straight.g),
linear_to_srgb_channel(straight.b),
a
);
}

2
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/raster_transform.wgsl
View File

@ -26,7 +26,7 @@ struct Params {
@group(0) @binding(0) var<uniform> params: Params;
@group(0) @binding(1) var src: texture_2d<f32>;
@group(0) @binding(2) var dst: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(2) var dst: texture_storage_2d<rgba16float, write>;
// Manual bilinear sample with clamp-to-edge (textureSample forbidden in compute shaders).
fn bilinear_sample(px: f32, py: f32) -> vec4<f32> {

2
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/warp_apply.wgsl
View File

@ -26,7 +26,7 @@ struct Params {
@group(0) @binding(0) var<uniform> params: Params;
@group(0) @binding(1) var src: texture_2d<f32>;
@group(0) @binding(2) var<storage, read> disp: array<vec2f>;
@group(0) @binding(3) var dst: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(3) var dst: texture_storage_2d<rgba16float, write>;
// Manual bilinear sample with clamp-to-edge (textureSample forbidden in compute shaders).
fn bilinear_sample(px: f32, py: f32) -> vec4<f32> {

32
lightningbeam-ui/lightningbeam-editor/src/panes/shaders/waveform.wgsl
View File

@ -77,27 +77,29 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let frames_per_pixel = params.sample_rate / params.pixels_per_second;
// Each mip level reduces by 4x in sample count (2x in each texture dimension)
let mip_f = max(0.0, log2(frames_per_pixel) / 2.0);
let max_mip = f32(textureNumLevels(peak_tex) - 1u);
let mip = min(mip_f, max_mip);
// Frame index at the chosen mip level
let mip_floor = u32(mip);
let reduction = pow(4.0, f32(mip_floor));
// Pick the NEAREST INTEGER LOD and read its exact texel. Sampling at a
// fractional mip (trilinear) blends level N and N+1, but each level has its
// own 1D→2D row-major linearization (width halves per level), so the two
// levels disagree on which audio frame a given screen column maps to. The
// blend then reads horizontally-offset neighbours, and because a 2x zoom step
// shifts mip_f by exactly 0.5, alternate zoom levels land on a clean integer
// (correct) vs a 50/50 blend (offset) — the "every other zoom level" artifact.
// textureLoad at one integer level keeps the frame→texel mapping exact.
let max_mip = i32(textureNumLevels(peak_tex)) - 1;
let mip_i = clamp(i32(mip_f + 0.5), 0, max_mip);
let reduction = pow(4.0, f32(mip_i));
let mip_frame = frame_f / reduction;
// Convert 1D mip-space index to 2D UV coordinates
// Use actual texture dimensions (not computed from total_frames) because the
// texture may be pre-allocated larger for live recording.
let mip_dims = textureDimensions(peak_tex, mip_floor);
// Convert 1D mip-space index to 2D texel coords using this level's actual
// dimensions (texture may be pre-allocated larger, e.g. for live recording).
let mip_dims = textureDimensions(peak_tex, mip_i);
let mip_tex_width = f32(mip_dims.x);
let mip_tex_height = f32(mip_dims.y);
let texel_x = mip_frame % mip_tex_width;
let texel_y = floor(mip_frame / mip_tex_width);
let uv = vec2((texel_x + 0.5) / mip_tex_width, (texel_y + 0.5) / mip_tex_height);
let texel_x = i32(mip_frame % mip_tex_width);
let texel_y = i32(floor(mip_frame / mip_tex_width));
// Sample the peak texture at computed mip level
// R = left_min, G = left_max, B = right_min, A = right_max
let peak = textureSampleLevel(peak_tex, peak_sampler, uv, mip);
let peak = textureLoad(peak_tex, vec2<i32>(texel_x, texel_y), mip_i);
let clip_height = params.clip_rect.w - params.clip_rect.y;
let clip_top = params.clip_rect.y;

779
lightningbeam-ui/lightningbeam-editor/src/panes/stage.rs
View File

File diff suppressed because it is too large Load Diff

390
lightningbeam-ui/lightningbeam-editor/src/panes/timeline.rs
View File

@ -99,6 +99,94 @@ fn clip_instance_y_bounds(row: usize, total_rows: usize) -> (f32, f32) {
}
}
/// Draw the strip of video thumbnails across a clip.
///
/// Thumbnails are tiled from the clip's **true** (unclamped) screen origin
/// `clip_origin_x` over its full width `clip_true_width_px`, and only the tiles
/// intersecting `visible_rect` are drawn. This makes the strip scroll naturally
/// with the timeline and show the correct content when the clip is partly off the
/// left edge (a tile's content time is derived from its offset from the true
/// origin, not from the clamped visible edge).
///
/// Each tile's GPU texture is cached by the **actual** thumbnail timestamp (from
/// [`VideoManager::get_thumbnail_at`]), so as closer thumbnails finish generating
/// the strip refreshes instead of freezing on whatever loaded first.
#[allow(clippy::too_many_arguments)]
fn draw_video_thumbnail_strip(
painter: &egui::Painter,
ui: &egui::Ui,
video_mgr: &lightningbeam_core::video::VideoManager,
textures: &mut std::collections::HashMap<(uuid::Uuid, i64), egui::TextureHandle>,
clip_id: uuid::Uuid,
trim_start: f64,
clip_origin_x: f32,
clip_true_width_px: f32,
visible_rect: egui::Rect,
thumb_top_y: f32,
thumb_height: f32,
pixels_per_second: f64,
) {
if clip_true_width_px <= 0.0 || thumb_height <= 0.0 {
return;
}
// Aspect ratio from any available thumbnail; sets the per-tile width.
let aspect = match video_mgr.get_thumbnail_at(&clip_id, 0.0) {
Some((_, tw, th, _)) if th > 0 => tw as f32 / th as f32,
_ => return,
};
let step = (thumb_height * aspect).max(1.0);
let num_thumbs = ((clip_true_width_px / step).ceil() as usize).max(1);
// Iterate only the tiles that intersect the visible rect.
let first = (((visible_rect.min.x - clip_origin_x) / step).floor() as i64).max(0) as usize;
let last = (((visible_rect.max.x - clip_origin_x) / step).ceil() as i64).max(0) as usize;
let last = last.min(num_thumbs);
for i in first..last {
let thumb_x = clip_origin_x + i as f32 * step;
// Content time at the tile centre, measured from the TRUE clip origin so
// it's stable as the clip scrolls.
let content_time =
trim_start + (i as f64 * step as f64 + step as f64 * 0.5) / pixels_per_second;
let Some((actual_ts, tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&clip_id, content_time)
else {
continue;
};
let ts_key = (actual_ts * 1000.0) as i64;
let texture = textures.entry((clip_id, ts_key)).or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied([tw as usize, th as usize], &rgba_data);
ui.ctx().load_texture(
format!("vthumb_{}_{}", clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(thumb_x, thumb_top_y),
egui::vec2(step, thumb_height),
);
let thumb_rect = full_rect.intersect(visible_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
);
}
}
}
/// Get the effective clip duration for a clip instance on a given layer.
/// For groups on vector layers, the duration spans all consecutive keyframes
/// where the group is present. For regular clips, returns the clip's internal duration.
@ -163,6 +251,11 @@ pub struct TimelinePane {
/// Vertical scroll offset (in pixels)
viewport_scroll_y: f32,
/// Clickable keyframe-diamond hit targets `(screen_rect, keyframe_time)` collected
/// during the last `render_layers`. Used by `handle_input` (next frame) to snap the
/// playhead exactly to a keyframe when its diamond is clicked.
keyframe_diamond_hits: Vec<(egui::Rect, f64)>,
/// Total duration of the animation
duration: f64,
@ -685,6 +778,7 @@ impl TimelinePane {
pixels_per_second: 100.0,
viewport_start_time: 0.0,
viewport_scroll_y: 0.0,
keyframe_diamond_hits: Vec::new(),
duration: 10.0, // Default 10 seconds
is_scrubbing: false,
is_panning: false,
@ -2632,18 +2726,21 @@ impl TimelinePane {
midi_event_cache: &std::collections::HashMap<u32, Vec<daw_backend::audio::midi::MidiEvent>>,
raw_audio_cache: &std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
waveform_gpu_dirty: &mut std::collections::HashSet<usize>,
waveform_minmax_pools: &std::collections::HashMap<usize, u32>,
target_format: wgpu::TextureFormat,
waveform_stereo: bool,
context_layers: &[&lightningbeam_core::layer::AnyLayer],
video_manager: &std::sync::Arc<std::sync::Mutex<lightningbeam_core::video::VideoManager>>,
audio_cache: &HashMap<uuid::Uuid, Vec<ClipInstance>>,
playback_time: f64,
) -> (Vec<(egui::Rect, uuid::Uuid, f64, f64)>, Vec<AutomationLaneRender>) {
) -> (Vec<(egui::Rect, uuid::Uuid, f64, f32)>, Vec<AutomationLaneRender>) {
let painter = ui.painter().clone();
let mut pending_lane_renders: Vec<AutomationLaneRender> = Vec::new();
// Rebuilt each frame; consumed by handle_input (next frame) for click-to-seek.
self.keyframe_diamond_hits.clear();
// Collect video clip rects for hover detection (to avoid borrow conflicts)
let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f64)> = Vec::new();
let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f32)> = Vec::new();
// Track visible video clip IDs for texture cache cleanup
let mut visible_video_clip_ids: std::collections::HashSet<uuid::Uuid> = std::collections::HashSet::new();
@ -2899,8 +2996,9 @@ impl TimelinePane {
theme.text_color(&["#timeline", ".group-bar"], ui.ctx(), egui::Color32::from_rgb(100, 220, 220))
};
for (s, e) in &merged {
let sx = self.time_to_x(*s);
let ex = self.time_to_x(*e).max(sx + MIN_CLIP_WIDTH_PX);
// `merged` ranges are in beats; convert to seconds for time_to_x.
let sx = self.time_to_x(document.tempo_map().transform(*s));
let ex = self.time_to_x(document.tempo_map().transform(*e)).max(sx + MIN_CLIP_WIDTH_PX);
if ex >= 0.0 && sx <= rect.width() {
let vsx = sx.max(0.0);
let vex = ex.min(rect.width());
@ -2940,8 +3038,9 @@ impl TimelinePane {
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
let ci_end = ci_start + ci_duration;
let sx = self.time_to_x(ci_start);
let ex = self.time_to_x(ci_end);
// ci_start/ci_end are in beats; convert to seconds for time_to_x.
let sx = self.time_to_x(document.tempo_map().transform(ci_start));
let ex = self.time_to_x(document.tempo_map().transform(ci_end));
if ex < 0.0 || sx > rect.width() { continue; }
let ci_rect = egui::Rect::from_min_max(
@ -2956,69 +3055,29 @@ impl TimelinePane {
egui::pos2(ci_rect.min.x, span_y_min),
egui::pos2(ci_rect.max.x, span_y_max),
);
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, ci_start));
// 4th elem = clip's TRUE (unclamped) origin x, for correct
// hover content time when scrolled partly off the left.
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, rect.min.x + sx));
let thumb_display_height = (thumb_y_max - span_y_min) - 4.0;
if thumb_display_height > 8.0 {
let video_mgr = video_manager.lock().unwrap();
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&ci.clip_id, 0.0) {
let aspect = tw as f32 / th as f32;
let thumb_display_width = thumb_display_height * aspect;
let ci_width = ci_rect.width();
let num_thumbs = ((ci_width / thumb_display_width).ceil() as usize).max(1);
for ti in 0..num_thumbs {
let x_offset = ti as f32 * thumb_display_width;
if x_offset >= ci_width { break; }
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5)
/ self.pixels_per_second as f64;
let content_time = ci.trim_start + time_offset;
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&ci.clip_id, content_time) {
let ts_key = (content_time * 1000.0) as i64;
let cache_key = (ci.clip_id, ts_key);
let texture = self.video_thumbnail_textures
.entry(cache_key)
.or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied(
[tw as usize, th as usize],
&rgba_data,
// Tile from the clip's true origin (sx unclamped; ci_rect is
// the clamped visible rect) — scrolls correctly off-screen.
draw_video_thumbnail_strip(
&painter,
ui,
&video_mgr,
&mut self.video_thumbnail_textures,
ci.clip_id,
ci.trim_start,
rect.min.x + sx,
ex - sx,
ci_rect,
ci_rect.min.y + 2.0,
thumb_display_height,
self.pixels_per_second as f64,
);
ui.ctx().load_texture(
format!("vthumb_{}_{}", ci.clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(ci_rect.min.x + x_offset, ci_rect.min.y + 2.0),
egui::vec2(thumb_display_width, thumb_display_height),
);
let thumb_rect = full_rect.intersect(ci_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
);
}
}
}
}
}
}
}
@ -3048,8 +3107,17 @@ impl TimelinePane {
None => continue,
};
let total_frames = samples.len() / (*ch).max(1) as usize;
let audio_file_duration = total_frames as f64 / *sr as f64;
// Min/max overview pools store 4 f32 per texel at the
// floor rate sr/B; raw pools store interleaved samples.
let minmax_b = waveform_minmax_pools.get(&audio_pool_index).copied();
let is_minmax = minmax_b.is_some();
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
let total_frames = samples.len() / frame_stride;
let eff_sr: f32 = match minmax_b {
Some(b) => *sr as f32 / b.max(1) as f32,
None => *sr as f32,
};
let audio_file_duration = total_frames as f64 / eff_sr as f64;
let clip_dur = audio_clip.duration;
let mut ci_start = ci.effective_start();
@ -3058,8 +3126,9 @@ impl TimelinePane {
}
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
let ci_screen_start = rect.min.x + self.time_to_x(ci_start);
let ci_screen_end = ci_screen_start + (ci_duration * self.pixels_per_second as f64) as f32;
// ci_start/ci_duration are in beats; convert to seconds for time_to_x.
let ci_screen_start = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start));
let ci_screen_end = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start + ci_duration));
let waveform_rect = egui::Rect::from_min_max(
egui::pos2(ci_screen_start.max(rect.min.x), wave_y_min),
@ -3081,9 +3150,10 @@ impl TimelinePane {
}
Some(crate::waveform_gpu::PendingUpload {
samples: samples.clone(),
sample_rate: *sr,
sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
channels: *ch,
frame_limit,
minmax: is_minmax,
})
} else {
None
@ -3098,7 +3168,7 @@ impl TimelinePane {
viewport_start_time: self.viewport_start_time as f32,
pixels_per_second: self.pixels_per_second as f32,
audio_duration: audio_file_duration as f32,
sample_rate: *sr as f32,
sample_rate: eff_sr,
clip_start_time: ci_screen_start,
trim_start: ci.trim_start as f32,
tex_width: crate::waveform_gpu::tex_width() as f32,
@ -3597,8 +3667,16 @@ impl TimelinePane {
// Sampled Audio: Draw waveform via GPU
lightningbeam_core::clip::AudioClipType::Sampled { audio_pool_index } => {
if let Some((samples, sr, ch)) = raw_audio_cache.get(audio_pool_index) {
let total_frames = samples.len() / (*ch).max(1) as usize;
let audio_file_duration = total_frames as f64 / *sr as f64;
// Min/max overview pools: 4 f32/texel at rate sr/B.
let minmax_b = waveform_minmax_pools.get(audio_pool_index).copied();
let is_minmax = minmax_b.is_some();
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
let total_frames = samples.len() / frame_stride;
let eff_sr: f32 = match minmax_b {
Some(b) => *sr as f32 / b.max(1) as f32,
None => *sr as f32,
};
let audio_file_duration = total_frames as f64 / eff_sr as f64;
let screen_size = ui.ctx().content_rect().size();
let pending_upload = if waveform_gpu_dirty.contains(audio_pool_index) {
@ -3620,9 +3698,10 @@ impl TimelinePane {
Some(crate::waveform_gpu::PendingUpload {
samples: samples.clone(),
sample_rate: *sr,
sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
channels: *ch,
frame_limit,
minmax: is_minmax,
})
} else {
None
@ -3684,7 +3763,7 @@ impl TimelinePane {
viewport_start_time: self.viewport_start_time as f32,
pixels_per_second: self.pixels_per_second as f32,
audio_duration: audio_file_duration as f32,
sample_rate: *sr as f32,
sample_rate: eff_sr,
clip_start_time: iter_screen_start,
trim_start: preview_trim_start as f32,
tex_width: crate::waveform_gpu::tex_width() as f32,
@ -3730,6 +3809,7 @@ impl TimelinePane {
sample_rate: *sr,
channels: *ch,
frame_limit: None, // recording uses incremental path
minmax: false,
})
} else {
None
@ -3794,75 +3874,31 @@ impl TimelinePane {
let thumb_display_height = clip_rect.height() - 4.0;
if thumb_display_height > 8.0 {
let video_mgr = video_manager.lock().unwrap();
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&clip_instance.clip_id, 0.0) {
let aspect = tw as f32 / th as f32;
let thumb_display_width = thumb_display_height * aspect;
let thumb_step_px = thumb_display_width;
let clip_width = clip_rect.width();
let num_thumbs = ((clip_width / thumb_step_px).ceil() as usize).max(1);
for i in 0..num_thumbs {
let x_offset = i as f32 * thumb_step_px;
if x_offset >= clip_width { break; }
// Map pixel position to content time
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5)
/ self.pixels_per_second as f64;
let content_time = clip_instance.trim_start + time_offset;
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(
&clip_instance.clip_id, content_time
) {
let ts_key = (content_time * 1000.0) as i64;
let cache_key = (clip_instance.clip_id, ts_key);
let texture = self.video_thumbnail_textures
.entry(cache_key)
.or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied(
[tw as usize, th as usize],
&rgba_data,
);
ui.ctx().load_texture(
format!("vthumb_{}_{}", clip_instance.clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(clip_rect.min.x + x_offset, clip_rect.min.y + 2.0),
egui::vec2(thumb_display_width, thumb_display_height),
);
let thumb_rect = full_rect.intersect(clip_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
// Tile from the clip's true origin (start_x is unclamped;
// clip_rect is the clamped visible rect) so the strip scrolls
// correctly and shows the right content when partly off-screen.
draw_video_thumbnail_strip(
&painter,
ui,
&video_mgr,
&mut self.video_thumbnail_textures,
clip_instance.clip_id,
clip_instance.trim_start,
rect.min.x + start_x,
end_x - start_x,
clip_rect,
clip_rect.min.y + 2.0,
thumb_display_height,
self.pixels_per_second as f64,
);
}
}
}
}
}
}
// VIDEO PREVIEW: Collect clip rect for hover detection
// VIDEO PREVIEW: Collect clip rect for hover detection. Store the
// clip's TRUE (unclamped) origin x so the hover content time is
// correct even when the clip is scrolled partly off the left.
if let lightningbeam_core::layer::AnyLayer::Video(_) = layer {
video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, instance_start));
video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, rect.min.x + start_x));
}
// Draw border per segment (per loop iteration for looping clips)
@ -3944,6 +3980,40 @@ impl TimelinePane {
color,
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
));
// Record a (slightly enlarged) clickable area for click-to-seek.
self.keyframe_diamond_hits.push((
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
kf.time,
));
}
}
}
// Draw keyframe markers for raster layers (same diamond as vector).
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
for kf in &rl.keyframes {
let x = self.time_to_x(kf.time);
if x >= 0.0 && x <= rect.width() {
let cx = rect.min.x + x;
let cy = y + LAYER_HEIGHT - 8.0;
let size = 5.0;
let diamond = [
egui::pos2(cx, cy - size),
egui::pos2(cx + size, cy),
egui::pos2(cx, cy + size),
egui::pos2(cx - size, cy),
];
let color = theme.bg_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(255, 220, 100));
painter.add(egui::Shape::convex_polygon(
diamond.to_vec(),
color,
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
));
// Record a (slightly enlarged) clickable area for click-to-seek.
self.keyframe_diamond_hits.push((
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
kf.time,
));
}
}
}
@ -4797,6 +4867,30 @@ impl TimelinePane {
}
}
// Pointing-hand cursor when hovering a clickable keyframe diamond.
if let Some(hover) = response.hover_pos() {
if self.keyframe_diamond_hits.iter().any(|(r, _)| r.contains(hover)) {
ui.ctx().set_cursor_icon(egui::CursorIcon::PointingHand);
}
}
// Click a keyframe diamond → snap the playhead exactly to that keyframe.
// (Hit targets come from the previous frame's render_layers; diamonds don't
// move between frames, so a click lands on the right one.)
if response.clicked() && !alt_held {
if let Some(pos) = response.interact_pointer_pos() {
if let Some(&(_, kf_time)) =
self.keyframe_diamond_hits.iter().find(|(r, _)| r.contains(pos))
{
*playback_time = kf_time;
if let Some(controller_arc) = audio_controller {
let mut controller = controller_arc.lock().unwrap();
controller.seek(kf_time);
}
}
}
}
// Get mouse position relative to content area
let mouse_pos = response.hover_pos().unwrap_or(content_rect.center());
let mouse_x = (mouse_pos.x - content_rect.min.x).max(0.0);
@ -5418,7 +5512,7 @@ impl PaneRenderer for TimelinePane {
// Render layer rows with clipping
ui.set_clip_rect(content_rect.intersect(original_clip_rect));
let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time);
let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.waveform_minmax_pools, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time);
// Render playhead on top (clip to timeline area)
ui.set_clip_rect(timeline_rect.intersect(original_clip_rect));
@ -5894,26 +5988,24 @@ impl PaneRenderer for TimelinePane {
// VIDEO HOVER DETECTION: Handle video clip hover tooltips AFTER input handling
// This ensures hover events aren't consumed by the main input handler
for (clip_rect, clip_id, trim_start, instance_start) in video_clip_hovers {
for (clip_rect, clip_id, trim_start, clip_origin_x) in video_clip_hovers {
let hover_response = ui.allocate_rect(clip_rect, egui::Sense::hover());
if hover_response.hovered() {
if let Some(hover_pos) = hover_response.hover_pos() {
// Calculate timestamp at hover position
let hover_offset_pixels = hover_pos.x - clip_rect.min.x;
let hover_offset_time = (hover_offset_pixels as f64) / (self.pixels_per_second as f64);
let hover_timestamp = instance_start + hover_offset_time;
// Remap to clip content time accounting for trim
let clip_content_time = trim_start + (hover_timestamp - instance_start);
// Content time from the clip's TRUE origin. `clip_rect` is clamped
// to the viewport, so using its left edge mislabels the frame when
// the clip is scrolled partly off the left (same bug the strip had).
let hover_offset_pixels = hover_pos.x - clip_origin_x;
let clip_content_time = trim_start + (hover_offset_pixels as f64) / (self.pixels_per_second as f64);
// Try to get thumbnail from video manager
let thumbnail_data: Option<(u32, u32, std::sync::Arc<Vec<u8>>)> = {
let thumbnail_data: Option<(f64, u32, u32, std::sync::Arc<Vec<u8>>)> = {
let video_mgr = shared.video_manager.lock().unwrap();
video_mgr.get_thumbnail_at(&clip_id, clip_content_time)
};
if let Some((thumb_width, thumb_height, ref thumb_data)) = thumbnail_data {
if let Some((_, thumb_width, thumb_height, ref thumb_data)) = thumbnail_data {
// Create texture from thumbnail
let color_image = egui::ColorImage::from_rgba_unmultiplied(
[thumb_width as usize, thumb_height as usize],

23
lightningbeam-ui/lightningbeam-editor/src/preferences/dialog.rs
View File

@ -9,6 +9,7 @@ use crate::config::AppConfig;
use crate::keymap::{self, AppAction, KeymapManager};
use crate::menu::{MenuSystem, Shortcut, ShortcutKey};
use crate::theme::{Theme, ThemeMode};
use lightningbeam_core::file_io::LargeMediaMode;
/// Which tab is selected in the preferences dialog
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@ -59,6 +60,7 @@ struct PreferencesState {
debug: bool,
waveform_stereo: bool,
theme_mode: ThemeMode,
large_media_default: LargeMediaMode,
}
impl From<(&AppConfig, &Theme)> for PreferencesState {
@ -75,6 +77,7 @@ impl From<(&AppConfig, &Theme)> for PreferencesState {
debug: config.debug,
waveform_stereo: config.waveform_stereo,
theme_mode: theme.mode(),
large_media_default: config.large_media_default,
}
}
}
@ -93,6 +96,7 @@ impl Default for PreferencesState {
debug: false,
waveform_stereo: false,
theme_mode: ThemeMode::System,
large_media_default: LargeMediaMode::default(),
}
}
}
@ -567,6 +571,24 @@ impl PreferencesDialog {
&mut self.working_prefs.waveform_stereo,
"Show waveforms as stacked stereo",
);
ui.horizontal(|ui| {
let threshold_gb = lightningbeam_core::beam_archive::LARGE_MEDIA_THRESHOLD
as f64
/ (1024.0 * 1024.0 * 1024.0);
ui.label(format!("Large media (>{:.0} GB):", threshold_gb));
let label = |m: LargeMediaMode| match m {
LargeMediaMode::Ask => "Ask each time",
LargeMediaMode::Pack => "Pack into project",
LargeMediaMode::Reference => "Reference external file",
};
egui::ComboBox::from_id_salt("large_media_default")
.selected_text(label(self.working_prefs.large_media_default))
.show_ui(ui, |ui| {
for mode in [LargeMediaMode::Ask, LargeMediaMode::Pack, LargeMediaMode::Reference] {
ui.selectable_value(&mut self.working_prefs.large_media_default, mode, label(mode));
}
});
});
});
}
@ -629,6 +651,7 @@ impl PreferencesDialog {
config.debug = self.working_prefs.debug;
config.waveform_stereo = self.working_prefs.waveform_stereo;
config.theme_mode = self.working_prefs.theme_mode.to_string_lower();
config.large_media_default = self.working_prefs.large_media_default;
config.keybindings = keybinding_config;
// Apply theme immediately

8
lightningbeam-ui/lightningbeam-editor/src/raster_tool.rs
View File

@ -8,7 +8,7 @@
//! | **B** | Write-only | Output / display. Compositor shows B while the tool is active. |
//! | **C** | Read+Write | Scratch. Dabs accumulate here across the stroke; composite A+C→B each frame. |
//!
//! All three are `Rgba8Unorm` with the same pixel dimensions. The framework
//! All three are `Rgba16Float` with the same pixel dimensions. The framework
//! allocates and validates them in [`begin_raster_workspace`]; tools only
//! dispatch shaders.
@ -45,11 +45,11 @@ pub enum WorkspaceSource {
/// commit or cancel.
#[derive(Debug)]
pub struct RasterWorkspace {
/// A canvas (Rgba8Unorm) — source pixels, uploaded at mousedown, read-only for tools.
/// A canvas (Rgba16Float) — source pixels, uploaded at mousedown, read-only for tools.
pub a_canvas_id: Uuid,
/// B canvas (Rgba8Unorm) — output / display; compositor shows this while active.
/// B canvas (Rgba16Float) — output / display; compositor shows this while active.
pub b_canvas_id: Uuid,
/// C canvas (Rgba8Unorm) — scratch; tools accumulate dabs here across the stroke.
/// C canvas (Rgba16Float) — scratch; tools accumulate dabs here across the stroke.
pub c_canvas_id: Uuid,
/// Optional R8Unorm selection mask (same pixel dimensions as A/B/C).
/// `None` means the entire workspace is selected.

73
lightningbeam-ui/lightningbeam-editor/src/waveform_gpu.rs
View File

@ -112,6 +112,37 @@ pub struct PendingUpload {
/// The texture is allocated at full size, but total_frames is set to
/// the limited count so subsequent calls use the incremental path.
pub frame_limit: Option<usize>,
/// When true, `samples` is interpreted as **pre-packed min/max texels**:
/// 4 floats per "frame" = `[l_min, l_max, r_min, r_max]` (a waveform-pyramid
/// floor level). The caller passes the *effective* `sample_rate` (`sr / B`)
/// so the shader's time→texel mapping covers `B` source samples per texel.
/// When false, `samples` is raw interleaved audio (min = max per texel).
pub minmax: bool,
}
/// Pack one source "frame" into an `Rgba16Float` texel `(Lmin,Lmax,Rmin,Rmax)`.
/// Raw audio sets min = max per channel; min/max input copies the 4 values.
#[inline]
fn pack_texel(samples: &[f32], global_frame: usize, channels: usize, minmax: bool) -> [half::f16; 4] {
if minmax {
let o = global_frame * 4;
[
half::f16::from_f32(samples.get(o).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 1).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 2).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 3).copied().unwrap_or(0.0)),
]
} else {
let so = global_frame * channels;
let left = samples.get(so).copied().unwrap_or(0.0);
let right = if channels >= 2 {
samples.get(so + 1).copied().unwrap_or(left)
} else {
left
};
let (l, r) = (half::f16::from_f32(left), half::f16::from_f32(right));
[l, l, r, r]
}
}
/// Maximum frames to convert and upload per frame (~250K frames ≈ 5.6s at 44.1kHz).
@ -291,8 +322,11 @@ impl WaveformGpuResources {
sample_rate: u32,
channels: u32,
frame_limit: Option<usize>,
minmax: bool,
) -> Vec<wgpu::CommandBuffer> {
let new_total_frames = samples.len() / channels.max(1) as usize;
// For min/max input each "frame" is 4 floats; for raw it's `channels`.
let frame_stride = if minmax { 4 } else { channels.max(1) as usize };
let new_total_frames = samples.len() / frame_stride;
if new_total_frames == 0 {
return Vec::new();
}
@ -329,22 +363,9 @@ impl WaveformGpuResources {
if global_frame >= effective_frames {
break;
}
let sample_offset = global_frame * channels as usize;
let left = if sample_offset < samples.len() {
samples[sample_offset]
} else {
0.0
};
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
samples[sample_offset + 1]
} else {
left
};
let texel_offset = frame * 4;
row_data[texel_offset] = half::f16::from_f32(left);
row_data[texel_offset + 1] = half::f16::from_f32(left);
row_data[texel_offset + 2] = half::f16::from_f32(right);
row_data[texel_offset + 3] = half::f16::from_f32(right);
let t = pack_texel(samples, global_frame, channels as usize, minmax);
row_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
}
let entry = self.entries.get(&pool_index).unwrap();
@ -466,24 +487,9 @@ impl WaveformGpuResources {
for frame in 0..seg_upload_count as usize {
let global_frame = seg_start_frame as usize + frame;
let sample_offset = global_frame * channels as usize;
let left = if sample_offset < samples.len() {
samples[sample_offset]
} else {
0.0
};
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
samples[sample_offset + 1]
} else {
left
};
let texel_offset = frame * 4;
mip0_data[texel_offset] = half::f16::from_f32(left);
mip0_data[texel_offset + 1] = half::f16::from_f32(left);
mip0_data[texel_offset + 2] = half::f16::from_f32(right);
mip0_data[texel_offset + 3] = half::f16::from_f32(right);
let t = pack_texel(samples, global_frame, channels as usize, minmax);
mip0_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
}
// Upload mip 0 (only rows with actual data)
@ -701,6 +707,7 @@ impl egui_wgpu::CallbackTrait for WaveformCallback {
upload.sample_rate,
upload.channels,
upload.frame_limit,
upload.minmax,
);
cmds.extend(new_cmds);
}