Compare commits

...

23 Commits

Author SHA1 Message Date
Skyler Lehmkuhl f585c370e8 docs: mark Phase 3e (playback prefetch) done 2026-06-20 22:46:23 -04:00
Skyler Lehmkuhl ed022995bd Keyframe diamonds: pointing-hand cursor + prefetch during playback
- Pointing-hand cursor when hovering a clickable keyframe diamond.
- Prefetch (Phase 3e, playback only): each update during playback, page in the next
  few upcoming keyframes (PREFETCH_AHEAD=4) per raster layer that aren't resident, via
  the existing async worker. Their full pixels land before the playhead reaches them,
  so playback shows full frames instead of the low-res proxy on every frame (the
  proxy→full pop was the "flicker"). Reactive faults still cover scrubbing.
2026-06-20 22:45:50 -04:00
Skyler Lehmkuhl 3188fc8bb6 Timeline: click a keyframe diamond to snap the playhead to it
render_layers now records each drawn keyframe diamond's screen rect + exact time in
`keyframe_diamond_hits`; handle_input hit-tests a click against them and sets the
playhead (and seeks the audio controller) to the keyframe's exact time. Uses the
previous frame's rects — diamonds don't move between frames, so the click lands
right — which sidesteps the input-before-render ordering and the drag/scroll Y math.
Works for both raster and vector keyframes.
2026-06-20 22:44:41 -04:00
Skyler Lehmkuhl 2cbaf67583 Raster keyframe timeline UI: display + explicit creation + no lazy create
Make raster layers behave like vector on the timeline.

- Timeline: draw a diamond per `RasterKeyframe` (mirrors the vector keyframe block).
- New Keyframe (K / menu): on a raster layer, insert a BLANK cel at the playhead via
  a new undoable `AddRasterKeyframeAction` (+ `RasterLayer::insert_blank_keyframe_at`
  / `remove_keyframe`). Vector path unchanged.
- Stop lazy creation: paint tools now edit the ACTIVE keyframe (at-or-before the
  playhead) instead of creating one. The brush captures the active keyframe's exact
  time; `RasterStroke`/`RasterFillAction` resolve via `keyframe_at_mut` (error if
  none); the tool-site `ensure_keyframe_at` blocks (brush/fill/bucket/wand/quick-
  select/floating-lift) are removed — each read already bails when no keyframe exists.
  New layers still seed a keyframe at the playhead, so there's normally one to paint
  into; painting before the first keyframe is now a no-op (as intended).

Next: onion skinning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 22:22:02 -04:00
Skyler Lehmkuhl 31e23b0fc7 Fix 3d: stroke vanished on commit (establish full buffer on first execute)
The brush commits via a GPU-canvas readback and relies on the action's first
execute() to SET raw_pixels from that readback — at that point raw_pixels is empty
(new keyframe) or the pre-stroke state, never something a dirty-rect diff can stamp
onto. My initial diff-only execute skipped (to avoid corruption), so the stroke
disappeared.

Fix: the action keeps the full post-edit buffer ONLY for the first execute (the
commit), assigning raw_pixels outright exactly like the old code; it's taken/dropped
immediately, so the action sitting in the undo stack still retains just the small
diff. Redo replays via the diff onto the now-resident base.

Also harden the diff itself for the blank-base case: `before_blank` lets apply_after
build from a transparent buffer (redo of a first stroke after undo-to-blank) and
apply_before restore to empty; the resident-base skip is kept only for non-blank
bases (faulted in before undo/redo). Tests cover commit/redo from empty.
2026-06-20 21:44:26 -04:00
Skyler Lehmkuhl c627533185 docs: mark Phase 3d (raster undo dirty-rect diffs) done 2026-06-20 21:31:57 -04:00
Skyler Lehmkuhl aae51e3b3c Phase 3d: dirty-rect diffs for raster undo (bound undo-history RAM)
`RasterStrokeAction`/`RasterFillAction` stored the whole before+after RGBA frame
(~16 MB/action at 1080p → up to ~1.6 GB at the 100-action cap). They now store a
`RasterDiff` — only the changed bounding box's pixels before and after — computed
once in `new()` from the full buffers, which are then dropped. A brush dab shrinks
from ~16 MB to tens of KB; a full-canvas fill is unchanged (its bbox is the frame).

Paging interaction: a diff overwrites just the bbox, so the keyframe's pixels must
be resident when undo/redo applies. A clean evicted frame's container bytes equal
its current logical state, so the editor faults the target frame in (synchronously)
before undo/redo via a new `Action::raster_resident_hint` + `peek_undo/redo_raster_hint`.
Dirty frames are never evicted, so they're already resident. If a base is somehow
not resident the apply is skipped (logged), never resized-and-corrupted.

Unit tests cover exact before/after round-trip, blank-first-stroke, no-op, and the
non-resident-base skip.
2026-06-20 21:31:38 -04:00
Skyler Lehmkuhl 4cb43d670b docs: add Phase 3e (prefetch raster frames) to the plan 2026-06-20 21:18:32 -04:00
Skyler Lehmkuhl a35cc6fa9f Bilinear-sample upscaled raster proxies (smooth, not blocky)
The canvas blit used a nearest sampler, so the upscaled low-res proxy looked
blocky. Added a Linear sampler + `CanvasBlitPipeline::blit_smooth`; the raster
render uses it only for the proxy path (full-res canvas stays nearest/crisp). The
bind-group layout already declares the canvas texture filterable, so no layout
change was needed.
2026-06-20 21:14:47 -04:00
Skyler Lehmkuhl 3c302f0215 docs: mark Phase 3a-3 (image proxy) done 2026-06-20 21:09:30 -04:00
Skyler Lehmkuhl 1bfd09f151 Phase 3a-3: low-res image proxy for cold-scrub raster frames
Scrubbing onto a paged-out raster keyframe flashed blank for the 1-2 frames its
full pixels took to page in. Now a low-res proxy is shown in that gap.

- core: `MediaKind::RasterProxy` (id derived from the keyframe id via
  `raster_proxy_media_id`); `brush_engine::encode_raster_proxy_png` downscales a full
  RGBA buffer to a ≤192px-long-edge PNG. Save writes a proxy beside each resident
  frame's full PNG (paged-out frames keep their existing proxy row, like the full).
  Load eagerly decodes proxies (small) into `RasterKeyframe::proxy`.
- editor: a separate `proxy_layer_cache` in the GPU brush (own recency LRU, budget 64
  since each is ~1/100th a full frame) + `ensure_proxy_texture`/`get_proxy_texture`.
  The raster render, when the full texture isn't resident, blits the proxy mapped to
  the keyframe's FULL logical dims so it upscales via the sampler. F3 VRAM figure now
  includes proxy textures.

When the full pixels land (async fault-in), the full path takes over automatically.
Proxies only exist after a save+reload; freshly-painted unsaved frames stay resident
so they need none.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 20:05:46 -04:00
Skyler Lehmkuhl a1e3cc841f docs: mark Phase 3c (raster GPU cache bound) done 2026-06-20 19:16:13 -04:00
Skyler Lehmkuhl 17d7395229 Phase 3c: bound the raster-layer GPU texture cache + show VRAM in F3
`raster_layer_cache` (one ~w·h·16-byte Rgba16Float CanvasPair per keyframe) had no
size cap — scrubbing a long timeline grew VRAM without bound (~33 MB/frame at 1080p),
the largest unbounded consumer. Added a recency LRU (RASTER_LAYER_CACHE_MAX = 12):
`ensure_layer_texture` bumps the frame to most-recent and evicts the oldest past the
budget; the shown frame (and any rendered this pass) is always most-recent so it's
never the victim. Evicted textures re-upload cheaply from the resident/faulted-in
pixels on revisit. `remove_layer_texture` keeps the LRU in sync.

F3 debug overlay now reports the tracked VRAM (raster cache MB + frame count), pushed
from the GPU brush whenever the cache changes (wgpu exposes no allocator query).
2026-06-20 19:15:16 -04:00
Skyler Lehmkuhl 62d6cd0c84 docs: mark Phase 3a-1/3a-2/3b done; note deferred raster-keyframe-UI bugs 2026-06-20 18:46:46 -04:00
Skyler Lehmkuhl d001ca1083 Fix save crash on zero/sparse-audio projects (phantom pool placeholder)
`AudioPool::load_from_serialized` sizes the slot Vec by pool_index and fills gaps
with empty `AudioFile::new(PathBuf::new(), …)` placeholders. Two bugs let a
placeholder reach the next save and abort it with "Is a directory":

- Off-by-one: `entries.max().unwrap_or(0) + 1` made an *empty* pool length 1, so a
  project with no audio still got one placeholder. Size by `max(pool_index + 1)`
  → empty entries yield length 0.
- `serialize()` emitted placeholder slots: an empty path round-trips to
  `relative_path = Some("")`, which `save_beam` resolves to the project directory
  (`join("")`) and tries to read as media. Skip empty-path / no-packed-media slots.
- Defense in `save_beam`: gate referenced-media packing on `full.is_file()` (not
  `exists()`), so any blank/dir path falls through to embedded data instead of
  reading a directory.

Pre-existing; surfaced by a save → reload → save cycle on a raster-only project.
2026-06-20 18:46:19 -04:00
Skyler Lehmkuhl 39dc402ba3 Phase 3b: bound resident raster pixels with an eviction LRU
Scrubbing a large paint project no longer accumulates every visited frame in RAM.
A fault-in-recency LRU keeps the most-recently-paged-in RASTER_RESIDENT_MAX (12)
keyframes resident and drops the pixels of older *clean* ones (re-arming their
fault-in so they re-page on revisit). The shown frame is always the most-recent
fault-in, so it's never evicted.

Data-loss safety: a new `dirty` flag marks any keyframe whose `raw_pixels` were
mutated by editing (stroke/fill/paint-bucket/floating-lift + their undo/redo) and
is NOT yet in the container. Dirty keyframes are NEVER evicted — they're only
unpinned from the LRU. The flag is cleared on a successful save, which also re-arms
the LRU for the now-clean resident frames so the bound still applies to frames
edited this session.

Also: the save loop now walks all layers (incl. nested) to match the load path's
recursive fault-in arming — evicted frames keep their existing container row
(media_exists), and nested raster keyframes are persisted + covered by live_media.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:56:37 -04:00
Skyler Lehmkuhl 2e07a88905 Phase 3a-2: async raster page-in (no UI block)
The fault-in drain no longer decodes on the UI thread. It now:
- dispatches each newly-requested keyframe's page-in to a background thread
  (deduped via an in-flight set, store cloned per request so path changes are
  picked up), and
- applies completed results from a channel at the top of update(), keeping the UI
  ticking while loads are outstanding.

Cold scrubs no longer freeze. The brief blank gap before a frame lands is removed
by the image proxy (3a-3); eviction to bound RAM while scrubbing is 3b.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:00:04 -04:00
Skyler Lehmkuhl 4228864259 Phase 3a-1: lazy fault-in of raster keyframe pixels
Raster keyframes are no longer eagerly decoded at load — `raw_pixels` stays empty
and is paged in on demand from the project container, so a big paint project opens
instantly and only touched frames hit RAM.

- core: `read_packed_media_readonly` (fresh read-only connection, can't conflict
  with an in-place save) + `RasterStore` (holds the container path; `load_pixels`
  reads+decodes a keyframe's PNG by id). `load_beam_sqlite` stops eager-decoding and
  instead marks every raster keyframe `needs_fault_in` (recursively, incl. nested);
  a freshly-created keyframe stays false (blank-resident, nothing to page). Added
  `Document::all_layers_mut`.
- editor: the canvas records a fault-in request when it needs a paged-out keyframe
  (empty pixels && needs_fault_in); the App drains the sink at the top of update(),
  pages the pixels in via the store, clears the flag, and repaints. Store path is set
  on load and after save. Export faults in synchronously per frame.

Cold-scrub still shows a 1-frame gap and the page-in is synchronous; the image proxy
(3a-2) and async load (3a-3) remove those next.
2026-06-19 16:59:46 -04:00
Skyler Lehmkuhl 6d386a884e Phase 5: fix broken unit tests; cargo test --lib green again
The lib unit tests had gone stale (time values became newtypes) and no longer
compiled. Updated the test code to the current API and fixed the few real issues
the now-running tests surfaced.

Test-only:
- Wrap raw f64 time literals in Beats(...) where the API now takes Beats
  (automation.rs); pass &TempoMap / Beats where signatures changed (clip.rs,
  effect_layer.rs).
- shape.rs: assert the documented no-fill default (fill_color None) instead of Some.
- add_clip_instance / trim_clip_instances tests: register a vector clip with the
  test's clip_id so the action's get_clip_duration lookup succeeds.

Production fix (delete_folder.rs):
- DeleteFolderAction(MoveToParent) reparented child subfolders to the deleted
  folder's parent but never restored them on undo, orphaning them. Track the moved
  subfolder ids and restore their parent on rollback.

Result: daw-backend lib 17 passed; lightningbeam-core lib 264 passed.
2026-06-19 16:45:23 -04:00
Skyler Lehmkuhl 4ad95e6755 Downmix surround to stereo + reload video audio via FFmpeg
Surround → stereo downmix:
- render_from_file folds multichannel sources (5.1/7.1/…) down to stereo with
  proper coefficients (full level for the matching front channel, 1/√2 for centre
  + each surround, LFE dropped), normalized per row to avoid clipping (matching
  ffmpeg's default). Applied uniformly to both the direct-copy and sinc-resample
  paths and to every storage type (PCM, compressed, video audio), only when
  dst==2 && src>2; unknown layouts fall back to front L/R. Previously it just took
  FL/FR, dropping centre dialog + surrounds.

Proper video-audio reload:
- A video's audio track is now stored as a path reference to the video (never
  packed/embedded as audio media) and re-probed via FFmpeg on load into a
  streaming VideoAudio entry, so multichannel audio survives reload (the old
  Symphonia reconstitution collapsed it, breaking the downmix). Driven by a new
  AudioPoolEntry.is_video_audio flag across serialize / save_beam / load. Also
  removes the decode-whole-video-to-RAM + temp-file path on load.

Fix video scaling:
- Any video with dimensions larger than the stage was being scaled down into the corner incorrectly; we now bake the frame-clip scale into the instance transform.
2026-06-17 18:30:42 -04:00
Skyler Lehmkuhl 097345be76 Fix video thumbnail strip bugs + persist thumbnails (resumable)
Thumbnail rendering fixes:
- Strip now tiles from each clip's true (unclamped) origin and draws only the
  tiles intersecting the visible rect, so it scrolls correctly and shows the
  right frames when a clip is scrolled partly off the left. Both render sites
  (collapsed group + expanded track) share one draw_video_thumbnail_strip helper.
- On-clip strip no longer freezes on the first thumbnail: get_thumbnail_at now
  returns the actual thumbnail timestamp and the GPU texture cache keys on it, so
  tiles refresh as closer thumbnails finish generating.
- Hover preview derives content time from the clip's true origin too (matches the
  strip when scrolled off-screen).
- insert_thumbnail keeps the cache sorted + deduped (fixes a latent unsorted
  binary_search bug, and makes concurrent restore + resume race-safe).

Thumbnail persistence (mirrors waveform persistence):
- MediaKind::Thumbnail rows, keyed by thumbnail_media_id(clip_id) (clip id XOR a
  sentinel). Each clip's thumbnails PNG-encoded into one opaque LBTN blob (editor
  owns the format), snapshotted cheaply (Arc clones) and encoded off the UI thread.
- Save writes the packs (kept in place on re-save); load reads them into
  LoadedProject.thumbnail_blobs; the editor decodes + inserts them on a background
  thread, so reload shows thumbnails instantly with no re-decode (even if the
  source video file is missing).
- Partial sets are persisted with a complete flag and RESUMED on load:
  generate_keyframe_thumbnails takes a should_skip predicate so a save made
  mid-generation continues from where it left off instead of redoing the work.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 15:36:56 -04:00
Skyler Lehmkuhl c784816615 Phase 2: bound video frame cache + stream the export mux
- VideoManager.frame_cache: unbounded HashMap (grew per distinct frame during
  playback) -> LruCache evicted by a 256MB byte budget. Byte-budget rather than
  frame count is robust across resolutions (a 4K frame is ~33MB vs ~2MB at
  800x600). unload_video pops per-clip keys (LruCache has no retain).
- mux_video_and_audio: stream-merge the two inputs by PTS with one pending
  packet per stream (O(1) memory) instead of collecting every packet into Vecs
  first (O(duration)). Output is byte-identical.
- export AAC: sanitize the planar-f32 path (non-finite -> 0, finite clamped to
  [-1,1]) like the integer paths, with a one-time warning. A stray NaN/Inf
  render sample no longer fails the whole export.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 14:30:32 -04:00
Skyler Lehmkuhl 3d7cff9ad0 Stream audio & video from .beam container; waveform LOD pyramid + persistence
Migrate the .beam container to SQLite and stream media from it instead of
decoding whole files into RAM on import/load.

Container & large files:
- SQLite .beam container (beam_archive) with in-place transactional saves and an
  incremental BlobReader; supports both packed (chunked blobs) and referenced
  (external path) media, with a user preference + first-import prompt for files
  over the large-media threshold.

Audio streaming:
- Stream packed compressed audio on load via an inversion-of-control blob factory
  (AudioBlobSourceFactory): daw-backend defines the trait, core implements it
  over BlobReader, so the audio engine stays container-agnostic.
- Bulk-activate disk streaming for all loaded clips after SetProject.
- Sample-accurate compressed seek (SeekMode::Accurate; Coarse mislands on VBR).

Video:
- Video frames decoded/streamed on demand; thumbnails generated asynchronously
  on a dedicated decoder so import/load never blocks the UI.
- The video's audio track is streamed on demand via an ffmpeg VideoAudioReader
  as a separate editable AudioClip (no /tmp WAV extraction).

Waveform overview:
- Streaming min/max LOD pyramid (waveform_pyramid), bounded memory, configurable
  floor B; serialized into the container and restored on load (or generated in
  the background from the packed blob when absent), so no re-decode on reload.
- GPU min/max upload path; integer-LOD textureLoad fixes zoom-dependent wobble.
2026-06-17 13:52:38 -04:00
48 changed files with 6509 additions and 1289 deletions

769
STREAMING_TO_DISK_PLAN.md Normal file
View File

@ -0,0 +1,769 @@
# Streaming Media To/From Disk — Plan
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
and to disk when the data is too large to fit comfortably in memory, rather than loading
the entire file regardless of size.
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
is impossible while the container forces a full decode, so the container decision (below)
is now part of this plan.
## Deferred bugs (do at the end)
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
## Deferred raster-keyframe-UI bugs (pre-existing; found during Phase 3 testing)
Both stem from the **timeline having no model for raster keyframes** — it renders *clip
instances* (`layer_clips`/`collect_clip_instances` return `&[ClipInstance]`), but raster layers use
`keyframes: Vec<RasterKeyframe>`; every raster arm in timeline.rs is a stub (`Raster => &[]`/`{}`).
- **(a) `Timeline > New Keyframe` doesn't refresh the canvas** until you draw. The menu action
creates a blank raster keyframe but doesn't trigger a canvas repaint / GPU-cache invalidation for
the new (different) `kf.id`, so the stale previous-frame texture stays until a stroke dirties it.
- **(b) Raster keyframes never render on the timeline** — no code walks `RasterLayer::keyframes` to
draw markers. Needs a raster-keyframe strip (display + click-to-navigate + insert).
Confirmed not paging regressions (same before 3a-1). A focused "raster keyframe timeline UI" task.
## Noted enhancements (later, after the phases)
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
Native multichannel support remains a separate, larger project.
- **Export speed:** a 1:14 1080p MP4 took ~9:06 to export (~7.4x slower than realtime). The video
export pipeline re-seeks + decodes per output frame (see `[Video Seek]`/`[Video Timing]` logs) and
does CPU YUV conversion; likely wins from sequential decode (avoid per-frame seeks), reusing the
decode cache, and/or GPU-side color conversion. Profile before optimizing.
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
decode) still worth chasing if it recurs.
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
2 h video keeps that work and continues from there on reload.
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
complete clips; a mid-generation save resumes from where it left off on reload.)*
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
on-disk: JPEG ~36 KB/thumb → **~6 MB/2 h**; PNG ~815 KB → ~14 MB/2 h. So persistence is cheap
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
absent). The 5 s interval already bounds count; no extra budget needed.
- **Progressive waveform on first import:** generation streams the whole file before the
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
the clip left-to-right instead of appearing all at once. Persistence saves only the final
complete pyramid.
## Guiding principle
Three subsystems already have the right streaming primitive; most of the work is wiring,
bounding caches, and adding a residency window. The recurring pattern:
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
> stable ID, and evict everything outside a window around the playhead.
---
## Audit summary (where we stand today)
### Correctly streaming / bounded
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame`
keyframe-index seek + decode-until-target, one frame resident).
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
frames in flight).
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
### Fully-loaded, unbounded by file length (the problems)
| Site | Issue |
|---|---|
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
---
## Container format decision: `.beam` → SQLite *(DECIDED)*
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
`.beam` extension). This is the foundation the rest of the plan builds on.
### Why
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
mutation** — every save and every raster frame write-back rewrites the whole archive — and
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
SQLite dissolves the single-file-vs-performance tension:
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
confusion; we have no bundle magic on Linux/Windows).
- **Streaming reads**`sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
atomicity).
- **Inspectable / scriptable**`sqlite3` CLI; `beam_inspector.py` can read it directly.
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
reads (or external paths); raster frames live in blob rows and write back transactionally.
### Large-media policy: packed OR referenced
Two storage modes per media item, both supported:
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
on a network drive, or media too large/volatile to pack).
**Default-mode preference for files over the per-blob limit (~2GB):**
- A user preference `large_media_default: Pack | Reference` controls what happens to
imports above the threshold.
- The **first time** the user imports a media file over the limit, **prompt** them
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
large imports (changeable later in settings).
- Files under the limit are packed by default (chunked only if needed).
### Schema sketch
```
media(
id BLOB PRIMARY KEY, -- stable Uuid
kind INTEGER, -- audio | video | raster | image-asset
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
storage INTEGER, -- 0 = packed, 1 = referenced
ext_path TEXT, -- set when storage = referenced
total_len INTEGER, -- bytes (packed) for chunk math
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
)
media_chunk(
media_id BLOB, chunk_index INTEGER, bytes BLOB,
PRIMARY KEY (media_id, chunk_index)
)
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
```
`project.json` stays the same serialized `BeamProject` for now — only its container and the
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
first open/save.
### Streaming reads from packed media
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
plain `File` exactly as `do_import_audio` already does for originals today.
---
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
### 1a. Turn on the compressed-audio disk reader
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
assign `AudioClip::read_ahead`.
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
from the ring buffer, not RAM.
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
already distinguishes.
### 1b. Stream on project load *(depends on the SQLite container)*
Three coupled changes (none works alone):
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
for them. After the engine installs the project, walk all audio clips and
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
`BlobReader` instead of a path for packed media.)
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
table instead of base64 `embedded_data`.
### 1c. Video's embedded audio track — stream from the video via ffmpeg
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
save/reload. **Superseded by the design below.**
**Proper design — stream the video's audio track on demand, never materialized.**
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
interface.
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
`sample_rate`/`channels`/`total_frames`.
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
entry `path` = the video file.
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
One ffmpeg context + 3 s buffer per active clip instance.
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)`
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
cleanup.
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
frames before target; for the frame straddling target, drop its leading samples). After
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
while here, or at least flag.
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
seek (both readers) → waveform-peaks pass.
---
## Phase 2 — Video: bound the caches *(small, isolated)*
### 2a. Bound `VideoManager.frame_cache`
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
### 2b. Stream the export mux
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
---
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
the container is opened only at load/save (no live handle).
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
### Drive-by (Arc pixels): DROPPED
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
clone). Not worth 64 edits. Start at 3a.
### 3a. Lazy async fault-in + image proxy
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
results applied at top of `update()`. No UI block on cold scrub.
- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written
beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`.
Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to
the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident.
*(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge
paint projects.)*
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
leave full `raw_pixels` empty.
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
upscaled proxy. Reused by the exporter (already frame-by-frame).
### 3b. Residency window + eviction **[DONE]**
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
rows via `media_exists` (no data loss) and walks all layers to match load.
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
### 3c. Bound the GPU cache **[DONE for raster_layer_cache]**
`raster_layer_cache` (`gpu_brush.rs`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
`w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in
`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay
now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom
if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello
`ImageCache` is image *assets* → Phase 4.)*
### 3d. Undo memory **[DONE]**
`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames.
Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in
`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor
faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`),
correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒
skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose
bbox is the whole frame.)*
### 3e. Prefetch frames **[DONE for playback]**
Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4`
upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so
full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during
playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a
prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction
prefetch still TODO.)*
Original note: *(future, after 3d — pure latency win, no correctness need)*
Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the
proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**:
on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of
playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker +
`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap
concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU
texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res
frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up.
### Build order & tests
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
---
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
**Data model.**
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
`Pixmap` map, keyed by asset id, **unbounded**.
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
**No asset→frame or frame→asset index exists today.**
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
the heavy one).
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
### 4b. Usage bookkeeping (the multi-frame problem)
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
paths in `vector_graph` and the relevant actions).
- count 0 → dead, fully evictable / GC candidate.
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
playhead**, not by count (a high-count asset far from the playhead is still evicted).
Residency decision: `resident = needed-now needed-soon`; beyond that, an **LRU with a byte
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
Eviction never touches an asset in needed-now.
### 4c. Bound the decoded tier
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
image-texture cache the same way, keyed to the residency window.
### Nested-clip prefetch (important)
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
each of which can reference its own image assets. Prefetch must therefore:
- Recurse through clip instances when computing both needed-now and needed-soon.
- Map outer playhead time → each nested clip's local time, and look ahead along the
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
keyframe are loaded before the nested clip reaches it.
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
once); the usage index handles refcounting.
---
## Cross-cutting: a shared residency abstraction
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
disk backing, residency = window/needed-set around playhead + LRU byte budget:
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|---|---|---|---|---|
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
enumeration + usage index is unique to Phase 4.
---
## Sequencing
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
2. **Phase 2** — small, isolated, independently shippable; container-independent.
3. **Phase 0 (container)**`.beam` ZIP → SQLite + `BlobReader` + large-media policy +
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
5. **Phase 1c** — depends on 1b's pool path.
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
---
## Status
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
buffer (was effectively silent before, since `read_ahead` was always `None`).
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
(bundled) — `lightningbeam-core/src/beam_archive.rs`
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
referenced-path, overwrite-replaces-chunks.
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
changed rows written; unchanged large media never rewritten); orphan cleanup via
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
must NOT copy+rename for existing SQLite files.
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
deleted (now packs original bytes with their codec).
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
`Ask` behaves as `Reference` at save time.
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
import_audio/video/image) queues a one-time modal; choice persists to config.
Threshold shown in the modal is derived from the constant.
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
skip unchanged frames on in-place save (Phase 3).
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
> build the lib in normal mode. Worth fixing separately.
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
+ proper video-audio reload remain as noted follow-ups.
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory**
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
violation; referenced-only-first = leaves packed <2GB in RAM.)
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
never fires for loaded clips (they bypass `AddAudioClip`).
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)``BeamBlobFactory`
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
by the editor between `load_audio_pool` and `set_project`.
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips`
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
from the full in-RAM decode, which no longer happens). Added
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
overview now fills in shortly after load.
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
then instantly on subsequent loads once saved; RAM stays flat on a big project.
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
`from_compressed` placeholder). Higher risk (touches the working referenced path); packed
covers the common <2GB case first.
- [ ] **Deferred (follow-up): packed video streaming.** Let small videos be packed into the `.beam`
(a `MediaKind::Video` blob, `VideoClip` referencing it by id) and stream **both frames and audio**
from the DB blob via FFmpeg. ffmpeg-next has no custom-I/O wrapper, so this needs an
`AVIOContext`-over-`BlobReader` shim via raw FFI. **Decision (user):** that FFI wrapper lives in
its **own crate, version-pinned to the ffmpeg version**, isolating the unsafe + the ABI coupling.
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
small temp partitions) and the temp path doesn't survive reload.
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
reference survives save/load.
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
empty channel layout → must `set_channel_layout` before resampling, else swr returns
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
until wired. `VideoAudioReader` made `pub` for the integration test.
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
daw-backend + editor compile clean; VideoAudioReader tests still pass.
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
regression) and that an activated VideoAudio clip produces sound.
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)`
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
**→ Feature is live end-to-end; ready for in-app testing.**
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
whole packet is discarded). So compressed clips no longer drift vs video audio after
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs`
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
Ragged last buckets reduce over available children (no value padding). Bounded
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
ordering, stereo channels, size bound, chunk-agnostic.
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
memory (one chunk + the pyramid). Test: envelope matches the signal through
both backends.
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
- [ ] Orchestration (with 1c).
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
drained in `update()``raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
- [x] Generation: on video-audio import Success, the same bg thread streams
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
main.rs construction) → `render_layers`**both** render sites (collapsed-group
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
features hold position at every zoom and line up with playback.
See memory `waveform-shader-fractional-mip-offset`.
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
`register_loaded_videos` only loads frames (not audio), so there is no redundant
regeneration to suppress. Compiles clean across all three crates.
### Waveform LOD pyramid design (step 6)
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
in → per-sample; seamless between.
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
stored.
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
temp before first save). `B` is stored with it (preference is just the default for new gen).
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
cleared only after its children") — so rendering can always walk up to a resident ancestor;
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 15 —
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
above the floor it's a disk read; below it the span is already small.
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
viewports of fine tiles; persist pyramid in `.beam`.
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
in-app export to confirm A/V sync.)*
- [ ] Phase 3a — lazy raster fault-in from blob store
- [ ] Phase 3b — raster residency window + eviction
- [ ] Phase 3c — bound raster GPU/CPU caches
- [ ] Phase 3d — spill undo snapshots
- [ ] Phase 4a — frame→asset enumeration (recursive)
- [ ] Phase 4b — usage bookkeeping + LRU residency
- [ ] Phase 4c — bound decoded image tier
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.

View File

@ -189,22 +189,22 @@ mod tests {
fn test_add_points_sorted() { fn test_add_points_sorted() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(2.0, 0.5, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(2.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(3.0, 0.8, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(3.0), 0.8, CurveType::Linear));
assert_eq!(lane.points().len(), 3); assert_eq!(lane.points().len(), 3);
assert_eq!(lane.points()[0].time, 1.0); assert_eq!(lane.points()[0].time, Beats(1.0));
assert_eq!(lane.points()[1].time, 2.0); assert_eq!(lane.points()[1].time, Beats(2.0));
assert_eq!(lane.points()[2].time, 3.0); assert_eq!(lane.points()[2].time, Beats(3.0));
} }
#[test] #[test]
fn test_replace_point_at_same_time() { fn test_replace_point_at_same_time() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
assert_eq!(lane.points().len(), 1); assert_eq!(lane.points().len(), 1);
assert_eq!(lane.points()[0].value, 0.5); assert_eq!(lane.points()[0].value, 0.5);
@ -214,59 +214,59 @@ mod tests {
fn test_linear_interpolation() { fn test_linear_interpolation() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.0, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(0.0), 0.0, CurveType::Linear));
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Linear));
assert_eq!(lane.evaluate(0.0), Some(0.0)); assert_eq!(lane.evaluate(Beats(0.0)), Some(0.0));
assert_eq!(lane.evaluate(0.5), Some(0.5)); assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
assert_eq!(lane.evaluate(1.0), Some(1.0)); assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
} }
#[test] #[test]
fn test_step_interpolation() { fn test_step_interpolation() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Step)); lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Step));
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Step)); lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Step));
assert_eq!(lane.evaluate(0.0), Some(0.5)); assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
assert_eq!(lane.evaluate(0.5), Some(0.5)); assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
assert_eq!(lane.evaluate(0.99), Some(0.5)); assert_eq!(lane.evaluate(Beats(0.99)), Some(0.5));
assert_eq!(lane.evaluate(1.0), Some(1.0)); assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
} }
#[test] #[test]
fn test_evaluate_outside_range() { fn test_evaluate_outside_range() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(2.0, 1.0, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(2.0), 1.0, CurveType::Linear));
// Before first point // Before first point
assert_eq!(lane.evaluate(0.0), Some(0.5)); assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
// After last point // After last point
assert_eq!(lane.evaluate(3.0), Some(1.0)); assert_eq!(lane.evaluate(Beats(3.0)), Some(1.0));
} }
#[test] #[test]
fn test_disabled_lane() { fn test_disabled_lane() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Linear));
lane.enabled = false; lane.enabled = false;
assert_eq!(lane.evaluate(0.0), None); assert_eq!(lane.evaluate(Beats(0.0)), None);
} }
#[test] #[test]
fn test_remove_point() { fn test_remove_point() {
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume); let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
lane.add_point(AutomationPoint::new(2.0, 0.8, CurveType::Linear)); lane.add_point(AutomationPoint::new(Beats(2.0), 0.8, CurveType::Linear));
assert!(lane.remove_point_at_time(1.0, 0.001)); assert!(lane.remove_point_at_time(Beats(1.0), Beats(0.001)));
assert_eq!(lane.points().len(), 1); assert_eq!(lane.points().len(), 1);
assert_eq!(lane.points()[0].time, 2.0); assert_eq!(lane.points()[0].time, Beats(2.0));
} }
} }

View File

@ -307,23 +307,96 @@ impl ReadAheadBuffer {
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Wraps a Symphonia decoder for streaming a single compressed audio file. /// Wraps a Symphonia decoder for streaming a single compressed audio file.
struct CompressedReader { ///
/// Public (like [`VideoAudioReader`]) only so integration tests can exercise it
/// directly; treat it as crate-internal.
pub struct CompressedReader {
format_reader: Box<dyn symphonia::core::formats::FormatReader>, format_reader: Box<dyn symphonia::core::formats::FormatReader>,
decoder: Box<dyn symphonia::core::codecs::Decoder>, decoder: Box<dyn symphonia::core::codecs::Decoder>,
track_id: u32, track_id: u32,
/// Current decoder position in frames. /// Current decoder position in frames.
current_frame: u64, current_frame: u64,
/// Frames still to drop from the front of decoded output, so that after a
/// (coarse) seek the next emitted sample lands exactly on the target frame.
pending_discard: u64,
sample_rate: u32, sample_rate: u32,
channels: u32, channels: u32,
#[allow(dead_code)]
total_frames: u64, total_frames: u64,
/// Temporary decode buffer. /// Temporary decode buffer.
sample_buf: Option<SampleBuffer<f32>>, sample_buf: Option<SampleBuffer<f32>>,
} }
/// A seekable byte stream for packed media held in the host's project container.
///
/// `daw-backend` stays container-agnostic: it never references the `.beam` SQLite
/// store directly. Instead the host (lightningbeam-core) implements this trait over
/// its incremental blob reader and installs a factory ([`AudioBlobSourceFactory`])
/// into the engine, so packed compressed audio can be stream-decoded without ever
/// being fully loaded into RAM.
pub trait MediaByteSource: std::io::Read + std::io::Seek + Send + Sync {
/// Total length of the stream in bytes (Symphonia needs this for seeking).
fn byte_len(&self) -> u64;
}
/// Opens fresh byte streams for packed media by id. Installed into the engine by
/// the host; invoked when activating a clip backed by container-packed audio.
/// (`Debug` so it can ride in the `Query` enum, which derives `Debug`.)
pub trait AudioBlobSourceFactory: Send + Sync + std::fmt::Debug {
/// Open a new independent reader for the packed media item `media_id`
/// (the UUID string stored on the audio pool entry).
fn open(&self, media_id: &str) -> Result<Box<dyn MediaByteSource>, String>;
}
/// Adapts a [`MediaByteSource`] to Symphonia's `MediaSource` (adds the seekable +
/// byte-length metadata Symphonia's probe/seek require).
struct SymphoniaByteSource(Box<dyn MediaByteSource>);
impl std::io::Read for SymphoniaByteSource {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.0.read(buf)
}
}
impl std::io::Seek for SymphoniaByteSource {
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
self.0.seek(pos)
}
}
impl symphonia::core::io::MediaSource for SymphoniaByteSource {
fn is_seekable(&self) -> bool {
true
}
fn byte_len(&self) -> Option<u64> {
Some(self.0.byte_len())
}
}
/// How to open a streaming audio source: a filesystem path (referenced media or a
/// video file) or a host-provided byte stream (container-packed media).
pub enum StreamOpen {
Path(PathBuf),
Source {
src: Box<dyn MediaByteSource>,
/// Codec/extension hint for the Symphonia probe (e.g. `"mp3"`, `"flac"`).
ext: Option<String>,
},
}
impl CompressedReader { impl CompressedReader {
pub fn sample_rate(&self) -> u32 {
self.sample_rate
}
pub fn channels(&self) -> u32 {
self.channels
}
/// Total frames from the codec header (0 if the format doesn't report it).
pub fn total_frames(&self) -> u64 {
self.total_frames
}
/// Open a compressed audio file and prepare for streaming decode. /// Open a compressed audio file and prepare for streaming decode.
fn open(path: &Path) -> Result<Self, String> { pub fn open(path: &Path) -> Result<Self, String> {
let file = let file =
std::fs::File::open(path).map_err(|e| format!("Failed to open file: {}", e))?; std::fs::File::open(path).map_err(|e| format!("Failed to open file: {}", e))?;
let mss = MediaSourceStream::new(Box::new(file), Default::default()); let mss = MediaSourceStream::new(Box::new(file), Default::default());
@ -332,7 +405,21 @@ impl CompressedReader {
if let Some(ext) = path.extension().and_then(|e| e.to_str()) { if let Some(ext) = path.extension().and_then(|e| e.to_str()) {
hint.with_extension(ext); hint.with_extension(ext);
} }
Self::from_mss(mss, hint)
}
/// Open a compressed stream from a host-provided byte source (packed media).
pub fn open_source(src: Box<dyn MediaByteSource>, ext: Option<&str>) -> Result<Self, String> {
let mss = MediaSourceStream::new(Box::new(SymphoniaByteSource(src)), Default::default());
let mut hint = Hint::new();
if let Some(ext) = ext {
hint.with_extension(ext);
}
Self::from_mss(mss, hint)
}
/// Shared probe + decoder setup over an already-built media stream.
fn from_mss(mss: MediaSourceStream, hint: Hint) -> Result<Self, String> {
let probed = symphonia::default::get_probe() let probed = symphonia::default::get_probe()
.format( .format(
&hint, &hint,
@ -368,6 +455,7 @@ impl CompressedReader {
decoder, decoder,
track_id, track_id,
current_frame: 0, current_frame: 0,
pending_discard: 0,
sample_rate, sample_rate,
channels, channels,
total_frames, total_frames,
@ -375,9 +463,17 @@ impl CompressedReader {
}) })
} }
/// Seek to a specific frame. Returns the actual frame reached (may differ /// Seek to `target_frame`, **sample-accurately**. Uses `SeekMode::Accurate`:
/// for compressed formats that can only seek to keyframes). /// for an elementary stream like MP3 a *coarse* seek byte-estimates the
fn seek(&mut self, target_frame: u64) -> Result<u64, String> { /// position and seeds the timestamp from that estimate — which for VBR (or a
/// file whose header padding the estimate ignores) lands off by up to ~1s.
/// Accurate mode instead counts frame *headers* (no decode) from a true anchor
/// (the current position, or a rewind to the start for backward seeks), so the
/// returned `actual_ts` is exact; the small residual to `target_frame` is then
/// dropped in `decode_next`. Container formats with seek tables (FLAC/OGG) seek
/// cheaply; a long MP3 walks headers from the anchor (I/O, not decode) — a
/// per-file seek index would make that O(1) (future work).
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
let seek_to = SeekTo::TimeStamp { let seek_to = SeekTo::TimeStamp {
ts: target_frame, ts: target_frame,
track_id: self.track_id, track_id: self.track_id,
@ -385,21 +481,23 @@ impl CompressedReader {
let seeked = self let seeked = self
.format_reader .format_reader
.seek(SeekMode::Coarse, seek_to) .seek(SeekMode::Accurate, seek_to)
.map_err(|e| format!("Seek failed: {}", e))?; .map_err(|e| format!("Seek failed: {}", e))?;
let actual_frame = seeked.actual_ts; let actual_frame = seeked.actual_ts;
self.current_frame = actual_frame; self.current_frame = actual_frame;
// Drop the frames between where the coarse seek landed and the target.
self.pending_discard = target_frame.saturating_sub(actual_frame);
// Reset the decoder after seeking. // Reset the decoder after seeking.
self.decoder.reset(); self.decoder.reset();
Ok(actual_frame) Ok(target_frame)
} }
/// Decode the next chunk of audio into `out`. Returns the number of frames /// Decode the next chunk of audio into `out`. Returns the number of frames
/// decoded. Returns `Ok(0)` at end-of-file. /// decoded. Returns `Ok(0)` at end-of-file.
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> { pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
out.clear(); out.clear();
loop { loop {
@ -428,10 +526,22 @@ impl CompressedReader {
if let Some(ref mut buf) = self.sample_buf { if let Some(ref mut buf) = self.sample_buf {
buf.copy_interleaved_ref(decoded); buf.copy_interleaved_ref(decoded);
let samples = buf.samples(); let samples = buf.samples();
out.extend_from_slice(samples); let ch = self.channels as usize;
let frames = samples.len() / self.channels as usize; let frames = samples.len() / ch;
self.current_frame += frames as u64;
return Ok(frames); // Drop leading frames for sample-accurate seek alignment.
let discard = self.pending_discard.min(frames as u64) as usize;
self.pending_discard -= discard as u64;
out.extend_from_slice(&samples[discard * ch..]);
let emitted = frames - discard;
self.current_frame += emitted as u64;
if emitted > 0 {
return Ok(emitted);
}
// Whole packet discarded for alignment — keep decoding so
// we never falsely report EOF (Ok(0)).
continue;
} }
return Ok(0); return Ok(0);
@ -445,16 +555,383 @@ impl CompressedReader {
} }
} }
// ---------------------------------------------------------------------------
// VideoAudioReader
// ---------------------------------------------------------------------------
/// Streams the audio track out of a media file (a video container, or any audio
/// file) using FFmpeg, decoding on demand. Mirrors [`CompressedReader`]'s
/// interface so the disk reader can drive either through [`StreamSource`].
///
/// Seeking is **sample-accurate**: after `seek(target)`, the next `decode_next`
/// yields samples beginning at exactly `target`. FFmpeg's container seek only
/// lands at-or-before the target, so we decode forward and discard the leading
/// samples to hit the frame precisely — this keeps video audio frame-synced with
/// other (mmap/in-memory) clips.
///
/// Public (vs. the private `CompressedReader`) only so integration tests can
/// exercise it directly; treat it as crate-internal.
pub struct VideoAudioReader {
input: ffmpeg_next::format::context::Input,
decoder: ffmpeg_next::decoder::Audio,
/// Built lazily from the first decoded frame's format/layout → interleaved f32.
resampler: Option<ffmpeg_next::software::resampling::Context>,
stream_index: usize,
/// Seconds per stream-timestamp unit.
time_base: f64,
sample_rate: u32,
channels: u32,
total_frames: u64,
/// Absolute frame index of the next sample `decode_next` will output.
current_frame: u64,
/// Frames still to drop from the front of decoded output (seek alignment).
pending_discard: u64,
/// When set, the next decoded frame establishes the discard needed to land on
/// this absolute target frame (sample-accurate seek).
align_to: Option<u64>,
}
impl VideoAudioReader {
pub fn sample_rate(&self) -> u32 {
self.sample_rate
}
pub fn channels(&self) -> u32 {
self.channels
}
/// Estimated total audio frames (from the stream/container duration).
pub fn total_frames(&self) -> u64 {
self.total_frames
}
pub fn open(path: &Path) -> Result<Self, String> {
ffmpeg_next::init().map_err(|e| e.to_string())?;
let input = ffmpeg_next::format::input(&path)
.map_err(|e| format!("Failed to open media: {}", e))?;
// Pull stream scalars + build the decoder inside a scope so the stream
// borrow of `input` ends before we use `input` again.
let (stream_index, time_base, stream_duration, decoder) = {
let stream = input
.streams()
.best(ffmpeg_next::media::Type::Audio)
.ok_or_else(|| "No audio stream found".to_string())?;
let stream_index = stream.index();
let time_base = f64::from(stream.time_base());
let stream_duration = stream.duration();
let ctx = ffmpeg_next::codec::context::Context::from_parameters(stream.parameters())
.map_err(|e| e.to_string())?;
let decoder = ctx.decoder().audio().map_err(|e| e.to_string())?;
(stream_index, time_base, stream_duration, decoder)
};
let sample_rate = decoder.rate();
let channels = decoder.channels() as u32;
let duration_secs = if stream_duration > 0 {
stream_duration as f64 * time_base
} else if input.duration() > 0 {
input.duration() as f64 / f64::from(ffmpeg_next::ffi::AV_TIME_BASE)
} else {
0.0
};
let total_frames = (duration_secs * sample_rate as f64).max(0.0) as u64;
Ok(Self {
input,
decoder,
resampler: None,
stream_index,
time_base,
sample_rate,
channels,
total_frames,
current_frame: 0,
pending_discard: 0,
align_to: None,
})
}
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
let seconds = target_frame as f64 / self.sample_rate.max(1) as f64;
let ts_av = (seconds * f64::from(ffmpeg_next::ffi::AV_TIME_BASE)) as i64;
// Seek to at-or-before the target (max_ts = ts_av) so we can decode
// forward to it exactly. ffmpeg-next's `seek` wants a bounded range.
self.input
.seek(ts_av, 0..(ts_av + 1))
.map_err(|e| format!("Seek failed: {}", e))?;
self.decoder.flush();
self.pending_discard = 0;
self.align_to = Some(target_frame);
self.current_frame = target_frame;
// We align to the exact frame below, so the effective position IS target.
Ok(target_frame)
}
pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
out.clear();
loop {
// Drain a decoded frame if one is ready.
let mut decoded = ffmpeg_next::frame::Audio::empty();
if self.decoder.receive_frame(&mut decoded).is_ok() {
self.ensure_layout(&mut decoded);
let n = self.emit(&decoded, out);
if n > 0 {
return Ok(n);
}
continue; // frame fully discarded by seek-alignment; keep going
}
// Read one packet (owned), releasing the `input` borrow before decoding.
let packet = self.input.packets().next().map(|(_, p)| p);
match packet {
Some(packet) => {
if packet.stream() == self.stream_index {
self.decoder
.send_packet(&packet)
.map_err(|e| e.to_string())?;
}
}
None => {
// EOF: flush and drain whatever remains.
let _ = self.decoder.send_eof();
let mut decoded = ffmpeg_next::frame::Audio::empty();
if self.decoder.receive_frame(&mut decoded).is_ok() {
self.ensure_layout(&mut decoded);
return Ok(self.emit(&decoded, out));
}
return Ok(0);
}
}
}
}
/// Decoders for some formats (e.g. raw mono WAV) leave the frame's channel
/// layout unset. The resampler needs a concrete layout that matches the
/// frame, so fill one in from the channel count when it's missing.
fn ensure_layout(&self, frame: &mut ffmpeg_next::frame::Audio) {
if frame.channel_layout().is_empty() {
frame.set_channel_layout(
ffmpeg_next::channel_layout::ChannelLayout::default(self.channels as i32),
);
}
}
/// Resample one decoded frame to interleaved f32, apply any pending
/// seek-alignment discard, append to `out`, return frames emitted.
fn emit(&mut self, frame: &ffmpeg_next::frame::Audio, out: &mut Vec<f32>) -> usize {
// `frame` already has a non-empty channel layout (set by `ensure_layout`
// before this call), so the resampler config and the actual frame agree
// — otherwise swr fails with AVERROR_INPUT_CHANGED.
if self.resampler.is_none() {
match ffmpeg_next::software::resampling::Context::get(
frame.format(),
frame.channel_layout(),
self.sample_rate,
ffmpeg_next::format::Sample::F32(ffmpeg_next::format::sample::Type::Packed),
frame.channel_layout(),
self.sample_rate,
) {
Ok(r) => self.resampler = Some(r),
Err(_) => return 0,
}
}
let mut resampled = ffmpeg_next::frame::Audio::empty();
if self
.resampler
.as_mut()
.unwrap()
.run(frame, &mut resampled)
.is_err()
{
return 0;
}
// The output is packed (interleaved) f32. Read it from the raw byte plane
// `data(0)` — its length is correct (`frames * channels * 4`), whereas
// `plane::<f32>(0)` is a known ffmpeg-next footgun that reports only
// `samples()` elements (ignoring channels) and would slice out of range
// for multi-channel audio.
let ch = self.channels.max(1) as usize;
let bytes = resampled.data(0);
let n_frames = (bytes.len() / 4) / ch;
if n_frames == 0 {
return 0;
}
// On the first frame after a seek, compute how many leading frames to
// drop so output begins exactly at the seek target.
if let Some(target) = self.align_to.take() {
let frame_start = self.pts_to_frame(frame.pts());
self.pending_discard = target.saturating_sub(frame_start);
}
let discard = (self.pending_discard.min(n_frames as u64)) as usize;
self.pending_discard -= discard as u64;
let start_byte = discard * ch * 4;
let end_byte = n_frames * ch * 4;
out.extend(
bytes[start_byte..end_byte]
.chunks_exact(4)
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]])),
);
let emitted = n_frames - discard;
self.current_frame += emitted as u64;
emitted
}
/// Convert a stream PTS to an absolute audio frame index.
fn pts_to_frame(&self, pts: Option<i64>) -> u64 {
match pts {
Some(p) if p >= 0 => {
((p as f64 * self.time_base) * self.sample_rate as f64).round() as u64
}
_ => self.current_frame,
}
}
}
// ---------------------------------------------------------------------------
// StreamSource — dispatches the disk reader over either decoder backend.
// (Wired into the reader thread in a later step.)
// ---------------------------------------------------------------------------
/// Which decoder backend a streaming source uses.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SourceKind {
/// Symphonia, for compressed audio files (MP3, FLAC, OGG, …).
CompressedAudio,
/// FFmpeg, for the audio track of a video container.
VideoAudio,
}
/// A streaming audio source backing one active clip: either Symphonia
/// ([`CompressedReader`]) or FFmpeg ([`VideoAudioReader`]).
enum StreamSource {
Compressed(CompressedReader),
Video(VideoAudioReader),
}
impl StreamSource {
fn open(open: StreamOpen, kind: SourceKind) -> Result<Self, String> {
match (kind, open) {
(SourceKind::CompressedAudio, StreamOpen::Path(p)) => {
Ok(StreamSource::Compressed(CompressedReader::open(&p)?))
}
(SourceKind::CompressedAudio, StreamOpen::Source { src, ext }) => {
Ok(StreamSource::Compressed(CompressedReader::open_source(src, ext.as_deref())?))
}
(SourceKind::VideoAudio, StreamOpen::Path(p)) => {
Ok(StreamSource::Video(VideoAudioReader::open(&p)?))
}
(SourceKind::VideoAudio, StreamOpen::Source { .. }) => {
Err("VideoAudio cannot be opened from a packed byte source".to_string())
}
}
}
fn sample_rate(&self) -> u32 {
match self {
StreamSource::Compressed(r) => r.sample_rate,
StreamSource::Video(r) => r.sample_rate,
}
}
fn channels(&self) -> u32 {
match self {
StreamSource::Compressed(r) => r.channels,
StreamSource::Video(r) => r.channels,
}
}
fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
match self {
StreamSource::Compressed(r) => r.seek(target_frame),
StreamSource::Video(r) => r.seek(target_frame),
}
}
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
match self {
StreamSource::Compressed(r) => r.decode_next(out),
StreamSource::Video(r) => r.decode_next(out),
}
}
fn total_frames(&self) -> u64 {
match self {
StreamSource::Compressed(r) => r.total_frames(),
StreamSource::Video(r) => r.total_frames(),
}
}
}
/// Decode a media source end-to-end and build its [`WaveformPyramid`] overview,
/// streaming — only one decode chunk plus the (bounded) pyramid are ever in
/// memory, never the full sample buffer. `floor_samples_per_texel` is the
/// finest-level resolution (see [`crate::audio::waveform_pyramid`]).
pub fn build_waveform_pyramid(
path: &Path,
kind: SourceKind,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
let src = StreamSource::open(StreamOpen::Path(path.to_path_buf()), kind)?;
build_pyramid_from_streamsource(src, floor_samples_per_texel)
}
/// Build a waveform pyramid from a host-provided byte source (container-packed
/// compressed audio) — the load-time counterpart of [`build_waveform_pyramid`]
/// for media that has no filesystem path.
pub fn build_waveform_pyramid_from_source(
src: Box<dyn MediaByteSource>,
ext: Option<&str>,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
let src = StreamSource::open(
StreamOpen::Source { src, ext: ext.map(|s| s.to_string()) },
SourceKind::CompressedAudio,
)?;
build_pyramid_from_streamsource(src, floor_samples_per_texel)
}
fn build_pyramid_from_streamsource(
mut src: StreamSource,
floor_samples_per_texel: u32,
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
use crate::audio::waveform_pyramid::WaveformPyramidBuilder;
let channels = src.channels();
let mut builder = WaveformPyramidBuilder::new(channels, floor_samples_per_texel);
builder.reserve_for_frames(src.total_frames());
let mut buf = Vec::new();
loop {
let frames = src.decode_next(&mut buf)?;
if frames == 0 {
break;
}
builder.push_interleaved(&buf);
}
Ok(builder.finish())
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// DiskReaderCommand // DiskReaderCommand
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Commands sent from the engine to the disk reader thread. /// Commands sent from the engine to the disk reader thread.
pub enum DiskReaderCommand { pub enum DiskReaderCommand {
/// Start streaming a compressed file for a clip instance. /// Start streaming a file for a clip instance, using the decoder backend
/// selected by `kind` (compressed audio vs. a video's audio track). `open`
/// is either a filesystem path (referenced media / video) or a host-provided
/// byte stream (container-packed media).
ActivateFile { ActivateFile {
reader_id: u64, reader_id: u64,
path: PathBuf, open: StreamOpen,
kind: SourceKind,
buffer: Arc<ReadAheadBuffer>, buffer: Arc<ReadAheadBuffer>,
}, },
/// Stop streaming for a clip instance. /// Stop streaming for a clip instance.
@ -529,7 +1006,7 @@ impl DiskReader {
mut command_rx: rtrb::Consumer<DiskReaderCommand>, mut command_rx: rtrb::Consumer<DiskReaderCommand>,
running: Arc<AtomicBool>, running: Arc<AtomicBool>,
) { ) {
let mut active_files: HashMap<u64, (CompressedReader, Arc<ReadAheadBuffer>)> = let mut active_files: HashMap<u64, (StreamSource, Arc<ReadAheadBuffer>)> =
HashMap::new(); HashMap::new();
let mut decode_buf = Vec::with_capacity(8192); let mut decode_buf = Vec::with_capacity(8192);
@ -539,18 +1016,19 @@ impl DiskReader {
match cmd { match cmd {
DiskReaderCommand::ActivateFile { DiskReaderCommand::ActivateFile {
reader_id, reader_id,
path, open,
kind,
buffer, buffer,
} => match CompressedReader::open(&path) { } => match StreamSource::open(open, kind) {
Ok(reader) => { Ok(reader) => {
eprintln!("[DiskReader] Activated reader={}, ch={}, sr={}, path={:?}", eprintln!("[DiskReader] Activated reader={}, kind={:?}, ch={}, sr={}",
reader_id, reader.channels, reader.sample_rate, path); reader_id, kind, reader.channels(), reader.sample_rate());
active_files.insert(reader_id, (reader, buffer)); active_files.insert(reader_id, (reader, buffer));
} }
Err(e) => { Err(e) => {
eprintln!( eprintln!(
"[DiskReader] Failed to open compressed file {:?}: {}", "[DiskReader] Failed to open reader={} ({:?}): {}",
path, e reader_id, kind, e
); );
} }
}, },
@ -588,7 +1066,7 @@ impl DiskReader {
// If the target has jumped behind or far ahead of the buffer, // If the target has jumped behind or far ahead of the buffer,
// seek the decoder and reset. // seek the decoder and reset.
if target < buf_start || target > buf_end + reader.sample_rate as u64 { if target < buf_start || target > buf_end + reader.sample_rate() as u64 {
buffer.reset(target); buffer.reset(target);
let _ = reader.seek(target); let _ = reader.seek(target);
continue; continue;
@ -607,7 +1085,7 @@ impl DiskReader {
let buf_valid = buffer.valid_frames_count(); let buf_valid = buffer.valid_frames_count();
let buf_end = buf_start + buf_valid; let buf_end = buf_start + buf_valid;
let prefetch_target = let prefetch_target =
target + (PREFETCH_SECONDS * reader.sample_rate as f64) as u64; target + (PREFETCH_SECONDS * reader.sample_rate() as f64) as u64;
if buf_end >= prefetch_target { if buf_end >= prefetch_target {
continue; // Already filled far enough ahead. continue; // Already filled far enough ahead.
@ -649,3 +1127,7 @@ impl Drop for DiskReader {
} }
} }
} }
// Tests for VideoAudioReader live in `daw-backend/tests/video_audio_stream.rs`
// (integration tests) so they build the lib in normal mode, independent of
// pre-existing breakage in the crate's `#[cfg(test)]` unit tests (automation.rs).

View File

@ -91,6 +91,10 @@ pub struct Engine {
// Disk reader for streaming playback of compressed files // Disk reader for streaming playback of compressed files
disk_reader: Option<crate::audio::disk_reader::DiskReader>, disk_reader: Option<crate::audio::disk_reader::DiskReader>,
// Host-installed factory for opening container-packed audio as a byte stream
// (set on load, before the project's clips are bulk-activated for streaming).
blob_source_factory: Option<Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>>,
// Input monitoring and metering // Input monitoring and metering
input_monitoring: bool, input_monitoring: bool,
input_gain: f32, input_gain: f32,
@ -176,6 +180,7 @@ impl Engine {
metronome: Metronome::new(sample_rate), metronome: Metronome::new(sample_rate),
recording_sample_buffer: Vec::with_capacity(4096), recording_sample_buffer: Vec::with_capacity(4096),
disk_reader: Some(disk_reader), disk_reader: Some(disk_reader),
blob_source_factory: None,
input_monitoring: false, input_monitoring: false,
input_gain: 1.0, input_gain: 1.0,
input_level_peak: 0.0, input_level_peak: 0.0,
@ -273,6 +278,97 @@ impl Engine {
/// Rebuild the clip snapshot from the current project state. /// Rebuild the clip snapshot from the current project state.
/// Call this after any command that adds, removes, or modifies clip instances. /// Call this after any command that adds, removes, or modifies clip instances.
/// Set up disk streaming for a clip backed by a streaming pool entry
/// (Compressed or VideoAudio). Sends `ActivateFile` to the disk reader and
/// returns the read-ahead buffer to attach to the clip. Returns `None` if the
/// entry isn't streamed, or a packed source can't be opened.
fn activate_streaming_for(
&mut self,
reader_id: u64,
pool_index: usize,
) -> Option<Arc<crate::audio::disk_reader::ReadAheadBuffer>> {
use crate::audio::pool::AudioStorage;
use crate::audio::disk_reader::{DiskReader, DiskReaderCommand, SourceKind, StreamOpen};
// Decide how to open the source. `Packed` ⇒ via the host factory (bytes in
// the container); otherwise stream from the file path. Extract owned values
// first so the immutable pool borrow ends before we touch the disk reader.
enum OpenDesc {
Path(std::path::PathBuf),
Packed { media_id: String, ext: Option<String> },
}
let (kind, sample_rate, channels, desc) = {
let file = self.audio_pool.get_file(pool_index)?;
let kind = match file.storage {
AudioStorage::Compressed { .. } => SourceKind::CompressedAudio,
AudioStorage::VideoAudio { .. } => SourceKind::VideoAudio,
_ => return None,
};
let desc = match &file.packed_media_id {
Some(id) => OpenDesc::Packed { media_id: id.clone(), ext: file.original_format.clone() },
None => OpenDesc::Path(file.path.clone()),
};
(kind, file.sample_rate, file.channels, desc)
};
let open = match desc {
OpenDesc::Path(p) => StreamOpen::Path(p),
OpenDesc::Packed { media_id, ext } => {
let factory = match self.blob_source_factory.as_ref() {
Some(f) => f,
None => {
eprintln!("[Engine] packed audio (pool {}) but no blob factory installed", pool_index);
return None;
}
};
match factory.open(&media_id) {
Ok(src) => StreamOpen::Source { src, ext },
Err(e) => {
eprintln!("[Engine] blob factory open({}) failed: {}", media_id, e);
return None;
}
}
}
};
let buffer = DiskReader::create_buffer(sample_rate, channels);
if let Some(ref mut dr) = self.disk_reader {
dr.send(DiskReaderCommand::ActivateFile { reader_id, open, kind, buffer: buffer.clone() });
}
Some(buffer)
}
/// Activate disk streaming for every loaded clip backed by a streaming pool
/// entry. Called after `SetProject` since loaded clips bypass `AddAudioClip`.
fn activate_all_streaming_clips(&mut self) {
use crate::audio::track::TrackNode;
// Collect (track, clip, pool) first via an immutable walk, then activate
// (needs &mut self) and attach the buffer back to the clip.
let targets: Vec<(TrackId, u64, usize)> = self
.project
.track_iter()
.filter_map(|(track_id, node)| match node {
TrackNode::Audio(t) => Some((track_id, t)),
_ => None,
})
.flat_map(|(track_id, t)| {
t.clips
.iter()
.map(move |c| (track_id, c.id as u64, c.audio_pool_index))
})
.collect();
for (track_id, clip_id, pool_index) in targets {
if let Some(buffer) = self.activate_streaming_for(clip_id, pool_index) {
if let Some(TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
if let Some(clip) = track.clips.iter_mut().find(|c| c.id as u64 == clip_id) {
clip.read_ahead = Some(buffer);
}
}
}
}
}
fn refresh_clip_snapshot(&self) { fn refresh_clip_snapshot(&self) {
let mut snap = self.clip_snapshot.write().unwrap(); let mut snap = self.clip_snapshot.write().unwrap();
snap.audio.clear(); snap.audio.clear();
@ -914,7 +1010,7 @@ impl Engine {
let start_secs = self.tempo_map.beats_to_seconds(start_beats); let start_secs = self.tempo_map.beats_to_seconds(start_beats);
let end_secs = self.tempo_map.beats_to_seconds(end_beats); let end_secs = self.tempo_map.beats_to_seconds(end_beats);
let content_dur_secs = (end_secs - start_secs).seconds_to_f64(); let content_dur_secs = (end_secs - start_secs).seconds_to_f64();
let clip = AudioClipInstance::new( let mut clip = AudioClipInstance::new(
clip_id, clip_id,
pool_index, pool_index,
Seconds(offset), Seconds(offset),
@ -923,6 +1019,13 @@ impl Engine {
Beats(duration), Beats(duration),
); );
// If the source is streamed (a compressed audio file, or a video's
// audio track), set up disk streaming instead of an in-memory decode.
// Each clip instance gets its own read-ahead buffer keyed by clip_id.
if let Some(buffer) = self.activate_streaming_for(clip_id as u64, pool_index) {
clip.read_ahead = Some(buffer);
}
// Add clip to track // Add clip to track
if let Some(crate::audio::track::TrackNode::Audio(track)) = self.project.get_track_mut(track_id) { if let Some(crate::audio::track::TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
track.clips.push(clip); track.clips.push(clip);
@ -2313,6 +2416,34 @@ impl Engine {
} }
} }
/// Add a video file's audio track as a streaming pool entry (FFmpeg, decoded
/// on demand — no extraction). Probes the audio track for channels/rate/frames
/// without decoding, and returns the pool index. Playback activation
/// (per-clip read-ahead) is wired separately when a clip references it.
fn do_add_video_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
use crate::audio::disk_reader::VideoAudioReader;
let reader = VideoAudioReader::open(path)?;
let channels = reader.channels();
let sample_rate = reader.sample_rate();
let total_frames = reader.total_frames();
drop(reader);
let audio_file = crate::audio::pool::AudioFile::from_video_audio(
path.to_path_buf(),
channels,
sample_rate,
total_frames,
);
let pool_index = self.audio_pool.add_file(audio_file);
eprintln!(
"[ENGINE] AddVideoAudio: ch={}, sr={}, total_frames={}, pool_index={}, path={:?}",
channels, sample_rate, total_frames, pool_index, path
);
Ok(pool_index)
}
/// Import an audio file into the pool: mmap for PCM, streaming for compressed. /// Import an audio file into the pool: mmap for PCM, streaming for compressed.
/// Returns the pool index on success. Emits AudioFileReady event. /// Returns the pool index on success. Emits AudioFileReady event.
fn do_import_audio(&mut self, path: &std::path::Path) -> Result<usize, String> { fn do_import_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
@ -2738,10 +2869,14 @@ impl Engine {
Query::GetPoolAudioSamples(pool_index) => { Query::GetPoolAudioSamples(pool_index) => {
match self.audio_pool.get_file(pool_index) { match self.audio_pool.get_file(pool_index) {
Some(file) => { Some(file) => {
// For Compressed storage, return decoded_for_waveform if available // For streamed (Compressed/VideoAudio) storage, return the
// progressively-decoded waveform overview if available.
let samples = match &file.storage { let samples = match &file.storage {
crate::audio::pool::AudioStorage::Compressed { crate::audio::pool::AudioStorage::Compressed {
decoded_for_waveform, decoded_frames, .. decoded_for_waveform, decoded_frames, ..
}
| crate::audio::pool::AudioStorage::VideoAudio {
decoded_for_waveform, decoded_frames, ..
} if *decoded_frames > 0 => { } if *decoded_frames > 0 => {
decoded_for_waveform.clone() decoded_for_waveform.clone()
} }
@ -2794,77 +2929,12 @@ impl Engine {
self.refresh_clip_snapshot(); self.refresh_clip_snapshot();
result result
} }
Query::AddAudioFileSync(path, data, channels, sample_rate) => {
// Add audio file to pool and return the pool index
// Detect original format from file extension
let path_buf = std::path::PathBuf::from(&path);
let original_format = path_buf.extension()
.and_then(|ext| ext.to_str())
.map(|s| s.to_lowercase());
// Create AudioFile and add to pool
let audio_file = crate::audio::pool::AudioFile::with_format(
path_buf.clone(),
data.clone(), // Clone data for background thread
channels,
sample_rate,
original_format,
);
let pool_index = self.audio_pool.add_file(audio_file);
// Generate Level 0 (overview) waveform chunks asynchronously in background thread
let chunk_tx = self.chunk_generation_tx.clone();
let duration = data.len() as f64 / (sample_rate as f64 * channels as f64);
println!("🔄 [ENGINE] Spawning background thread to generate Level 0 chunks for pool {}", pool_index);
std::thread::spawn(move || {
// Create temporary AudioFile for chunk generation
let temp_audio_file = crate::audio::pool::AudioFile::with_format(
path_buf,
data,
channels,
sample_rate,
None,
);
// Generate Level 0 chunks
let chunk_count = crate::audio::waveform_cache::WaveformCache::calculate_chunk_count(duration, 0);
println!("🔄 [BACKGROUND] Generating {} Level 0 chunks for pool {}", chunk_count, pool_index);
let chunks = crate::audio::waveform_cache::WaveformCache::generate_chunks(
&temp_audio_file,
pool_index,
0, // Level 0 (overview)
&(0..chunk_count).collect::<Vec<_>>(),
);
// Send chunks via MPSC channel (will be forwarded by audio thread)
if !chunks.is_empty() {
println!("📤 [BACKGROUND] Generated {} chunks, sending to audio thread (pool {})", chunks.len(), pool_index);
let event_chunks: Vec<(u32, (f64, f64), Vec<crate::io::WaveformPeak>)> = chunks
.into_iter()
.map(|chunk| (chunk.chunk_index, chunk.time_range, chunk.peaks))
.collect();
match chunk_tx.send(AudioEvent::WaveformChunksReady {
pool_index,
detail_level: 0,
chunks: event_chunks,
}) {
Ok(_) => println!("✅ [BACKGROUND] Chunks sent successfully for pool {}", pool_index),
Err(e) => eprintln!("❌ [BACKGROUND] Failed to send chunks: {}", e),
}
} else {
eprintln!("⚠️ [BACKGROUND] No chunks generated for pool {}", pool_index);
}
});
// Notify UI about the new audio file (for event listeners)
let _ = self.event_tx.push(AudioEvent::AudioFileAdded(pool_index, path));
QueryResponse::AudioFileAddedSync(Ok(pool_index))
}
Query::ImportAudioSync(path) => { Query::ImportAudioSync(path) => {
QueryResponse::AudioImportedSync(self.do_import_audio(&path)) QueryResponse::AudioImportedSync(self.do_import_audio(&path))
} }
Query::AddVideoAudioSync(path) => {
QueryResponse::AudioImportedSync(self.do_add_video_audio(&path))
}
Query::GetProject => { Query::GetProject => {
// Save graph presets before cloning — AudioTrack::clone() creates // Save graph presets before cloning — AudioTrack::clone() creates
// a fresh default graph (not a copy), so the preset must be populated // a fresh default graph (not a copy), so the preset must be populated
@ -2879,11 +2949,19 @@ impl Engine {
match project.rebuild_audio_graphs(self.buffer_pool.buffer_size()) { match project.rebuild_audio_graphs(self.buffer_pool.buffer_size()) {
Ok(()) => { Ok(()) => {
self.project = project; self.project = project;
// Loaded clips bypass AddAudioClip, so their disk streaming was
// never activated — do it now for every streaming-backed clip.
self.activate_all_streaming_clips();
self.refresh_clip_snapshot();
QueryResponse::ProjectSet(Ok(())) QueryResponse::ProjectSet(Ok(()))
} }
Err(e) => QueryResponse::ProjectSet(Err(format!("Failed to rebuild audio graphs: {}", e))), Err(e) => QueryResponse::ProjectSet(Err(format!("Failed to rebuild audio graphs: {}", e))),
} }
} }
Query::SetBlobSourceFactory(factory) => {
self.blob_source_factory = Some(factory);
QueryResponse::BlobSourceFactorySet(Ok(()))
}
Query::DuplicateMidiClipSync(clip_id) => { Query::DuplicateMidiClipSync(clip_id) => {
match self.project.midi_clip_pool.duplicate_clip(clip_id) { match self.project.midi_clip_pool.duplicate_clip(clip_id) {
Some(new_id) => QueryResponse::MidiClipDuplicated(Ok(new_id)), Some(new_id) => QueryResponse::MidiClipDuplicated(Ok(new_id)),
@ -3395,16 +3473,6 @@ impl EngineController {
} }
} }
/// Add an audio file to the pool synchronously and get the pool index
/// Returns the pool index where the audio file was added
pub fn add_audio_file_sync(&mut self, path: String, data: Vec<f32>, channels: u32, sample_rate: u32) -> Result<usize, String> {
let query = Query::AddAudioFileSync(path, data, channels, sample_rate);
match self.send_query(query)? {
QueryResponse::AudioFileAddedSync(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Import an audio file asynchronously. The engine will memory-map WAV/AIFF /// Import an audio file asynchronously. The engine will memory-map WAV/AIFF
/// files for instant availability, or set up stream decoding for compressed /// files for instant availability, or set up stream decoding for compressed
/// formats. Listen for `AudioEvent::AudioFileReady` to get the pool index. /// formats. Listen for `AudioEvent::AudioFileReady` to get the pool index.
@ -3427,6 +3495,30 @@ impl EngineController {
} }
} }
/// Add a video file's audio track as a streaming pool entry (decoded on
/// demand via FFmpeg — no extraction to disk or RAM). Probes the audio track
/// and returns the pool index. Use this for a video clip's embedded audio.
pub fn add_video_audio_sync(&mut self, path: std::path::PathBuf) -> Result<usize, String> {
let query = Query::AddVideoAudioSync(path);
match self.send_query(query)? {
QueryResponse::AudioImportedSync(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Install the host's packed-media byte-source factory. Must be called before
/// loading a project so its container-packed audio can be streamed (the disk
/// reader opens packed entries through this factory).
pub fn set_blob_source_factory(
&mut self,
factory: std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>,
) -> Result<(), String> {
match self.send_query(Query::SetBlobSourceFactory(factory))? {
QueryResponse::BlobSourceFactorySet(result) => result,
_ => Err("Unexpected query response".to_string()),
}
}
/// Generate the next unique audio clip instance ID (atomic, thread-safe) /// Generate the next unique audio clip instance ID (atomic, thread-safe)
pub fn next_audio_clip_id(&self) -> AudioClipInstanceId { pub fn next_audio_clip_id(&self) -> AudioClipInstanceId {
self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed) self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed)

View File

@ -671,14 +671,39 @@ fn convert_chunk_to_planar_i16(interleaved: &[f32], channels: u32) -> Vec<Vec<i1
planar planar
} }
/// Convert a chunk of interleaved f32 samples to planar f32 format /// Convert a chunk of interleaved f32 samples to planar f32 format.
///
/// Non-finite samples (NaN/±Inf) are replaced with `0.0` and finite samples are
/// clamped to `[-1.0, 1.0]`: the float encoders (e.g. AAC, which takes `fltp`)
/// reject a frame outright on "(near) NaN/+-Inf", failing the whole export, so we
/// sanitize here exactly as the integer paths already clamp.
fn convert_chunk_to_planar_f32(interleaved: &[f32], channels: u32) -> Vec<Vec<f32>> { fn convert_chunk_to_planar_f32(interleaved: &[f32], channels: u32) -> Vec<Vec<f32>> {
let num_frames = interleaved.len() / channels as usize; let num_frames = interleaved.len() / channels as usize;
let mut planar = vec![vec![0.0f32; num_frames]; channels as usize]; let mut planar = vec![vec![0.0f32; num_frames]; channels as usize];
let mut non_finite = 0u64;
for (i, chunk) in interleaved.chunks(channels as usize).enumerate() { for (i, chunk) in interleaved.chunks(channels as usize).enumerate() {
for (ch, &sample) in chunk.iter().enumerate() { for (ch, &sample) in chunk.iter().enumerate() {
planar[ch][i] = sample; planar[ch][i] = if sample.is_finite() {
sample.clamp(-1.0, 1.0)
} else {
non_finite += 1;
0.0
};
}
}
if non_finite > 0 {
// One-time warning: we sanitized rather than failed, but a non-finite
// sample reaching here means something upstream (an effect, automation,
// or a source decode) produced NaN/Inf — worth chasing if audio is wrong.
use std::sync::atomic::{AtomicBool, Ordering};
static WARNED: AtomicBool = AtomicBool::new(false);
if !WARNED.swap(true, Ordering::Relaxed) {
eprintln!(
"⚠️ [EXPORT] sanitized {} non-finite (NaN/Inf) audio sample(s) in a chunk — \
check effects/automation/source decode",
non_finite
);
} }
} }

View File

@ -15,10 +15,12 @@ pub mod recording;
pub mod sample_loader; pub mod sample_loader;
pub mod track; pub mod track;
pub mod waveform_cache; pub mod waveform_cache;
pub mod waveform_pyramid;
pub use automation::{AutomationLane, AutomationLaneId, AutomationPoint, CurveType, ParameterId}; pub use automation::{AutomationLane, AutomationLaneId, AutomationPoint, CurveType, ParameterId};
pub use buffer_pool::BufferPool; pub use buffer_pool::BufferPool;
pub use clip::{AudioClipInstance, AudioClipInstanceId, Clip, ClipId}; pub use clip::{AudioClipInstance, AudioClipInstanceId, Clip, ClipId};
pub use disk_reader::{AudioBlobSourceFactory, MediaByteSource};
pub use engine::{AudioClipSnapshot, Engine, EngineController}; pub use engine::{AudioClipSnapshot, Engine, EngineController};
pub use export::{export_audio, ExportFormat, ExportSettings}; pub use export::{export_audio, ExportFormat, ExportSettings};
pub use metronome::Metronome; pub use metronome::Metronome;

View File

@ -4,6 +4,46 @@ use std::f32::consts::PI;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::time::Seconds; use crate::time::Seconds;
/// Per-output-channel mix coefficients to fold a multichannel source down to
/// stereo, indexed `[out_channel(0=L,1=R)][src_channel]`.
///
/// Assumes the conventional interleave order for each channel count (FL, FR, FC,
/// LFE, BL, BR, SL, SR …). Uses standard ITU/AC-3-style coefficients: full level
/// for the matching front channel, `1/√2` (≈ 3 dB) for centre and each surround,
/// LFE dropped. Each row is then normalized so its absolute-coefficient sum ≤ 1,
/// which prevents clipping (matching FFmpeg's default `normalize` behaviour).
///
/// Returns `None` for layouts we don't special-case (caller falls back to taking
/// the front L/R pair).
fn stereo_downmix_matrix(src_channels: usize) -> Option<[Vec<f32>; 2]> {
const C: f32 = std::f32::consts::FRAC_1_SQRT_2; // ≈ 0.7071
// (L row, R row); each entry is the gain applied to that source channel.
let (l, r): (Vec<f32>, Vec<f32>) = match src_channels {
3 => (vec![1.0, 0.0, C], vec![0.0, 1.0, C]), // FL FR FC
4 => (vec![1.0, 0.0, C, 0.0], vec![0.0, 1.0, 0.0, C]), // quad: FL FR BL BR
5 => (vec![1.0, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C]), // FL FR FC BL BR
// 5.1: FL FR FC LFE BL BR (LFE dropped)
6 => (vec![1.0, 0.0, C, 0.0, C, 0.0], vec![0.0, 1.0, C, 0.0, 0.0, C]),
// 6.1: FL FR FC LFE BC SL SR (BC → both)
7 => (vec![1.0, 0.0, C, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C, 0.0, C]),
// 7.1: FL FR FC LFE BL BR SL SR
8 => (
vec![1.0, 0.0, C, 0.0, C, 0.0, C, 0.0],
vec![0.0, 1.0, C, 0.0, 0.0, C, 0.0, C],
),
_ => return None,
};
let normalize = |row: Vec<f32>| -> Vec<f32> {
let sum: f32 = row.iter().map(|c| c.abs()).sum();
if sum > 1.0 {
row.into_iter().map(|c| c / sum).collect()
} else {
row
}
};
Some([normalize(l), normalize(r)])
}
/// Windowed sinc interpolation for high-quality time stretching /// Windowed sinc interpolation for high-quality time stretching
/// This is stateless and can handle arbitrary fractional positions /// This is stateless and can handle arbitrary fractional positions
#[inline] #[inline]
@ -83,6 +123,16 @@ pub enum AudioStorage {
decoded_frames: u64, decoded_frames: u64,
total_frames: u64, total_frames: u64,
}, },
/// Audio track of a video container, decoded on demand via FFmpeg
/// (`VideoAudioReader`). The source video file is `AudioFile::path`. Like
/// `Compressed`, playback is streamed through the disk reader and
/// `decoded_for_waveform` is filled progressively for the overview.
VideoAudio {
decoded_for_waveform: Vec<f32>,
decoded_frames: u64,
total_frames: u64,
},
} }
/// Audio file stored in the pool /// Audio file stored in the pool
@ -98,6 +148,10 @@ pub struct AudioFile {
pub original_format: Option<String>, pub original_format: Option<String>,
/// Original compressed file bytes (preserved across save/load to avoid re-encoding) /// Original compressed file bytes (preserved across save/load to avoid re-encoding)
pub original_bytes: Option<Vec<u8>>, pub original_bytes: Option<Vec<u8>>,
/// When `Some`, this entry's bytes are packed in the project container (not on
/// disk at `path`); the disk reader opens them via the host's
/// `AudioBlobSourceFactory` using this media id. `None` ⇒ stream from `path`.
pub packed_media_id: Option<String>,
} }
impl AudioFile { impl AudioFile {
@ -112,6 +166,7 @@ impl AudioFile {
frames, frames,
original_format: None, original_format: None,
original_bytes: None, original_bytes: None,
packed_media_id: None,
} }
} }
@ -126,6 +181,7 @@ impl AudioFile {
frames, frames,
original_format, original_format,
original_bytes: None, original_bytes: None,
packed_media_id: None,
} }
} }
@ -158,6 +214,7 @@ impl AudioFile {
frames: total_frames, frames: total_frames,
original_format: Some("wav".to_string()), original_format: Some("wav".to_string()),
original_bytes: None, original_bytes: None,
packed_media_id: None,
} }
} }
@ -181,6 +238,32 @@ impl AudioFile {
frames: total_frames, frames: total_frames,
original_format, original_format,
original_bytes: None, original_bytes: None,
packed_media_id: None,
}
}
/// Create a placeholder AudioFile for a video's audio track. `path` is the
/// source video file; the audio is streamed on demand by the disk reader's
/// FFmpeg-backed `VideoAudioReader`.
pub fn from_video_audio(
path: PathBuf,
channels: u32,
sample_rate: u32,
total_frames: u64,
) -> Self {
Self {
path,
storage: AudioStorage::VideoAudio {
decoded_for_waveform: Vec::new(),
decoded_frames: 0,
total_frames,
},
channels,
sample_rate,
frames: total_frames,
original_format: None,
original_bytes: None,
packed_media_id: None,
} }
} }
@ -274,8 +357,8 @@ impl AudioFile {
} }
written written
} }
AudioStorage::Compressed { .. } => { AudioStorage::Compressed { .. } | AudioStorage::VideoAudio { .. } => {
// Compressed files are read through the disk reader // Streamed through the disk reader, not via read_samples().
0 0
} }
} }
@ -537,6 +620,15 @@ impl AudioClipPool {
let dst_channels = engine_channels as usize; let dst_channels = engine_channels as usize;
let output_frames = output.len() / dst_channels; let output_frames = output.len() / dst_channels;
// Fold a multichannel source (5.1, 7.1, …) down to stereo with proper
// coefficients (centre + surrounds mixed in, LFE dropped) instead of just
// taking the front L/R pair. `None` ⇒ no downmix needed / unknown layout.
let downmix = if dst_channels == 2 && src_channels > 2 {
stereo_downmix_matrix(src_channels)
} else {
None
};
let src_start_position = start_time_seconds * audio_file.sample_rate as f64; let src_start_position = start_time_seconds * audio_file.sample_rate as f64;
// Tell the disk reader where we're reading so it buffers the right region. // Tell the disk reader where we're reading so it buffers the right region.
@ -582,6 +674,15 @@ impl AudioClipPool {
sum += get_sample!(sf, src_ch); sum += get_sample!(sf, src_ch);
} }
sum / src_channels as f32 sum / src_channels as f32
} else if let Some(ref mat) = downmix {
// Surround → stereo with proper coefficients.
let mut s = 0.0f32;
for (src_ch, &c) in mat[dst_ch].iter().enumerate() {
if c != 0.0 {
s += c * get_sample!(sf, src_ch);
}
}
s
} else { } else {
get_sample!(sf, dst_ch % src_channels) get_sample!(sf, dst_ch % src_channels)
}; };
@ -606,39 +707,45 @@ impl AudioClipPool {
break; break;
} }
// Sinc-interpolate a single source channel at the current position.
macro_rules! sinc_ch {
($ch:expr) => {{
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, $ch);
}
}
windowed_sinc_interpolate(&channel_samples, frac)
}};
}
for dst_ch in 0..dst_channels { for dst_ch in 0..dst_channels {
let sample = if let Some(ref mat) = downmix {
// Surround → stereo: interpolate each contributing channel.
let mut s = 0.0f32;
for (ch, &c) in mat[dst_ch].iter().enumerate() {
if c != 0.0 {
s += c * sinc_ch!(ch);
}
}
s
} else if dst_channels == 1 {
let mut sum = 0.0;
for ch in 0..src_channels {
sum += sinc_ch!(ch);
}
sum / src_channels as f32
} else {
let src_ch = if src_channels == dst_channels { let src_ch = if src_channels == dst_channels {
dst_ch dst_ch
} else if src_channels == 1 { } else if src_channels == 1 {
0 0
} else if dst_channels == 1 {
usize::MAX // sentinel: average all channels below
} else { } else {
dst_ch % src_channels dst_ch % src_channels
}; };
sinc_ch!(src_ch)
let sample = if src_ch == usize::MAX {
let mut sum = 0.0;
for ch in 0..src_channels {
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, ch);
}
}
sum += windowed_sinc_interpolate(&channel_samples, frac);
}
sum / src_channels as f32
} else {
let mut channel_samples = [0.0f32; KERNEL_SIZE];
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
let idx = src_frame + i;
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
channel_samples[j] = get_sample!(idx as usize, src_ch);
}
}
windowed_sinc_interpolate(&channel_samples, frac)
}; };
output[output_frame * dst_channels + dst_ch] += sample * gain; output[output_frame * dst_channels + dst_ch] += sample * gain;
@ -786,6 +893,25 @@ pub struct AudioPoolEntry {
pub channels: u32, pub channels: u32,
/// Embedded audio data (for files < 10MB) /// Embedded audio data (for files < 10MB)
pub embedded_data: Option<EmbeddedAudioData>, pub embedded_data: Option<EmbeddedAudioData>,
/// Stable media id (UUID string) for the SQLite `.beam` container. When set,
/// the audio bytes live in the container's `media` table keyed by this id
/// (packed storage). `None` for referenced entries (use `relative_path`) or
/// legacy ZIP-loaded entries. Populated by the file_io save/load layer.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub media_id: Option<String>,
/// Transient carrier for this entry's serialized waveform-pyramid blob (LBWF
/// bytes). Never serialized into project.json — the bytes live in the
/// container's `media` table (kind `Waveform`). Set by the file_io save layer
/// (in) and load layer (out); `None` everywhere else.
#[serde(skip)]
pub waveform_blob: Option<Vec<u8>>,
/// This entry is a video container's audio track (`relative_path` points at the
/// video file). It is always stored as a path reference (never packed/embedded
/// — the `VideoClip` already references the file) and reloaded by re-probing
/// the video via FFmpeg, so multichannel (5.1/7.1) audio survives the round-trip
/// (Symphonia reconstitution would otherwise collapse it).
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub is_video_audio: bool,
} }
impl AudioClipPool { impl AudioClipPool {
@ -800,6 +926,66 @@ impl AudioClipPool {
let mut entries = Vec::new(); let mut entries = Vec::new();
for (index, file) in self.files.iter().enumerate() { for (index, file) in self.files.iter().enumerate() {
// Skip placeholder pool slots: `load_from_serialized` resizes the pool to
// `max_index + 1` filled with empty `AudioFile::new(PathBuf::new(), …)` to
// cover index gaps (and creates one even when there are no entries at all).
// Such a slot has an empty path and no packed media — there's nothing to
// persist, and emitting it yields an entry whose empty `relative_path`
// resolves to the project directory itself (unreadable on the next save).
if file.path.as_os_str().is_empty() && file.packed_media_id.is_none() {
continue;
}
// Video's audio track: reference the video file (it's also referenced
// by the VideoClip) and re-probe it via FFmpeg on load. Never pack or
// embed it as audio media — that both wastes space and loses the 5.1+
// layout when Symphonia later decodes it.
if matches!(file.storage, AudioStorage::VideoAudio { .. }) {
let relative_path = pathdiff::diff_paths(&file.path, project_dir)
.map(|r| r.to_string_lossy().to_string())
.or_else(|| Some(file.path.to_string_lossy().to_string()));
entries.push(AudioPoolEntry {
pool_index: index,
is_video_audio: true,
waveform_blob: None,
name: file
.path
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| format!("file_{}", index)),
relative_path,
duration: file.duration_seconds(),
sample_rate: file.sample_rate,
channels: file.channels,
embedded_data: None,
media_id: None,
});
continue;
}
// Packed-in-container streaming entry: its bytes already live in the
// `.beam` media table (kept in place across re-saves). Emit just the
// media id — no path, no embedded bytes, nothing to decode.
if let Some(media_id) = &file.packed_media_id {
entries.push(AudioPoolEntry {
pool_index: index,
is_video_audio: false,
waveform_blob: None,
name: file
.path
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| format!("file_{}", index)),
relative_path: None,
duration: file.duration_seconds(),
sample_rate: file.sample_rate,
channels: file.channels,
embedded_data: None,
media_id: Some(media_id.clone()),
});
continue;
}
let file_path = &file.path; let file_path = &file.path;
let file_path_str = file_path.to_string_lossy(); let file_path_str = file_path.to_string_lossy();
@ -830,6 +1016,8 @@ impl AudioClipPool {
let entry = AudioPoolEntry { let entry = AudioPoolEntry {
pool_index: index, pool_index: index,
is_video_audio: false,
waveform_blob: None,
name: file_path name: file_path
.file_name() .file_name()
.map(|n| n.to_string_lossy().to_string()) .map(|n| n.to_string_lossy().to_string())
@ -839,6 +1027,7 @@ impl AudioClipPool {
sample_rate: file.sample_rate, sample_rate: file.sample_rate,
channels: file.channels, channels: file.channels,
embedded_data, embedded_data,
media_id: None,
}; };
entries.push(entry); entries.push(entry);
@ -962,22 +1151,86 @@ impl AudioClipPool {
self.files.clear(); self.files.clear();
eprintln!("📊 [LOAD_SERIALIZED] Clear pool took {:.2}ms", clear_start.elapsed().as_secs_f64() * 1000.0); eprintln!("📊 [LOAD_SERIALIZED] Clear pool took {:.2}ms", clear_start.elapsed().as_secs_f64() * 1000.0);
// Find the maximum pool index to determine required size // Size the pool to hold the highest pool_index (slots are addressed by index,
let max_index = entries.iter() // so gaps are filled with placeholders). No entries → length 0, NOT 1: the old
.map(|e| e.pool_index) // `max().unwrap_or(0) + 1` produced a spurious placeholder for an empty pool.
let pool_size = entries.iter()
.map(|e| e.pool_index + 1)
.max() .max()
.unwrap_or(0); .unwrap_or(0);
// Ensure we have space for all entries // Ensure we have space for all entries
let resize_start = std::time::Instant::now(); let resize_start = std::time::Instant::now();
self.files.resize(max_index + 1, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100)); self.files.resize(pool_size, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100));
eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", max_index + 1, resize_start.elapsed().as_secs_f64() * 1000.0); eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", pool_size, resize_start.elapsed().as_secs_f64() * 1000.0);
for (i, entry) in entries.iter().enumerate() { for (i, entry) in entries.iter().enumerate() {
let entry_start = std::time::Instant::now(); let entry_start = std::time::Instant::now();
eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name); eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name);
let success = if let Some(ref embedded) = entry.embedded_data { let success = if entry.is_video_audio {
// Re-probe the video's audio track via FFmpeg → a streaming
// VideoAudio entry (keeps full 5.1/7.1; no decode-to-RAM).
match entry.relative_path.as_ref() {
Some(rel) => {
let full = if std::path::Path::new(rel).is_absolute() {
PathBuf::from(rel)
} else {
project_dir.join(rel)
};
if full.exists() {
match crate::audio::disk_reader::VideoAudioReader::open(&full) {
Ok(reader) => {
let file = AudioFile::from_video_audio(
full,
reader.channels(),
reader.sample_rate(),
reader.total_frames(),
);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;
true
} else {
false
}
}
Err(e) => {
eprintln!("[AudioPool] Failed to reopen video audio {:?}: {}", full, e);
false
}
}
} else {
eprintln!("[AudioPool] Video file not found for audio: {:?}", full);
false
}
}
None => false,
}
} else if entry.media_id.is_some() && entry.embedded_data.is_none() {
// Packed-in-container streaming entry: build a Compressed placeholder
// backed by the host blob factory (opened at clip-activation time).
// No decode here — playback streams through the disk reader.
let media_id = entry.media_id.clone().unwrap();
let ext = std::path::Path::new(&entry.name)
.extension()
.and_then(|e| e.to_str())
.map(|s| s.to_lowercase());
let total_frames = (entry.duration * entry.sample_rate as f64).ceil() as u64;
let mut file = AudioFile::from_compressed(
PathBuf::from(&entry.name),
entry.channels,
entry.sample_rate,
total_frames,
ext,
);
file.packed_media_id = Some(media_id);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;
true
} else {
false
}
} else if let Some(ref embedded) = entry.embedded_data {
// Load from embedded data // Load from embedded data
eprintln!("📊 [LOAD_SERIALIZED] Entry has embedded data (format: {})", embedded.format); eprintln!("📊 [LOAD_SERIALIZED] Entry has embedded data (format: {})", embedded.format);
match Self::load_from_embedded_into_pool(self, entry.pool_index, embedded.clone(), &entry.name) { match Self::load_from_embedded_into_pool(self, entry.pool_index, embedded.clone(), &entry.name) {

View File

@ -0,0 +1,292 @@
//! Streaming min/max waveform LOD pyramid.
//!
//! A waveform pyramid is a tree of zoom levels. **Index = tree depth:**
//! `levels[0]` is the **root** (a single texel — the min/max envelope of the
//! whole file, lowest resolution); each deeper level is `BRANCH`× finer, and
//! `levels.last()` is the **floor** (one texel per `floor_samples_per_texel`
//! source frames — the finest *persisted* level). A node's children live at
//! `index + 1`, so the residency invariant ("a node is cleared only after its
//! children") reads straight off the index.
//!
//! Below the floor (finer than the floor bucket) is *not* stored; the caller
//! re-decodes the source window on demand for true per-sample detail.
//!
//! The builder is **streaming**: samples are pushed once, in order, and only the
//! finest level is accumulated (~`total_frames / floor` texels); the coarser
//! levels are derived by repeated `BRANCH:1` min/max reduction in [`finish`].
//! This yields the identical pyramid to an in-stream cascade (each parent = the
//! min/max of its children) without ever holding the full sample buffer.
//!
//! **Ragged edges are handled by reducing over available children:** a bucket
//! whose group is partial (1..BRANCH children, or `< floor` samples at the floor)
//! simply takes the min/max of what's there — no value padding. Padding to a
//! regular shape, if ever needed, is a GPU-texture/tile concern, not the data's.
//!
//! Each texel carries per-channel min/max for up to two channels
//! (`Lmin,Lmax,Rmin,Rmax`), matching the GPU waveform texture; mono mirrors the
//! left channel into the right.
//!
//! [`finish`]: WaveformPyramidBuilder::finish
/// Reduction factor between adjacent pyramid levels.
pub const BRANCH: u32 = 4;
/// Default finest-level resolution (source frames per floor texel). Trades
/// on-disk pyramid size against how soon zoom-in must re-decode the source.
pub const DEFAULT_FLOOR_SAMPLES_PER_TEXEL: u32 = 256;
/// One waveform texel: per-channel min/max (stereo; mono duplicates left→right).
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Texel {
pub l_min: f32,
pub l_max: f32,
pub r_min: f32,
pub r_max: f32,
}
impl Texel {
const EMPTY: Texel = Texel {
l_min: f32::INFINITY,
l_max: f32::NEG_INFINITY,
r_min: f32::INFINITY,
r_max: f32::NEG_INFINITY,
};
#[inline]
fn include_sample(&mut self, l: f32, r: f32) {
self.l_min = self.l_min.min(l);
self.l_max = self.l_max.max(l);
self.r_min = self.r_min.min(r);
self.r_max = self.r_max.max(r);
}
#[inline]
fn include_texel(&mut self, c: &Texel) {
self.l_min = self.l_min.min(c.l_min);
self.l_max = self.l_max.max(c.l_max);
self.r_min = self.r_min.min(c.r_min);
self.r_max = self.r_max.max(c.r_max);
}
}
/// A built min/max LOD pyramid, **root-first**: `levels[0]` is the coarsest
/// (whole-file envelope), `levels.last()` is the finest persisted (floor).
#[derive(Clone, Debug)]
pub struct WaveformPyramid {
pub floor_samples_per_texel: u32,
pub branch: u32,
pub channels: u32,
pub total_frames: u64,
pub levels: Vec<Vec<Texel>>,
}
impl WaveformPyramid {
/// Coarsest level — a single texel (whole-file envelope), or empty if no
/// samples were pushed.
pub fn root(&self) -> &[Texel] {
self.levels.first().map_or(&[][..], |v| v)
}
/// Finest persisted level (`floor_samples_per_texel` frames per texel).
pub fn floor(&self) -> &[Texel] {
self.levels.last().map_or(&[][..], |v| v)
}
/// Number of levels (tree depth + 1).
pub fn depth(&self) -> usize {
self.levels.len()
}
/// Serialize to a compact binary blob (for persisting in the `.beam`
/// container). Header carries `B`/branch/channels/total_frames + per-level
/// lengths, then root-first texel data (`f32` min/max).
pub fn to_bytes(&self) -> Vec<u8> {
let total_texels: usize = self.levels.iter().map(|l| l.len()).sum();
let mut out = Vec::with_capacity(32 + self.levels.len() * 4 + total_texels * 16);
out.extend_from_slice(b"LBWF");
out.extend_from_slice(&1u32.to_le_bytes()); // format version
out.extend_from_slice(&self.floor_samples_per_texel.to_le_bytes());
out.extend_from_slice(&self.branch.to_le_bytes());
out.extend_from_slice(&self.channels.to_le_bytes());
out.extend_from_slice(&self.total_frames.to_le_bytes());
out.extend_from_slice(&(self.levels.len() as u32).to_le_bytes());
for level in &self.levels {
out.extend_from_slice(&(level.len() as u32).to_le_bytes());
}
for level in &self.levels {
for t in level {
out.extend_from_slice(&t.l_min.to_le_bytes());
out.extend_from_slice(&t.l_max.to_le_bytes());
out.extend_from_slice(&t.r_min.to_le_bytes());
out.extend_from_slice(&t.r_max.to_le_bytes());
}
}
out
}
/// Reconstruct from [`WaveformPyramid::to_bytes`].
pub fn from_bytes(data: &[u8]) -> Result<WaveformPyramid, String> {
let mut r = ByteReader::new(data);
if r.take(4)? != b"LBWF" {
return Err("Not a waveform pyramid blob".to_string());
}
let version = r.u32()?;
if version != 1 {
return Err(format!("Unsupported waveform pyramid version {}", version));
}
let floor_samples_per_texel = r.u32()?;
let branch = r.u32()?;
let channels = r.u32()?;
let total_frames = r.u64()?;
let num_levels = r.u32()? as usize;
let mut level_lens = Vec::with_capacity(num_levels);
for _ in 0..num_levels {
level_lens.push(r.u32()? as usize);
}
let mut levels = Vec::with_capacity(num_levels);
for &len in &level_lens {
let mut level = Vec::with_capacity(len);
for _ in 0..len {
level.push(Texel {
l_min: r.f32()?,
l_max: r.f32()?,
r_min: r.f32()?,
r_max: r.f32()?,
});
}
levels.push(level);
}
Ok(WaveformPyramid {
floor_samples_per_texel,
branch,
channels,
total_frames,
levels,
})
}
}
/// Minimal little-endian byte cursor for [`WaveformPyramid::from_bytes`].
struct ByteReader<'a> {
data: &'a [u8],
pos: usize,
}
impl<'a> ByteReader<'a> {
fn new(data: &'a [u8]) -> Self {
Self { data, pos: 0 }
}
fn take(&mut self, n: usize) -> Result<&'a [u8], String> {
let end = self.pos.checked_add(n).ok_or("overflow")?;
if end > self.data.len() {
return Err("Truncated waveform pyramid blob".to_string());
}
let s = &self.data[self.pos..end];
self.pos = end;
Ok(s)
}
fn u32(&mut self) -> Result<u32, String> {
Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap()))
}
fn u64(&mut self) -> Result<u64, String> {
Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap()))
}
fn f32(&mut self) -> Result<f32, String> {
Ok(f32::from_le_bytes(self.take(4)?.try_into().unwrap()))
}
}
/// Streaming builder for a [`WaveformPyramid`]. See the module docs.
pub struct WaveformPyramidBuilder {
floor: u32,
branch: u32,
channels: u32,
total_frames: u64,
floor_level: Vec<Texel>,
acc: Texel,
acc_count: u32,
}
impl WaveformPyramidBuilder {
pub fn new(channels: u32, floor_samples_per_texel: u32) -> Self {
Self {
floor: floor_samples_per_texel.max(1),
branch: BRANCH,
channels: channels.max(1),
total_frames: 0,
floor_level: Vec::new(),
acc: Texel::EMPTY,
acc_count: 0,
}
}
/// Pre-reserve the floor `Vec` from an estimated total frame count (e.g. the
/// probe's `total_frames`), to avoid reallocations during streaming. Purely a
/// hint — the final size is set by the actual number of frames pushed.
pub fn reserve_for_frames(&mut self, estimated_frames: u64) {
let est_texels = (estimated_frames / self.floor as u64).saturating_add(1);
self.floor_level.reserve(est_texels.min(usize::MAX as u64) as usize);
}
/// Push a block of interleaved samples (`channels` per frame). Partial
/// trailing frames (fewer than `channels`) are ignored.
pub fn push_interleaved(&mut self, samples: &[f32]) {
let ch = self.channels as usize;
for frame in samples.chunks_exact(ch) {
let l = frame[0];
let r = if ch >= 2 { frame[1] } else { l };
self.push_frame(l, r);
}
}
#[inline]
fn push_frame(&mut self, l: f32, r: f32) {
self.total_frames += 1;
self.acc.include_sample(l, r);
self.acc_count += 1;
if self.acc_count >= self.floor {
self.floor_level.push(std::mem::replace(&mut self.acc, Texel::EMPTY));
self.acc_count = 0;
}
}
/// Flush the trailing partial bucket and reduce up to the root.
pub fn finish(mut self) -> WaveformPyramid {
if self.acc_count > 0 {
self.floor_level.push(self.acc);
}
// Build finest-first by repeated BRANCH:1 reduction until one texel.
// The shape is fully determined by the floor texel count; the last group
// at each level may be ragged (1..BRANCH children) and reduces over what
// it has.
let mut levels = vec![std::mem::take(&mut self.floor_level)];
let branch = self.branch as usize;
while levels.last().map_or(0, |l| l.len()) > 1 {
let prev = levels.last().unwrap();
let mut next = Vec::with_capacity(prev.len().div_ceil(branch));
for chunk in prev.chunks(branch) {
let mut t = Texel::EMPTY;
for c in chunk {
t.include_texel(c);
}
next.push(t);
}
levels.push(next);
}
// Output is root-first (convention B): levels[0] = root, last = floor.
levels.reverse();
WaveformPyramid {
floor_samples_per_texel: self.floor,
branch: self.branch,
channels: self.channels,
total_frames: self.total_frames,
levels,
}
}
}
// Tests live in `daw-backend/tests/waveform_pyramid.rs` (integration tests) so
// they build the lib in normal mode, independent of the crate's pre-existing
// broken `#[cfg(test)]` unit tests (automation.rs).

View File

@ -430,8 +430,6 @@ pub enum Query {
/// Add a MIDI clip instance to a track synchronously (track_id, instance) - returns instance ID /// Add a MIDI clip instance to a track synchronously (track_id, instance) - returns instance ID
/// The clip must already exist in the MidiClipPool /// The clip must already exist in the MidiClipPool
AddMidiClipInstanceSync(TrackId, crate::audio::midi::MidiClipInstance), AddMidiClipInstanceSync(TrackId, crate::audio::midi::MidiClipInstance),
/// Add an audio file to the pool synchronously (path, data, channels, sample_rate) - returns pool index
AddAudioFileSync(String, Vec<f32>, u32, u32),
/// Import an audio file synchronously (path) - returns pool index. /// Import an audio file synchronously (path) - returns pool index.
/// Does the same work as Command::ImportAudio (mmap for PCM, streaming /// Does the same work as Command::ImportAudio (mmap for PCM, streaming
/// setup for compressed) but returns the real pool index in the response. /// setup for compressed) but returns the real pool index in the response.
@ -440,12 +438,20 @@ pub enum Query {
/// problem for very large files, switch to async import with event-based /// problem for very large files, switch to async import with event-based
/// pool index reconciliation. /// pool index reconciliation.
ImportAudioSync(std::path::PathBuf), ImportAudioSync(std::path::PathBuf),
/// Add the audio track of a video file as a streaming pool entry (FFmpeg,
/// decoded on demand — no extraction). Probes the audio track and returns
/// the pool index. Response: `AudioImportedSync`.
AddVideoAudioSync(std::path::PathBuf),
/// Get raw audio samples from pool (pool_index) - returns (samples, sample_rate, channels) /// Get raw audio samples from pool (pool_index) - returns (samples, sample_rate, channels)
GetPoolAudioSamples(usize), GetPoolAudioSamples(usize),
/// Get a clone of the current project for serialization /// Get a clone of the current project for serialization
GetProject, GetProject,
/// Set the project (replaces current project state) /// Set the project (replaces current project state)
SetProject(Box<crate::audio::project::Project>), SetProject(Box<crate::audio::project::Project>),
/// Install the host's packed-media byte-source factory (for streaming
/// container-packed audio on load). Sent before `SetProject` so bulk
/// activation can open packed sources.
SetBlobSourceFactory(std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>),
/// Duplicate a MIDI clip in the pool, returning the new clip's ID /// Duplicate a MIDI clip in the pool, returning the new clip's ID
DuplicateMidiClipSync(MidiClipId), DuplicateMidiClipSync(MidiClipId),
/// Get whether a track's graph is still the auto-generated default /// Get whether a track's graph is still the auto-generated default
@ -516,10 +522,10 @@ pub enum QueryResponse {
AudioExported(Result<(), String>), AudioExported(Result<(), String>),
/// MIDI clip instance added (returns instance ID) /// MIDI clip instance added (returns instance ID)
MidiClipInstanceAdded(Result<MidiClipInstanceId, String>), MidiClipInstanceAdded(Result<MidiClipInstanceId, String>),
/// Audio file added to pool (returns pool index)
AudioFileAddedSync(Result<usize, String>),
/// Audio file imported to pool (returns pool index) /// Audio file imported to pool (returns pool index)
AudioImportedSync(Result<usize, String>), AudioImportedSync(Result<usize, String>),
/// Packed-media byte-source factory installed
BlobSourceFactorySet(Result<(), String>),
/// Raw audio samples from pool (samples, sample_rate, channels) /// Raw audio samples from pool (samples, sample_rate, channels)
PoolAudioSamples(Result<(Vec<f32>, u32, u32), String>), PoolAudioSamples(Result<(Vec<f32>, u32, u32), String>),
/// Project retrieved /// Project retrieved

View File

@ -0,0 +1,89 @@
//! Integration test for `CompressedReader::open_source` — decoding a streaming
//! audio source from an in-memory byte stream (the packed-in-container path)
//! rather than a filesystem path. Proves the `MediaByteSource` adapter feeds
//! Symphonia correctly (probe + decode + seekable byte length).
use std::io::{Cursor, Read, Seek, SeekFrom};
use daw_backend::audio::disk_reader::{CompressedReader, MediaByteSource};
/// A `MediaByteSource` over an in-memory buffer (stands in for core's BlobReader).
struct VecSource(Cursor<Vec<u8>>);
impl Read for VecSource {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.0.read(buf)
}
}
impl Seek for VecSource {
fn seek(&mut self, pos: SeekFrom) -> std::io::Result<u64> {
self.0.seek(pos)
}
}
impl MediaByteSource for VecSource {
fn byte_len(&self) -> u64 {
self.0.get_ref().len() as u64
}
}
/// Build a minimal PCM16 mono WAV byte buffer holding `frames` samples of a ramp.
fn make_wav(sample_rate: u32, frames: u32) -> Vec<u8> {
let channels: u16 = 1;
let bits: u16 = 16;
let block_align: u16 = channels * bits / 8;
let byte_rate: u32 = sample_rate * block_align as u32;
let data_len: u32 = frames * block_align as u32;
let mut v = Vec::new();
v.extend_from_slice(b"RIFF");
v.extend_from_slice(&(36 + data_len).to_le_bytes());
v.extend_from_slice(b"WAVE");
v.extend_from_slice(b"fmt ");
v.extend_from_slice(&16u32.to_le_bytes());
v.extend_from_slice(&1u16.to_le_bytes()); // PCM
v.extend_from_slice(&channels.to_le_bytes());
v.extend_from_slice(&sample_rate.to_le_bytes());
v.extend_from_slice(&byte_rate.to_le_bytes());
v.extend_from_slice(&block_align.to_le_bytes());
v.extend_from_slice(&bits.to_le_bytes());
v.extend_from_slice(b"data");
v.extend_from_slice(&data_len.to_le_bytes());
for i in 0..frames {
// A ramp from -16000..16000 so values are recognizable.
let s = (((i % 1000) as i32 - 500) * 32) as i16;
v.extend_from_slice(&s.to_le_bytes());
}
v
}
#[test]
fn open_source_decodes_in_memory_wav() {
let sample_rate = 8000;
let frames = 4096;
let bytes = make_wav(sample_rate, frames);
let src = Box::new(VecSource(Cursor::new(bytes)));
let mut reader = CompressedReader::open_source(src, Some("wav"))
.expect("open_source should probe the in-memory WAV");
assert_eq!(reader.sample_rate(), sample_rate);
assert_eq!(reader.channels(), 1);
// Decode the whole stream and count emitted frames.
let mut buf = Vec::new();
let mut decoded = 0usize;
loop {
let n = reader.decode_next(&mut buf).expect("decode_next");
if n == 0 {
break;
}
decoded += n;
}
// Should recover (approximately) all frames — codec frame counts can round.
assert!(
(decoded as i64 - frames as i64).abs() < 64,
"decoded {} vs expected {}",
decoded,
frames
);
}

View File

@ -0,0 +1,253 @@
//! Integration tests for `VideoAudioReader` (FFmpeg streaming audio source).
//!
//! These build the daw-backend lib in normal mode, so they're independent of
//! the crate's pre-existing broken `#[cfg(test)]` unit tests (automation.rs).
//! They synthesize a mono 32-bit-float WAV whose sample `i` has value `i/n`, so
//! a decoded sample's value identifies its frame index — letting us assert both
//! in-order decoding and **sample-accurate seeking** (the property video audio
//! needs to stay synced with other clips).
use daw_backend::audio::disk_reader::{
build_waveform_pyramid, CompressedReader, SourceKind, VideoAudioReader,
};
use std::io::Write;
use std::path::Path;
fn write_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
let channels = 1u16;
let bytes_per_sample = 4u32;
let data_size = n * bytes_per_sample;
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
buf.extend_from_slice(b"RIFF");
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
buf.extend_from_slice(b"WAVE");
buf.extend_from_slice(b"fmt ");
buf.extend_from_slice(&16u32.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
buf.extend_from_slice(&channels.to_le_bytes());
buf.extend_from_slice(&sample_rate.to_le_bytes());
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
buf.extend_from_slice(&32u16.to_le_bytes());
buf.extend_from_slice(b"data");
buf.extend_from_slice(&data_size.to_le_bytes());
for i in 0..n {
buf.extend_from_slice(&((i as f32) / (n as f32)).to_le_bytes());
}
let mut f = std::fs::File::create(path).unwrap();
f.write_all(&buf).unwrap();
}
/// Stereo ramp: frame `i` has left = `i/n`, right = `0.5 - i/n` (distinct per
/// channel), interleaved `[L0,R0,L1,R1,…]`. Exercises the channels>1 path.
fn write_stereo_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
let channels = 2u16;
let bytes_per_sample = 4u32;
let data_size = n * channels as u32 * bytes_per_sample;
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
buf.extend_from_slice(b"RIFF");
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
buf.extend_from_slice(b"WAVE");
buf.extend_from_slice(b"fmt ");
buf.extend_from_slice(&16u32.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
buf.extend_from_slice(&channels.to_le_bytes());
buf.extend_from_slice(&sample_rate.to_le_bytes());
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
buf.extend_from_slice(&32u16.to_le_bytes());
buf.extend_from_slice(b"data");
buf.extend_from_slice(&data_size.to_le_bytes());
for i in 0..n {
let l = i as f32 / n as f32;
let r = 0.5 - i as f32 / n as f32;
buf.extend_from_slice(&l.to_le_bytes());
buf.extend_from_slice(&r.to_le_bytes());
}
let mut f = std::fs::File::create(path).unwrap();
f.write_all(&buf).unwrap();
}
fn temp_path(tag: &str) -> std::path::PathBuf {
let mut p = std::env::temp_dir();
p.push(format!("lb_videoaudio_test_{}_{}.wav", std::process::id(), tag));
p
}
#[test]
fn decodes_samples_in_order() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("seq");
write_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
assert_eq!(reader.channels(), 1);
assert_eq!(reader.sample_rate(), sr);
// Probe estimate (used by add_video_audio_sync) should be ~n frames.
let tf = reader.total_frames() as f64;
assert!(
(tf - n as f64).abs() < n as f64 * 0.1,
"total_frames {} not ~{}",
tf, n
);
let mut all = Vec::new();
let mut buf = Vec::new();
loop {
let frames = reader.decode_next(&mut buf).unwrap();
if frames == 0 {
break;
}
all.extend_from_slice(&buf);
}
// Allow a couple of priming/flush samples of slack at the very end.
assert!(all.len() + 4 >= n as usize, "decoded too few samples: {}", all.len());
for (i, &v) in all.iter().enumerate().take(n as usize) {
let expected = i as f32 / n as f32;
assert!((v - expected).abs() < 1e-3, "sample {} = {}, expected {}", i, v, expected);
}
let _ = std::fs::remove_file(&path);
}
/// CompressedReader (symphonia) must seek **sample-accurately** too, so compressed
/// audio stays frame-synced with video audio. Symphonia decodes WAV via the same
/// path; its coarse seek lands on packet boundaries, exercising the decode-discard.
#[test]
fn compressed_reader_seek_is_sample_accurate() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("comp_seek");
write_ramp_wav(&path, n, sr);
let mut reader = CompressedReader::open(&path).unwrap();
assert_eq!(reader.channels(), 1);
assert_eq!(reader.sample_rate(), sr);
for &target in &[2000u64, 137, 3500, 0] {
let actual = reader.seek(target).unwrap();
assert_eq!(actual, target, "seek should report the exact target");
let mut buf = Vec::new();
let mut frames = 0;
for _ in 0..128 {
frames = reader.decode_next(&mut buf).unwrap();
if frames > 0 {
break;
}
}
assert!(frames > 0, "no samples after seek to {}", target);
let expected = target as f32 / n as f32;
assert!(
(buf[0] - expected).abs() < 1e-3,
"compressed seek to {}: first sample = {}, expected {}",
target, buf[0], expected
);
}
let _ = std::fs::remove_file(&path);
}
/// The decode→pyramid bridge should produce an envelope matching the signal,
/// through both reader backends (symphonia + ffmpeg), with bounded memory.
#[test]
fn waveform_pyramid_from_decode_matches_signal() {
let n = 5000u32;
let sr = 8000u32;
let path = temp_path("pyr");
write_ramp_wav(&path, n, sr); // ramp 0 .. (n-1)/n, all positive
for kind in [SourceKind::CompressedAudio, SourceKind::VideoAudio] {
let p = build_waveform_pyramid(&path, kind, 256).unwrap();
assert_eq!(p.channels, 1);
assert_eq!(p.root().len(), 1, "{:?}: root should be one texel", kind);
let root = p.root()[0];
assert!(root.l_min.abs() < 1e-2, "{:?}: root min {} ~ 0", kind, root.l_min);
let expected_max = (n - 1) as f32 / n as f32;
assert!(
(root.l_max - expected_max).abs() < 1e-2,
"{:?}: root max {} ~ {}", kind, root.l_max, expected_max
);
// Frame count is approximate across decoders (priming/resampler overhead);
// the envelope above is the real check. Just confirm it's about right.
assert!((p.total_frames as i64 - n as i64).abs() < 128, "{:?}: frames {}", kind, p.total_frames);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn decodes_stereo_interleaved() {
let n = 2000u32;
let sr = 8000u32;
let path = temp_path("stereo");
write_stereo_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
assert_eq!(reader.channels(), 2);
let mut all = Vec::new();
let mut buf = Vec::new();
loop {
let frames = reader.decode_next(&mut buf).unwrap();
if frames == 0 {
break;
}
// Each decode_next returns whole interleaved frames.
assert_eq!(buf.len() % 2, 0, "stereo decode returned a partial frame");
all.extend_from_slice(&buf);
}
// Interleaved L/R, ~n frames.
assert!(all.len() + 8 >= (n * 2) as usize, "decoded too few samples: {}", all.len());
for i in 0..n as usize {
let l = all[2 * i];
let r = all[2 * i + 1];
assert!((l - i as f32 / n as f32).abs() < 1e-3, "L[{}]={} expected {}", i, l, i as f32 / n as f32);
assert!(
(r - (0.5 - i as f32 / n as f32)).abs() < 1e-3,
"R[{}]={} expected {}", i, r, 0.5 - i as f32 / n as f32
);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn seek_is_sample_accurate() {
let n = 4000u32;
let sr = 8000u32;
let path = temp_path("seek");
write_ramp_wav(&path, n, sr);
let mut reader = VideoAudioReader::open(&path).unwrap();
for &target in &[2000u64, 137, 3500, 0] {
let actual = reader.seek(target).unwrap();
assert_eq!(actual, target);
// Pull the first non-empty decode after the seek.
let mut buf = Vec::new();
let mut frames = 0;
for _ in 0..64 {
frames = reader.decode_next(&mut buf).unwrap();
if frames > 0 {
break;
}
}
assert!(frames > 0, "no samples after seek to {}", target);
let expected = target as f32 / n as f32;
assert!(
(buf[0] - expected).abs() < 1e-3,
"after seek to {}: first sample = {}, expected {}",
target,
buf[0],
expected
);
// And the next few advance in order.
for k in 0..frames.min(8) {
let exp = (target as usize + k) as f32 / n as f32;
assert!((buf[k] - exp).abs() < 1e-3, "seek {}+{}: {} vs {}", target, k, buf[k], exp);
}
}
let _ = std::fs::remove_file(&path);
}

View File

@ -0,0 +1,135 @@
//! Integration tests for the streaming waveform LOD pyramid builder.
//!
//! Convention B: `levels[0]` is the root (coarsest), `levels.last()` the floor
//! (finest). Tests use the `.root()` / `.floor()` accessors so they don't depend
//! on the raw index ordering.
use daw_backend::audio::waveform_pyramid::{Texel, WaveformPyramid, WaveformPyramidBuilder};
fn build_mono(samples: &[f32], floor: u32) -> WaveformPyramid {
let mut b = WaveformPyramidBuilder::new(1, floor);
b.push_interleaved(samples);
b.finish()
}
#[test]
fn floor_level_min_max_per_bucket() {
// 8 samples, floor 4 → two floor texels covering [0..4) and [4..8).
let s: Vec<f32> = (0..8).map(|i| i as f32).collect();
let p = build_mono(&s, 4);
assert_eq!(p.floor().len(), 2);
assert_eq!(p.floor()[0], Texel { l_min: 0.0, l_max: 3.0, r_min: 0.0, r_max: 3.0 });
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 7.0, r_min: 4.0, r_max: 7.0 });
// Root reduces the two floor texels into the envelope [0..8).
assert_eq!(p.root().len(), 1);
assert_eq!(p.root()[0], Texel { l_min: 0.0, l_max: 7.0, r_min: 0.0, r_max: 7.0 });
}
#[test]
fn partial_trailing_bucket_is_flushed() {
// 6 samples, floor 4 → texels [0..4) and a ragged [4..6).
let s: Vec<f32> = (0..6).map(|i| i as f32).collect();
let p = build_mono(&s, 4);
assert_eq!(p.floor().len(), 2);
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 5.0, r_min: 4.0, r_max: 5.0 });
assert_eq!(p.total_frames, 6);
}
#[test]
fn multi_level_envelope_matches_global_min_max() {
let s: Vec<f32> = (0..1000).map(|i| ((i as f32) * 0.01).sin()).collect();
let g_min = s.iter().cloned().fold(f32::INFINITY, f32::min);
let g_max = s.iter().cloned().fold(f32::NEG_INFINITY, f32::max);
let p = build_mono(&s, 16);
assert_eq!(p.root().len(), 1);
assert!((p.root()[0].l_min - g_min).abs() < 1e-6);
assert!((p.root()[0].l_max - g_max).abs() < 1e-6);
// Every level's overall min/max equals the global (extremes are lossless).
for level in &p.levels {
let lmin = level.iter().map(|t| t.l_min).fold(f32::INFINITY, f32::min);
let lmax = level.iter().map(|t| t.l_max).fold(f32::NEG_INFINITY, f32::max);
assert!((lmin - g_min).abs() < 1e-6);
assert!((lmax - g_max).abs() < 1e-6);
}
}
#[test]
fn levels_are_root_first_and_get_finer() {
let s: Vec<f32> = (0..1000).map(|i| i as f32).collect();
let p = build_mono(&s, 16);
// Root first, floor last; strictly finer (more texels) as depth increases.
assert_eq!(p.root().len(), 1);
assert!(p.depth() >= 3);
for w in p.levels.windows(2) {
assert!(w[1].len() >= w[0].len(), "deeper level should be finer");
}
// Floor has ceil(1000/16) = 63 texels.
assert_eq!(p.floor().len(), 63);
}
#[test]
fn stereo_channels_tracked_separately() {
// L ramps up, R ramps down; interleaved.
let n = 64;
let mut s = Vec::new();
for i in 0..n {
s.push(i as f32); // L
s.push(-(i as f32)); // R
}
let mut b = WaveformPyramidBuilder::new(2, 16);
b.push_interleaved(&s);
let p = b.finish();
assert_eq!(p.root().len(), 1);
assert_eq!(p.root()[0].l_min, 0.0);
assert_eq!(p.root()[0].l_max, (n - 1) as f32);
assert_eq!(p.root()[0].r_min, -((n - 1) as f32));
assert_eq!(p.root()[0].r_max, 0.0);
}
#[test]
fn pyramid_size_is_bounded() {
let n = 100_000usize;
let s: Vec<f32> = (0..n).map(|i| (i % 7) as f32).collect();
let floor = 256u32;
let p = build_mono(&s, floor);
let total: usize = p.levels.iter().map(|l| l.len()).sum();
let floor_texels = (n as u32).div_ceil(floor) as usize;
// Geometric bound: < floor_texels * branch/(branch-1) + small per-level slack.
let bound = floor_texels * 4 / 3 + p.depth() + 2;
assert!(total <= bound, "pyramid too big: {} > {}", total, bound);
}
#[test]
fn bytes_round_trip() {
let s: Vec<f32> = (0..3333).map(|i| ((i as f32) * 0.013).sin()).collect();
let p = build_mono(&s, 64);
let bytes = p.to_bytes();
let q = WaveformPyramid::from_bytes(&bytes).unwrap();
assert_eq!(p.floor_samples_per_texel, q.floor_samples_per_texel);
assert_eq!(p.branch, q.branch);
assert_eq!(p.channels, q.channels);
assert_eq!(p.total_frames, q.total_frames);
assert_eq!(p.levels, q.levels);
// Truncated/garbage input is rejected, not panicking.
assert!(WaveformPyramid::from_bytes(&bytes[..bytes.len() - 4]).is_err());
assert!(WaveformPyramid::from_bytes(b"nope").is_err());
}
#[test]
fn pushing_in_arbitrary_chunks_matches() {
// The streaming builder must be agnostic to how samples are chunked.
let s: Vec<f32> = (0..5000).map(|i| ((i * 13) % 97) as f32 - 48.0).collect();
let whole = build_mono(&s, 32);
let mut b = WaveformPyramidBuilder::new(1, 32);
b.reserve_for_frames(5000);
for chunk in s.chunks(37) {
b.push_interleaved(chunk);
}
let chunked = b.finish();
assert_eq!(whole.depth(), chunked.depth());
for (a, c) in whole.levels.iter().zip(chunked.levels.iter()) {
assert_eq!(a, c);
}
}

View File

@ -2160,6 +2160,18 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365" checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
[[package]]
name = "fallible-iterator"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
[[package]]
name = "fallible-streaming-iterator"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]] [[package]]
name = "fancy-regex" name = "fancy-regex"
version = "0.16.2" version = "0.16.2"
@ -2925,6 +2937,9 @@ name = "hashbrown"
version = "0.14.5" version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
dependencies = [
"ahash 0.8.12",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
@ -2946,6 +2961,15 @@ dependencies = [
"foldhash 0.2.0", "foldhash 0.2.0",
] ]
[[package]]
name = "hashlink"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
dependencies = [
"hashbrown 0.14.5",
]
[[package]] [[package]]
name = "heapless" name = "heapless"
version = "0.8.0" version = "0.8.0"
@ -3415,6 +3439,17 @@ dependencies = [
"redox_syscall 0.5.18", "redox_syscall 0.5.18",
] ]
[[package]]
name = "libsqlite3-sys"
version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f"
dependencies = [
"cc",
"pkg-config",
"vcpkg",
]
[[package]] [[package]]
name = "libxdo" name = "libxdo"
version = "0.6.0" version = "0.6.0"
@ -3455,6 +3490,7 @@ dependencies = [
"objc2-foundation 0.3.2", "objc2-foundation 0.3.2",
"pathdiff", "pathdiff",
"rstar", "rstar",
"rusqlite",
"serde", "serde",
"serde_json", "serde_json",
"tiny-skia", "tiny-skia",
@ -3469,7 +3505,7 @@ dependencies = [
[[package]] [[package]]
name = "lightningbeam-editor" name = "lightningbeam-editor"
version = "1.0.3-alpha" version = "1.0.4-alpha"
dependencies = [ dependencies = [
"beamdsp", "beamdsp",
"bytemuck", "bytemuck",
@ -5478,6 +5514,20 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba" checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba"
[[package]]
name = "rusqlite"
version = "0.31.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae"
dependencies = [
"bitflags 2.10.0",
"fallible-iterator",
"fallible-streaming-iterator",
"hashlink",
"libsqlite3-sys",
"smallvec",
]
[[package]] [[package]]
name = "rustc-hash" name = "rustc-hash"
version = "1.1.0" version = "1.1.0"

View File

@ -36,6 +36,9 @@ zip = "0.6"
chrono = "0.4" chrono = "0.4"
base64 = "0.21" base64 = "0.21"
pathdiff = "0.2" pathdiff = "0.2"
# .beam container: SQLite database file. `bundled` compiles SQLite from source
# (no system dependency); `blob` enables incremental blob I/O for streaming.
rusqlite = { version = "0.31", features = ["bundled", "blob"] }
# Audio encoding for embedded files # Audio encoding for embedded files
flacenc = "0.4" # For FLAC encoding (lossless) flacenc = "0.4" # For FLAC encoding (lossless)

View File

@ -71,6 +71,15 @@ pub trait Action: Send {
/// Get a human-readable description of this action (for UI display) /// Get a human-readable description of this action (for UI display)
fn description(&self) -> String; fn description(&self) -> String;
/// For raster actions that store dirty-rect diffs: the `(layer_id, time)` of the
/// keyframe whose full pixels must be resident before `execute`/`rollback` can
/// apply the diff. The editor faults the frame in (synchronously) before undo/redo
/// so a paged-out clean frame is restored to its container state first. Non-raster
/// actions (and full-buffer ones) return `None`.
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
None
}
/// Execute backend operations after document changes /// Execute backend operations after document changes
/// ///
/// Called AFTER execute() succeeds. If this returns an error, execute() /// Called AFTER execute() succeeds. If this returns an error, execute()
@ -290,6 +299,18 @@ impl ActionExecutor {
self.undo_stack.last().map(|a| a.description()) self.undo_stack.last().map(|a| a.description())
} }
/// `(layer_id, time)` of the raster keyframe the next undo needs resident, if any.
/// The editor faults it in before calling `undo()` so a paged-out clean frame is
/// restored to its container state, giving the diff a correct base to apply onto.
pub fn peek_undo_raster_hint(&self) -> Option<(Uuid, f64)> {
self.undo_stack.last().and_then(|a| a.raster_resident_hint())
}
/// `(layer_id, time)` of the raster keyframe the next redo needs resident, if any.
pub fn peek_redo_raster_hint(&self) -> Option<(Uuid, f64)> {
self.redo_stack.last().and_then(|a| a.raster_resident_hint())
}
/// Get MIDI cache data from the last action on the undo stack (after redo). /// Get MIDI cache data from the last action on the undo stack (after redo).
/// Returns the notes reflecting execute state. /// Returns the notes reflecting execute state.
pub fn last_undo_midi_notes(&self) -> Option<(u32, &[(f64, u8, u8, f64)])> { pub fn last_undo_midi_notes(&self) -> Option<(u32, &[(f64, u8, u8, f64)])> {

View File

@ -235,11 +235,22 @@ impl Action for AddClipInstanceAction {
} }
} }
AudioClipType::Sampled { audio_pool_index } => { AudioClipType::Sampled { audio_pool_index } => {
// `trim_*` / `clip.duration` are in SECONDS (audio content time),
// while `timeline_*` and the backend's `duration` are in BEATS.
let internal_start = self.clip_instance.trim_start; let internal_start = self.clip_instance.trim_start;
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration); let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
let effective_duration = self.clip_instance.timeline_duration
.unwrap_or(internal_end - internal_start);
let start_time = self.clip_instance.timeline_start; let start_time = self.clip_instance.timeline_start;
// `effective_duration` is in BEATS. When `timeline_duration` is set
// it already is; otherwise the clip occupies its natural content
// length, so convert that seconds-span to beats at the clip's start
// (NOT `internal_end - internal_start`, which is seconds — that was
// the seconds-as-beats bug that made clips stop early off 60 BPM).
let effective_duration = self.clip_instance.timeline_duration.unwrap_or_else(|| {
let tempo_map = document.tempo_map();
let content_secs = internal_end - internal_start;
tempo_map.inverse_transform(tempo_map.transform(start_time) + content_secs)
- start_time
});
let instance_id = controller.add_audio_clip( let instance_id = controller.add_audio_clip(
*backend_track_id, *backend_track_id,
@ -305,8 +316,11 @@ mod tests {
let layer_id = layer.layer.id; let layer_id = layer.layer.id;
document.root_mut().add_child(AnyLayer::Vector(layer)); document.root_mut().add_child(AnyLayer::Vector(layer));
// Create a clip instance (using a fake clip_id since we're just testing the action) // Register a vector clip so get_clip_duration succeeds
let clip_id = Uuid::new_v4(); let clip_id = Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 5.0);
document.vector_clips.insert(clip_id, clip);
let clip_instance = ClipInstance::new(clip_id); let clip_instance = ClipInstance::new(clip_id);
let instance_id = clip_instance.id; let instance_id = clip_instance.id;

View File

@ -0,0 +1,52 @@
//! Add a blank raster keyframe at the playhead (the explicit "New Keyframe" command
//! for raster layers — mirrors `SetKeyframeAction` for vector, but inserts an empty
//! cel rather than copying the current graph).
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use uuid::Uuid;
pub struct AddRasterKeyframeAction {
layer_id: Uuid,
time: f64,
width: u32,
height: u32,
/// Id of the keyframe created by the last `execute` (so `rollback` can remove
/// exactly that one). `None` if a keyframe already existed at `time` (no-op).
created_id: Option<Uuid>,
}
impl AddRasterKeyframeAction {
pub fn new(layer_id: Uuid, time: f64, width: u32, height: u32) -> Self {
Self { layer_id, time, width, height, created_id: None }
}
}
impl Action for AddRasterKeyframeAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
let rl = match layer {
AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()),
};
// Inserts a blank cel only if one doesn't already exist at this time.
self.created_id = rl.insert_blank_keyframe_at(self.time, self.width, self.height);
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if let Some(id) = self.created_id.take() {
if let Some(AnyLayer::Raster(rl)) = document.get_layer_mut(&self.layer_id) {
rl.remove_keyframe(id);
}
}
Ok(())
}
fn description(&self) -> String {
"New keyframe".to_string()
}
}

View File

@ -37,6 +37,9 @@ pub struct DeleteFolderAction {
/// Asset IDs that were deleted (for DeleteRecursive strategy) /// Asset IDs that were deleted (for DeleteRecursive strategy)
deleted_asset_ids: Vec<Uuid>, deleted_asset_ids: Vec<Uuid>,
/// Subfolder IDs that were reparented to the deleted folder's parent (for MoveToParent strategy)
moved_subfolder_ids: Vec<Uuid>,
} }
impl DeleteFolderAction { impl DeleteFolderAction {
@ -55,6 +58,7 @@ impl DeleteFolderAction {
removed_folders: Vec::new(), removed_folders: Vec::new(),
moved_asset_ids: Vec::new(), moved_asset_ids: Vec::new(),
deleted_asset_ids: Vec::new(), deleted_asset_ids: Vec::new(),
moved_subfolder_ids: Vec::new(),
} }
} }
} }
@ -130,6 +134,7 @@ impl Action for DeleteFolderAction {
for subfolder_id in subfolder_ids { for subfolder_id in subfolder_ids {
if let Some(subfolder) = tree.folders.get_mut(&subfolder_id) { if let Some(subfolder) = tree.folders.get_mut(&subfolder_id) {
subfolder.parent_id = parent_id; subfolder.parent_id = parent_id;
self.moved_subfolder_ids.push(subfolder_id);
} }
} }
} }
@ -259,6 +264,13 @@ impl Action for DeleteFolderAction {
tree.add_folder(folder.clone()); tree.add_folder(folder.clone());
} }
// Restore reparented subfolders back under the deleted folder
for subfolder_id in &self.moved_subfolder_ids {
if let Some(subfolder) = tree.folders.get_mut(subfolder_id) {
subfolder.parent_id = Some(self.folder_id);
}
}
match self.strategy { match self.strategy {
DeleteStrategy::MoveToParent => { DeleteStrategy::MoveToParent => {
// Restore folder_id for moved assets // Restore folder_id for moved assets
@ -312,6 +324,7 @@ impl Action for DeleteFolderAction {
self.removed_folders.clear(); self.removed_folders.clear();
self.moved_asset_ids.clear(); self.moved_asset_ids.clear();
self.deleted_asset_ids.clear(); self.deleted_asset_ids.clear();
self.moved_subfolder_ids.clear();
Ok(()) Ok(())
} }

View File

@ -34,8 +34,10 @@ pub mod convert_to_movie_clip;
pub mod region_split; pub mod region_split;
pub mod toggle_group_expansion; pub mod toggle_group_expansion;
pub mod group_layers; pub mod group_layers;
pub mod raster_diff;
pub mod raster_stroke; pub mod raster_stroke;
pub mod raster_fill; pub mod raster_fill;
pub mod add_raster_keyframe;
pub mod move_layer; pub mod move_layer;
pub mod set_fill_paint; pub mod set_fill_paint;
@ -70,6 +72,7 @@ pub use toggle_group_expansion::ToggleGroupExpansionAction;
pub use group_layers::GroupLayersAction; pub use group_layers::GroupLayersAction;
pub use raster_stroke::RasterStrokeAction; pub use raster_stroke::RasterStrokeAction;
pub use raster_fill::RasterFillAction; pub use raster_fill::RasterFillAction;
pub use add_raster_keyframe::AddRasterKeyframeAction;
pub use move_layer::MoveLayerAction; pub use move_layer::MoveLayerAction;
pub use set_fill_paint::SetFillPaintAction; pub use set_fill_paint::SetFillPaintAction;
pub use change_bpm::ChangeBpmAction; pub use change_bpm::ChangeBpmAction;

View File

@ -0,0 +1,232 @@
//! Dirty-rect diff for raster undo/redo.
//!
//! Brush strokes and fills used to store the *entire* before/after RGBA frame in the
//! undo stack (~8 MB each at 1080p → up to ~1.6 GB at the 100-action cap). A
//! [`RasterDiff`] instead stores only the changed bounding box's pixels before and
//! after, which for a typical brush dab is a few tens of KB.
//!
//! Applying a diff overwrites just the bbox of the keyframe's `raw_pixels`, so the
//! buffer **must be resident** (full length `w*h*4`) when `apply_*` runs. The editor
//! guarantees this by faulting the target frame in before undo/redo (a clean evicted
//! frame's container bytes equal its current logical state, so the restored base is
//! correct). If the base is somehow not resident we skip rather than corrupt.
/// Normalize a buffer to full length `n`; an empty/short buffer becomes transparent.
fn normalize(buf: &[u8], n: usize) -> std::borrow::Cow<[u8]> {
if buf.len() == n {
std::borrow::Cow::Borrowed(buf)
} else {
std::borrow::Cow::Owned(vec![0u8; n])
}
}
/// A minimal before/after record of the region a raster edit changed.
#[derive(Clone, Debug)]
pub struct RasterDiff {
full_width: u32,
full_height: u32,
/// Changed bounding box `(x, y, w, h)`; `None` when before == after (no-op).
bbox: Option<(u32, u32, u32, u32)>,
/// bbox-sized RGBA (`w*h*4`) of the region before the edit.
before_region: Vec<u8>,
/// bbox-sized RGBA (`w*h*4`) of the region after the edit.
after_region: Vec<u8>,
/// The pre-edit buffer was blank (empty/unallocated) — i.e. this was the first
/// edit on a fresh keyframe. Lets `apply_after` build from a transparent base
/// (the commit/redo path often starts with empty `raw_pixels`) and `apply_before`
/// restore to blank, instead of requiring a resident base.
before_blank: bool,
}
impl RasterDiff {
/// Build a diff from full before/after buffers. `after` is expected to be the
/// resident post-edit buffer (`width*height*4`); `before` may be empty (a blank
/// keyframe's first stroke), treated as fully transparent.
pub fn compute(before: &[u8], after: &[u8], width: u32, height: u32) -> Self {
let n = width as usize * height as usize * 4;
let before_blank = before.len() != n;
// Normalize both sides to full length; empty/short ⇒ transparent.
let before_full = normalize(before, n);
let after_full = normalize(after, n);
// Find the tight bbox of differing pixels (compare 4-byte texels).
let (w, h) = (width as usize, height as usize);
let (mut min_x, mut min_y, mut max_x, mut max_y) = (usize::MAX, usize::MAX, 0usize, 0usize);
let mut any = false;
for y in 0..h {
let row = y * w * 4;
for x in 0..w {
let i = row + x * 4;
if before_full[i..i + 4] != after_full[i..i + 4] {
any = true;
if x < min_x { min_x = x; }
if x > max_x { max_x = x; }
if y < min_y { min_y = y; }
if y > max_y { max_y = y; }
}
}
}
if !any {
return Self { full_width: width, full_height: height, bbox: None,
before_region: Vec::new(), after_region: Vec::new(), before_blank };
}
let bw = max_x - min_x + 1;
let bh = max_y - min_y + 1;
let crop = |full: &[u8]| -> Vec<u8> {
let mut out = Vec::with_capacity(bw * bh * 4);
for row in 0..bh {
let src = ((min_y + row) * w + min_x) * 4;
out.extend_from_slice(&full[src..src + bw * 4]);
}
out
};
Self {
full_width: width,
full_height: height,
bbox: Some((min_x as u32, min_y as u32, bw as u32, bh as u32)),
before_region: crop(&before_full),
after_region: crop(&after_full),
before_blank,
}
}
/// Approximate retained size in bytes (the two cropped regions).
pub fn byte_size(&self) -> usize {
self.before_region.len() + self.after_region.len()
}
/// Restore the pre-edit pixels into `raw` (undo / first-execute rollback).
pub fn apply_before(&self, raw: &mut Vec<u8>) {
if self.bbox.is_none() {
return; // no change
}
if self.before_blank {
// The frame was blank before this edit (it was the first stroke); undoing
// it returns to blank regardless of the current buffer.
raw.clear();
return;
}
self.stamp_resident(&self.before_region, raw);
}
/// Apply the post-edit pixels into `raw` (commit / redo).
pub fn apply_after(&self, raw: &mut Vec<u8>) {
if self.bbox.is_none() {
return; // no change
}
if self.before_blank {
// Base was blank: build a full transparent buffer then stamp the bbox. The
// commit/redo path frequently starts from empty `raw_pixels` here.
let n = self.full_width as usize * self.full_height as usize * 4;
raw.clear();
raw.resize(n, 0);
}
self.stamp_resident(&self.after_region, raw);
}
/// Stamp a bbox-sized region into `raw`, which must already be full-size. If it
/// isn't (a non-blank base that the editor failed to fault in), skip rather than
/// resize-and-corrupt — the frame will re-page to its container state.
fn stamp_resident(&self, region: &[u8], raw: &mut [u8]) {
let n = self.full_width as usize * self.full_height as usize * 4;
let (x, y, bw, bh) = match self.bbox {
Some(b) => b,
None => return,
};
if raw.len() != n {
eprintln!(
"⚠️ [RASTER_DIFF] base not resident ({} != {}); skipping undo/redo apply",
raw.len(), n
);
return;
}
let (x, y, bw, bh) = (x as usize, y as usize, bw as usize, bh as usize);
let fw = self.full_width as usize;
for row in 0..bh {
let dst = ((y + row) * fw + x) * 4;
let src = row * bw * 4;
raw[dst..dst + bw * 4].copy_from_slice(&region[src..src + bw * 4]);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn solid(w: u32, h: u32, px: [u8; 4]) -> Vec<u8> {
px.iter().copied().cycle().take((w * h * 4) as usize).collect()
}
#[test]
fn roundtrip_reproduces_buffers_exactly() {
let (w, h) = (8, 6);
let before = solid(w, h, [10, 20, 30, 255]);
let mut after = before.clone();
// Change a 2x2 region at (3,2).
for (dy, dx) in [(0, 0), (0, 1), (1, 0), (1, 1)] {
let i = (((2 + dy) * w + (3 + dx)) * 4) as usize;
after[i..i + 4].copy_from_slice(&[200, 100, 50, 255]);
}
let diff = RasterDiff::compute(&before, &after, w, h);
assert_eq!(diff.bbox, Some((3, 2, 2, 2)));
let mut buf = after.clone();
diff.apply_before(&mut buf);
assert_eq!(buf, before, "undo must reproduce the pre-edit buffer exactly");
diff.apply_after(&mut buf);
assert_eq!(buf, after, "redo must reproduce the post-edit buffer exactly");
}
#[test]
fn blank_before_first_stroke() {
let (w, h) = (4, 4);
let n = (w * h * 4) as usize;
let before: Vec<u8> = Vec::new(); // blank keyframe
let mut after = vec![0u8; n];
let i = ((1 * w + 1) * 4) as usize;
after[i..i + 4].copy_from_slice(&[255, 0, 0, 255]);
let diff = RasterDiff::compute(&before, &after, w, h);
assert_eq!(diff.bbox, Some((1, 1, 1, 1)));
// First execute / redo from EMPTY raw_pixels (the real commit path): builds
// the full buffer from transparent + the stroke.
let mut buf: Vec<u8> = Vec::new();
diff.apply_after(&mut buf);
assert_eq!(buf, after, "commit/redo must build the frame from a blank base");
// Undo the first stroke → back to blank (empty).
diff.apply_before(&mut buf);
assert!(buf.is_empty(), "undoing the first stroke restores the blank keyframe");
// Redo again from the now-empty buffer.
diff.apply_after(&mut buf);
assert_eq!(buf, after);
}
#[test]
fn no_change_is_noop() {
let (w, h) = (4, 4);
let buf = solid(w, h, [1, 2, 3, 4]);
let diff = RasterDiff::compute(&buf, &buf, w, h);
assert_eq!(diff.bbox, None);
assert_eq!(diff.byte_size(), 0);
let mut b = buf.clone();
diff.apply_before(&mut b);
assert_eq!(b, buf);
}
#[test]
fn not_resident_base_is_skipped_not_corrupted() {
let (w, h) = (4, 4);
let before = solid(w, h, [9, 9, 9, 255]);
let after = solid(w, h, [1, 2, 3, 255]);
let diff = RasterDiff::compute(&before, &after, w, h);
let mut empty: Vec<u8> = Vec::new();
diff.apply_before(&mut empty); // base not resident
assert!(empty.is_empty(), "must not resize/corrupt a non-resident base");
}
}

View File

@ -1,6 +1,7 @@
//! Raster flood-fill action — records and undoes a paint bucket fill on a RasterLayer. //! Raster flood-fill action — records and undoes a paint bucket fill on a RasterLayer.
use crate::action::Action; use crate::action::Action;
use crate::actions::raster_diff::RasterDiff;
use crate::document::Document; use crate::document::Document;
use crate::layer::AnyLayer; use crate::layer::AnyLayer;
use uuid::Uuid; use uuid::Uuid;
@ -8,11 +9,13 @@ use uuid::Uuid;
pub struct RasterFillAction { pub struct RasterFillAction {
layer_id: Uuid, layer_id: Uuid,
time: f64, time: f64,
buffer_before: Vec<u8>,
buffer_after: Vec<u8>,
width: u32, width: u32,
height: u32, height: u32,
name: String, name: String,
diff: RasterDiff,
/// Full post-fill buffer, kept only for the first `execute` (commit); see
/// `RasterStrokeAction::full_after`.
full_after: Option<Vec<u8>>,
} }
impl RasterFillAction { impl RasterFillAction {
@ -24,7 +27,9 @@ impl RasterFillAction {
width: u32, width: u32,
height: u32, height: u32,
) -> Self { ) -> Self {
Self { layer_id, time, buffer_before, buffer_after, width, height, name: "Flood fill".to_string() } let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
Self { layer_id, time, width, height, name: "Flood fill".to_string(),
diff, full_after: Some(buffer_after) }
} }
pub fn with_description(mut self, name: &str) -> Self { pub fn with_description(mut self, name: &str) -> Self {
@ -41,9 +46,17 @@ impl Action for RasterFillAction {
AnyLayer::Raster(rl) => rl, AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()), _ => return Err("Not a raster layer".to_string()),
}; };
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height); let _ = (self.width, self.height);
kf.raw_pixels = self.buffer_after.clone(); let kf = raster
.keyframe_at_mut(self.time)
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
if let Some(full) = self.full_after.take() {
kf.raw_pixels = full;
} else {
self.diff.apply_after(&mut kf.raw_pixels);
}
kf.texture_dirty = true; kf.texture_dirty = true;
kf.dirty = true;
Ok(()) Ok(())
} }
@ -54,13 +67,21 @@ impl Action for RasterFillAction {
AnyLayer::Raster(rl) => rl, AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()), _ => return Err("Not a raster layer".to_string()),
}; };
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height); let _ = (self.width, self.height);
kf.raw_pixels = self.buffer_before.clone(); let kf = raster
.keyframe_at_mut(self.time)
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
self.diff.apply_before(&mut kf.raw_pixels);
kf.texture_dirty = true; kf.texture_dirty = true;
kf.dirty = true;
Ok(()) Ok(())
} }
fn description(&self) -> String { fn description(&self) -> String {
self.name.clone() self.name.clone()
} }
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
Some((self.layer_id, self.time))
}
} }

View File

@ -9,6 +9,7 @@
//! `rollback` → swap in `buffer_before` //! `rollback` → swap in `buffer_before`
use crate::action::Action; use crate::action::Action;
use crate::actions::raster_diff::RasterDiff;
use crate::document::Document; use crate::document::Document;
use crate::layer::AnyLayer; use crate::layer::AnyLayer;
use uuid::Uuid; use uuid::Uuid;
@ -16,16 +17,20 @@ use uuid::Uuid;
/// Action that records a single brush stroke for undo/redo. /// Action that records a single brush stroke for undo/redo.
/// ///
/// The stroke must already be painted into the document's `raw_pixels` before /// The stroke must already be painted into the document's `raw_pixels` before
/// this action is executed for the first time. /// this action is executed for the first time. Only the changed bounding box is
/// retained (see [`RasterDiff`]) rather than two full frame buffers.
pub struct RasterStrokeAction { pub struct RasterStrokeAction {
layer_id: Uuid, layer_id: Uuid,
time: f64, time: f64,
/// Raw RGBA pixels *before* the stroke (for rollback / undo)
buffer_before: Vec<u8>,
/// Raw RGBA pixels *after* the stroke (for execute / redo)
buffer_after: Vec<u8>,
width: u32, width: u32,
height: u32, height: u32,
diff: RasterDiff,
/// The full post-stroke buffer, kept ONLY for the first `execute` (the commit),
/// which establishes `raw_pixels` exactly like the old code did — robust no matter
/// what state the working buffer is in (empty new keyframe, GPU-canvas readback,
/// etc.). Taken (dropped) on first execute, so the action sitting in the undo stack
/// retains only the small `diff`; redo then replays via the diff.
full_after: Option<Vec<u8>>,
} }
impl RasterStrokeAction { impl RasterStrokeAction {
@ -33,6 +38,8 @@ impl RasterStrokeAction {
/// ///
/// * `buffer_before` raw RGBA pixels captured just before the stroke began. /// * `buffer_before` raw RGBA pixels captured just before the stroke began.
/// * `buffer_after` raw RGBA pixels captured just after the stroke finished. /// * `buffer_after` raw RGBA pixels captured just after the stroke finished.
///
/// The full buffers are diffed down to the changed bbox here and then dropped.
pub fn new( pub fn new(
layer_id: Uuid, layer_id: Uuid,
time: f64, time: f64,
@ -41,28 +48,41 @@ impl RasterStrokeAction {
width: u32, width: u32,
height: u32, height: u32,
) -> Self { ) -> Self {
Self { layer_id, time, buffer_before, buffer_after, width, height } let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
Self { layer_id, time, width, height, diff, full_after: Some(buffer_after) }
} }
} }
impl Action for RasterStrokeAction { impl Action for RasterStrokeAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> { fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?; let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
kf.raw_pixels = self.buffer_after.clone(); if let Some(full) = self.full_after.take() {
// First execute (commit): assign the full buffer outright.
kf.raw_pixels = full;
} else {
// Redo: replay via the diff onto the (resident) base.
self.diff.apply_after(&mut kf.raw_pixels);
}
kf.texture_dirty = true; kf.texture_dirty = true;
kf.dirty = true;
Ok(()) Ok(())
} }
fn rollback(&mut self, document: &mut Document) -> Result<(), String> { fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?; let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
kf.raw_pixels = self.buffer_before.clone(); self.diff.apply_before(&mut kf.raw_pixels);
kf.texture_dirty = true; kf.texture_dirty = true;
kf.dirty = true;
Ok(()) Ok(())
} }
fn description(&self) -> String { fn description(&self) -> String {
"Paint stroke".to_string() "Paint stroke".to_string()
} }
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
Some((self.layer_id, self.time))
}
} }
fn get_keyframe_mut<'a>( fn get_keyframe_mut<'a>(
@ -79,5 +99,10 @@ fn get_keyframe_mut<'a>(
AnyLayer::Raster(rl) => rl, AnyLayer::Raster(rl) => rl,
_ => return Err("Not a raster layer".to_string()), _ => return Err("Not a raster layer".to_string()),
}; };
Ok(raster.ensure_keyframe_at(time, width, height)) let _ = (width, height);
// Edit the ACTIVE keyframe (at-or-before `time`); never create one — keyframes
// are made explicitly via "New Keyframe".
raster
.keyframe_at_mut(time)
.ok_or_else(|| format!("No raster keyframe at/before t={time}"))
} }

View File

@ -547,6 +547,8 @@ mod tests {
// Create a clip ID (ClipInstance references clip by ID) // Create a clip ID (ClipInstance references clip by ID)
let clip_id = uuid::Uuid::new_v4(); let clip_id = uuid::Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
document.vector_clips.insert(clip_id, clip);
let mut vector_layer = VectorLayer::new("Layer 1"); let mut vector_layer = VectorLayer::new("Layer 1");
@ -607,6 +609,8 @@ mod tests {
// Create a clip ID (ClipInstance references clip by ID) // Create a clip ID (ClipInstance references clip by ID)
let clip_id = uuid::Uuid::new_v4(); let clip_id = uuid::Uuid::new_v4();
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
document.vector_clips.insert(clip_id, clip);
let mut vector_layer = VectorLayer::new("Layer 1"); let mut vector_layer = VectorLayer::new("Layer 1");

View File

@ -0,0 +1,813 @@
//! SQLite-backed `.beam` project container.
//!
//! The `.beam` format is a single SQLite database file. It replaces the older
//! ZIP-archive format. SQLite gives us, in one file:
//!
//! - **Streaming reads** — packed media is split into chunk rows and read on
//! demand through [`BlobReader`] (`Read + Seek`), so arbitrary-length audio /
//! video never has to be fully decoded into RAM on load.
//! - **In-place, crash-safe mutation** — raster frame write-back and re-save are
//! transactional `UPDATE`s rather than rewriting a whole archive.
//! - **Single-file UX** — behaves like a file on every platform.
//!
//! ## Media storage
//!
//! Each media item is one row in `media` plus, when *packed*, N rows in
//! `media_chunk`:
//!
//! - **Packed** (`MediaStorage::Packed`) — bytes live in the database, split
//! into [`CHUNK_SIZE`]-byte chunks. Chunking keeps each blob well under
//! SQLite's ~2 GB per-blob ceiling and bounds the working set of a streaming
//! reader to a single chunk.
//! - **Referenced** (`MediaStorage::Referenced`) — only an external path is
//! stored; the bytes stay on disk (useful for shared media on a network drive,
//! or media too large/volatile to pack). Callers open the path directly.
//!
//! `project.json` (the serialized `BeamProject`) is stored verbatim in the
//! single-row `project_json` table; only the container and media storage change
//! relative to the legacy format.
use rusqlite::blob::Blob;
use rusqlite::{Connection, DatabaseName, OpenFlags, OptionalExtension};
use std::io::{self, Read, Seek, SeekFrom};
use std::path::Path;
use uuid::Uuid;
/// Default packed-media chunk size: 4 MiB.
///
/// Small enough to bound a streaming reader's per-chunk work and any
/// whole-chunk buffering, large enough to keep row counts modest (a 1 GB file
/// is 256 rows). Comfortably under SQLite's ~2 GB per-blob limit.
pub const CHUNK_SIZE: u64 = 4 * 1024 * 1024;
/// Files at or above this size prompt the user to pick packed vs referenced
/// (and the choice is then persisted as the default). Matches SQLite's
/// practical large-blob threshold.
pub const LARGE_MEDIA_THRESHOLD: u64 = 2 * 1024 * 1024 * 1024;
/// Kind of media stored in the `media` table.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MediaKind {
Audio = 0,
Video = 1,
Raster = 2,
ImageAsset = 3,
/// A precomputed waveform LOD pyramid blob for an audio item (keyed by the
/// same id as the audio it describes). See `daw_backend::audio::waveform_pyramid`.
Waveform = 4,
/// A pack of precomputed video thumbnails for a video clip (keyed by a
/// sentinel-derived id from the clip id). Opaque blob; format owned by the editor.
Thumbnail = 5,
/// A low-res PNG proxy of a raster keyframe (keyed by a sentinel-derived id from
/// the keyframe id). Decoded eagerly on load and shown while the full-res pixels
/// page in, so cold scrubs don't flash blank. See `raster_proxy_media_id`.
RasterProxy = 6,
}
impl MediaKind {
fn from_i64(v: i64) -> Option<Self> {
match v {
0 => Some(Self::Audio),
1 => Some(Self::Video),
2 => Some(Self::Raster),
3 => Some(Self::ImageAsset),
4 => Some(Self::Waveform),
5 => Some(Self::Thumbnail),
6 => Some(Self::RasterProxy),
_ => None,
}
}
}
/// How a media item's bytes are stored.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MediaStorage {
/// Bytes are chunked into `media_chunk` rows inside the database.
Packed = 0,
/// Only an external path is stored; bytes live on disk.
Referenced = 1,
}
impl MediaStorage {
fn from_i64(v: i64) -> Option<Self> {
match v {
0 => Some(Self::Packed),
1 => Some(Self::Referenced),
_ => None,
}
}
}
/// Metadata row for a media item (no bytes).
#[derive(Debug, Clone)]
pub struct MediaInfo {
pub id: Uuid,
pub kind: MediaKind,
/// Original codec / container extension, e.g. `"flac"`, `"mp3"`, `"png"`.
pub codec: String,
pub storage: MediaStorage,
/// Set when `storage == Referenced`.
pub ext_path: Option<String>,
/// Total byte length of the media payload (packed only; 0 for referenced).
pub total_len: u64,
// Kind-specific metadata (nullable; meaning depends on `kind`).
pub channels: Option<u32>,
pub sample_rate: Option<u32>,
pub width: Option<u32>,
pub height: Option<u32>,
}
/// Optional kind-specific metadata supplied when writing a media item.
#[derive(Debug, Clone, Copy, Default)]
pub struct MediaMeta {
pub channels: Option<u32>,
pub sample_rate: Option<u32>,
pub width: Option<u32>,
pub height: Option<u32>,
}
/// A `.beam` project container backed by a SQLite database.
pub struct BeamArchive {
conn: Connection,
chunk_size: u64,
}
impl BeamArchive {
/// Schema version stored in `meta` under `"schema_version"`.
pub const SCHEMA_VERSION: i64 = 1;
/// Create a new (empty) archive at `path`, replacing any existing file.
pub fn create(path: &Path) -> Result<Self, String> {
// Remove any existing file so we start from a clean schema.
if path.exists() {
std::fs::remove_file(path).map_err(|e| format!("Failed to replace {:?}: {}", path, e))?;
}
let conn = Connection::open(path).map_err(map_sql)?;
let mut archive = Self { conn, chunk_size: CHUNK_SIZE };
archive.init_schema()?;
Ok(archive)
}
/// Open an existing archive for read/write.
pub fn open(path: &Path) -> Result<Self, String> {
let conn = Connection::open(path).map_err(map_sql)?;
let archive = Self { conn, chunk_size: CHUNK_SIZE };
archive.verify_schema()?;
Ok(archive)
}
/// Quick check: does `path` look like a SQLite database (vs. a legacy ZIP)?
/// Reads the 16-byte SQLite header magic. Used to route between the SQLite
/// loader and the legacy-ZIP migration path.
pub fn is_sqlite(path: &Path) -> bool {
use std::io::Read as _;
let mut f = match std::fs::File::open(path) {
Ok(f) => f,
Err(_) => return false,
};
let mut magic = [0u8; 16];
if f.read_exact(&mut magic).is_err() {
return false;
}
&magic == b"SQLite format 3\0"
}
fn init_schema(&mut self) -> Result<(), String> {
self.conn
.execute_batch(
"BEGIN;
CREATE TABLE media (
id BLOB PRIMARY KEY, -- 16-byte Uuid
kind INTEGER NOT NULL,
codec TEXT NOT NULL,
storage INTEGER NOT NULL,
ext_path TEXT,
total_len INTEGER NOT NULL DEFAULT 0,
channels INTEGER,
sample_rate INTEGER,
width INTEGER,
height INTEGER
);
CREATE TABLE media_chunk (
id INTEGER PRIMARY KEY, -- rowid, for blob_open
media_id BLOB NOT NULL,
chunk_index INTEGER NOT NULL,
bytes BLOB NOT NULL,
UNIQUE (media_id, chunk_index)
);
CREATE TABLE project_json (
id INTEGER PRIMARY KEY CHECK (id = 0),
data TEXT NOT NULL
);
CREATE TABLE meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
COMMIT;",
)
.map_err(map_sql)?;
self.set_meta("schema_version", &Self::SCHEMA_VERSION.to_string())?;
Ok(())
}
fn verify_schema(&self) -> Result<(), String> {
let v: Option<String> = self.get_meta("schema_version")?;
match v.as_deref().and_then(|s| s.parse::<i64>().ok()) {
Some(n) if n <= Self::SCHEMA_VERSION => Ok(()),
Some(n) => Err(format!(
"Unsupported .beam schema version {} (this build supports up to {})",
n,
Self::SCHEMA_VERSION
)),
None => Err("Not a valid .beam archive (missing schema_version)".to_string()),
}
}
/// Begin a write transaction grouping multiple media/json writes into one
/// atomic, crash-safe commit. Used by saves so unchanged (large) media is
/// never rewritten — only dirty rows are touched, in place.
pub fn transaction(&mut self) -> Result<BeamTxn<'_>, String> {
let tx = self.conn.transaction().map_err(map_sql)?;
Ok(BeamTxn { tx, chunk_size: self.chunk_size })
}
// -- meta key/value --------------------------------------------------
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
set_meta_conn(&self.conn, key, value)
}
pub fn get_meta(&self, key: &str) -> Result<Option<String>, String> {
self.conn
.query_row("SELECT value FROM meta WHERE key = ?1", [key], |r| r.get(0))
.optional()
.map_err(map_sql)
}
// -- project.json ----------------------------------------------------
/// Store the serialized `project.json` (single row).
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
set_project_json_conn(&self.conn, json)
}
/// Read the serialized `project.json`.
pub fn get_project_json(&self) -> Result<String, String> {
self.conn
.query_row("SELECT data FROM project_json WHERE id = 0", [], |r| r.get(0))
.optional()
.map_err(map_sql)?
.ok_or_else(|| "Archive has no project.json".to_string())
}
// -- media write -----------------------------------------------------
/// Write a media item whose bytes are packed (chunked) into the database.
/// Replaces any existing rows for `id`.
pub fn put_media_packed(
&mut self,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
let tx = self.conn.transaction().map_err(map_sql)?;
write_media_packed(&tx, self.chunk_size, id, kind, codec, bytes, meta)?;
tx.commit().map_err(map_sql)?;
Ok(())
}
/// Write a media item that references an external file by path (no bytes
/// stored). Replaces any existing rows for `id`.
pub fn put_media_referenced(
&mut self,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
let tx = self.conn.transaction().map_err(map_sql)?;
write_media_referenced(&tx, id, kind, codec, ext_path, meta)?;
tx.commit().map_err(map_sql)?;
Ok(())
}
// -- media read ------------------------------------------------------
/// Look up a media item's metadata.
pub fn media_info(&self, id: Uuid) -> Result<Option<MediaInfo>, String> {
let id_bytes = id.as_bytes().to_vec();
self.conn
.query_row(
"SELECT kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height
FROM media WHERE id = ?1",
[&id_bytes],
|r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, String>(1)?,
r.get::<_, i64>(2)?,
r.get::<_, Option<String>>(3)?,
r.get::<_, i64>(4)?,
r.get::<_, Option<u32>>(5)?,
r.get::<_, Option<u32>>(6)?,
r.get::<_, Option<u32>>(7)?,
r.get::<_, Option<u32>>(8)?,
))
},
)
.optional()
.map_err(map_sql)?
.map(|(kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)| {
Ok(MediaInfo {
id,
kind: MediaKind::from_i64(kind)
.ok_or_else(|| format!("Unknown media kind {}", kind))?,
codec,
storage: MediaStorage::from_i64(storage)
.ok_or_else(|| format!("Unknown media storage {}", storage))?,
ext_path,
total_len: total_len.max(0) as u64,
channels,
sample_rate,
width,
height,
})
})
.transpose()
}
/// List every media item of a given kind.
pub fn media_ids_of_kind(&self, kind: MediaKind) -> Result<Vec<Uuid>, String> {
let mut stmt = self
.conn
.prepare("SELECT id FROM media WHERE kind = ?1")
.map_err(map_sql)?;
let rows = stmt
.query_map([kind as i64], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::new();
for row in rows {
let bytes = row.map_err(map_sql)?;
out.push(uuid_from_bytes(&bytes)?);
}
Ok(out)
}
/// Read an entire packed media item into memory. Convenience for small
/// media (raster frames, image assets); large media should stream via
/// [`BeamArchive::open_blob_reader`] instead.
pub fn read_media_full(&self, id: Uuid) -> Result<Vec<u8>, String> {
let info = self
.media_info(id)?
.ok_or_else(|| format!("Media {} not found", id))?;
if info.storage != MediaStorage::Packed {
return Err(format!("Media {} is referenced, not packed", id));
}
let id_bytes = id.as_bytes().to_vec();
let mut stmt = self
.conn
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::with_capacity(info.total_len as usize);
for row in rows {
out.extend_from_slice(&row.map_err(map_sql)?);
}
Ok(out)
}
/// Open a streaming reader over a packed media item. The reader owns its own
/// SQLite connection (read-only) so it can live on a separate thread (e.g.
/// the audio disk reader) independent of this archive handle.
pub fn open_blob_reader(&self, db_path: &Path, id: Uuid) -> Result<BlobReader, String> {
let info = self
.media_info(id)?
.ok_or_else(|| format!("Media {} not found", id))?;
if info.storage != MediaStorage::Packed {
return Err(format!("Media {} is referenced, not packed", id));
}
BlobReader::open(db_path, id, info.total_len, self.chunk_size)
}
/// Override the chunk size (testing / tuning). Affects subsequent writes.
#[doc(hidden)]
pub fn set_chunk_size(&mut self, chunk_size: u64) {
assert!(chunk_size > 0);
self.chunk_size = chunk_size;
}
}
/// Streaming reader (`Read + Seek`) over a packed media item's chunk rows.
///
/// Owns a dedicated read-only SQLite connection so it is independent of the
/// writing [`BeamArchive`] handle and can be moved to another thread. Each
/// `read` opens a blob handle on the current chunk's row via `blob_open` (no
/// per-read query — chunk rowids are resolved once up front) and reads up to the
/// chunk boundary; callers that issue many tiny reads should wrap this in a
/// `BufReader`.
pub struct BlobReader {
conn: Connection,
chunk_rowids: Vec<i64>,
chunk_size: u64,
total_len: u64,
pos: u64,
}
impl BlobReader {
fn open(db_path: &Path, id: Uuid, total_len: u64, chunk_size: u64) -> Result<Self, String> {
let conn = Connection::open_with_flags(
db_path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)
.map_err(map_sql)?;
let id_bytes = id.as_bytes().to_vec();
let mut stmt = conn
.prepare("SELECT id FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, i64>(0))
.map_err(map_sql)?;
let mut chunk_rowids = Vec::new();
for row in rows {
chunk_rowids.push(row.map_err(map_sql)?);
}
drop(stmt);
Ok(Self { conn, chunk_rowids, chunk_size, total_len, pos: 0 })
}
/// Total length of the media payload in bytes.
pub fn len(&self) -> u64 {
self.total_len
}
pub fn is_empty(&self) -> bool {
self.total_len == 0
}
fn chunk_blob(&self, chunk_index: usize) -> io::Result<Blob<'_>> {
let rowid = *self
.chunk_rowids
.get(chunk_index)
.ok_or_else(|| io::Error::new(io::ErrorKind::UnexpectedEof, "chunk index out of range"))?;
self.conn
.blob_open(DatabaseName::Main, "media_chunk", "bytes", rowid, true)
.map_err(|e| io::Error::new(io::ErrorKind::Other, e))
}
}
impl Read for BlobReader {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
if self.pos >= self.total_len || buf.is_empty() {
return Ok(0);
}
let chunk_index = (self.pos / self.chunk_size) as usize;
let off_in_chunk = self.pos % self.chunk_size;
// The chunk's length is derivable from total_len/chunk_size, so we don't
// depend on Blob::len(): every chunk but the last is exactly chunk_size.
let chunk_start = chunk_index as u64 * self.chunk_size;
let chunk_len = (self.total_len - chunk_start).min(self.chunk_size);
let avail_in_chunk = (chunk_len - off_in_chunk) as usize;
let avail_total = (self.total_len - self.pos) as usize;
let want = buf.len().min(avail_in_chunk).min(avail_total);
// Scope the blob borrow (it borrows `self.conn`) so it ends before we
// mutate `self.pos`.
let n = {
let mut blob = self.chunk_blob(chunk_index)?;
blob.seek(SeekFrom::Start(off_in_chunk))?;
blob.read(&mut buf[..want])?
};
self.pos += n as u64;
Ok(n)
}
}
impl Seek for BlobReader {
fn seek(&mut self, pos: SeekFrom) -> io::Result<u64> {
let new_pos = match pos {
SeekFrom::Start(n) => n as i64,
SeekFrom::End(n) => self.total_len as i64 + n,
SeekFrom::Current(n) => self.pos as i64 + n,
};
if new_pos < 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"seek before start of media",
));
}
// Allow seeking to/past end (reads then return 0), matching File semantics.
self.pos = new_pos as u64;
Ok(self.pos)
}
}
/// A write transaction over a [`BeamArchive`]. All writes are buffered until
/// [`BeamTxn::commit`]; dropping without committing rolls back. Lets a save
/// touch only the rows that changed, in place, without rewriting unchanged media.
pub struct BeamTxn<'a> {
tx: rusqlite::Transaction<'a>,
chunk_size: u64,
}
impl BeamTxn<'_> {
pub fn put_media_packed(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
write_media_packed(&self.tx, self.chunk_size, id, kind, codec, bytes, meta)
}
pub fn put_media_referenced(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
write_media_referenced(&self.tx, id, kind, codec, ext_path, meta)
}
/// Like [`BeamTxn::put_media_packed`] but streams the bytes from a file on
/// disk chunk-by-chunk, so an arbitrarily large file is never fully loaded
/// into memory. `total_len` is taken from the bytes actually read.
pub fn put_media_packed_from_path(
&self,
id: Uuid,
kind: MediaKind,
codec: &str,
path: &Path,
meta: MediaMeta,
) -> Result<(), String> {
write_media_packed_from_path(&self.tx, self.chunk_size, id, kind, codec, path, meta)
}
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
set_project_json_conn(&self.tx, json)
}
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
set_meta_conn(&self.tx, key, value)
}
/// Does a media row with this id already exist?
pub fn media_exists(&self, id: Uuid) -> Result<bool, String> {
let id_bytes = id.as_bytes().to_vec();
let n: i64 = self
.tx
.query_row("SELECT COUNT(*) FROM media WHERE id = ?1", [&id_bytes], |r| r.get(0))
.map_err(map_sql)?;
Ok(n > 0)
}
/// Every media id currently in the archive.
pub fn all_media_ids(&self) -> Result<Vec<Uuid>, String> {
let mut stmt = self.tx.prepare("SELECT id FROM media").map_err(map_sql)?;
let rows = stmt.query_map([], |r| r.get::<_, Vec<u8>>(0)).map_err(map_sql)?;
let mut out = Vec::new();
for row in rows {
out.push(uuid_from_bytes(&row.map_err(map_sql)?)?);
}
Ok(out)
}
/// Delete a media row (and its chunks).
pub fn delete_media(&self, id: Uuid) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
self.tx
.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
self.tx
.execute("DELETE FROM media WHERE id = ?1", [&id_bytes])
.map_err(map_sql)?;
Ok(())
}
/// Delete every media row whose id is not in `keep` (orphan cleanup).
pub fn retain_media(&self, keep: &std::collections::HashSet<Uuid>) -> Result<usize, String> {
let mut removed = 0;
for id in self.all_media_ids()? {
if !keep.contains(&id) {
self.delete_media(id)?;
removed += 1;
}
}
Ok(removed)
}
pub fn commit(self) -> Result<(), String> {
self.tx.commit().map_err(map_sql)
}
}
// -- shared write helpers (used by both BeamArchive and BeamTxn) --------------
fn write_media_packed(
conn: &Connection,
chunk_size: u64,
id: Uuid,
kind: MediaKind,
codec: &str,
bytes: &[u8],
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Packed as i64,
bytes.len() as i64,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
for (chunk_index, chunk) in bytes.chunks(chunk_size as usize).enumerate() {
conn.execute(
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
rusqlite::params![id_bytes, chunk_index as i64, chunk],
)
.map_err(map_sql)?;
}
Ok(())
}
fn write_media_packed_from_path(
conn: &Connection,
chunk_size: u64,
id: Uuid,
kind: MediaKind,
codec: &str,
path: &Path,
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
let file = std::fs::File::open(path).map_err(|e| format!("Failed to open {:?}: {}", path, e))?;
let mut reader = std::io::BufReader::new(file);
let mut buf = vec![0u8; chunk_size as usize];
let mut chunk_index: i64 = 0;
let mut total_len: u64 = 0;
loop {
// Fill `buf` up to chunk_size, tolerating short reads.
let mut filled = 0usize;
while filled < buf.len() {
let n = reader
.read(&mut buf[filled..])
.map_err(|e| format!("Failed to read {:?}: {}", path, e))?;
if n == 0 {
break;
}
filled += n;
}
if filled == 0 {
break;
}
conn.execute(
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
rusqlite::params![id_bytes, chunk_index, &buf[..filled]],
)
.map_err(map_sql)?;
chunk_index += 1;
total_len += filled as u64;
if filled < buf.len() {
break; // reached EOF
}
}
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Packed as i64,
total_len as i64,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
Ok(())
}
fn write_media_referenced(
conn: &Connection,
id: Uuid,
kind: MediaKind,
codec: &str,
ext_path: &str,
meta: MediaMeta,
) -> Result<(), String> {
let id_bytes = id.as_bytes().to_vec();
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
.map_err(map_sql)?;
conn.execute(
"INSERT INTO media
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
VALUES (?1, ?2, ?3, ?4, ?5, 0, ?6, ?7, ?8, ?9)",
rusqlite::params![
id_bytes,
kind as i64,
codec,
MediaStorage::Referenced as i64,
ext_path,
meta.channels,
meta.sample_rate,
meta.width,
meta.height,
],
)
.map_err(map_sql)?;
Ok(())
}
fn set_project_json_conn(conn: &Connection, json: &str) -> Result<(), String> {
conn.execute(
"INSERT INTO project_json (id, data) VALUES (0, ?1)
ON CONFLICT(id) DO UPDATE SET data = excluded.data",
[json],
)
.map_err(map_sql)?;
Ok(())
}
fn set_meta_conn(conn: &Connection, key: &str, value: &str) -> Result<(), String> {
conn.execute(
"INSERT INTO meta (key, value) VALUES (?1, ?2)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
rusqlite::params![key, value],
)
.map_err(map_sql)?;
Ok(())
}
/// Read a packed media item's full bytes via a fresh **read-only** connection,
/// without an open [`BeamArchive`] handle. Returns `Ok(None)` if the item has no
/// chunk rows. Used for on-demand raster fault-in: a read-only connection can't
/// conflict with an in-place save and needs no long-lived handle.
pub fn read_packed_media_readonly(db_path: &Path, id: Uuid) -> Result<Option<Vec<u8>>, String> {
let conn = Connection::open_with_flags(
db_path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)
.map_err(map_sql)?;
let id_bytes = id.as_bytes().to_vec();
let mut stmt = conn
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
.map_err(map_sql)?;
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::new();
let mut any = false;
for row in rows {
out.extend_from_slice(&row.map_err(map_sql)?);
any = true;
}
Ok(if any { Some(out) } else { None })
}
fn map_sql(e: rusqlite::Error) -> String {
format!("SQLite error: {}", e)
}
fn uuid_from_bytes(bytes: &[u8]) -> Result<Uuid, String> {
let arr: [u8; 16] = bytes
.try_into()
.map_err(|_| format!("Invalid uuid blob length {}", bytes.len()))?;
Ok(Uuid::from_bytes(arr))
}
// Tests live in `tests/beam_archive.rs` (integration tests), so they compile the
// library in non-test mode and don't depend on the crate's other `#[cfg(test)]`
// modules.

View File

@ -575,6 +575,27 @@ pub fn encode_png(img: &RgbaImage) -> Result<Vec<u8>, String> {
Ok(buf.into_inner()) Ok(buf.into_inner())
} }
/// Long-edge cap (pixels) for raster keyframe proxies — low-res page-in placeholders.
pub const RASTER_PROXY_MAX_EDGE: u32 = 192;
/// Downscale a full RGBA buffer to a low-res proxy PNG (long edge ≤
/// `RASTER_PROXY_MAX_EDGE`). Returns `None` on invalid dims/length or encode failure.
pub fn encode_raster_proxy_png(raw: &[u8], width: u32, height: u32) -> Option<Vec<u8>> {
if width == 0 || height == 0 || raw.len() != (width as usize * height as usize * 4) {
return None;
}
let long = width.max(height);
let (pw, ph) = if long <= RASTER_PROXY_MAX_EDGE {
(width, height)
} else {
let s = RASTER_PROXY_MAX_EDGE as f32 / long as f32;
(((width as f32 * s).round() as u32).max(1), ((height as f32 * s).round() as u32).max(1))
};
let img = RgbaImage::from_raw(width, height, raw.to_vec())?;
let resized = image::imageops::resize(&img, pw, ph, image::imageops::FilterType::Triangle);
encode_png(&resized).ok()
}
/// Decode PNG bytes into an `RgbaImage` /// Decode PNG bytes into an `RgbaImage`
pub fn decode_png(data: &[u8]) -> Result<RgbaImage, String> { pub fn decode_png(data: &[u8]) -> Result<RgbaImage, String> {
image::load_from_memory(data) image::load_from_memory(data)

View File

@ -902,7 +902,7 @@ mod tests {
#[test] #[test]
fn test_audio_clip_midi() { fn test_audio_clip_midi() {
let clip = AudioClip::new_midi("Piano Melody", 1, 60.0); let clip = AudioClip::new_midi("Piano Melody", 1, daw_backend::Beats(60.0));
assert_eq!(clip.name, "Piano Melody"); assert_eq!(clip.name, "Piano Melody");
assert_eq!(clip.duration, 60.0); assert_eq!(clip.duration, 60.0);
match &clip.clip_type { match &clip.clip_type {
@ -952,7 +952,10 @@ mod tests {
assert_eq!(instance.trim_start, 2.0); assert_eq!(instance.trim_start, 2.0);
assert_eq!(instance.trim_end, Some(8.0)); assert_eq!(instance.trim_end, Some(8.0));
assert_eq!(instance.effective_duration(10.0), 6.0); // At 60 BPM the tempo map is identity (1 beat == 1 second), so the
// beats-domain effective duration equals the seconds content window.
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(10.0, &tempo_map), 6.0);
} }
#[test] #[test]
@ -963,7 +966,9 @@ mod tests {
assert_eq!(instance.trim_start, 2.0); assert_eq!(instance.trim_start, 2.0);
assert_eq!(instance.trim_end, None); assert_eq!(instance.trim_end, None);
assert_eq!(instance.effective_duration(10.0), 8.0); // At 60 BPM the tempo map is identity (1 beat == 1 second).
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
assert_eq!(instance.effective_duration(10.0, &tempo_map), 8.0);
} }
#[test] #[test]

View File

@ -650,6 +650,29 @@ impl Document {
layers layers
} }
/// Get mutable references to all layers across the entire document
/// (root + nested groups + inside all vector clips). Mirrors [`all_layers`].
pub fn all_layers_mut(&mut self) -> Vec<&mut AnyLayer> {
let mut layers: Vec<&mut AnyLayer> = Vec::new();
// Iterative walk with an explicit stack of child slices. Group layers are
// descended into but not themselves collected (they hold no keyframes).
let mut stack: Vec<&mut [AnyLayer]> = vec![&mut self.root.children];
while let Some(list) = stack.pop() {
for layer in list {
match layer {
AnyLayer::Group(g) => {
stack.push(&mut g.children);
}
other => layers.push(other),
}
}
}
for clip in self.vector_clips.values_mut() {
layers.extend(clip.layers.root_data_mut());
}
layers
}
// === CLIP LIBRARY METHODS === // === CLIP LIBRARY METHODS ===
/// Add a vector clip to the library /// Add a vector clip to the library

View File

@ -240,14 +240,18 @@ mod tests {
let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end
layer.add_clip_instance(effect2); layer.add_clip_instance(effect2);
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
// query time in seconds equals the effect's beats-domain extents.
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
// At time 2: only effect1 active // At time 2: only effect1 active
assert_eq!(layer.active_clip_instances_at(2.0, 60.0).len(), 1); assert_eq!(layer.active_clip_instances_at(2.0, &tempo_map).len(), 1);
// At time 4: both effects active // At time 4: both effects active
assert_eq!(layer.active_clip_instances_at(4.0, 60.0).len(), 2); assert_eq!(layer.active_clip_instances_at(4.0, &tempo_map).len(), 2);
// At time 7: only effect2 active // At time 7: only effect2 active
assert_eq!(layer.active_clip_instances_at(7.0, 60.0).len(), 1); assert_eq!(layer.active_clip_instances_at(7.0, &tempo_map).len(), 1);
} }
#[test] #[test]

View File

@ -1,20 +1,25 @@
//! File I/O for .beam project files //! File I/O for .beam project files
//! //!
//! This module handles saving and loading Lightningbeam projects in the .beam format, //! The `.beam` format is a single **SQLite database** (see [`crate::beam_archive`]):
//! which is a ZIP archive containing: //! - `project_json` table — serialized project metadata and structure
//! - project.json (compressed) - Project metadata and structure //! - `media` / `media_chunk` tables — audio and raster media (packed as chunked
//! - media/ directory (uncompressed) - Embedded media files (FLAC for audio) //! blobs, or referenced by external path)
//!
//! Older `.beam` files are ZIP archives; [`load_beam`] detects and reads those
//! too (via [`load_beam_zip_legacy`]). Saving always writes the SQLite form, so
//! opening a legacy file and saving migrates it.
use crate::beam_archive::{BeamArchive, MediaKind, MediaMeta, LARGE_MEDIA_THRESHOLD};
use crate::document::Document; use crate::document::Document;
use daw_backend::audio::pool::AudioPoolEntry; use daw_backend::audio::pool::AudioPoolEntry;
use daw_backend::audio::project::Project as AudioProject; use daw_backend::audio::project::Project as AudioProject;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use std::fs::File; use std::fs::File;
use std::io::{Read, Write}; use std::io::Read;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use zip::write::FileOptions; use uuid::Uuid;
use zip::{CompressionMethod, ZipArchive, ZipWriter}; use zip::ZipArchive;
use flacenc::error::Verify;
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
/// File format version /// File format version
@ -51,9 +56,7 @@ pub struct SerializedAudioBackend {
/// Audio project (tracks, MIDI clips, etc.) /// Audio project (tracks, MIDI clips, etc.)
pub project: AudioProject, pub project: AudioProject,
/// Audio pool entries (metadata and paths for audio files) /// Audio pool entries (metadata and media references for audio files)
/// Note: embedded_data field from daw-backend is ignored; embedded files
/// are stored as FLAC in the ZIP's media/audio/ directory instead
pub audio_pool_entries: Vec<AudioPoolEntry>, pub audio_pool_entries: Vec<AudioPoolEntry>,
/// Mapping from UI layer UUIDs to backend TrackIds /// Mapping from UI layer UUIDs to backend TrackIds
@ -63,6 +66,25 @@ pub struct SerializedAudioBackend {
} }
/// How to store a media file at or above [`LARGE_MEDIA_THRESHOLD`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum LargeMediaMode {
/// Not yet decided — prompt the user the first time a large file is imported.
/// Treated as [`LargeMediaMode::Reference`] at save time. Resetting the
/// preference to `Ask` re-triggers the prompt (useful for testing).
Ask,
/// Pack the bytes into the `.beam` container (chunked, streamed from disk).
Pack,
/// Keep the file external and store only a path reference.
Reference,
}
impl Default for LargeMediaMode {
fn default() -> Self {
LargeMediaMode::Ask
}
}
/// Settings for saving a project /// Settings for saving a project
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct SaveSettings { pub struct SaveSettings {
@ -74,6 +96,10 @@ pub struct SaveSettings {
/// Force linking all media files (don't embed any) /// Force linking all media files (don't embed any)
pub force_link_all: bool, pub force_link_all: bool,
/// How to store files at/above [`LARGE_MEDIA_THRESHOLD`] (pack vs reference).
/// `Ask` behaves as `Reference` here (safe default: don't bloat the DB).
pub large_media_mode: LargeMediaMode,
} }
impl Default for SaveSettings { impl Default for SaveSettings {
@ -82,6 +108,7 @@ impl Default for SaveSettings {
auto_embed_threshold_bytes: 10_000_000, // 10 MB auto_embed_threshold_bytes: 10_000_000, // 10 MB
force_embed_all: false, force_embed_all: false,
force_link_all: false, force_link_all: false,
large_media_mode: LargeMediaMode::Ask,
} }
} }
} }
@ -100,6 +127,11 @@ pub struct LoadedProject {
/// Loaded audio pool entries /// Loaded audio pool entries
pub audio_pool_entries: Vec<AudioPoolEntry>, pub audio_pool_entries: Vec<AudioPoolEntry>,
/// Persisted video-thumbnail packs by clip id (opaque LBTN blobs; decoded and
/// inserted into the VideoManager by the editor). Clips present here don't need
/// their thumbnails regenerated on load.
pub thumbnail_blobs: std::collections::HashMap<uuid::Uuid, Vec<u8>>,
/// List of files that couldn't be found /// List of files that couldn't be found
pub missing_files: Vec<MissingFileInfo>, pub missing_files: Vec<MissingFileInfo>,
} }
@ -125,282 +157,319 @@ pub enum MediaFileType {
Image, Image,
} }
/// Save a project to a .beam file /// Save a project to a `.beam` file (SQLite container).
/// ///
/// This function: /// Re-saving an existing SQLite `.beam` updates it **in place** inside a single
/// 1. Prepares audio project for save (saves AudioGraph presets) /// transaction: unchanged (large) media is never rewritten, only changed rows
/// 2. Serializes project data to JSON /// are touched, and the commit is atomic/crash-safe. A brand-new file or a
/// 3. Creates ZIP archive with compressed project.json /// legacy-ZIP migration is written to a temp file and atomically renamed (there
/// 4. Embeds media files as FLAC (for audio) in media/ directory /// is no large existing container to copy in that case).
/// ///
/// # Arguments /// Audio and raster media become rows in the `media` table — packed as chunked
/// * `path` - Path to save the .beam file /// blobs, or referenced by external path for files at/above
/// * `document` - UI document state /// [`LARGE_MEDIA_THRESHOLD`]. `project.json` goes in the `project_json` table.
/// * `audio_project` - Audio backend project /// Whether a stored media codec is an audio format the disk reader (Symphonia)
/// * `audio_pool_entries` - Serialized audio pool entries /// can stream directly from a packed blob. Video-container audio tracks and any
/// * `settings` - Save settings (embedding preferences) /// unknown formats fall back to the legacy reconstitution-and-decode path.
/// fn is_streamable_audio_codec(codec: &str) -> bool {
/// # Returns matches!(
/// Ok(()) on success, or error message codec.to_lowercase().as_str(),
"mp3" | "flac" | "ogg" | "oga" | "wav" | "wave" | "aiff" | "aif"
| "aac" | "m4a" | "opus" | "alac" | "caf"
)
}
/// A `Sync` wrapper over core's `BlobReader` so it satisfies Symphonia's
/// `MediaSource: Send + Sync`. `BlobReader` holds a rusqlite `Connection`
/// (`Send` but `!Sync`); the disk reader uses it single-threaded, so the
/// hot Read/Seek path goes through `Mutex::get_mut` (no runtime locking).
struct SyncBlobReader {
inner: std::sync::Mutex<crate::beam_archive::BlobReader>,
len: u64,
}
impl std::io::Read for SyncBlobReader {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.inner.get_mut().unwrap().read(buf)
}
}
impl std::io::Seek for SyncBlobReader {
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
self.inner.get_mut().unwrap().seek(pos)
}
}
impl daw_backend::audio::MediaByteSource for SyncBlobReader {
fn byte_len(&self) -> u64 {
self.len
}
}
/// The host's packed-media byte-source factory: opens an independent streaming
/// reader over a `.beam` container's packed audio by media id. Installed into the
/// engine on load so container-packed audio streams without a full decode.
#[derive(Debug)]
struct BeamBlobFactory {
db_path: PathBuf,
}
impl daw_backend::audio::AudioBlobSourceFactory for BeamBlobFactory {
fn open(
&self,
media_id: &str,
) -> Result<Box<dyn daw_backend::audio::MediaByteSource>, String> {
let id = Uuid::parse_str(media_id).map_err(|e| format!("bad media id {}: {}", media_id, e))?;
let archive = BeamArchive::open(&self.db_path)?;
let reader = archive.open_blob_reader(&self.db_path, id)?;
let len = reader.len();
Ok(Box::new(SyncBlobReader { inner: std::sync::Mutex::new(reader), len }))
}
}
/// Build a packed-media byte-source factory for a `.beam` file, to install into
/// the engine (`EngineController::set_blob_source_factory`) before loading so its
/// packed audio can be streamed.
pub fn blob_source_factory(
beam_path: &Path,
) -> std::sync::Arc<dyn daw_backend::audio::AudioBlobSourceFactory> {
std::sync::Arc::new(BeamBlobFactory { db_path: beam_path.to_path_buf() })
}
/// Deterministic id for the waveform-pyramid media row of audio pool entry
/// `pool_index`, within a single project container. Stable across saves (so an
/// in-place re-save reuses the row instead of orphaning/rewriting it) and
/// independent of how the audio bytes are stored. The top 32 bits are a fixed
/// "LBWF" sentinel so it can't collide with the random v4 ids used elsewhere.
fn waveform_media_id(pool_index: usize) -> Uuid {
const SENTINEL: u128 = 0x4C42_5746u128 << 96; // "LBWF" in the high 32 bits
Uuid::from_u128(SENTINEL | (pool_index as u128))
}
/// Deterministic id for the thumbnail-pack media row of a video clip. Derived from
/// the clip id by XOR with a fixed constant — bijective and stable across saves, so
/// it reuses the row in place, and it can't (in practice) collide with the random
/// v4 ids used for other media of a different kind.
fn thumbnail_media_id(clip_id: Uuid) -> Uuid {
// "LBTN" repeated — moves clip ids into a distinct region of the id space.
const SENTINEL: u128 = 0x4C42_544E_4C42_544E_4C42_544E_4C42_544E;
Uuid::from_u128(clip_id.as_u128() ^ SENTINEL)
}
/// Derived id for a raster keyframe's low-res proxy row (distinct from the keyframe's
/// own full-res `Raster` row, which is keyed by the raw keyframe id).
fn raster_proxy_media_id(kf_id: Uuid) -> Uuid {
// "LBPX" repeated — distinct id region from the full raster + thumbnail rows.
const SENTINEL: u128 = 0x4C42_5058_4C42_5058_4C42_5058_4C42_5058;
Uuid::from_u128(kf_id.as_u128() ^ SENTINEL)
}
pub fn save_beam( pub fn save_beam(
path: &Path, path: &Path,
document: &Document, document: &Document,
audio_project: &mut AudioProject, audio_project: &mut AudioProject,
audio_pool_entries: Vec<AudioPoolEntry>, audio_pool_entries: Vec<AudioPoolEntry>,
layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>, layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>,
thumbnail_blobs: &std::collections::HashMap<uuid::Uuid, Vec<u8>>,
_settings: &SaveSettings, _settings: &SaveSettings,
) -> Result<(), String> { ) -> Result<(), String> {
let fn_start = std::time::Instant::now(); let fn_start = std::time::Instant::now();
eprintln!("📊 [SAVE_BEAM] Starting save_beam()..."); eprintln!("📊 [SAVE_BEAM] Starting save_beam() (SQLite container)...");
// 1. Create backup if file exists and open it for reading old audio files let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
let step1_start = std::time::Instant::now(); let in_place = path.exists() && BeamArchive::is_sqlite(path);
let mut old_zip = if path.exists() {
let backup_path = path.with_extension("beam.backup");
std::fs::copy(path, &backup_path)
.map_err(|e| format!("Failed to create backup: {}", e))?;
// Open the backup as a ZIP archive for reading // In-place for an existing SQLite container (don't rewrite unchanged media);
match File::open(&backup_path) { // temp + atomic rename for new files / legacy-ZIP migration.
Ok(file) => match ZipArchive::new(file) { let tmp_path = path.with_extension("beam.tmp");
Ok(archive) => { let mut archive = if in_place {
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup and open for reading took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0); BeamArchive::open(path)?
Some(archive)
}
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup as ZIP: {}, will not copy old audio files", e);
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
None
}
},
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup: {}, will not copy old audio files", e);
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
None
}
}
} else { } else {
eprintln!("📊 [SAVE_BEAM] Step 1: No backup needed (new file)"); BeamArchive::create(&tmp_path)?
None
}; };
// 2. Graph presets are already populated by the engine thread (in GetProject handler) let now = chrono::Utc::now().to_rfc3339();
// before cloning. Do NOT call prepare_for_save() here — the cloned project has let created = if in_place {
// default empty graphs (AudioTrack::clone() doesn't copy the graph), so calling archive.get_meta("created").ok().flatten().unwrap_or_else(|| now.clone())
// prepare_for_save() would overwrite the good presets with empty ones. } else {
let step2_start = std::time::Instant::now(); now.clone()
eprintln!("📊 [SAVE_BEAM] Step 2: (graph presets already prepared) took {:.2}ms", step2_start.elapsed().as_secs_f64() * 1000.0); };
// 3. Create ZIP writer let txn = archive.transaction()?;
let step3_start = std::time::Instant::now();
let file = File::create(path)
.map_err(|e| format!("Failed to create file: {}", e))?;
let mut zip = ZipWriter::new(file);
eprintln!("📊 [SAVE_BEAM] Step 3: Create ZIP writer took {:.2}ms", step3_start.elapsed().as_secs_f64() * 1000.0);
// 4. Process audio pool entries and write embedded audio files to ZIP // --- audio pool entries -> media rows (packed) or external references ---
// Priority: old ZIP file > external file > encode PCM as FLAC let mut modified_entries = Vec::with_capacity(audio_pool_entries.len());
let step4_start = std::time::Instant::now(); let mut live_media: HashSet<Uuid> = HashSet::new();
let mut modified_entries = Vec::new();
let mut flac_encode_time = 0.0;
let mut zip_write_time = 0.0;
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
for entry in &audio_pool_entries { for entry in &audio_pool_entries {
let mut modified_entry = entry.clone(); let mut e = entry.clone();
let existing_id = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok());
// Try to get audio data from various sources (in priority order) // Already packed in this archive (in-place re-save): leave the bytes
let audio_source: Option<(Vec<u8>, String)> = if let Some(ref rel_path) = entry.relative_path { // untouched, just keep the reference.
// Priority 1: Check if file is in the old ZIP if let Some(id) = existing_id {
if rel_path.starts_with("media/audio/") { if txn.media_exists(id)? {
if let Some(ref mut old_zip_archive) = old_zip { live_media.insert(id);
match old_zip_archive.by_name(rel_path) { e.media_id = Some(id.to_string());
Ok(mut file) => { e.relative_path = None;
let mut bytes = Vec::new(); e.embedded_data = None;
if file.read_to_end(&mut bytes).is_ok() { modified_entries.push(e);
let extension = rel_path.split('.').last().unwrap_or("bin").to_string(); continue;
eprintln!("📊 [SAVE_BEAM] Copying from old ZIP: {}", rel_path);
Some((bytes, extension))
} else {
eprintln!("⚠️ [SAVE_BEAM] Failed to read {} from old ZIP", rel_path);
None
} }
} }
Err(_) => {
eprintln!("⚠️ [SAVE_BEAM] File {} not found in old ZIP", rel_path); // Otherwise resolve the source: external file (Priority 2, streamed from
None // disk so a huge file is never fully loaded), or embedded data (Priority 3).
} let meta = MediaMeta {
} channels: Some(entry.channels),
} else { sample_rate: Some(entry.sample_rate),
None ..Default::default()
}
}
// Priority 2: Check external filesystem
else {
let full_path = project_dir.join(rel_path);
if full_path.exists() {
match std::fs::read(&full_path) {
Ok(bytes) => {
let extension = full_path.extension()
.and_then(|e| e.to_str())
.unwrap_or("bin")
.to_string();
eprintln!("📊 [SAVE_BEAM] Using external file: {:?}", full_path);
Some((bytes, extension))
}
Err(e) => {
eprintln!("⚠️ [SAVE_BEAM] Failed to read {:?}: {}", full_path, e);
None
}
}
} else {
eprintln!("⚠️ [SAVE_BEAM] External file not found: {:?}", full_path);
None
}
}
} else {
None
}; };
let mut wrote_packed: Option<Uuid> = None;
let mut referenced: Option<String> = None;
if let Some((audio_bytes, extension)) = audio_source { if let Some(rel) = entry.relative_path.as_ref() {
// We have the original file - copy it directly let full = if Path::new(rel).is_absolute() {
let zip_filename = format!("media/audio/{}.{}", entry.pool_index, extension); PathBuf::from(rel)
} else {
let file_options = FileOptions::default() project_dir.join(rel)
.compression_method(CompressionMethod::Stored); };
// Require an actual file: an empty/blank `relative_path` resolves to the
zip.start_file(&zip_filename, file_options) // project directory itself (`join("")` == dir), which `exists()` accepts
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?; // but can't be read as media. `is_file()` skips dirs + missing paths, so
// such an entry correctly falls through to embedded data below.
let write_start = std::time::Instant::now(); if full.is_file() {
zip.write_all(&audio_bytes) let size = std::fs::metadata(&full).map(|m| m.len()).unwrap_or(0);
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?; let codec = full
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0; .extension()
.and_then(|x| x.to_str())
// Update entry to point to ZIP file .unwrap_or("bin")
modified_entry.embedded_data = None; .to_lowercase();
modified_entry.relative_path = Some(zip_filename); // Video-audio entries are always referenced (the video is already
// referenced by its VideoClip; reloaded by re-probing via FFmpeg).
} else if let Some(ref embedded_data) = entry.embedded_data { // Otherwise large files honor the user's pack-vs-reference choice
// Priority 3: No original file - encode PCM as FLAC // (`Ask` == reference); smaller files are always packed.
eprintln!("📊 [SAVE_BEAM] Encoding PCM to FLAC for pool {} (no original file)", entry.pool_index); let reference_it = entry.is_video_audio
// Embedded data is always PCM - encode as FLAC || (size >= LARGE_MEDIA_THRESHOLD
let audio_bytes = BASE64_STANDARD.decode(&embedded_data.data_base64) && _settings.large_media_mode != LargeMediaMode::Pack);
.map_err(|e| format!("Failed to decode base64 audio data for pool index {}: {}", entry.pool_index, e))?; if reference_it {
referenced = Some(rel.clone());
let zip_filename = format!("media/audio/{}.flac", entry.pool_index); } else {
let id = existing_id.unwrap_or_else(Uuid::new_v4);
let file_options = FileOptions::default() txn.put_media_packed_from_path(id, MediaKind::Audio, &codec, &full, meta)?;
.compression_method(CompressionMethod::Stored); wrote_packed = Some(id);
}
zip.start_file(&zip_filename, file_options) }
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?;
// Encode PCM samples to FLAC
let flac_start = std::time::Instant::now();
// The audio_bytes are raw PCM samples (interleaved f32 little-endian)
let samples: Vec<f32> = audio_bytes
.chunks_exact(4)
.map(|chunk| f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]))
.collect();
// Convert f32 samples to i32 for FLAC encoding
let samples_i32: Vec<i32> = samples
.iter()
.map(|&s| {
let clamped = s.clamp(-1.0, 1.0);
(clamped * 8388607.0) as i32
})
.collect();
// Configure FLAC encoder
let config = flacenc::config::Encoder::default()
.into_verified()
.map_err(|(_, e)| format!("FLAC encoder config error: {:?}", e))?;
let source = flacenc::source::MemSource::from_samples(
&samples_i32,
entry.channels as usize,
24,
entry.sample_rate as usize,
);
// Encode to FLAC
let flac_stream = flacenc::encode_with_fixed_block_size(
&config,
source,
config.block_size,
).map_err(|e| format!("FLAC encoding failed: {:?}", e))?;
// Convert stream to bytes
use flacenc::component::BitRepr;
let mut sink = flacenc::bitsink::ByteSink::new();
flac_stream.write(&mut sink)
.map_err(|e| format!("Failed to write FLAC stream: {:?}", e))?;
let flac_bytes = sink.as_slice();
flac_encode_time += flac_start.elapsed().as_secs_f64() * 1000.0;
let write_start = std::time::Instant::now();
zip.write_all(flac_bytes)
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?;
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0;
// Update entry to point to ZIP file instead of embedding data
modified_entry.embedded_data = None;
modified_entry.relative_path = Some(zip_filename);
} }
modified_entries.push(modified_entry); if wrote_packed.is_none() && referenced.is_none() {
if let Some(ed) = entry.embedded_data.as_ref() {
if let Ok(bytes) = BASE64_STANDARD.decode(&ed.data_base64) {
let id = existing_id.unwrap_or_else(Uuid::new_v4);
txn.put_media_packed(id, MediaKind::Audio, &ed.format.to_lowercase(), &bytes, meta)?;
wrote_packed = Some(id);
} }
eprintln!("📊 [SAVE_BEAM] Step 4: Process audio pool ({} entries) took {:.2}ms",
audio_pool_entries.len(), step4_start.elapsed().as_secs_f64() * 1000.0);
if flac_encode_time > 0.0 {
eprintln!("📊 [SAVE_BEAM] - FLAC encoding: {:.2}ms", flac_encode_time);
} }
if zip_write_time > 0.0 {
eprintln!("📊 [SAVE_BEAM] - ZIP writing: {:.2}ms", zip_write_time);
} }
// 4b. Write raster layer PNG buffers to ZIP (media/raster/<keyframe-uuid>.png) if let Some(id) = wrote_packed {
let step4b_start = std::time::Instant::now(); live_media.insert(id);
let raster_file_options = FileOptions::default() e.media_id = Some(id.to_string());
.compression_method(CompressionMethod::Stored); // PNG is already compressed e.relative_path = None;
e.embedded_data = None;
} else if let Some(rel) = referenced {
e.media_id = None;
e.relative_path = Some(rel);
e.embedded_data = None;
} // else: nothing available — keep original references (reported missing on load)
// Persist this entry's waveform pyramid (keyed by pool index, independent
// of the audio storage above). Reuse the row in place on re-save.
let wf_id = waveform_media_id(entry.pool_index);
if let Some(blob) = entry.waveform_blob.as_ref() {
txn.put_media_packed(wf_id, MediaKind::Waveform, "lbwf", blob, MediaMeta::default())?;
live_media.insert(wf_id);
} else if txn.media_exists(wf_id)? {
// Unchanged this save — keep the stored waveform row.
live_media.insert(wf_id);
}
modified_entries.push(e);
}
// --- raster keyframes -> media rows (PNG), keyed by keyframe id ---
// (Phase 0 writes all resident frames each save; a disk-dirty flag to skip
// unchanged frames in place is deferred to Phase 3.)
// Walk ALL layers (incl. nested in groups/clips) so nested raster keyframes
// are persisted too, and so `live_media` covers them — matching the load path,
// which arms `needs_fault_in` recursively. Top-level-only projects are unaffected.
let mut raster_count = 0usize; let mut raster_count = 0usize;
for layer in &document.root.children { for layer in document.all_layers() {
if let crate::layer::AnyLayer::Raster(rl) = layer { if let crate::layer::AnyLayer::Raster(rl) = layer {
for kf in &rl.keyframes { for kf in &rl.keyframes {
if !kf.raw_pixels.is_empty() { if !kf.raw_pixels.is_empty() {
// Encode raw RGBA to PNG for storage let img =
let img = crate::brush_engine::image_from_raw( crate::brush_engine::image_from_raw(kf.raw_pixels.clone(), kf.width, kf.height);
kf.raw_pixels.clone(), kf.width, kf.height,
);
match crate::brush_engine::encode_png(&img) { match crate::brush_engine::encode_png(&img) {
Ok(png_bytes) => { Ok(png_bytes) => {
let zip_path = kf.media_path.clone(); txn.put_media_packed(
zip.start_file(&zip_path, raster_file_options) kf.id,
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_path, e))?; MediaKind::Raster,
zip.write_all(&png_bytes) "png",
.map_err(|e| format!("Failed to write {}: {}", zip_path, e))?; &png_bytes,
MediaMeta {
width: Some(kf.width),
height: Some(kf.height),
..Default::default()
},
)?;
live_media.insert(kf.id);
raster_count += 1; raster_count += 1;
} }
Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster PNG {}: {}", kf.media_path, e), Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster {}: {}", kf.id, e),
}
// Low-res proxy alongside the full PNG (shown while the full pages
// in on a later load). Regenerated from the resident pixels.
let proxy_id = raster_proxy_media_id(kf.id);
if let Some(proxy_png) =
crate::brush_engine::encode_raster_proxy_png(&kf.raw_pixels, kf.width, kf.height)
{
txn.put_media_packed(
proxy_id, MediaKind::RasterProxy, "png", &proxy_png, MediaMeta::default(),
)?;
live_media.insert(proxy_id);
}
} else if txn.media_exists(kf.id)? {
// Pixels not resident but already stored — keep both rows.
live_media.insert(kf.id);
let proxy_id = raster_proxy_media_id(kf.id);
if txn.media_exists(proxy_id)? {
live_media.insert(proxy_id);
} }
} }
} }
} }
} }
eprintln!("📊 [SAVE_BEAM] Step 4b: Write {} raster PNG buffers took {:.2}ms",
raster_count, step4b_start.elapsed().as_secs_f64() * 1000.0);
// 5. Build BeamProject structure with modified entries // --- video thumbnail packs -> media rows (opaque LBTN blob), keyed by a
let step5_start = std::time::Instant::now(); // sentinel-derived id from the video clip id ---
let now = chrono::Utc::now().to_rfc3339(); for clip_id in document.video_clips.keys() {
let tn_id = thumbnail_media_id(*clip_id);
if let Some(blob) = thumbnail_blobs.get(clip_id) {
txn.put_media_packed(tn_id, MediaKind::Thumbnail, "lbtn", blob, MediaMeta::default())?;
live_media.insert(tn_id);
} else if txn.media_exists(tn_id)? {
// Not regenerated this session — keep the stored pack.
live_media.insert(tn_id);
}
}
// --- orphan cleanup: drop media for removed clips/keyframes ---
let removed = txn.retain_media(&live_media)?;
// --- project.json + meta ---
let beam_project = BeamProject { let beam_project = BeamProject {
version: BEAM_VERSION.to_string(), version: BEAM_VERSION.to_string(),
created: now.clone(), created: created.clone(),
modified: now, modified: now.clone(),
ui_state: document.clone(), ui_state: document.clone(),
audio_backend: SerializedAudioBackend { audio_backend: SerializedAudioBackend {
sample_rate: 48000, // TODO: Get from audio engine sample_rate: 48000, // TODO: Get from audio engine
@ -409,52 +478,217 @@ pub fn save_beam(
layer_to_track_map: layer_to_track_map.clone(), layer_to_track_map: layer_to_track_map.clone(),
}, },
}; };
eprintln!("📊 [SAVE_BEAM] Step 5: Build BeamProject structure took {:.2}ms", step5_start.elapsed().as_secs_f64() * 1000.0); let json = serde_json::to_string(&beam_project)
// 6. Write project.json (compressed with DEFLATE)
let step6_start = std::time::Instant::now();
let json_options = FileOptions::default()
.compression_method(CompressionMethod::Deflated)
.compression_level(Some(6));
zip.start_file("project.json", json_options)
.map_err(|e| format!("Failed to create project.json in ZIP: {}", e))?;
let json = serde_json::to_string_pretty(&beam_project)
.map_err(|e| format!("JSON serialization failed: {}", e))?; .map_err(|e| format!("JSON serialization failed: {}", e))?;
txn.set_project_json(&json)?;
txn.set_meta("version", BEAM_VERSION)?;
txn.set_meta("created", &created)?;
txn.set_meta("modified", &now)?;
txn.commit()?;
zip.write_all(json.as_bytes()) // Close the connection before renaming (required on Windows; harmless elsewhere).
.map_err(|e| format!("Failed to write project.json: {}", e))?; drop(archive);
eprintln!("📊 [SAVE_BEAM] Step 6: Write project.json ({} bytes) took {:.2}ms", json.len(), step6_start.elapsed().as_secs_f64() * 1000.0); if !in_place {
std::fs::rename(&tmp_path, path)
// 7. Finalize ZIP .map_err(|e| format!("Failed to finalize {:?}: {}", path, e))?;
let step7_start = std::time::Instant::now(); }
zip.finish()
.map_err(|e| format!("Failed to finalize ZIP: {}", e))?;
eprintln!("📊 [SAVE_BEAM] Step 7: Finalize ZIP took {:.2}ms", step7_start.elapsed().as_secs_f64() * 1000.0);
eprintln!("📊 [SAVE_BEAM] ✅ Total save_beam() time: {:.2}ms", fn_start.elapsed().as_secs_f64() * 1000.0);
eprintln!(
"📊 [SAVE_BEAM] ✅ Saved {} audio + {} raster media, {} orphans removed, in {:.2}ms",
audio_pool_entries.len(),
raster_count,
removed,
fn_start.elapsed().as_secs_f64() * 1000.0
);
Ok(()) Ok(())
} }
/// Load a project from a .beam file /// Load a project from a `.beam` file.
/// ///
/// This function: /// Detects the container format: SQLite (current) or legacy ZIP, and dispatches
/// 1. Opens ZIP archive and reads project.json /// accordingly. Both produce an identical [`LoadedProject`].
/// 2. Deserializes project data
/// 3. Loads embedded media files from archive
/// 4. Attempts to load external media files
/// 5. Rebuilds AudioGraphs from presets with correct sample_rate
///
/// # Arguments
/// * `path` - Path to the .beam file
///
/// # Returns
/// LoadedProject on success (with missing_files list), or error message
pub fn load_beam(path: &Path) -> Result<LoadedProject, String> { pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
if BeamArchive::is_sqlite(path) {
load_beam_sqlite(path)
} else {
load_beam_zip_legacy(path)
}
}
/// Load a project from a SQLite `.beam` container.
///
/// Phase 0 reconstitutes packed audio into each entry's `embedded_data` so the
/// existing (full-decode) audio pool loader keeps working unchanged; Phase 1b
/// replaces this with streaming reads via `BlobReader`.
fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
let fn_start = std::time::Instant::now(); let fn_start = std::time::Instant::now();
eprintln!("📊 [LOAD_BEAM] Starting load_beam()..."); eprintln!("📊 [LOAD_BEAM] Starting load_beam() (SQLite container)...");
let archive = BeamArchive::open(path)?;
let json = archive.get_project_json()?;
let beam_project: BeamProject = serde_json::from_str(&json)
.map_err(|e| format!("Failed to deserialize project.json: {}", e))?;
if beam_project.version != BEAM_VERSION {
return Err(format!(
"Unsupported file version: {} (expected {})",
beam_project.version, BEAM_VERSION
));
}
let mut document = beam_project.ui_state;
document.tempo_map_mut().rebuild_seconds();
let mut audio_project = beam_project.audio_backend.project;
audio_project
.rebuild_audio_graphs(DEFAULT_BUFFER_SIZE)
.map_err(|e| format!("Failed to rebuild audio graphs: {}", e))?;
let layer_to_track_map = beam_project.audio_backend.layer_to_track_map;
// For each packed audio item: stream it (leave `embedded_data` empty so the
// pool builds a Compressed placeholder backed by the blob factory) when it's a
// recognized audio codec; otherwise fall back to the legacy reconstitution
// (whole bytes → base64 → decode), which still covers video-container audio
// tracks symphonia can't stream and any unknown formats.
let mut restored_entries = Vec::with_capacity(beam_project.audio_backend.audio_pool_entries.len());
for entry in &beam_project.audio_backend.audio_pool_entries {
let mut e = entry.clone();
if let Some(id) = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok()) {
match archive.media_info(id) {
Ok(Some(info)) => {
if is_streamable_audio_codec(&info.codec) {
// Stream: keep media_id, no embedded bytes. The engine opens
// the packed blob via the factory at activation time.
e.embedded_data = None;
e.relative_path = None;
} else {
match archive.read_media_full(id) {
Ok(bytes) => {
e.embedded_data = Some(daw_backend::audio::pool::EmbeddedAudioData {
data_base64: BASE64_STANDARD.encode(&bytes),
format: info.codec,
});
e.relative_path = None;
}
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read audio media {}: {}", id, err),
}
}
}
Ok(None) => eprintln!("⚠️ [LOAD_BEAM] Audio media {} missing from archive", id),
Err(err) => eprintln!("⚠️ [LOAD_BEAM] media_info({}) failed: {}", id, err),
}
}
// Restore this entry's persisted waveform pyramid, if present — avoids
// re-decoding the source media just to redraw the overview.
let wf_id = waveform_media_id(entry.pool_index);
if let Ok(Some(_)) = archive.media_info(wf_id) {
match archive.read_media_full(wf_id) {
Ok(bytes) => e.waveform_blob = Some(bytes),
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read waveform {}: {}", wf_id, err),
}
}
restored_entries.push(e);
}
// Raster keyframes are NOT eagerly decoded (Phase 3 paging): `raw_pixels` stays
// empty and is faulted in on demand from the container's `Raster` rows via the
// editor's `RasterStore` (keyed by `kf.id`). Loading a big paint project is now
// instant and only the resident window lives in RAM. Mark every keyframe
// `needs_fault_in` (recursively, incl. nested layers) so the renderer requests a
// page-in; a freshly-created keyframe stays `false` (blank-resident, nothing to load).
// Proxies (low-res, ~tens of KB each) ARE decoded eagerly so a cold scrub onto a
// not-yet-paged frame shows the proxy instantly instead of flashing blank.
let mut raster_load_count = 0usize;
let mut proxy_load_count = 0usize;
for layer in document.all_layers_mut() {
if let crate::layer::AnyLayer::Raster(rl) = layer {
for kf in &mut rl.keyframes {
kf.needs_fault_in = true;
raster_load_count += 1;
let proxy_id = raster_proxy_media_id(kf.id);
if let Ok(Some(_)) = archive.media_info(proxy_id) {
if let Ok(bytes) = archive.read_media_full(proxy_id) {
if let Ok(img) = crate::brush_engine::decode_png(&bytes) {
let (w, h) = (img.width(), img.height());
kf.proxy = Some(crate::raster_layer::RasterProxy {
width: w,
height: h,
pixels: img.into_raw(),
});
proxy_load_count += 1;
}
}
}
}
}
}
let _ = proxy_load_count;
// Missing external files (referenced entries whose file no longer exists).
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
let missing_files: Vec<MissingFileInfo> = restored_entries
.iter()
.enumerate()
.filter_map(|(idx, entry)| {
if entry.embedded_data.is_none() && entry.media_id.is_none() {
if let Some(rel) = entry.relative_path.as_ref() {
let full = if Path::new(rel).is_absolute() {
PathBuf::from(rel)
} else {
project_dir.join(rel)
};
if !full.exists() {
return Some(MissingFileInfo {
pool_index: idx,
original_path: full,
file_type: MediaFileType::Audio,
});
}
}
}
None
})
.collect();
// Persisted video thumbnail packs (opaque LBTN blobs), keyed by clip id. The
// editor decodes + inserts them and skips regeneration for these clips.
let mut thumbnail_blobs = std::collections::HashMap::new();
for clip_id in document.video_clips.keys() {
let tn_id = thumbnail_media_id(*clip_id);
if let Ok(Some(info)) = archive.media_info(tn_id) {
if info.kind == MediaKind::Thumbnail {
match archive.read_media_full(tn_id) {
Ok(bytes) => { thumbnail_blobs.insert(*clip_id, bytes); }
Err(e) => eprintln!("⚠️ [LOAD_BEAM] Failed to read thumbnails for {}: {}", clip_id, e),
}
}
}
}
eprintln!(
"📊 [LOAD_BEAM] ✅ Loaded {} audio entries, {} raster frames, {} thumbnail packs in {:.2}ms",
restored_entries.len(),
raster_load_count,
thumbnail_blobs.len(),
fn_start.elapsed().as_secs_f64() * 1000.0
);
Ok(LoadedProject {
document,
audio_project,
layer_to_track_map,
audio_pool_entries: restored_entries,
thumbnail_blobs,
missing_files,
})
}
/// Load a project from a legacy ZIP `.beam` archive (pre-SQLite format).
/// Retained for backward compatibility; saving converts to SQLite.
fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
let fn_start = std::time::Instant::now();
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (legacy ZIP)...");
// 1. Open ZIP archive // 1. Open ZIP archive
let step1_start = std::time::Instant::now(); let step1_start = std::time::Instant::now();
@ -673,6 +907,7 @@ pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
audio_project, audio_project,
layer_to_track_map, layer_to_track_map,
audio_pool_entries: restored_entries, audio_pool_entries: restored_entries,
thumbnail_blobs: std::collections::HashMap::new(), // legacy ZIP has no thumbnail packs
missing_files, missing_files,
}) })
} }

View File

@ -42,6 +42,7 @@ pub mod segment_builder;
pub mod planar_graph; pub mod planar_graph;
pub mod file_types; pub mod file_types;
pub mod file_io; pub mod file_io;
pub mod beam_archive;
pub mod export; pub mod export;
pub mod clipboard; pub mod clipboard;
pub(crate) mod clipboard_platform; pub(crate) mod clipboard_platform;
@ -53,6 +54,7 @@ pub mod svg_export;
pub mod snap; pub mod snap;
pub mod webcam; pub mod webcam;
pub mod raster_layer; pub mod raster_layer;
pub mod raster_store;
pub mod brush_settings; pub mod brush_settings;
pub mod brush_engine; pub mod brush_engine;
pub mod raster_draw; pub mod raster_draw;

View File

@ -92,6 +92,16 @@ impl Default for TweenType {
} }
} }
/// A low-res decoded RGBA proxy of a keyframe's pixels, shown while the full-res
/// buffer pages in from the container so cold scrubs don't flash blank.
#[derive(Clone, Debug)]
pub struct RasterProxy {
pub width: u32,
pub height: u32,
/// RGBA, `width * height * 4` bytes.
pub pixels: Vec<u8>,
}
/// A single keyframe of a raster layer /// A single keyframe of a raster layer
#[derive(Clone, Debug, Serialize, Deserialize)] #[derive(Clone, Debug, Serialize, Deserialize)]
pub struct RasterKeyframe { pub struct RasterKeyframe {
@ -117,6 +127,24 @@ pub struct RasterKeyframe {
/// Always `true` after load; cleared by the renderer after uploading. /// Always `true` after load; cleared by the renderer after uploading.
#[serde(skip, default = "default_true")] #[serde(skip, default = "default_true")]
pub texture_dirty: bool, pub texture_dirty: bool,
/// Phase 3 paging: the keyframe's pixels live in the container and must be
/// faulted in (`raw_pixels` empty *and* this true ⇒ page in from the store).
/// A *new* keyframe is `false` (intentionally blank/resident, nothing to load);
/// set true on load and again when evicted. Never serialized.
#[serde(skip)]
pub needs_fault_in: bool,
/// Phase 3a eviction: set `true` whenever user editing mutates `raw_pixels`
/// (brush, fill, paint-bucket, floating-selection commit/lift, undo/redo of
/// those). A dirty keyframe's current pixels are NOT yet persisted in the
/// container, so it must NEVER be evicted (doing so would silently lose the
/// unsaved edit). Cleared on a successful save. Never serialized.
#[serde(skip)]
pub dirty: bool,
/// Phase 3a-3: low-res proxy decoded from the container on load, rendered while
/// the full pixels page in (removes the cold-scrub blank flash). `None` if the
/// keyframe has no persisted proxy yet (new/unsaved, or pre-proxy project).
#[serde(skip)]
pub proxy: Option<RasterProxy>,
} }
fn default_true() -> bool { true } fn default_true() -> bool { true }
@ -140,6 +168,9 @@ impl RasterKeyframe {
tween_after: TweenType::Hold, tween_after: TweenType::Hold,
raw_pixels: Vec::new(), raw_pixels: Vec::new(),
texture_dirty: true, texture_dirty: true,
needs_fault_in: false,
dirty: false,
proxy: None,
} }
} }
} }
@ -204,6 +235,35 @@ impl RasterLayer {
&mut self.keyframes[insert_idx] &mut self.keyframes[insert_idx]
} }
/// Insert a blank keyframe at `time` if none exists there (within tolerance).
/// Returns the new keyframe's id if one was created, `None` if a keyframe already
/// existed. Used by the explicit "New Keyframe" command (blank cel).
pub fn insert_blank_keyframe_at(&mut self, time: f64, width: u32, height: u32) -> Option<Uuid> {
if self.keyframe_index_at_exact(time, 0.001).is_some() {
return None;
}
let (w, h) = if width == 0 || height == 0 {
self.keyframe_at(time)
.map(|kf| (kf.width, kf.height))
.unwrap_or((1920, 1080))
} else {
(width, height)
};
let insert_idx = self.keyframes.partition_point(|kf| kf.time < time);
let kf = RasterKeyframe::new(time, w, h);
let id = kf.id;
self.keyframes.insert(insert_idx, kf);
Some(id)
}
/// Remove the keyframe with the given id, returning it if found.
pub fn remove_keyframe(&mut self, id: Uuid) -> Option<RasterKeyframe> {
self.keyframes
.iter()
.position(|kf| kf.id == id)
.map(|pos| self.keyframes.remove(pos))
}
/// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`. /// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`.
pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> { pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> {
self.keyframe_at(time).map(|kf| kf.media_path.as_str()) self.keyframe_at(time).map(|kf| kf.media_path.as_str())

View File

@ -0,0 +1,56 @@
//! On-demand loader for raster keyframe pixels backed by the project `.beam`
//! container (Phase 3 paging).
//!
//! Raster keyframes are no longer eagerly decoded at load; `raw_pixels` stays
//! empty until something needs the frame, then it is faulted in from the
//! container's `Raster` media row (keyed by the keyframe id). The store holds only
//! the container path and reads through a fresh **read-only** connection per call,
//! so it never conflicts with an in-place save and keeps no long-lived handle.
//! `None` path = an unsaved document (nothing to fault in).
use std::path::PathBuf;
use uuid::Uuid;
/// Faults in raster keyframe pixels from the project container on demand.
#[derive(Default, Clone)]
pub struct RasterStore {
path: Option<PathBuf>,
}
impl RasterStore {
pub fn new(path: Option<PathBuf>) -> Self {
Self { path }
}
/// Point the store at a (possibly new) container path, or `None` for an
/// unsaved document. Call on load and on save-as.
pub fn set_path(&mut self, path: Option<PathBuf>) {
self.path = path;
}
pub fn has_path(&self) -> bool {
self.path.is_some()
}
/// Decode the keyframe's full RGBA pixels from the container, or `None` if the
/// container has no row for it (or decoding fails). The returned buffer is the
/// working `raw_pixels` representation (`width*height*4` sRGB-premultiplied RGBA).
pub fn load_pixels(&self, kf_id: Uuid) -> Option<Vec<u8>> {
let path = self.path.as_ref()?;
let png = match crate::beam_archive::read_packed_media_readonly(path, kf_id) {
Ok(Some(bytes)) => bytes,
Ok(None) => return None,
Err(e) => {
eprintln!("[RasterStore] read {} failed: {}", kf_id, e);
return None;
}
};
match crate::brush_engine::decode_png(&png) {
Ok(img) => Some(img.into_raw()),
Err(e) => {
eprintln!("[RasterStore] decode {} failed: {}", kf_id, e);
None
}
}
}
}

View File

@ -423,11 +423,25 @@ pub fn render_layer_isolated(
* Affine::scale_non_uniform(scale_x, scale_y) * Affine::scale_non_uniform(scale_x, scale_y)
* skew_transform; * skew_transform;
// The decoded frame is scaled down to fit the document (decoder caps
// at the canvas size), so its pixel size is smaller than the clip's
// native dimensions. The instance is blitted treating the texture as
// `frame.width × frame.height`, while `clip_transform` is expressed in
// the clip's native space — so scale frame-px → clip-native-px first,
// else the frame renders small in a corner with its edges streaked.
let frame_to_clip = if frame.width > 0 && frame.height > 0 {
Affine::scale_non_uniform(
video_clip.width / frame.width as f64,
video_clip.height / frame.height as f64,
)
} else {
Affine::IDENTITY
};
instances.push(VideoRenderInstance { instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(), rgba_data: frame.rgba_data.clone(),
width: frame.width, width: frame.width,
height: frame.height, height: frame.height,
transform: base_transform * clip_transform, transform: base_transform * clip_transform * frame_to_clip,
opacity: (layer_opacity * inst_opacity) as f32, opacity: (layer_opacity * inst_opacity) as f32,
}); });
} }
@ -1028,12 +1042,26 @@ fn render_video_layer(
// Create rectangle path for the video frame // Create rectangle path for the video frame
let video_rect = Rect::new(0.0, 0.0, video_clip.width, video_clip.height); let video_rect = Rect::new(0.0, 0.0, video_clip.width, video_clip.height);
// The decoded frame is scaled down to fit the document (the decoder caps at
// the canvas size to bound memory), so its pixel dimensions are smaller than
// the clip's native display size. Scale the image brush from frame-pixel
// space to the clip rect; without this the image is drawn 1:1 in a corner
// and its edge pixels pad the rest (small frame with "stretched corners").
let brush_transform = if frame.width > 0 && frame.height > 0 {
Affine::scale_non_uniform(
video_clip.width / frame.width as f64,
video_clip.height / frame.height as f64,
)
} else {
Affine::IDENTITY
};
// Render video frame as image fill // Render video frame as image fill
scene.fill( scene.fill(
Fill::NonZero, Fill::NonZero,
instance_transform, instance_transform,
&image_with_alpha, &image_with_alpha,
None, Some(brush_transform),
&video_rect, &video_rect,
); );
clip_rendered = true; clip_rendered = true;

View File

@ -395,7 +395,7 @@ mod tests {
let shape = Shape::new(path); let shape = Shape::new(path);
assert_eq!(shape.versions.len(), 1); assert_eq!(shape.versions.len(), 1);
assert!(shape.fill_color.is_some()); assert!(shape.fill_color.is_none());
} }
#[test] #[test]

View File

@ -97,7 +97,7 @@ impl VideoDecoder {
// Optionally build keyframe index for fast seeking // Optionally build keyframe index for fast seeking
let keyframe_positions = if build_keyframes { let keyframe_positions = if build_keyframes {
eprintln!("[Video Decoder] Building keyframe index for {}", path); eprintln!("[Video Decoder] Building keyframe index for {}", path);
let positions = Self::build_keyframe_index(&path, stream_index)?; let positions = Self::scan_keyframes(&path, stream_index)?;
eprintln!("[Video Decoder] Found {} keyframes", positions.len()); eprintln!("[Video Decoder] Found {} keyframes", positions.len());
positions positions
} else { } else {
@ -125,14 +125,19 @@ impl VideoDecoder {
}) })
} }
/// Build keyframe index for this decoder /// Source file path this decoder reads from.
/// This can be called asynchronously after decoder creation pub fn path(&self) -> &str {
fn build_and_set_keyframe_index(&mut self) -> Result<(), String> { &self.path
eprintln!("[Video Decoder] Building keyframe index for {}", self.path); }
let positions = Self::build_keyframe_index(&self.path, self.stream_index)?;
eprintln!("[Video Decoder] Found {} keyframes", positions.len()); /// Parameters needed to scan keyframes off-thread (path + video stream index).
pub fn keyframe_scan_params(&self) -> (String, usize) {
(self.path.clone(), self.stream_index)
}
/// Replace the keyframe index (built off-thread via [`VideoDecoder::scan_keyframes`]).
pub fn set_keyframe_index(&mut self, positions: Vec<i64>) {
self.keyframe_positions = positions; self.keyframe_positions = positions;
Ok(())
} }
/// Get the output width (scaled dimensions) /// Get the output width (scaled dimensions)
@ -150,9 +155,10 @@ impl VideoDecoder {
self.get_frame(timestamp) self.get_frame(timestamp)
} }
/// Build an index of all keyframe positions in the video /// Build an index of all keyframe positions in the video by scanning packets
/// This enables fast seeking by knowing exactly where keyframes are /// from a fresh input. Does not touch `self` — call it off-thread (it is slow
fn build_keyframe_index(path: &str, stream_index: usize) -> Result<Vec<i64>, String> { /// on long videos) and hand the result to [`VideoDecoder::set_keyframe_index`].
pub fn scan_keyframes(path: &str, stream_index: usize) -> Result<Vec<i64>, String> {
let mut input = ffmpeg::format::input(path) let mut input = ffmpeg::format::input(path)
.map_err(|e| format!("Failed to open video for indexing: {}", e))?; .map_err(|e| format!("Failed to open video for indexing: {}", e))?;
@ -340,6 +346,58 @@ impl VideoDecoder {
} }
} }
/// Generate timeline thumbnails for a video using a **dedicated** decoder that
/// is independent of any shared playback decoder — so thumbnail work never holds
/// a lock the UI/playback needs.
///
/// Thumbnails are sampled at keyframes ~`interval_secs` apart. Decoding at a
/// keyframe is cheap (≈one frame) versus decoding forward to an arbitrary
/// timestamp (the whole GOP). Frames are decoded directly at `thumb_width` (so
/// `get_thumbnail_at`'s 128-wide assumption holds) and tightly packed RGBA is
/// handed to `on_thumb` as `(timestamp_secs, data)`.
pub fn generate_keyframe_thumbnails(
path: &str,
interval_secs: f64,
thumb_width: u32,
mut should_skip: impl FnMut(f64) -> bool,
mut on_thumb: impl FnMut(f64, Arc<Vec<u8>>),
) -> Result<(), String> {
// Own decoder at thumbnail resolution; builds its own keyframe index. The
// large max-height lets width be the constraining dimension, so output width
// is exactly `thumb_width`.
let mut decoder = VideoDecoder::new(
path.to_string(),
4,
Some(thumb_width),
Some(100_000),
true, // build keyframe index (needed to sample at keyframes)
)?;
let keyframe_secs: Vec<f64> = decoder
.keyframe_positions
.iter()
.map(|&ts| ts as f64 * decoder.time_base)
.collect();
let mut last_emitted = f64::NEG_INFINITY;
for ks in keyframe_secs {
if ks - last_emitted < interval_secs {
continue;
}
// This keyframe is a target slot; advance regardless of skip so the chosen
// slots are deterministic (lets a resumed pass target the same timestamps).
last_emitted = ks;
// Skip slots already covered (resume after a partial save / dedup).
if should_skip(ks) {
continue;
}
if let Ok(rgba) = decoder.get_frame(ks) {
on_thumb(ks, Arc::new(rgba));
}
}
Ok(())
}
/// Probe video file for metadata without creating a full decoder /// Probe video file for metadata without creating a full decoder
pub fn probe_video(path: &str) -> Result<VideoMetadata, String> { pub fn probe_video(path: &str) -> Result<VideoMetadata, String> {
ffmpeg::init().map_err(|e| e.to_string())?; ffmpeg::init().map_err(|e| e.to_string())?;
@ -407,18 +465,33 @@ pub struct VideoManager {
/// Pool of video decoders, one per clip /// Pool of video decoders, one per clip
decoders: HashMap<Uuid, Arc<Mutex<VideoDecoder>>>, decoders: HashMap<Uuid, Arc<Mutex<VideoDecoder>>>,
/// Frame cache: (clip_id, timestamp_ms) -> frame /// Frame cache: (clip_id, timestamp_ms) -> frame. Stores decoded RGBA for
/// Stores raw RGBA data for zero-copy rendering /// zero-copy rendering. Bounded by a **byte budget** (not a frame count, which
frame_cache: HashMap<(Uuid, i64), Arc<VideoFrame>>, /// would be unsafe across resolutions — a 4K frame is ~33MB vs ~2MB at 800x600)
/// so playback of arbitrarily long video never grows unbounded.
frame_cache: LruCache<(Uuid, i64), Arc<VideoFrame>>,
/// Running total of bytes held in `frame_cache` (sum of each frame's RGBA len),
/// kept in sync on insert/evict/remove so eviction is O(1) per frame.
frame_cache_bytes: usize,
/// Thumbnail cache: clip_id -> Vec of (timestamp, rgba_data) /// Thumbnail cache: clip_id -> Vec of (timestamp, rgba_data)
/// Low-resolution (64px width) thumbnails for scrubbing /// Low-resolution (64px width) thumbnails for scrubbing
thumbnail_cache: HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>>, thumbnail_cache: HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>>,
/// Clips whose thumbnail generation finished. Only complete sets are worth
/// persisting — a partial set (saved mid-generation) is dropped so the load
/// regenerates it fully rather than leaving it permanently incomplete.
thumbnails_complete: std::collections::HashSet<Uuid>,
/// Maximum number of frames to cache per decoder /// Maximum number of frames to cache per decoder
cache_size: usize, cache_size: usize,
} }
/// Byte budget for [`VideoManager::frame_cache`] (decoded full-resolution frames).
/// At ~2MB/frame (800x600) this holds ~128 frames; at ~33MB/frame (4K) ~8 — in
/// both cases enough for the current frame plus a scrub window, while bounding RAM.
const FRAME_CACHE_BYTE_BUDGET: usize = 256 * 1024 * 1024;
impl VideoManager { impl VideoManager {
/// Create a new video manager with default cache size /// Create a new video manager with default cache size
pub fn new() -> Self { pub fn new() -> Self {
@ -429,8 +502,10 @@ impl VideoManager {
pub fn with_cache_size(cache_size: usize) -> Self { pub fn with_cache_size(cache_size: usize) -> Self {
Self { Self {
decoders: HashMap::new(), decoders: HashMap::new(),
frame_cache: HashMap::new(), frame_cache: LruCache::unbounded(),
frame_cache_bytes: 0,
thumbnail_cache: HashMap::new(), thumbnail_cache: HashMap::new(),
thumbnails_complete: std::collections::HashSet::new(),
cache_size, cache_size,
} }
} }
@ -440,8 +515,9 @@ impl VideoManager {
/// `target_width` and `target_height` specify the maximum dimensions /// `target_width` and `target_height` specify the maximum dimensions
/// for decoded frames. Video will be scaled down if larger. /// for decoded frames. Video will be scaled down if larger.
/// ///
/// The keyframe index is NOT built during this call - use `build_keyframe_index_async` /// The keyframe index is NOT built during this call — scan it off-thread via
/// in a background thread to build it asynchronously. /// [`VideoDecoder::scan_keyframes`] and store it with
/// [`VideoDecoder::set_keyframe_index`] so the slow scan never blocks playback.
pub fn load_video( pub fn load_video(
&mut self, &mut self,
clip_id: Uuid, clip_id: Uuid,
@ -467,20 +543,6 @@ impl VideoManager {
Ok(metadata) Ok(metadata)
} }
/// Build keyframe index for a loaded video asynchronously
///
/// This should be called from a background thread after load_video()
/// to avoid blocking the UI during import.
pub fn build_keyframe_index(&self, clip_id: &Uuid) -> Result<(), String> {
let decoder_arc = self.decoders.get(clip_id)
.ok_or_else(|| format!("Video clip {} not found", clip_id))?;
let mut decoder = decoder_arc.lock()
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
decoder.build_and_set_keyframe_index()
}
/// Get a decoded frame for a specific clip at a specific timestamp /// Get a decoded frame for a specific clip at a specific timestamp
/// ///
/// Returns None if the clip is not loaded or decoding fails. /// Returns None if the clip is not loaded or decoding fails.
@ -495,14 +557,16 @@ impl VideoManager {
return Some(Arc::clone(cached_frame)); return Some(Arc::clone(cached_frame));
} }
// Get decoder for this clip // Get decoder for this clip. Clone the Arc so we don't hold a borrow of
let decoder_arc = self.decoders.get(clip_id)?; // `self.decoders` across the `&mut self` cache insert below.
let decoder_arc = Arc::clone(self.decoders.get(clip_id)?);
let mut decoder = decoder_arc.lock().ok()?; let mut decoder = decoder_arc.lock().ok()?;
// Decode the frame // Decode the frame
let rgba_data = decoder.get_frame(timestamp).ok()?; let rgba_data = decoder.get_frame(timestamp).ok()?;
let width = decoder.output_width; let width = decoder.output_width;
let height = decoder.output_height; let height = decoder.output_height;
drop(decoder); // release the lock before touching `self`
// Create VideoFrame and cache it // Create VideoFrame and cache it
let frame = Arc::new(VideoFrame { let frame = Arc::new(VideoFrame {
@ -512,65 +576,27 @@ impl VideoManager {
timestamp, timestamp,
}); });
self.frame_cache.insert(cache_key, Arc::clone(&frame)); self.cache_frame(cache_key, Arc::clone(&frame));
Some(frame) Some(frame)
} }
/// Generate thumbnails for a video clip (single batch version - use generate_thumbnails_progressive instead) /// Insert a frame into the byte-budgeted cache, evicting least-recently-used
/// /// frames until the total is within [`FRAME_CACHE_BYTE_BUDGET`].
/// Thumbnails are generated every 5 seconds at 128px width. fn cache_frame(&mut self, key: (Uuid, i64), frame: Arc<VideoFrame>) {
/// This should be called in a background thread to avoid blocking. let bytes = frame.rgba_data.len();
/// Thumbnails are inserted into the cache progressively as they're generated, if let Some(old) = self.frame_cache.put(key, frame) {
/// allowing the UI to display them immediately. self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(old.rgba_data.len());
/// }
/// DEPRECATED: Use generate_thumbnails_progressive which releases the lock between thumbnails. self.frame_cache_bytes += bytes;
pub fn generate_thumbnails(&mut self, clip_id: &Uuid, duration: f64) -> Result<(), String> { // Keep at least one frame resident even if it alone exceeds the budget.
let decoder_arc = self.decoders.get(clip_id) while self.frame_cache_bytes > FRAME_CACHE_BYTE_BUDGET && self.frame_cache.len() > 1 {
.ok_or("Clip not loaded")? if let Some((_, evicted)) = self.frame_cache.pop_lru() {
.clone(); self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(evicted.rgba_data.len());
} else {
let mut decoder = decoder_arc.lock() break;
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
// Initialize thumbnail cache entry with empty vec
self.thumbnail_cache.insert(*clip_id, Vec::new());
let interval = 5.0; // Generate thumbnail every 5 seconds
let mut t = 0.0;
while t < duration {
// Decode frame at this timestamp
if let Ok(rgba_data) = decoder.get_frame(t) {
// Decode already scaled to output dimensions, but we want 128px width for thumbnails
// We need to scale down further
let current_width = decoder.output_width;
let current_height = decoder.output_height;
// Calculate thumbnail dimensions (128px width, maintain aspect ratio)
let thumb_width = 128u32;
let aspect_ratio = current_height as f32 / current_width as f32;
let thumb_height = (thumb_width as f32 * aspect_ratio) as u32;
// Simple nearest-neighbor downsampling for thumbnails
let thumb_data = downsample_rgba(
&rgba_data,
current_width,
current_height,
thumb_width,
thumb_height,
);
// Insert thumbnail into cache immediately so UI can display it
if let Some(thumbnails) = self.thumbnail_cache.get_mut(clip_id) {
thumbnails.push((t, Arc::new(thumb_data)));
} }
} }
t += interval;
}
Ok(())
} }
/// Get the decoder Arc for a clip (for external thumbnail generation) /// Get the decoder Arc for a clip (for external thumbnail generation)
@ -579,18 +605,57 @@ impl VideoManager {
self.decoders.get(clip_id).cloned() self.decoders.get(clip_id).cloned()
} }
/// Insert a thumbnail into the cache (for external thumbnail generation) /// Snapshot all cached thumbnails for persistence (clip id -> sorted
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) { /// (timestamp, rgba) pairs). Cheap: clones the `Arc`s, not the pixel data.
self.thumbnail_cache /// Partial sets are persisted too — pair with [`complete_thumbnail_clips`] so
.entry(*clip_id) /// the load knows which clips still need generation resumed.
.or_insert_with(Vec::new) pub fn snapshot_all_thumbnails(&self) -> HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>> {
.push((timestamp, data)); self.thumbnail_cache.clone()
} }
/// Get the thumbnail closest to the specified timestamp /// The set of clips whose thumbnail generation has finished (a full keyframe
/// pass). A persisted set flagged incomplete is resumed on load.
pub fn complete_thumbnail_clips(&self) -> std::collections::HashSet<Uuid> {
self.thumbnails_complete.clone()
}
/// Mark a clip's thumbnail generation as complete (called when the background
/// generator finishes the full keyframe pass).
pub fn mark_thumbnails_complete(&mut self, clip_id: &Uuid) {
self.thumbnails_complete.insert(*clip_id);
}
/// Whether the clip already has a thumbnail within `tol` seconds of `ts`.
/// Lets the generator skip keyframes already covered (resume / dedup).
pub fn has_thumbnail_near(&self, clip_id: &Uuid, ts: f64, tol: f64) -> bool {
self.thumbnail_cache
.get(clip_id)
.map_or(false, |v| v.iter().any(|(t, _)| (t - ts).abs() < tol))
}
/// Insert a thumbnail into the cache, keeping it **sorted by timestamp** and
/// **deduped** (an existing entry at the same timestamp is replaced). Sorted
/// order is required by `get_thumbnail_at`'s binary search, and dedup makes
/// concurrent restore + resumed generation idempotent (no double inserts).
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) {
let vec = self.thumbnail_cache.entry(*clip_id).or_default();
match vec.binary_search_by(|(t, _)| {
t.partial_cmp(&timestamp).unwrap_or(std::cmp::Ordering::Equal)
}) {
Ok(i) => vec[i] = (timestamp, data),
Err(i) => vec.insert(i, (timestamp, data)),
}
}
/// Get the thumbnail closest to the specified timestamp.
/// ///
/// Returns `(actual_timestamp, width, height, data)` — `actual_timestamp` is
/// the time of the thumbnail actually chosen (which may differ from the
/// requested `timestamp`, and changes as closer thumbnails finish generating).
/// Callers key their GPU texture cache on it so the on-clip strip refreshes as
/// better thumbnails load instead of freezing on the first one.
/// Returns None if no thumbnails have been generated for this clip. /// Returns None if no thumbnails have been generated for this clip.
pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(u32, u32, Arc<Vec<u8>>)> { pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(f64, u32, u32, Arc<Vec<u8>>)> {
let thumbnails = self.thumbnail_cache.get(clip_id)?; let thumbnails = self.thumbnail_cache.get(clip_id)?;
if thumbnails.is_empty() { if thumbnails.is_empty() {
@ -618,30 +683,43 @@ impl VideoManager {
} }
}); });
let (_, rgba_data) = &thumbnails[idx]; let (actual_ts, rgba_data) = &thumbnails[idx];
// Return (width, height, data) // Return (actual_timestamp, width, height, data)
// Thumbnails are always 128px width // Thumbnails are always 128px width
let thumb_width = 128; let thumb_width = 128;
let thumb_height = (rgba_data.len() / (thumb_width * 4)) as u32; let thumb_height = (rgba_data.len() / (thumb_width * 4)) as u32;
Some((thumb_width as u32, thumb_height, Arc::clone(rgba_data))) Some((*actual_ts, thumb_width as u32, thumb_height, Arc::clone(rgba_data)))
} }
/// Remove a video clip and its cached data /// Remove a video clip and its cached data
pub fn unload_video(&mut self, clip_id: &Uuid) { pub fn unload_video(&mut self, clip_id: &Uuid) {
self.decoders.remove(clip_id); self.decoders.remove(clip_id);
// Remove all cached frames for this clip // Remove all cached frames for this clip (LruCache has no retain; collect
self.frame_cache.retain(|(id, _), _| id != clip_id); // matching keys, then pop each, keeping the byte total in sync).
let keys: Vec<(Uuid, i64)> = self
.frame_cache
.iter()
.filter(|((id, _), _)| id == clip_id)
.map(|(k, _)| *k)
.collect();
for key in keys {
if let Some(frame) = self.frame_cache.pop(&key) {
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(frame.rgba_data.len());
}
}
// Remove thumbnails // Remove thumbnails
self.thumbnail_cache.remove(clip_id); self.thumbnail_cache.remove(clip_id);
self.thumbnails_complete.remove(clip_id);
} }
/// Clear all frame caches (useful for memory management) /// Clear all frame caches (useful for memory management)
pub fn clear_frame_cache(&mut self) { pub fn clear_frame_cache(&mut self) {
self.frame_cache.clear(); self.frame_cache.clear();
self.frame_cache_bytes = 0;
} }
} }
@ -650,211 +728,3 @@ impl Default for VideoManager {
Self::new() Self::new()
} }
} }
/// Simple nearest-neighbor downsampling for RGBA images
pub fn downsample_rgba_public(
src: &[u8],
src_width: u32,
src_height: u32,
dst_width: u32,
dst_height: u32,
) -> Vec<u8> {
downsample_rgba(src, src_width, src_height, dst_width, dst_height)
}
/// Simple nearest-neighbor downsampling for RGBA images (internal)
fn downsample_rgba(
src: &[u8],
src_width: u32,
src_height: u32,
dst_width: u32,
dst_height: u32,
) -> Vec<u8> {
let mut dst = Vec::with_capacity((dst_width * dst_height * 4) as usize);
let x_ratio = src_width as f32 / dst_width as f32;
let y_ratio = src_height as f32 / dst_height as f32;
for y in 0..dst_height {
for x in 0..dst_width {
let src_x = (x as f32 * x_ratio) as u32;
let src_y = (y as f32 * y_ratio) as u32;
let src_idx = ((src_y * src_width + src_x) * 4) as usize;
// Copy RGBA bytes
dst.push(src[src_idx]); // R
dst.push(src[src_idx + 1]); // G
dst.push(src[src_idx + 2]); // B
dst.push(src[src_idx + 3]); // A
}
}
dst
}
/// Extracted audio data from a video file
#[derive(Debug, Clone)]
pub struct ExtractedAudio {
pub samples: Vec<f32>,
pub channels: u32,
pub sample_rate: u32,
pub duration: f64,
}
/// Extract audio from a video file
///
/// This function performs the slow FFmpeg decoding without holding any locks.
/// The caller can then quickly add the audio to the DAW backend in a background thread.
///
/// Returns None if the video has no audio stream.
pub fn extract_audio_from_video(path: &str) -> Result<Option<ExtractedAudio>, String> {
ffmpeg::init().map_err(|e| e.to_string())?;
// Open video file
let mut input = ffmpeg::format::input(path)
.map_err(|e| format!("Failed to open video: {}", e))?;
// Find audio stream
let audio_stream_opt = input.streams()
.best(ffmpeg::media::Type::Audio);
// Return None if no audio stream
if audio_stream_opt.is_none() {
return Ok(None);
}
let audio_stream = audio_stream_opt.unwrap();
let audio_index = audio_stream.index();
// Get audio properties
let context_decoder = ffmpeg::codec::context::Context::from_parameters(
audio_stream.parameters()
).map_err(|e| e.to_string())?;
let mut audio_decoder = context_decoder.decoder().audio()
.map_err(|e| e.to_string())?;
let sample_rate = audio_decoder.rate();
let channels = audio_decoder.channels() as u32;
// Decode all audio frames
let mut audio_samples: Vec<f32> = Vec::new();
for (stream, packet) in input.packets() {
if stream.index() == audio_index {
audio_decoder.send_packet(&packet)
.map_err(|e| e.to_string())?;
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
// Convert audio to f32 packed format
let format = audio_frame.format();
let frame_channels = audio_frame.channels() as usize;
// Create resampler to convert to f32 packed
let mut resampler = ffmpeg::software::resampling::context::Context::get(
format,
audio_frame.channel_layout(),
sample_rate,
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
audio_frame.channel_layout(),
sample_rate,
).map_err(|e| e.to_string())?;
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
resampler.run(&audio_frame, &mut resampled_frame)
.map_err(|e| e.to_string())?;
// Extract f32 samples (interleaved format)
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
let total_samples = resampled_frame.samples() * frame_channels;
// Safety checks before creating slice from FFmpeg data
// 1. Verify f32 alignment (required: 4 bytes)
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
}
// 2. Verify the frame actually has enough data
let byte_size = resampled_frame.data(0).len();
let expected_bytes = total_samples * std::mem::size_of::<f32>();
if byte_size < expected_bytes {
return Err(format!(
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
byte_size, expected_bytes
));
}
// SAFETY: We verified alignment and bounds above.
// The slice lifetime is tied to resampled_frame which lives until
// after extend_from_slice completes.
let samples_slice = unsafe {
std::slice::from_raw_parts(data_ptr, total_samples)
};
audio_samples.extend_from_slice(samples_slice);
}
}
}
// Flush audio decoder
audio_decoder.send_eof().map_err(|e| e.to_string())?;
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
let format = audio_frame.format();
let frame_channels = audio_frame.channels() as usize;
let mut resampler = ffmpeg::software::resampling::context::Context::get(
format,
audio_frame.channel_layout(),
sample_rate,
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
audio_frame.channel_layout(),
sample_rate,
).map_err(|e| e.to_string())?;
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
resampler.run(&audio_frame, &mut resampled_frame)
.map_err(|e| e.to_string())?;
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
let total_samples = resampled_frame.samples() * frame_channels;
// Safety checks before creating slice from FFmpeg data
// 1. Verify f32 alignment (required: 4 bytes)
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
}
// 2. Verify the frame actually has enough data
let byte_size = resampled_frame.data(0).len();
let expected_bytes = total_samples * std::mem::size_of::<f32>();
if byte_size < expected_bytes {
return Err(format!(
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
byte_size, expected_bytes
));
}
// SAFETY: We verified alignment and bounds above.
// The slice lifetime is tied to resampled_frame which lives until
// after extend_from_slice completes.
let samples_slice = unsafe {
std::slice::from_raw_parts(data_ptr, total_samples)
};
audio_samples.extend_from_slice(samples_slice);
}
// Calculate duration
let total_samples_per_channel = audio_samples.len() / channels as usize;
let duration = total_samples_per_channel as f64 / sample_rate as f64;
Ok(Some(ExtractedAudio {
samples: audio_samples,
channels,
sample_rate,
duration,
}))
}

View File

@ -0,0 +1,207 @@
//! Integration tests for the SQLite-backed `.beam` container.
//!
//! These are integration tests (not `#[cfg(test)]` unit tests) so they build the
//! library in normal mode and exercise only the public API — independent of any
//! pre-existing breakage in the crate's internal test modules.
use lightningbeam_core::beam_archive::{BeamArchive, MediaKind, MediaMeta, MediaStorage};
use std::io::{Read, Seek, SeekFrom};
use std::sync::atomic::{AtomicU64, Ordering};
use uuid::Uuid;
fn temp_db_path(tag: &str) -> std::path::PathBuf {
static N: AtomicU64 = AtomicU64::new(0);
let n = N.fetch_add(1, Ordering::Relaxed);
let mut p = std::env::temp_dir();
p.push(format!("beam_archive_it_{}_{}_{}.beam", std::process::id(), tag, n));
let _ = std::fs::remove_file(&p);
p
}
#[test]
fn project_json_roundtrip() {
let path = temp_db_path("json");
let archive = BeamArchive::create(&path).unwrap();
archive.set_project_json("{\"hello\":\"world\"}").unwrap();
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
drop(archive);
let archive = BeamArchive::open(&path).unwrap();
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
assert!(BeamArchive::is_sqlite(&path));
let _ = std::fs::remove_file(&path);
}
#[test]
fn packed_media_roundtrip_full() {
let path = temp_db_path("full");
let mut archive = BeamArchive::create(&path).unwrap();
let id = Uuid::from_u128(0x1234);
archive.set_chunk_size(1000);
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
archive
.put_media_packed(
id,
MediaKind::Audio,
"flac",
&data,
MediaMeta { channels: Some(2), sample_rate: Some(44100), ..Default::default() },
)
.unwrap();
let info = archive.media_info(id).unwrap().unwrap();
assert_eq!(info.kind, MediaKind::Audio);
assert_eq!(info.codec, "flac");
assert_eq!(info.storage, MediaStorage::Packed);
assert_eq!(info.total_len, 3500);
assert_eq!(info.channels, Some(2));
assert_eq!(info.sample_rate, Some(44100));
assert_eq!(archive.read_media_full(id).unwrap(), data);
assert_eq!(archive.media_ids_of_kind(MediaKind::Audio).unwrap(), vec![id]);
let _ = std::fs::remove_file(&path);
}
#[test]
fn blob_reader_streams_and_seeks() {
let path = temp_db_path("stream");
let mut archive = BeamArchive::create(&path).unwrap();
archive.set_chunk_size(1000);
let id = Uuid::from_u128(0xBEEF);
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
archive
.put_media_packed(id, MediaKind::Audio, "mp3", &data, MediaMeta::default())
.unwrap();
let mut reader = archive.open_blob_reader(&path, id).unwrap();
assert_eq!(reader.len(), 3500);
// Sequential read in odd-sized buffers crosses chunk boundaries.
let mut got = Vec::new();
let mut buf = [0u8; 333];
loop {
let n = reader.read(&mut buf).unwrap();
if n == 0 {
break;
}
got.extend_from_slice(&buf[..n]);
}
assert_eq!(got, data);
// Seek to a mid-chunk position and read across a boundary.
reader.seek(SeekFrom::Start(990)).unwrap();
let mut window = [0u8; 20];
let mut filled = 0;
while filled < window.len() {
let n = reader.read(&mut window[filled..]).unwrap();
assert!(n > 0);
filled += n;
}
assert_eq!(&window[..], &data[990..1010]);
// Seek from end and read the tail.
reader.seek(SeekFrom::End(-10)).unwrap();
let mut tail = Vec::new();
reader.read_to_end(&mut tail).unwrap();
assert_eq!(tail, &data[3490..]);
let _ = std::fs::remove_file(&path);
}
#[test]
fn referenced_media_records_path() {
let path = temp_db_path("ref");
let mut archive = BeamArchive::create(&path).unwrap();
let id = Uuid::from_u128(0xCAFE);
archive
.put_media_referenced(
id,
MediaKind::Video,
"mp4",
"/mnt/share/big.mp4",
MediaMeta { width: Some(3840), height: Some(2160), ..Default::default() },
)
.unwrap();
let info = archive.media_info(id).unwrap().unwrap();
assert_eq!(info.storage, MediaStorage::Referenced);
assert_eq!(info.ext_path.as_deref(), Some("/mnt/share/big.mp4"));
assert_eq!(info.width, Some(3840));
// Streaming a referenced item is an error (caller opens the path directly).
assert!(archive.open_blob_reader(&path, id).is_err());
let _ = std::fs::remove_file(&path);
}
#[test]
fn transaction_groups_writes_and_orphan_cleanup() {
let path = temp_db_path("txn");
let keep = Uuid::from_u128(1);
let orphan = Uuid::from_u128(2);
// First save: two media items committed in one transaction.
{
let mut archive = BeamArchive::create(&path).unwrap();
let txn = archive.transaction().unwrap();
txn.put_media_packed(keep, MediaKind::Audio, "flac", &vec![9u8; 10], MediaMeta::default())
.unwrap();
txn.put_media_packed(orphan, MediaKind::Audio, "mp3", &vec![8u8; 10], MediaMeta::default())
.unwrap();
txn.set_project_json("{}").unwrap();
txn.commit().unwrap();
}
{
let archive = BeamArchive::open(&path).unwrap();
assert!(archive.media_info(keep).unwrap().is_some());
assert!(archive.media_info(orphan).unwrap().is_some());
}
// Second save (in place): keep only `keep`; `orphan` should be retained-out.
{
let mut archive = BeamArchive::open(&path).unwrap();
let txn = archive.transaction().unwrap();
// `keep` already present → in-place save leaves it untouched.
assert!(txn.media_exists(keep).unwrap());
let mut live = std::collections::HashSet::new();
live.insert(keep);
let removed = txn.retain_media(&live).unwrap();
assert_eq!(removed, 1);
txn.commit().unwrap();
}
{
let archive = BeamArchive::open(&path).unwrap();
assert!(archive.media_info(keep).unwrap().is_some());
assert!(archive.media_info(orphan).unwrap().is_none());
// `keep`'s bytes survived untouched.
assert_eq!(archive.read_media_full(keep).unwrap(), vec![9u8; 10]);
}
let _ = std::fs::remove_file(&path);
}
#[test]
fn rolled_back_transaction_writes_nothing() {
let path = temp_db_path("rollback");
let id = Uuid::from_u128(42);
let mut archive = BeamArchive::create(&path).unwrap();
{
let txn = archive.transaction().unwrap();
txn.put_media_packed(id, MediaKind::Audio, "flac", &vec![1u8; 5], MediaMeta::default())
.unwrap();
// Drop without commit → rollback.
}
assert!(archive.media_info(id).unwrap().is_none());
let _ = std::fs::remove_file(&path);
}
#[test]
fn overwrite_media_replaces_chunks() {
let path = temp_db_path("overwrite");
let mut archive = BeamArchive::create(&path).unwrap();
archive.set_chunk_size(100);
let id = Uuid::from_u128(7);
archive
.put_media_packed(id, MediaKind::Raster, "png", &vec![1u8; 250], MediaMeta::default())
.unwrap();
// Overwrite with shorter data — stale chunks must be gone.
archive
.put_media_packed(id, MediaKind::Raster, "png", &vec![2u8; 50], MediaMeta::default())
.unwrap();
assert_eq!(archive.read_media_full(id).unwrap(), vec![2u8; 50]);
let _ = std::fs::remove_file(&path);
}

View File

@ -1,6 +1,7 @@
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::path::PathBuf; use std::path::PathBuf;
use crate::keymap::KeybindingConfig; use crate::keymap::KeybindingConfig;
use lightningbeam_core::file_io::LargeMediaMode;
/// Application configuration (persistent) /// Application configuration (persistent)
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
@ -57,6 +58,18 @@ pub struct AppConfig {
/// Custom keyboard shortcut overrides (sparse — only non-default bindings stored) /// Custom keyboard shortcut overrides (sparse — only non-default bindings stored)
#[serde(default)] #[serde(default)]
pub keybindings: KeybindingConfig, pub keybindings: KeybindingConfig,
/// How to store media files at/above the large-media threshold (~2GB).
/// `Ask` (default) prompts the first time such a file is imported, then the
/// chosen mode is persisted here. Reset to `Ask` to be prompted again.
#[serde(default)]
pub large_media_default: LargeMediaMode,
/// Finest-level resolution of the waveform LOD pyramid: source frames per
/// floor texel (`B`). Smaller = larger on-disk pyramid but zoom-in re-decodes
/// sooner; larger = smaller pyramid, wider re-decode span. Default 256.
#[serde(default = "defaults::waveform_floor_samples_per_texel")]
pub waveform_floor_samples_per_texel: u32,
} }
impl Default for AppConfig { impl Default for AppConfig {
@ -75,6 +88,8 @@ impl Default for AppConfig {
waveform_stereo: defaults::waveform_stereo(), waveform_stereo: defaults::waveform_stereo(),
theme_mode: defaults::theme_mode(), theme_mode: defaults::theme_mode(),
keybindings: KeybindingConfig::default(), keybindings: KeybindingConfig::default(),
large_media_default: LargeMediaMode::default(),
waveform_floor_samples_per_texel: defaults::waveform_floor_samples_per_texel(),
} }
} }
} }
@ -276,4 +291,5 @@ mod defaults {
pub fn debug() -> bool { false } pub fn debug() -> bool { false }
pub fn waveform_stereo() -> bool { false } pub fn waveform_stereo() -> bool { false }
pub fn theme_mode() -> String { "system".to_string() } pub fn theme_mode() -> String { "system".to_string() }
pub fn waveform_floor_samples_per_texel() -> u32 { 256 }
} }

View File

@ -39,6 +39,33 @@ pub fn update_prepare_timing(
t.composite_ms = composite_ms; t.composite_ms = composite_ms;
} }
} }
/// GPU memory the editor tracks itself (wgpu has no allocator query). Currently the
/// raster-layer texture cache — the only unbounded-by-default VRAM consumer.
#[derive(Debug, Clone, Default)]
pub struct GpuMemoryStats {
pub raster_cache_entries: usize,
pub raster_cache_bytes: usize,
}
static GPU_MEMORY: OnceLock<Mutex<GpuMemoryStats>> = OnceLock::new();
/// Called by the GPU brush whenever the raster-layer cache changes.
pub fn update_gpu_memory(raster_cache_entries: usize, raster_cache_bytes: usize) {
let cell = GPU_MEMORY.get_or_init(|| Mutex::new(GpuMemoryStats::default()));
if let Ok(mut s) = cell.lock() {
s.raster_cache_entries = raster_cache_entries;
s.raster_cache_bytes = raster_cache_bytes;
}
}
fn get_gpu_memory() -> GpuMemoryStats {
GPU_MEMORY
.get_or_init(|| Mutex::new(GpuMemoryStats::default()))
.lock()
.map(|s| s.clone())
.unwrap_or_default()
}
const DEVICE_REFRESH_INTERVAL: Duration = Duration::from_secs(2); // Refresh devices every 2 seconds const DEVICE_REFRESH_INTERVAL: Duration = Duration::from_secs(2); // Refresh devices every 2 seconds
const MEMORY_REFRESH_INTERVAL: Duration = Duration::from_millis(500); // Refresh memory every 500ms const MEMORY_REFRESH_INTERVAL: Duration = Duration::from_millis(500); // Refresh memory every 500ms
@ -52,6 +79,7 @@ pub struct DebugStats {
pub frame_time_ms: f32, // Current frame time in milliseconds pub frame_time_ms: f32, // Current frame time in milliseconds
pub memory_physical_mb: usize, pub memory_physical_mb: usize,
pub memory_virtual_mb: usize, pub memory_virtual_mb: usize,
pub gpu_memory: GpuMemoryStats,
pub gpu_name: String, pub gpu_name: String,
pub gpu_backend: String, pub gpu_backend: String,
pub gpu_driver: String, pub gpu_driver: String,
@ -218,6 +246,7 @@ impl DebugStatsCollector {
frame_time_ms, frame_time_ms,
memory_physical_mb, memory_physical_mb,
memory_virtual_mb, memory_virtual_mb,
gpu_memory: get_gpu_memory(),
gpu_name, gpu_name,
gpu_backend, gpu_backend,
gpu_driver, gpu_driver,
@ -286,6 +315,11 @@ pub fn render_debug_overlay(ctx: &egui::Context, stats: &DebugStats) {
ui.colored_label(egui::Color32::YELLOW, format!("Memory: ({}µs)", stats.timing_memory_us)); ui.colored_label(egui::Color32::YELLOW, format!("Memory: ({}µs)", stats.timing_memory_us));
ui.label(format!("Physical: {} MB", stats.memory_physical_mb)); ui.label(format!("Physical: {} MB", stats.memory_physical_mb));
ui.label(format!("Virtual: {} MB", stats.memory_virtual_mb)); ui.label(format!("Virtual: {} MB", stats.memory_virtual_mb));
ui.label(format!(
"VRAM (raster cache): {:.1} MB ({} frames)",
stats.gpu_memory.raster_cache_bytes as f64 / (1024.0 * 1024.0),
stats.gpu_memory.raster_cache_entries,
));
ui.add_space(8.0); ui.add_space(8.0);

View File

@ -384,90 +384,90 @@ impl ExportOrchestrator {
println!("🎵 [MUX] Audio stream - Input TB: {}/{}, Output TB: {}/{}", println!("🎵 [MUX] Audio stream - Input TB: {}/{}, Output TB: {}/{}",
audio_input_tb.0, audio_input_tb.1, audio_output_tb.0, audio_output_tb.1); audio_input_tb.0, audio_input_tb.1, audio_output_tb.0, audio_output_tb.1);
// Collect all packets with their stream info and timestamps // Stream-merge the two inputs by PTS, writing each packet as it's read —
let mut video_packets = Vec::new(); // O(1) memory (one pending packet per stream) instead of collecting every
for (stream, packet) in video_input.packets() { // packet first, so muxing a long export never grows unbounded.
if stream.index() == video_stream_index { let video_idx = video_stream_index;
video_packets.push(packet); let audio_idx = audio_stream_index;
let mut v_iter = video_input.packets();
let mut a_iter = audio_input.packets();
// Pull the next packet belonging to the desired stream from each input.
let mut next_video = move || -> Option<ffmpeg::Packet> {
loop {
match v_iter.next() {
Some((stream, packet)) => {
if stream.index() == video_idx {
return Some(packet);
} }
} }
None => return None,
let mut audio_packets = Vec::new();
for (stream, packet) in audio_input.packets() {
if stream.index() == audio_stream_index {
audio_packets.push(packet);
} }
} }
};
println!("🎬 [MUX] Collected {} video packets, {} audio packets", let mut next_audio = move || -> Option<ffmpeg::Packet> {
video_packets.len(), audio_packets.len()); loop {
match a_iter.next() {
// Report first and last timestamps Some((stream, packet)) => {
if !video_packets.is_empty() { if stream.index() == audio_idx {
println!("🎬 [MUX] Video PTS range: {} to {}", return Some(packet);
video_packets[0].pts().unwrap_or(0),
video_packets[video_packets.len()-1].pts().unwrap_or(0));
} }
if !audio_packets.is_empty() {
println!("🎵 [MUX] Audio PTS range: {} to {}",
audio_packets[0].pts().unwrap_or(0),
audio_packets[audio_packets.len()-1].pts().unwrap_or(0));
} }
None => return None,
}
}
};
// Interleave packets by comparing timestamps in a common time base (use microseconds) let mut pending_v = next_video();
let mut v_idx = 0; let mut pending_a = next_audio();
let mut a_idx = 0; let mut v_count = 0usize;
let mut interleave_log_count = 0; let mut a_count = 0usize;
let mut log_count = 0;
while v_idx < video_packets.len() || a_idx < audio_packets.len() { loop {
let write_video = if v_idx >= video_packets.len() { // Write whichever pending packet has the earlier PTS (in a common
false // No more video // microsecond base); when one stream is exhausted, drain the other.
} else if a_idx >= audio_packets.len() { let write_video = match (&pending_v, &pending_a) {
true // No more audio, write video (None, None) => break,
} else { (Some(_), None) => true,
// Compare timestamps - convert both to microseconds (None, Some(_)) => false,
let v_pts = video_packets[v_idx].pts().unwrap_or(0); (Some(v), Some(a)) => {
let a_pts = audio_packets[a_idx].pts().unwrap_or(0); let v_us = v.pts().unwrap_or(0) * 1_000_000 * video_input_tb.0 as i64
/ video_input_tb.1 as i64;
// Convert to microseconds: pts * 1000000 * tb.num / tb.den let a_us = a.pts().unwrap_or(0) * 1_000_000 * audio_input_tb.0 as i64
let v_us = v_pts * 1_000_000 * video_input_tb.0 as i64 / video_input_tb.1 as i64; / audio_input_tb.1 as i64;
let a_us = a_pts * 1_000_000 * audio_input_tb.0 as i64 / audio_input_tb.1 as i64; v_us <= a_us
}
v_us <= a_us // Write video if it comes before or at same time as audio
}; };
if write_video { if write_video {
let mut packet = video_packets[v_idx].clone(); let mut packet = pending_v.take().unwrap();
packet.set_stream(0); packet.set_stream(0);
packet.rescale_ts(video_input_tb, video_output_tb); packet.rescale_ts(video_input_tb, video_output_tb);
if log_count < 10 {
if interleave_log_count < 10 { println!("🎬 [MUX] Writing V packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
println!("🎬 [MUX] Writing V packet {} - PTS={:?}, DTS={:?}, Duration={:?}", log_count += 1;
v_idx, packet.pts(), packet.dts(), packet.duration());
interleave_log_count += 1;
} }
packet.write_interleaved(&mut output) packet.write_interleaved(&mut output)
.map_err(|e| format!("Failed to write video packet: {}", e))?; .map_err(|e| format!("Failed to write video packet: {}", e))?;
v_idx += 1; v_count += 1;
pending_v = next_video();
} else { } else {
let mut packet = audio_packets[a_idx].clone(); let mut packet = pending_a.take().unwrap();
packet.set_stream(1); packet.set_stream(1);
packet.rescale_ts(audio_input_tb, audio_output_tb); packet.rescale_ts(audio_input_tb, audio_output_tb);
if log_count < 10 {
if interleave_log_count < 10 { println!("🎵 [MUX] Writing A packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
println!("🎵 [MUX] Writing A packet {} - PTS={:?}, DTS={:?}, Duration={:?}", log_count += 1;
a_idx, packet.pts(), packet.dts(), packet.duration());
interleave_log_count += 1;
} }
packet.write_interleaved(&mut output) packet.write_interleaved(&mut output)
.map_err(|e| format!("Failed to write audio packet: {}", e))?; .map_err(|e| format!("Failed to write audio packet: {}", e))?;
a_idx += 1; a_count += 1;
pending_a = next_audio();
} }
} }
println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_idx, a_idx); println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_count, a_count);
// Write trailer // Write trailer
output.write_trailer().map_err(|e| format!("Failed to write trailer: {}", e))?; output.write_trailer().map_err(|e| format!("Failed to write trailer: {}", e))?;
@ -528,6 +528,7 @@ impl ExportOrchestrator {
image_cache: &mut ImageCache, image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>, video_manager: &Arc<std::sync::Mutex<VideoManager>>,
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>, floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<bool, String> { ) -> Result<bool, String> {
if self.cancel_flag.load(Ordering::Relaxed) { if self.cancel_flag.load(Ordering::Relaxed) {
self.image_state = None; self.image_state = None;
@ -575,6 +576,7 @@ impl ExportOrchestrator {
output_view, output_view,
floating_selection, floating_selection,
state.settings.allow_transparency, state.settings.allow_transparency,
raster_store,
)?; )?;
queue.submit(Some(encoder.finish())); queue.submit(Some(encoder.finish()));
@ -1029,6 +1031,7 @@ impl ExportOrchestrator {
renderer: &mut vello::Renderer, renderer: &mut vello::Renderer,
image_cache: &mut ImageCache, image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>, video_manager: &Arc<std::sync::Mutex<VideoManager>>,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<bool, String> { ) -> Result<bool, String> {
use std::time::Instant; use std::time::Instant;
@ -1126,6 +1129,7 @@ impl ExportOrchestrator {
gpu_resources, &acquired.rgba_texture_view, gpu_resources, &acquired.rgba_texture_view,
None, // No floating selection during video export None, // No floating selection during video export
false, // Video export is never transparent false, // Video export is never transparent
raster_store,
)?; )?;
let render_end = Instant::now(); let render_end = Instant::now();

View File

@ -1179,6 +1179,39 @@ pub fn render_frame_to_rgba_hdr(
/// ///
/// # Returns /// # Returns
/// Command encoder ready for submission (caller submits via ReadbackPipeline) /// Command encoder ready for submission (caller submits via ReadbackPipeline)
/// Fault in raster keyframe pixels needed to composite the document at its current
/// time, decoding them from the project `.beam` container via `raster_store`.
///
/// Mutates the document in place: for every raster layer's active keyframe whose
/// `raw_pixels` are empty, loads + sets them (and marks `texture_dirty`). A no-op
/// when `raster_store` is `None`/unsaved or everything is already resident.
fn fault_in_raster_for_frame(
document: &mut Document,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) {
let store = match raster_store {
Some(s) if s.has_path() => s,
_ => return,
};
let now = document.current_time;
for layer in document.all_layers_mut() {
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
// Resolve the active keyframe id at the current time, then fault it in.
let kf_id = match rl.keyframe_at(now) {
Some(kf) if kf.raw_pixels.is_empty() && kf.needs_fault_in => kf.id,
_ => continue,
};
if let Some(kf) = rl.keyframes.iter_mut().find(|kf| kf.id == kf_id) {
if let Some(pixels) = store.load_pixels(kf_id) {
kf.raw_pixels = pixels;
kf.texture_dirty = true;
}
kf.needs_fault_in = false;
}
}
}
}
pub fn render_frame_to_gpu_rgba( pub fn render_frame_to_gpu_rgba(
document: &mut Document, document: &mut Document,
timestamp: f64, timestamp: f64,
@ -1193,12 +1226,19 @@ pub fn render_frame_to_gpu_rgba(
rgba_texture_view: &wgpu::TextureView, rgba_texture_view: &wgpu::TextureView,
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>, floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
allow_transparency: bool, allow_transparency: bool,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<wgpu::CommandEncoder, String> { ) -> Result<wgpu::CommandEncoder, String> {
use vello::kurbo::Affine; use vello::kurbo::Affine;
// Set document time to the frame timestamp // Set document time to the frame timestamp
document.current_time = timestamp; document.current_time = timestamp;
// Fault in raster keyframe pixels for this frame (Phase 3 paging). Offline
// export renders synchronously with no "next frame", so unlike the live canvas
// we must page the pixels in here, before compositing. Cheap no-op when every
// keyframe is already resident or when the document is unsaved (no store path).
fault_in_raster_for_frame(document, raster_store);
// Scale the document to the export resolution. The core renderer bakes this // Scale the document to the export resolution. The core renderer bakes this
// base transform into every layer (vector scenes, raster and video layer // base transform into every layer (vector scenes, raster and video layer
// transforms), so the whole stage scales up/down to fill the output. When the // transforms), so the whole stage scales up/down to fill the output. When the

View File

@ -1049,6 +1049,17 @@ pub struct GpuBrushEngine {
/// once when `RasterKeyframe::texture_dirty` is set, then reused every frame. /// once when `RasterKeyframe::texture_dirty` is set, then reused every frame.
/// Separate from `canvases` so tool teardown never accidentally removes them. /// Separate from `canvases` so tool teardown never accidentally removes them.
pub raster_layer_cache: HashMap<Uuid, CanvasPair>, pub raster_layer_cache: HashMap<Uuid, CanvasPair>,
/// Recency order for `raster_layer_cache` (least-recent first), bumped on every
/// `ensure_layer_texture` so the visible frames stay at the back. Scrubbing a long
/// timeline would otherwise grow this map without bound (~w·h·16 bytes of VRAM per
/// entry); we evict the oldest once it exceeds `RASTER_LAYER_CACHE_MAX`.
raster_layer_lru: Vec<Uuid>,
/// Low-res proxy textures (one small `CanvasPair` per keyframe), shown while the
/// full-res pixels page in. Keyed by keyframe id; separate from
/// `raster_layer_cache` so a proxy and its full frame never collide. Bounded by
/// its own (generous, since each is tiny) recency LRU.
proxy_layer_cache: HashMap<Uuid, CanvasPair>,
proxy_layer_lru: Vec<Uuid>,
} }
/// CPU-side parameters uniform for the compute shader. /// CPU-side parameters uniform for the compute shader.
@ -1066,6 +1077,16 @@ struct DabParams {
} }
impl GpuBrushEngine { impl GpuBrushEngine {
/// Max number of idle raster-layer textures kept resident in VRAM. Scrubbing a
/// long timeline would otherwise cache one ~`w·h·16`-byte pair per visited frame
/// without bound; the least-recently-used are evicted past this (re-uploaded on
/// revisit from the faulted-in pixels).
const RASTER_LAYER_CACHE_MAX: usize = 12;
/// Proxies are ~1/100th the VRAM of a full frame, so we can keep many resident
/// for instant cold-scrub display before evicting the least-recently-used.
const RASTER_PROXY_CACHE_MAX: usize = 64;
/// Create the pipeline. Returns `Err` if the device lacks the required /// Create the pipeline. Returns `Err` if the device lacks the required
/// storage-texture capability for `Rgba16Float`. /// storage-texture capability for `Rgba16Float`.
pub fn new(device: &wgpu::Device) -> Self { pub fn new(device: &wgpu::Device) -> Self {
@ -1160,6 +1181,9 @@ impl GpuBrushEngine {
canvases: HashMap::new(), canvases: HashMap::new(),
displacement_bufs: HashMap::new(), displacement_bufs: HashMap::new(),
raster_layer_cache: HashMap::new(), raster_layer_cache: HashMap::new(),
raster_layer_lru: Vec::new(),
proxy_layer_cache: HashMap::new(),
proxy_layer_lru: Vec::new(),
} }
} }
@ -1533,6 +1557,33 @@ impl GpuBrushEngine {
} }
self.raster_layer_cache.insert(kf_id, canvas); self.raster_layer_cache.insert(kf_id, canvas);
} }
// Bump recency (the frame was used this render) and evict the least-recently
// used textures over budget. The current frame is pushed to the back, so it
// (and every other frame rendered this pass) is never the eviction victim.
if let Some(pos) = self.raster_layer_lru.iter().position(|id| *id == kf_id) {
self.raster_layer_lru.remove(pos);
}
self.raster_layer_lru.push(kf_id);
let mut evicted = false;
while self.raster_layer_lru.len() > Self::RASTER_LAYER_CACHE_MAX {
let old = self.raster_layer_lru.remove(0);
self.raster_layer_cache.remove(&old);
evicted = true;
}
if needs_new || evicted {
self.report_raster_cache_vram();
}
}
/// Estimated VRAM footprint of `raster_layer_cache` (two `Rgba16Float` textures =
/// `w·h·16` bytes per entry), published to the F3 debug overlay.
fn report_raster_cache_vram(&self) {
let bytes = |cache: &HashMap<Uuid, CanvasPair>| -> usize {
cache.values().map(|c| (c.width as usize) * (c.height as usize) * 16).sum()
};
let total = bytes(&self.raster_layer_cache) + bytes(&self.proxy_layer_cache);
let count = self.raster_layer_cache.len() + self.proxy_layer_cache.len();
crate::debug_overlay::update_gpu_memory(count, total);
} }
/// Get the cached display texture for a raster layer keyframe. /// Get the cached display texture for a raster layer keyframe.
@ -1540,9 +1591,55 @@ impl GpuBrushEngine {
self.raster_layer_cache.get(kf_id) self.raster_layer_cache.get(kf_id)
} }
/// Ensure a low-res proxy texture exists for `kf_id` (uploaded once; proxies are
/// immutable). Bumps recency and evicts the least-recently-used past the budget.
/// `pixels` is sRGB-premultiplied RGBA of length `w * h * 4`.
pub fn ensure_proxy_texture(
&mut self,
device: &wgpu::Device,
queue: &wgpu::Queue,
kf_id: Uuid,
pixels: &[u8],
w: u32,
h: u32,
) {
if pixels.len() != (w * h * 4) as usize {
return; // malformed proxy — skip rather than panic in the render loop
}
let mut changed = false;
if !self.proxy_layer_cache.contains_key(&kf_id) {
let canvas = CanvasPair::new(device, w, h);
canvas.upload(queue, pixels);
self.proxy_layer_cache.insert(kf_id, canvas);
changed = true;
}
if let Some(pos) = self.proxy_layer_lru.iter().position(|id| *id == kf_id) {
self.proxy_layer_lru.remove(pos);
}
self.proxy_layer_lru.push(kf_id);
while self.proxy_layer_lru.len() > Self::RASTER_PROXY_CACHE_MAX {
let old = self.proxy_layer_lru.remove(0);
self.proxy_layer_cache.remove(&old);
changed = true;
}
if changed {
self.report_raster_cache_vram();
}
}
/// Get the cached low-res proxy texture for a raster keyframe.
pub fn get_proxy_texture(&self, kf_id: &Uuid) -> Option<&CanvasPair> {
self.proxy_layer_cache.get(kf_id)
}
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted). /// Remove the cached texture for a raster layer keyframe (e.g. when deleted).
pub fn remove_layer_texture(&mut self, kf_id: &Uuid) { pub fn remove_layer_texture(&mut self, kf_id: &Uuid) {
self.raster_layer_cache.remove(kf_id); if self.raster_layer_cache.remove(kf_id).is_some() {
if let Some(pos) = self.raster_layer_lru.iter().position(|id| id == kf_id) {
self.raster_layer_lru.remove(pos);
}
self.report_raster_cache_vram();
}
} }
/// Composite the accumulated-dab scratch buffer C over the source A, writing the /// Composite the accumulated-dab scratch buffer C over the source A, writing the
@ -1873,6 +1970,9 @@ pub struct CanvasBlitPipeline {
pub pipeline: wgpu::RenderPipeline, pub pipeline: wgpu::RenderPipeline,
pub bg_layout: wgpu::BindGroupLayout, pub bg_layout: wgpu::BindGroupLayout,
pub sampler: wgpu::Sampler, pub sampler: wgpu::Sampler,
/// Bilinear sampler for smooth upscaling (used by `blit_smooth`, e.g. low-res
/// proxies). The default `sampler` stays nearest to keep the real canvas crisp.
pub linear_sampler: wgpu::Sampler,
/// Nearest-neighbour sampler used for the selection mask texture. /// Nearest-neighbour sampler used for the selection mask texture.
pub mask_sampler: wgpu::Sampler, pub mask_sampler: wgpu::Sampler,
} }
@ -2048,6 +2148,17 @@ impl CanvasBlitPipeline {
..Default::default() ..Default::default()
}); });
let linear_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("canvas_blit_linear_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
address_mode_v: wgpu::AddressMode::ClampToEdge,
address_mode_w: wgpu::AddressMode::ClampToEdge,
mag_filter: wgpu::FilterMode::Linear,
min_filter: wgpu::FilterMode::Linear,
mipmap_filter: wgpu::FilterMode::Linear,
..Default::default()
});
let mask_sampler = device.create_sampler(&wgpu::SamplerDescriptor { let mask_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("canvas_mask_sampler"), label: Some("canvas_mask_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge, address_mode_u: wgpu::AddressMode::ClampToEdge,
@ -2059,7 +2170,7 @@ impl CanvasBlitPipeline {
..Default::default() ..Default::default()
}); });
Self { pipeline, bg_layout, sampler, mask_sampler } Self { pipeline, bg_layout, sampler, linear_sampler, mask_sampler }
} }
/// Render the canvas texture into `target_view` (Rgba16Float) with the given camera. /// Render the canvas texture into `target_view` (Rgba16Float) with the given camera.
@ -2067,6 +2178,7 @@ impl CanvasBlitPipeline {
/// `target_view` is cleared to transparent before writing. /// `target_view` is cleared to transparent before writing.
/// `mask_view` is an R8Unorm texture in canvas-pixel space: 255 = keep, 0 = discard. /// `mask_view` is an R8Unorm texture in canvas-pixel space: 255 = keep, 0 = discard.
/// Pass `None` to use the built-in 1×1 all-white default (no masking). /// Pass `None` to use the built-in 1×1 all-white default (no masking).
/// Blit with the default nearest-neighbour sampler (crisp; for real canvas pixels).
pub fn blit( pub fn blit(
&self, &self,
device: &wgpu::Device, device: &wgpu::Device,
@ -2075,6 +2187,32 @@ impl CanvasBlitPipeline {
target_view: &wgpu::TextureView, target_view: &wgpu::TextureView,
transform: &BlitTransform, transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>, mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.sampler);
}
/// Blit with a bilinear sampler — smooth upscaling for low-res sources (proxies).
pub fn blit_smooth(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
canvas_view: &wgpu::TextureView,
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler);
}
fn blit_with(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
canvas_view: &wgpu::TextureView,
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
canvas_sampler: &wgpu::Sampler,
) { ) {
// When no mask is provided, create a temporary 1×1 all-white texture. // When no mask is provided, create a temporary 1×1 all-white texture.
// (queue is already available here, unlike in new()) // (queue is already available here, unlike in new())
@ -2127,7 +2265,7 @@ impl CanvasBlitPipeline {
}, },
wgpu::BindGroupEntry { wgpu::BindGroupEntry {
binding: 1, binding: 1,
resource: wgpu::BindingResource::Sampler(&self.sampler), resource: wgpu::BindingResource::Sampler(canvas_sampler),
}, },
wgpu::BindGroupEntry { wgpu::BindGroupEntry {
binding: 2, binding: 2,

File diff suppressed because it is too large Load Diff

View File

@ -242,6 +242,16 @@ pub struct SharedPaneState<'a> {
pub raw_audio_cache: &'a std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>, pub raw_audio_cache: &'a std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
/// Pool indices needing GPU waveform texture upload /// Pool indices needing GPU waveform texture upload
pub waveform_gpu_dirty: &'a mut std::collections::HashSet<usize>, pub waveform_gpu_dirty: &'a mut std::collections::HashSet<usize>,
/// Pools whose `raw_audio_cache` entry is a packed min/max floor rather than
/// raw samples (pool_index -> `B`, floor frames-per-texel). Drives the GPU
/// min/max upload path and the floor's effective rate `sr/B` in the renderer.
pub waveform_minmax_pools: &'a std::collections::HashMap<usize, u32>,
/// Miss-sink for on-demand raster keyframe pixel faulting (Phase 3 paging).
/// The canvas inserts the id of any raster keyframe it wants to upload whose
/// `raw_pixels` aren't resident; the App drains this at the top of the next
/// `update()` and faults the pixels in from the project container.
pub raster_fault_requests:
&'a std::sync::Arc<std::sync::Mutex<std::collections::HashSet<uuid::Uuid>>>,
/// Effect ID to load into shader editor (set by asset library, consumed by shader editor) /// Effect ID to load into shader editor (set by asset library, consumed by shader editor)
pub effect_to_load: &'a mut Option<Uuid>, pub effect_to_load: &'a mut Option<Uuid>,
/// Queue for effect thumbnail requests (effect IDs to generate thumbnails for) /// Queue for effect thumbnail requests (effect IDs to generate thumbnails for)

View File

@ -77,27 +77,29 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
let frames_per_pixel = params.sample_rate / params.pixels_per_second; let frames_per_pixel = params.sample_rate / params.pixels_per_second;
// Each mip level reduces by 4x in sample count (2x in each texture dimension) // Each mip level reduces by 4x in sample count (2x in each texture dimension)
let mip_f = max(0.0, log2(frames_per_pixel) / 2.0); let mip_f = max(0.0, log2(frames_per_pixel) / 2.0);
let max_mip = f32(textureNumLevels(peak_tex) - 1u);
let mip = min(mip_f, max_mip);
// Frame index at the chosen mip level // Pick the NEAREST INTEGER LOD and read its exact texel. Sampling at a
let mip_floor = u32(mip); // fractional mip (trilinear) blends level N and N+1, but each level has its
let reduction = pow(4.0, f32(mip_floor)); // own 1D2D row-major linearization (width halves per level), so the two
// levels disagree on which audio frame a given screen column maps to. The
// blend then reads horizontally-offset neighbours, and because a 2x zoom step
// shifts mip_f by exactly 0.5, alternate zoom levels land on a clean integer
// (correct) vs a 50/50 blend (offset) the "every other zoom level" artifact.
// textureLoad at one integer level keeps the frametexel mapping exact.
let max_mip = i32(textureNumLevels(peak_tex)) - 1;
let mip_i = clamp(i32(mip_f + 0.5), 0, max_mip);
let reduction = pow(4.0, f32(mip_i));
let mip_frame = frame_f / reduction; let mip_frame = frame_f / reduction;
// Convert 1D mip-space index to 2D UV coordinates // Convert 1D mip-space index to 2D texel coords using this level's actual
// Use actual texture dimensions (not computed from total_frames) because the // dimensions (texture may be pre-allocated larger, e.g. for live recording).
// texture may be pre-allocated larger for live recording. let mip_dims = textureDimensions(peak_tex, mip_i);
let mip_dims = textureDimensions(peak_tex, mip_floor);
let mip_tex_width = f32(mip_dims.x); let mip_tex_width = f32(mip_dims.x);
let mip_tex_height = f32(mip_dims.y); let texel_x = i32(mip_frame % mip_tex_width);
let texel_x = mip_frame % mip_tex_width; let texel_y = i32(floor(mip_frame / mip_tex_width));
let texel_y = floor(mip_frame / mip_tex_width);
let uv = vec2((texel_x + 0.5) / mip_tex_width, (texel_y + 0.5) / mip_tex_height);
// Sample the peak texture at computed mip level
// R = left_min, G = left_max, B = right_min, A = right_max // R = left_min, G = left_max, B = right_min, A = right_max
let peak = textureSampleLevel(peak_tex, peak_sampler, uv, mip); let peak = textureLoad(peak_tex, vec2<i32>(texel_x, texel_y), mip_i);
let clip_height = params.clip_rect.w - params.clip_rect.y; let clip_height = params.clip_rect.w - params.clip_rect.y;
let clip_top = params.clip_rect.y; let clip_top = params.clip_rect.y;

View File

@ -523,6 +523,12 @@ struct VelloRenderContext {
/// When `Some`, readback this B-canvas into `RASTER_READBACK_RESULTS` after /// When `Some`, readback this B-canvas into `RASTER_READBACK_RESULTS` after
/// dispatching GPU tool work. Set on mouseup by the unified raster tool commit path. /// dispatching GPU tool work. Set on mouseup by the unified raster tool commit path.
pending_tool_readback_b: Option<uuid::Uuid>, pending_tool_readback_b: Option<uuid::Uuid>,
/// Miss-sink for on-demand raster keyframe pixel faulting (Phase 3 paging).
/// When the compositor wants to upload an idle raster keyframe whose `raw_pixels`
/// aren't resident, it inserts the keyframe id here; the App drains this at the
/// top of the next `update()` and faults the pixels in from the project container.
raster_fault_requests:
std::sync::Arc<std::sync::Mutex<std::collections::HashSet<uuid::Uuid>>>,
} }
/// Callback for Vello rendering within egui /// Callback for Vello rendering within egui
@ -1169,6 +1175,9 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
// 4. Raster layer texture cache: for idle raster layers (no active tool canvas). // 4. Raster layer texture cache: for idle raster layers (no active tool canvas).
// Upload raw_pixels to the cache if texture_dirty; then use the cache entry. // Upload raw_pixels to the cache if texture_dirty; then use the cache entry.
// If the full pixels aren't resident, fall back to the low-res proxy:
// (kf_id, logical_w, logical_h) so the blit upscales it to full size.
let mut raster_proxy_blit: Option<(uuid::Uuid, u32, u32)> = None;
let raster_cache_kf: Option<uuid::Uuid> = if gpu_canvas_kf.is_none() { let raster_cache_kf: Option<uuid::Uuid> = if gpu_canvas_kf.is_none() {
// Find the active keyframe for this raster layer. // Find the active keyframe for this raster layer.
let doc = &self.ctx.document; let doc = &self.ctx.document;
@ -1206,6 +1215,24 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
); );
Some(kf_id) Some(kf_id)
} else { } else {
// Empty pixels: if the frame is paged out (lives in the
// container), record a fault-in request so the App pages it
// in at the top of the next frame. A new blank keyframe
// (needs_fault_in == false) has nothing to load — skip it.
if kf.needs_fault_in {
if let Ok(mut reqs) = self.ctx.raster_fault_requests.lock() {
reqs.insert(kf_id);
}
}
// Show the low-res proxy (if decoded) while the full
// pages in, so the cold scrub doesn't flash blank.
if let Some(proxy) = &kf.proxy {
gpu_brush.ensure_proxy_texture(
device, queue, kf_id,
&proxy.pixels, proxy.width, proxy.height,
);
raster_proxy_blit = Some((kf_id, kf.width, kf.height));
}
None None
} }
} else { } else {
@ -1221,7 +1248,9 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
None None
}; };
if !rendered_layer.has_content && gpu_canvas_kf.is_none() && raster_cache_kf.is_none() { if !rendered_layer.has_content && gpu_canvas_kf.is_none() && raster_cache_kf.is_none()
&& raster_proxy_blit.is_none()
{
continue; continue;
} }
@ -1268,27 +1297,47 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
} }
RenderedLayerType::Raster { transform: layer_transform, .. } => { RenderedLayerType::Raster { transform: layer_transform, .. } => {
// Raster layer — GPU canvas blit directly to HDR (bypasses Vello). // Raster layer — GPU canvas blit directly to HDR (bypasses Vello).
// Tool override canvas (gpu_canvas_kf) takes priority over cached texture. // Tool override canvas (gpu_canvas_kf) takes priority over cached
if let Some(use_kf_id) = gpu_canvas_kf.or(raster_cache_kf) { // texture; if neither full-res source is present, the low-res proxy.
let full_kf = gpu_canvas_kf.or(raster_cache_kf);
if full_kf.is_some() || raster_proxy_blit.is_some() {
let hdr_layer_handle = buffer_pool.acquire(device, hdr_spec); let hdr_layer_handle = buffer_pool.acquire(device, hdr_spec);
if let (Some(hdr_layer_view), Some(hdr_view)) = ( if let (Some(hdr_layer_view), Some(hdr_view)) = (
buffer_pool.get_view(hdr_layer_handle), buffer_pool.get_view(hdr_layer_handle),
&instance_resources.hdr_texture_view, &instance_resources.hdr_texture_view,
) { ) {
if let Ok(gpu_brush) = shared.gpu_brush.lock() { if let Ok(gpu_brush) = shared.gpu_brush.lock() {
let canvas = gpu_brush.canvases.get(&use_kf_id) // Pick the source texture and the LOGICAL dims the blit
.or_else(|| gpu_brush.raster_layer_cache.get(&use_kf_id)); // should map it to. A full texture uses its own dims; a
if let Some(canvas) = canvas { // proxy uses the keyframe's full dims so it upscales.
let blit = if let Some(use_kf_id) = full_kf {
gpu_brush.canvases.get(&use_kf_id)
.or_else(|| gpu_brush.raster_layer_cache.get(&use_kf_id))
.map(|c| (c, c.width, c.height, false))
} else if let Some((pkf, lw, lh)) = raster_proxy_blit {
gpu_brush.get_proxy_texture(&pkf).map(|c| (c, lw, lh, true))
} else {
None
};
if let Some((canvas, logical_w, logical_h, is_proxy)) = blit {
let bt = crate::gpu_brush::BlitTransform::new( let bt = crate::gpu_brush::BlitTransform::new(
*layer_transform, *layer_transform,
canvas.width, canvas.height, logical_w, logical_h,
width, height, width, height,
); );
// Proxies are upscaled, so sample them bilinearly;
// the real canvas stays nearest (crisp pixels).
if is_proxy {
shared.canvas_blit.blit_smooth(
device, queue, canvas.src_view(), hdr_layer_view, &bt, None,
);
} else {
shared.canvas_blit.blit( shared.canvas_blit.blit(
device, queue, canvas.src_view(), hdr_layer_view, &bt, None, device, queue, canvas.src_view(), hdr_layer_view, &bt, None,
); );
} }
} }
}
let compositor_layer = lightningbeam_core::gpu::CompositorLayer::new( let compositor_layer = lightningbeam_core::gpu::CompositorLayer::new(
hdr_layer_handle, hdr_layer_handle,
rendered_layer.opacity, rendered_layer.opacity,
@ -5575,15 +5624,8 @@ impl StagePane {
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
// Ensure the keyframe exists before reading its ID. // Don't create a keyframe — explicit "New Keyframe" only. The read below
{ // returns None (no workspace) if there's no active keyframe to lift from.
let doc = shared.action_executor.document_mut();
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
rl.ensure_keyframe_at(time, doc_w, doc_h);
} else {
return None; // not a raster layer
}
}
// Read keyframe id and pixels. // Read keyframe id and pixels.
let (kf_id, w, h, pixels) = { let (kf_id, w, h, pixels) = {
@ -5806,6 +5848,9 @@ impl StagePane {
kf.raw_pixels[si..si + 4].fill(0); kf.raw_pixels[si..si + 4].fill(0);
} }
} }
// Punching the hole edits raw_pixels but is NOT committed through an
// action yet — mark dirty so eviction can't drop the un-persisted hole.
kf.dirty = true;
} }
// Re-set selection (commit_raster_floating_now cleared it) and create float. // Re-set selection (commit_raster_floating_now cleared it) and create float.
@ -6014,21 +6059,12 @@ impl StagePane {
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
// Ensure the keyframe exists BEFORE reading its ID, so we always get // Paint into the ACTIVE keyframe (the one at-or-before the playhead) —
// the real UUID. Previously we read the ID first and fell back to a // do NOT create one. Keyframes are made explicitly via "New Keyframe"
// randomly-generated UUID when no keyframe existed; that fake UUID was // (a new layer already seeds one). If none exists at-or-before the
// stored in painting_canvas but subsequent drag frames used the real UUID // playhead, there's nothing to paint into; bail.
// from keyframe_at(), causing the GPU canvas to be a different object from let _ = (doc_width, doc_height);
// the one being composited. let (keyframe_id, kf_time, canvas_width, canvas_height, buffer_before, initial_pixels) = {
{
let doc = shared.action_executor.document_mut();
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&active_layer_id) {
rl.ensure_keyframe_at(*shared.playback_time, doc_width, doc_height);
}
}
// Now read the guaranteed-to-exist keyframe to get the real UUID.
let (keyframe_id, canvas_width, canvas_height, buffer_before, initial_pixels) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
if let Some(AnyLayer::Raster(rl)) = doc.get_layer(&active_layer_id) { if let Some(AnyLayer::Raster(rl)) = doc.get_layer(&active_layer_id) {
if let Some(kf) = rl.keyframe_at(*shared.playback_time) { if let Some(kf) = rl.keyframe_at(*shared.playback_time) {
@ -6038,9 +6074,9 @@ impl StagePane {
} else { } else {
raw.clone() raw.clone()
}; };
(kf.id, kf.width, kf.height, raw, init) (kf.id, kf.time, kf.width, kf.height, raw, init)
} else { } else {
return; // shouldn't happen after ensure_keyframe_at return; // no keyframe at/before the playhead — nothing to paint
} }
} else { } else {
return; return;
@ -6072,7 +6108,7 @@ impl StagePane {
self.painting_canvas = Some((active_layer_id, keyframe_id)); self.painting_canvas = Some((active_layer_id, keyframe_id));
self.pending_undo_before = Some(( self.pending_undo_before = Some((
active_layer_id, active_layer_id,
*shared.playback_time, kf_time,
canvas_width, canvas_width,
canvas_height, canvas_height,
buffer_before, buffer_before,
@ -6080,7 +6116,7 @@ impl StagePane {
self.pending_raster_dabs = Some(PendingRasterDabs { self.pending_raster_dabs = Some(PendingRasterDabs {
keyframe_id, keyframe_id,
layer_id: active_layer_id, layer_id: active_layer_id,
time: *shared.playback_time, time: kf_time,
canvas_width, canvas_width,
canvas_height, canvas_height,
initial_pixels: Some(initial_pixels), initial_pixels: Some(initial_pixels),
@ -6090,7 +6126,7 @@ impl StagePane {
}); });
self.raster_stroke_state = Some(( self.raster_stroke_state = Some((
active_layer_id, active_layer_id,
*shared.playback_time, kf_time,
stroke_state, stroke_state,
Vec::new(), // buffer_before now lives in pending_undo_before Vec::new(), // buffer_before now lives in pending_undo_before
)); ));
@ -6316,17 +6352,13 @@ impl StagePane {
let time = *shared.playback_time; let time = *shared.playback_time;
// Canvas dimensions (to create keyframe if needed). // Canvas dimensions (to create keyframe if needed).
let (doc_w, doc_h) = { let (_doc_w, _doc_h) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
// Ensure a keyframe exists at the current time. // Ensure a keyframe exists at the current time.
{ // Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
let doc = shared.action_executor.document_mut(); // The snapshot below edits the active keyframe and bails if none exists.
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
rl.ensure_keyframe_at(time, doc_w, doc_h);
}
}
// Snapshot the pixel buffer before drawing. // Snapshot the pixel buffer before drawing.
let (buffer_before, w, h) = { let (buffer_before, w, h) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
@ -6669,16 +6701,12 @@ impl StagePane {
let time = *shared.playback_time; let time = *shared.playback_time;
// Ensure a keyframe exists at the current time. // Ensure a keyframe exists at the current time.
let (doc_w, doc_h) = { let (_doc_w, _doc_h) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
{ // Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
let doc = shared.action_executor.document_mut(); // The snapshot below edits the active keyframe and bails if none exists.
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
rl.ensure_keyframe_at(time, doc_w, doc_h);
}
}
// Snapshot current pixels. // Snapshot current pixels.
let (buffer_before, width, height) = { let (buffer_before, width, height) = {
@ -6750,16 +6778,12 @@ impl StagePane {
let time = *shared.playback_time; let time = *shared.playback_time;
// Ensure keyframe exists. // Ensure keyframe exists.
let (doc_w, doc_h) = { let (_doc_w, _doc_h) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
{ // Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
let doc = shared.action_executor.document_mut(); // The snapshot below edits the active keyframe and bails if none exists.
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
rl.ensure_keyframe_at(time, doc_w, doc_h);
}
}
let (pixels, width, height) = { let (pixels, width, height) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
@ -6836,16 +6860,11 @@ impl StagePane {
Self::commit_raster_floating_now(shared); Self::commit_raster_floating_now(shared);
// Ensure the keyframe exists. // Ensure the keyframe exists.
let (doc_w, doc_h) = { let (_doc_w, _doc_h) = {
let doc = shared.action_executor.document(); let doc = shared.action_executor.document();
(doc.width as u32, doc.height as u32) (doc.width as u32, doc.height as u32)
}; };
{ // Don't create a keyframe — explicit "New Keyframe" only; bail below if none.
let doc = shared.action_executor.document_mut();
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
rl.ensure_keyframe_at(time, doc_w, doc_h);
}
}
// Snapshot canvas pixels. // Snapshot canvas pixels.
let (pixels, width, height) = { let (pixels, width, height) = {
@ -11902,6 +11921,7 @@ impl PaneRenderer for StagePane {
.and_then(|(tool, _)| tool.take_pending_gpu_work()), .and_then(|(tool, _)| tool.take_pending_gpu_work()),
pending_layer_cache_removals: std::mem::take(&mut self.pending_layer_cache_removals), pending_layer_cache_removals: std::mem::take(&mut self.pending_layer_cache_removals),
pending_tool_readback_b: self.pending_tool_readback_b.take(), pending_tool_readback_b: self.pending_tool_readback_b.take(),
raster_fault_requests: std::sync::Arc::clone(shared.raster_fault_requests),
}}; }};
let cb = egui_wgpu::Callback::new_paint_callback( let cb = egui_wgpu::Callback::new_paint_callback(

View File

@ -99,6 +99,94 @@ fn clip_instance_y_bounds(row: usize, total_rows: usize) -> (f32, f32) {
} }
} }
/// Draw the strip of video thumbnails across a clip.
///
/// Thumbnails are tiled from the clip's **true** (unclamped) screen origin
/// `clip_origin_x` over its full width `clip_true_width_px`, and only the tiles
/// intersecting `visible_rect` are drawn. This makes the strip scroll naturally
/// with the timeline and show the correct content when the clip is partly off the
/// left edge (a tile's content time is derived from its offset from the true
/// origin, not from the clamped visible edge).
///
/// Each tile's GPU texture is cached by the **actual** thumbnail timestamp (from
/// [`VideoManager::get_thumbnail_at`]), so as closer thumbnails finish generating
/// the strip refreshes instead of freezing on whatever loaded first.
#[allow(clippy::too_many_arguments)]
fn draw_video_thumbnail_strip(
painter: &egui::Painter,
ui: &egui::Ui,
video_mgr: &lightningbeam_core::video::VideoManager,
textures: &mut std::collections::HashMap<(uuid::Uuid, i64), egui::TextureHandle>,
clip_id: uuid::Uuid,
trim_start: f64,
clip_origin_x: f32,
clip_true_width_px: f32,
visible_rect: egui::Rect,
thumb_top_y: f32,
thumb_height: f32,
pixels_per_second: f64,
) {
if clip_true_width_px <= 0.0 || thumb_height <= 0.0 {
return;
}
// Aspect ratio from any available thumbnail; sets the per-tile width.
let aspect = match video_mgr.get_thumbnail_at(&clip_id, 0.0) {
Some((_, tw, th, _)) if th > 0 => tw as f32 / th as f32,
_ => return,
};
let step = (thumb_height * aspect).max(1.0);
let num_thumbs = ((clip_true_width_px / step).ceil() as usize).max(1);
// Iterate only the tiles that intersect the visible rect.
let first = (((visible_rect.min.x - clip_origin_x) / step).floor() as i64).max(0) as usize;
let last = (((visible_rect.max.x - clip_origin_x) / step).ceil() as i64).max(0) as usize;
let last = last.min(num_thumbs);
for i in first..last {
let thumb_x = clip_origin_x + i as f32 * step;
// Content time at the tile centre, measured from the TRUE clip origin so
// it's stable as the clip scrolls.
let content_time =
trim_start + (i as f64 * step as f64 + step as f64 * 0.5) / pixels_per_second;
let Some((actual_ts, tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&clip_id, content_time)
else {
continue;
};
let ts_key = (actual_ts * 1000.0) as i64;
let texture = textures.entry((clip_id, ts_key)).or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied([tw as usize, th as usize], &rgba_data);
ui.ctx().load_texture(
format!("vthumb_{}_{}", clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(thumb_x, thumb_top_y),
egui::vec2(step, thumb_height),
);
let thumb_rect = full_rect.intersect(visible_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
);
}
}
}
/// Get the effective clip duration for a clip instance on a given layer. /// Get the effective clip duration for a clip instance on a given layer.
/// For groups on vector layers, the duration spans all consecutive keyframes /// For groups on vector layers, the duration spans all consecutive keyframes
/// where the group is present. For regular clips, returns the clip's internal duration. /// where the group is present. For regular clips, returns the clip's internal duration.
@ -163,6 +251,11 @@ pub struct TimelinePane {
/// Vertical scroll offset (in pixels) /// Vertical scroll offset (in pixels)
viewport_scroll_y: f32, viewport_scroll_y: f32,
/// Clickable keyframe-diamond hit targets `(screen_rect, keyframe_time)` collected
/// during the last `render_layers`. Used by `handle_input` (next frame) to snap the
/// playhead exactly to a keyframe when its diamond is clicked.
keyframe_diamond_hits: Vec<(egui::Rect, f64)>,
/// Total duration of the animation /// Total duration of the animation
duration: f64, duration: f64,
@ -685,6 +778,7 @@ impl TimelinePane {
pixels_per_second: 100.0, pixels_per_second: 100.0,
viewport_start_time: 0.0, viewport_start_time: 0.0,
viewport_scroll_y: 0.0, viewport_scroll_y: 0.0,
keyframe_diamond_hits: Vec::new(),
duration: 10.0, // Default 10 seconds duration: 10.0, // Default 10 seconds
is_scrubbing: false, is_scrubbing: false,
is_panning: false, is_panning: false,
@ -2632,18 +2726,21 @@ impl TimelinePane {
midi_event_cache: &std::collections::HashMap<u32, Vec<daw_backend::audio::midi::MidiEvent>>, midi_event_cache: &std::collections::HashMap<u32, Vec<daw_backend::audio::midi::MidiEvent>>,
raw_audio_cache: &std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>, raw_audio_cache: &std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
waveform_gpu_dirty: &mut std::collections::HashSet<usize>, waveform_gpu_dirty: &mut std::collections::HashSet<usize>,
waveform_minmax_pools: &std::collections::HashMap<usize, u32>,
target_format: wgpu::TextureFormat, target_format: wgpu::TextureFormat,
waveform_stereo: bool, waveform_stereo: bool,
context_layers: &[&lightningbeam_core::layer::AnyLayer], context_layers: &[&lightningbeam_core::layer::AnyLayer],
video_manager: &std::sync::Arc<std::sync::Mutex<lightningbeam_core::video::VideoManager>>, video_manager: &std::sync::Arc<std::sync::Mutex<lightningbeam_core::video::VideoManager>>,
audio_cache: &HashMap<uuid::Uuid, Vec<ClipInstance>>, audio_cache: &HashMap<uuid::Uuid, Vec<ClipInstance>>,
playback_time: f64, playback_time: f64,
) -> (Vec<(egui::Rect, uuid::Uuid, f64, f64)>, Vec<AutomationLaneRender>) { ) -> (Vec<(egui::Rect, uuid::Uuid, f64, f32)>, Vec<AutomationLaneRender>) {
let painter = ui.painter().clone(); let painter = ui.painter().clone();
let mut pending_lane_renders: Vec<AutomationLaneRender> = Vec::new(); let mut pending_lane_renders: Vec<AutomationLaneRender> = Vec::new();
// Rebuilt each frame; consumed by handle_input (next frame) for click-to-seek.
self.keyframe_diamond_hits.clear();
// Collect video clip rects for hover detection (to avoid borrow conflicts) // Collect video clip rects for hover detection (to avoid borrow conflicts)
let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f64)> = Vec::new(); let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f32)> = Vec::new();
// Track visible video clip IDs for texture cache cleanup // Track visible video clip IDs for texture cache cleanup
let mut visible_video_clip_ids: std::collections::HashSet<uuid::Uuid> = std::collections::HashSet::new(); let mut visible_video_clip_ids: std::collections::HashSet<uuid::Uuid> = std::collections::HashSet::new();
@ -2899,8 +2996,9 @@ impl TimelinePane {
theme.text_color(&["#timeline", ".group-bar"], ui.ctx(), egui::Color32::from_rgb(100, 220, 220)) theme.text_color(&["#timeline", ".group-bar"], ui.ctx(), egui::Color32::from_rgb(100, 220, 220))
}; };
for (s, e) in &merged { for (s, e) in &merged {
let sx = self.time_to_x(*s); // `merged` ranges are in beats; convert to seconds for time_to_x.
let ex = self.time_to_x(*e).max(sx + MIN_CLIP_WIDTH_PX); let sx = self.time_to_x(document.tempo_map().transform(*s));
let ex = self.time_to_x(document.tempo_map().transform(*e)).max(sx + MIN_CLIP_WIDTH_PX);
if ex >= 0.0 && sx <= rect.width() { if ex >= 0.0 && sx <= rect.width() {
let vsx = sx.max(0.0); let vsx = sx.max(0.0);
let vex = ex.min(rect.width()); let vex = ex.min(rect.width());
@ -2940,8 +3038,9 @@ impl TimelinePane {
let ci_duration = ci.total_duration(clip_dur, document.tempo_map()); let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
let ci_end = ci_start + ci_duration; let ci_end = ci_start + ci_duration;
let sx = self.time_to_x(ci_start); // ci_start/ci_end are in beats; convert to seconds for time_to_x.
let ex = self.time_to_x(ci_end); let sx = self.time_to_x(document.tempo_map().transform(ci_start));
let ex = self.time_to_x(document.tempo_map().transform(ci_end));
if ex < 0.0 || sx > rect.width() { continue; } if ex < 0.0 || sx > rect.width() { continue; }
let ci_rect = egui::Rect::from_min_max( let ci_rect = egui::Rect::from_min_max(
@ -2956,69 +3055,29 @@ impl TimelinePane {
egui::pos2(ci_rect.min.x, span_y_min), egui::pos2(ci_rect.min.x, span_y_min),
egui::pos2(ci_rect.max.x, span_y_max), egui::pos2(ci_rect.max.x, span_y_max),
); );
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, ci_start)); // 4th elem = clip's TRUE (unclamped) origin x, for correct
// hover content time when scrolled partly off the left.
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, rect.min.x + sx));
let thumb_display_height = (thumb_y_max - span_y_min) - 4.0; let thumb_display_height = (thumb_y_max - span_y_min) - 4.0;
if thumb_display_height > 8.0 { if thumb_display_height > 8.0 {
let video_mgr = video_manager.lock().unwrap(); let video_mgr = video_manager.lock().unwrap();
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&ci.clip_id, 0.0) { // Tile from the clip's true origin (sx unclamped; ci_rect is
let aspect = tw as f32 / th as f32; // the clamped visible rect) — scrolls correctly off-screen.
let thumb_display_width = thumb_display_height * aspect; draw_video_thumbnail_strip(
let ci_width = ci_rect.width(); &painter,
let num_thumbs = ((ci_width / thumb_display_width).ceil() as usize).max(1); ui,
&video_mgr,
for ti in 0..num_thumbs { &mut self.video_thumbnail_textures,
let x_offset = ti as f32 * thumb_display_width; ci.clip_id,
if x_offset >= ci_width { break; } ci.trim_start,
rect.min.x + sx,
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5) ex - sx,
/ self.pixels_per_second as f64; ci_rect,
let content_time = ci.trim_start + time_offset; ci_rect.min.y + 2.0,
thumb_display_height,
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&ci.clip_id, content_time) { self.pixels_per_second as f64,
let ts_key = (content_time * 1000.0) as i64;
let cache_key = (ci.clip_id, ts_key);
let texture = self.video_thumbnail_textures
.entry(cache_key)
.or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied(
[tw as usize, th as usize],
&rgba_data,
); );
ui.ctx().load_texture(
format!("vthumb_{}_{}", ci.clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(ci_rect.min.x + x_offset, ci_rect.min.y + 2.0),
egui::vec2(thumb_display_width, thumb_display_height),
);
let thumb_rect = full_rect.intersect(ci_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
);
}
}
}
}
} }
} }
} }
@ -3048,8 +3107,17 @@ impl TimelinePane {
None => continue, None => continue,
}; };
let total_frames = samples.len() / (*ch).max(1) as usize; // Min/max overview pools store 4 f32 per texel at the
let audio_file_duration = total_frames as f64 / *sr as f64; // floor rate sr/B; raw pools store interleaved samples.
let minmax_b = waveform_minmax_pools.get(&audio_pool_index).copied();
let is_minmax = minmax_b.is_some();
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
let total_frames = samples.len() / frame_stride;
let eff_sr: f32 = match minmax_b {
Some(b) => *sr as f32 / b.max(1) as f32,
None => *sr as f32,
};
let audio_file_duration = total_frames as f64 / eff_sr as f64;
let clip_dur = audio_clip.duration; let clip_dur = audio_clip.duration;
let mut ci_start = ci.effective_start(); let mut ci_start = ci.effective_start();
@ -3058,8 +3126,9 @@ impl TimelinePane {
} }
let ci_duration = ci.total_duration(clip_dur, document.tempo_map()); let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
let ci_screen_start = rect.min.x + self.time_to_x(ci_start); // ci_start/ci_duration are in beats; convert to seconds for time_to_x.
let ci_screen_end = ci_screen_start + (ci_duration * self.pixels_per_second as f64) as f32; let ci_screen_start = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start));
let ci_screen_end = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start + ci_duration));
let waveform_rect = egui::Rect::from_min_max( let waveform_rect = egui::Rect::from_min_max(
egui::pos2(ci_screen_start.max(rect.min.x), wave_y_min), egui::pos2(ci_screen_start.max(rect.min.x), wave_y_min),
@ -3081,9 +3150,10 @@ impl TimelinePane {
} }
Some(crate::waveform_gpu::PendingUpload { Some(crate::waveform_gpu::PendingUpload {
samples: samples.clone(), samples: samples.clone(),
sample_rate: *sr, sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
channels: *ch, channels: *ch,
frame_limit, frame_limit,
minmax: is_minmax,
}) })
} else { } else {
None None
@ -3098,7 +3168,7 @@ impl TimelinePane {
viewport_start_time: self.viewport_start_time as f32, viewport_start_time: self.viewport_start_time as f32,
pixels_per_second: self.pixels_per_second as f32, pixels_per_second: self.pixels_per_second as f32,
audio_duration: audio_file_duration as f32, audio_duration: audio_file_duration as f32,
sample_rate: *sr as f32, sample_rate: eff_sr,
clip_start_time: ci_screen_start, clip_start_time: ci_screen_start,
trim_start: ci.trim_start as f32, trim_start: ci.trim_start as f32,
tex_width: crate::waveform_gpu::tex_width() as f32, tex_width: crate::waveform_gpu::tex_width() as f32,
@ -3597,8 +3667,16 @@ impl TimelinePane {
// Sampled Audio: Draw waveform via GPU // Sampled Audio: Draw waveform via GPU
lightningbeam_core::clip::AudioClipType::Sampled { audio_pool_index } => { lightningbeam_core::clip::AudioClipType::Sampled { audio_pool_index } => {
if let Some((samples, sr, ch)) = raw_audio_cache.get(audio_pool_index) { if let Some((samples, sr, ch)) = raw_audio_cache.get(audio_pool_index) {
let total_frames = samples.len() / (*ch).max(1) as usize; // Min/max overview pools: 4 f32/texel at rate sr/B.
let audio_file_duration = total_frames as f64 / *sr as f64; let minmax_b = waveform_minmax_pools.get(audio_pool_index).copied();
let is_minmax = minmax_b.is_some();
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
let total_frames = samples.len() / frame_stride;
let eff_sr: f32 = match minmax_b {
Some(b) => *sr as f32 / b.max(1) as f32,
None => *sr as f32,
};
let audio_file_duration = total_frames as f64 / eff_sr as f64;
let screen_size = ui.ctx().content_rect().size(); let screen_size = ui.ctx().content_rect().size();
let pending_upload = if waveform_gpu_dirty.contains(audio_pool_index) { let pending_upload = if waveform_gpu_dirty.contains(audio_pool_index) {
@ -3620,9 +3698,10 @@ impl TimelinePane {
Some(crate::waveform_gpu::PendingUpload { Some(crate::waveform_gpu::PendingUpload {
samples: samples.clone(), samples: samples.clone(),
sample_rate: *sr, sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
channels: *ch, channels: *ch,
frame_limit, frame_limit,
minmax: is_minmax,
}) })
} else { } else {
None None
@ -3684,7 +3763,7 @@ impl TimelinePane {
viewport_start_time: self.viewport_start_time as f32, viewport_start_time: self.viewport_start_time as f32,
pixels_per_second: self.pixels_per_second as f32, pixels_per_second: self.pixels_per_second as f32,
audio_duration: audio_file_duration as f32, audio_duration: audio_file_duration as f32,
sample_rate: *sr as f32, sample_rate: eff_sr,
clip_start_time: iter_screen_start, clip_start_time: iter_screen_start,
trim_start: preview_trim_start as f32, trim_start: preview_trim_start as f32,
tex_width: crate::waveform_gpu::tex_width() as f32, tex_width: crate::waveform_gpu::tex_width() as f32,
@ -3730,6 +3809,7 @@ impl TimelinePane {
sample_rate: *sr, sample_rate: *sr,
channels: *ch, channels: *ch,
frame_limit: None, // recording uses incremental path frame_limit: None, // recording uses incremental path
minmax: false,
}) })
} else { } else {
None None
@ -3794,75 +3874,31 @@ impl TimelinePane {
let thumb_display_height = clip_rect.height() - 4.0; let thumb_display_height = clip_rect.height() - 4.0;
if thumb_display_height > 8.0 { if thumb_display_height > 8.0 {
let video_mgr = video_manager.lock().unwrap(); let video_mgr = video_manager.lock().unwrap();
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&clip_instance.clip_id, 0.0) { // Tile from the clip's true origin (start_x is unclamped;
let aspect = tw as f32 / th as f32; // clip_rect is the clamped visible rect) so the strip scrolls
let thumb_display_width = thumb_display_height * aspect; // correctly and shows the right content when partly off-screen.
let thumb_step_px = thumb_display_width; draw_video_thumbnail_strip(
&painter,
let clip_width = clip_rect.width(); ui,
let num_thumbs = ((clip_width / thumb_step_px).ceil() as usize).max(1); &video_mgr,
&mut self.video_thumbnail_textures,
for i in 0..num_thumbs { clip_instance.clip_id,
let x_offset = i as f32 * thumb_step_px; clip_instance.trim_start,
if x_offset >= clip_width { break; } rect.min.x + start_x,
end_x - start_x,
// Map pixel position to content time clip_rect,
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5) clip_rect.min.y + 2.0,
/ self.pixels_per_second as f64; thumb_display_height,
let content_time = clip_instance.trim_start + time_offset; self.pixels_per_second as f64,
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(
&clip_instance.clip_id, content_time
) {
let ts_key = (content_time * 1000.0) as i64;
let cache_key = (clip_instance.clip_id, ts_key);
let texture = self.video_thumbnail_textures
.entry(cache_key)
.or_insert_with(|| {
let image = egui::ColorImage::from_rgba_unmultiplied(
[tw as usize, th as usize],
&rgba_data,
);
ui.ctx().load_texture(
format!("vthumb_{}_{}", clip_instance.clip_id, ts_key),
image,
egui::TextureOptions::LINEAR,
)
});
let full_rect = egui::Rect::from_min_size(
egui::pos2(clip_rect.min.x + x_offset, clip_rect.min.y + 2.0),
egui::vec2(thumb_display_width, thumb_display_height),
);
let thumb_rect = full_rect.intersect(clip_rect);
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
let uv_min = egui::pos2(
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
);
let uv_max = egui::pos2(
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
);
painter.image(
texture.id(),
thumb_rect,
egui::Rect::from_min_max(uv_min, uv_max),
egui::Color32::WHITE,
); );
} }
} }
}
}
}
}
// VIDEO PREVIEW: Collect clip rect for hover detection // VIDEO PREVIEW: Collect clip rect for hover detection. Store the
// clip's TRUE (unclamped) origin x so the hover content time is
// correct even when the clip is scrolled partly off the left.
if let lightningbeam_core::layer::AnyLayer::Video(_) = layer { if let lightningbeam_core::layer::AnyLayer::Video(_) = layer {
video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, instance_start)); video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, rect.min.x + start_x));
} }
// Draw border per segment (per loop iteration for looping clips) // Draw border per segment (per loop iteration for looping clips)
@ -3944,6 +3980,40 @@ impl TimelinePane {
color, color,
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))), egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
)); ));
// Record a (slightly enlarged) clickable area for click-to-seek.
self.keyframe_diamond_hits.push((
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
kf.time,
));
}
}
}
// Draw keyframe markers for raster layers (same diamond as vector).
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
for kf in &rl.keyframes {
let x = self.time_to_x(kf.time);
if x >= 0.0 && x <= rect.width() {
let cx = rect.min.x + x;
let cy = y + LAYER_HEIGHT - 8.0;
let size = 5.0;
let diamond = [
egui::pos2(cx, cy - size),
egui::pos2(cx + size, cy),
egui::pos2(cx, cy + size),
egui::pos2(cx - size, cy),
];
let color = theme.bg_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(255, 220, 100));
painter.add(egui::Shape::convex_polygon(
diamond.to_vec(),
color,
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
));
// Record a (slightly enlarged) clickable area for click-to-seek.
self.keyframe_diamond_hits.push((
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
kf.time,
));
} }
} }
} }
@ -4797,6 +4867,30 @@ impl TimelinePane {
} }
} }
// Pointing-hand cursor when hovering a clickable keyframe diamond.
if let Some(hover) = response.hover_pos() {
if self.keyframe_diamond_hits.iter().any(|(r, _)| r.contains(hover)) {
ui.ctx().set_cursor_icon(egui::CursorIcon::PointingHand);
}
}
// Click a keyframe diamond → snap the playhead exactly to that keyframe.
// (Hit targets come from the previous frame's render_layers; diamonds don't
// move between frames, so a click lands on the right one.)
if response.clicked() && !alt_held {
if let Some(pos) = response.interact_pointer_pos() {
if let Some(&(_, kf_time)) =
self.keyframe_diamond_hits.iter().find(|(r, _)| r.contains(pos))
{
*playback_time = kf_time;
if let Some(controller_arc) = audio_controller {
let mut controller = controller_arc.lock().unwrap();
controller.seek(kf_time);
}
}
}
}
// Get mouse position relative to content area // Get mouse position relative to content area
let mouse_pos = response.hover_pos().unwrap_or(content_rect.center()); let mouse_pos = response.hover_pos().unwrap_or(content_rect.center());
let mouse_x = (mouse_pos.x - content_rect.min.x).max(0.0); let mouse_x = (mouse_pos.x - content_rect.min.x).max(0.0);
@ -5418,7 +5512,7 @@ impl PaneRenderer for TimelinePane {
// Render layer rows with clipping // Render layer rows with clipping
ui.set_clip_rect(content_rect.intersect(original_clip_rect)); ui.set_clip_rect(content_rect.intersect(original_clip_rect));
let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time); let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.waveform_minmax_pools, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time);
// Render playhead on top (clip to timeline area) // Render playhead on top (clip to timeline area)
ui.set_clip_rect(timeline_rect.intersect(original_clip_rect)); ui.set_clip_rect(timeline_rect.intersect(original_clip_rect));
@ -5894,26 +5988,24 @@ impl PaneRenderer for TimelinePane {
// VIDEO HOVER DETECTION: Handle video clip hover tooltips AFTER input handling // VIDEO HOVER DETECTION: Handle video clip hover tooltips AFTER input handling
// This ensures hover events aren't consumed by the main input handler // This ensures hover events aren't consumed by the main input handler
for (clip_rect, clip_id, trim_start, instance_start) in video_clip_hovers { for (clip_rect, clip_id, trim_start, clip_origin_x) in video_clip_hovers {
let hover_response = ui.allocate_rect(clip_rect, egui::Sense::hover()); let hover_response = ui.allocate_rect(clip_rect, egui::Sense::hover());
if hover_response.hovered() { if hover_response.hovered() {
if let Some(hover_pos) = hover_response.hover_pos() { if let Some(hover_pos) = hover_response.hover_pos() {
// Calculate timestamp at hover position // Content time from the clip's TRUE origin. `clip_rect` is clamped
let hover_offset_pixels = hover_pos.x - clip_rect.min.x; // to the viewport, so using its left edge mislabels the frame when
let hover_offset_time = (hover_offset_pixels as f64) / (self.pixels_per_second as f64); // the clip is scrolled partly off the left (same bug the strip had).
let hover_timestamp = instance_start + hover_offset_time; let hover_offset_pixels = hover_pos.x - clip_origin_x;
let clip_content_time = trim_start + (hover_offset_pixels as f64) / (self.pixels_per_second as f64);
// Remap to clip content time accounting for trim
let clip_content_time = trim_start + (hover_timestamp - instance_start);
// Try to get thumbnail from video manager // Try to get thumbnail from video manager
let thumbnail_data: Option<(u32, u32, std::sync::Arc<Vec<u8>>)> = { let thumbnail_data: Option<(f64, u32, u32, std::sync::Arc<Vec<u8>>)> = {
let video_mgr = shared.video_manager.lock().unwrap(); let video_mgr = shared.video_manager.lock().unwrap();
video_mgr.get_thumbnail_at(&clip_id, clip_content_time) video_mgr.get_thumbnail_at(&clip_id, clip_content_time)
}; };
if let Some((thumb_width, thumb_height, ref thumb_data)) = thumbnail_data { if let Some((_, thumb_width, thumb_height, ref thumb_data)) = thumbnail_data {
// Create texture from thumbnail // Create texture from thumbnail
let color_image = egui::ColorImage::from_rgba_unmultiplied( let color_image = egui::ColorImage::from_rgba_unmultiplied(
[thumb_width as usize, thumb_height as usize], [thumb_width as usize, thumb_height as usize],

View File

@ -9,6 +9,7 @@ use crate::config::AppConfig;
use crate::keymap::{self, AppAction, KeymapManager}; use crate::keymap::{self, AppAction, KeymapManager};
use crate::menu::{MenuSystem, Shortcut, ShortcutKey}; use crate::menu::{MenuSystem, Shortcut, ShortcutKey};
use crate::theme::{Theme, ThemeMode}; use crate::theme::{Theme, ThemeMode};
use lightningbeam_core::file_io::LargeMediaMode;
/// Which tab is selected in the preferences dialog /// Which tab is selected in the preferences dialog
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@ -59,6 +60,7 @@ struct PreferencesState {
debug: bool, debug: bool,
waveform_stereo: bool, waveform_stereo: bool,
theme_mode: ThemeMode, theme_mode: ThemeMode,
large_media_default: LargeMediaMode,
} }
impl From<(&AppConfig, &Theme)> for PreferencesState { impl From<(&AppConfig, &Theme)> for PreferencesState {
@ -75,6 +77,7 @@ impl From<(&AppConfig, &Theme)> for PreferencesState {
debug: config.debug, debug: config.debug,
waveform_stereo: config.waveform_stereo, waveform_stereo: config.waveform_stereo,
theme_mode: theme.mode(), theme_mode: theme.mode(),
large_media_default: config.large_media_default,
} }
} }
} }
@ -93,6 +96,7 @@ impl Default for PreferencesState {
debug: false, debug: false,
waveform_stereo: false, waveform_stereo: false,
theme_mode: ThemeMode::System, theme_mode: ThemeMode::System,
large_media_default: LargeMediaMode::default(),
} }
} }
} }
@ -567,6 +571,24 @@ impl PreferencesDialog {
&mut self.working_prefs.waveform_stereo, &mut self.working_prefs.waveform_stereo,
"Show waveforms as stacked stereo", "Show waveforms as stacked stereo",
); );
ui.horizontal(|ui| {
let threshold_gb = lightningbeam_core::beam_archive::LARGE_MEDIA_THRESHOLD
as f64
/ (1024.0 * 1024.0 * 1024.0);
ui.label(format!("Large media (>{:.0} GB):", threshold_gb));
let label = |m: LargeMediaMode| match m {
LargeMediaMode::Ask => "Ask each time",
LargeMediaMode::Pack => "Pack into project",
LargeMediaMode::Reference => "Reference external file",
};
egui::ComboBox::from_id_salt("large_media_default")
.selected_text(label(self.working_prefs.large_media_default))
.show_ui(ui, |ui| {
for mode in [LargeMediaMode::Ask, LargeMediaMode::Pack, LargeMediaMode::Reference] {
ui.selectable_value(&mut self.working_prefs.large_media_default, mode, label(mode));
}
});
});
}); });
} }
@ -629,6 +651,7 @@ impl PreferencesDialog {
config.debug = self.working_prefs.debug; config.debug = self.working_prefs.debug;
config.waveform_stereo = self.working_prefs.waveform_stereo; config.waveform_stereo = self.working_prefs.waveform_stereo;
config.theme_mode = self.working_prefs.theme_mode.to_string_lower(); config.theme_mode = self.working_prefs.theme_mode.to_string_lower();
config.large_media_default = self.working_prefs.large_media_default;
config.keybindings = keybinding_config; config.keybindings = keybinding_config;
// Apply theme immediately // Apply theme immediately

View File

@ -112,6 +112,37 @@ pub struct PendingUpload {
/// The texture is allocated at full size, but total_frames is set to /// The texture is allocated at full size, but total_frames is set to
/// the limited count so subsequent calls use the incremental path. /// the limited count so subsequent calls use the incremental path.
pub frame_limit: Option<usize>, pub frame_limit: Option<usize>,
/// When true, `samples` is interpreted as **pre-packed min/max texels**:
/// 4 floats per "frame" = `[l_min, l_max, r_min, r_max]` (a waveform-pyramid
/// floor level). The caller passes the *effective* `sample_rate` (`sr / B`)
/// so the shader's time→texel mapping covers `B` source samples per texel.
/// When false, `samples` is raw interleaved audio (min = max per texel).
pub minmax: bool,
}
/// Pack one source "frame" into an `Rgba16Float` texel `(Lmin,Lmax,Rmin,Rmax)`.
/// Raw audio sets min = max per channel; min/max input copies the 4 values.
#[inline]
fn pack_texel(samples: &[f32], global_frame: usize, channels: usize, minmax: bool) -> [half::f16; 4] {
if minmax {
let o = global_frame * 4;
[
half::f16::from_f32(samples.get(o).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 1).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 2).copied().unwrap_or(0.0)),
half::f16::from_f32(samples.get(o + 3).copied().unwrap_or(0.0)),
]
} else {
let so = global_frame * channels;
let left = samples.get(so).copied().unwrap_or(0.0);
let right = if channels >= 2 {
samples.get(so + 1).copied().unwrap_or(left)
} else {
left
};
let (l, r) = (half::f16::from_f32(left), half::f16::from_f32(right));
[l, l, r, r]
}
} }
/// Maximum frames to convert and upload per frame (~250K frames ≈ 5.6s at 44.1kHz). /// Maximum frames to convert and upload per frame (~250K frames ≈ 5.6s at 44.1kHz).
@ -291,8 +322,11 @@ impl WaveformGpuResources {
sample_rate: u32, sample_rate: u32,
channels: u32, channels: u32,
frame_limit: Option<usize>, frame_limit: Option<usize>,
minmax: bool,
) -> Vec<wgpu::CommandBuffer> { ) -> Vec<wgpu::CommandBuffer> {
let new_total_frames = samples.len() / channels.max(1) as usize; // For min/max input each "frame" is 4 floats; for raw it's `channels`.
let frame_stride = if minmax { 4 } else { channels.max(1) as usize };
let new_total_frames = samples.len() / frame_stride;
if new_total_frames == 0 { if new_total_frames == 0 {
return Vec::new(); return Vec::new();
} }
@ -329,22 +363,9 @@ impl WaveformGpuResources {
if global_frame >= effective_frames { if global_frame >= effective_frames {
break; break;
} }
let sample_offset = global_frame * channels as usize;
let left = if sample_offset < samples.len() {
samples[sample_offset]
} else {
0.0
};
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
samples[sample_offset + 1]
} else {
left
};
let texel_offset = frame * 4; let texel_offset = frame * 4;
row_data[texel_offset] = half::f16::from_f32(left); let t = pack_texel(samples, global_frame, channels as usize, minmax);
row_data[texel_offset + 1] = half::f16::from_f32(left); row_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
row_data[texel_offset + 2] = half::f16::from_f32(right);
row_data[texel_offset + 3] = half::f16::from_f32(right);
} }
let entry = self.entries.get(&pool_index).unwrap(); let entry = self.entries.get(&pool_index).unwrap();
@ -466,24 +487,9 @@ impl WaveformGpuResources {
for frame in 0..seg_upload_count as usize { for frame in 0..seg_upload_count as usize {
let global_frame = seg_start_frame as usize + frame; let global_frame = seg_start_frame as usize + frame;
let sample_offset = global_frame * channels as usize;
let left = if sample_offset < samples.len() {
samples[sample_offset]
} else {
0.0
};
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
samples[sample_offset + 1]
} else {
left
};
let texel_offset = frame * 4; let texel_offset = frame * 4;
mip0_data[texel_offset] = half::f16::from_f32(left); let t = pack_texel(samples, global_frame, channels as usize, minmax);
mip0_data[texel_offset + 1] = half::f16::from_f32(left); mip0_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
mip0_data[texel_offset + 2] = half::f16::from_f32(right);
mip0_data[texel_offset + 3] = half::f16::from_f32(right);
} }
// Upload mip 0 (only rows with actual data) // Upload mip 0 (only rows with actual data)
@ -701,6 +707,7 @@ impl egui_wgpu::CallbackTrait for WaveformCallback {
upload.sample_rate, upload.sample_rate,
upload.channels, upload.channels,
upload.frame_limit, upload.frame_limit,
upload.minmax,
); );
cmds.extend(new_cmds); cmds.extend(new_cmds);
} }