Compare commits
No commits in common. "f585c370e8c692c562fdc74127ed8c4a7cf34fc7" and "83609cc9dc8ffcc1f372ffc4a5e0eeb8e2769d2f" have entirely different histories.
f585c370e8
...
83609cc9dc
|
|
@ -1,769 +0,0 @@
|
||||||
# Streaming Media To/From Disk — Plan
|
|
||||||
|
|
||||||
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
|
|
||||||
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
|
|
||||||
and to disk when the data is too large to fit comfortably in memory, rather than loading
|
|
||||||
the entire file regardless of size.
|
|
||||||
|
|
||||||
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
|
|
||||||
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
|
|
||||||
is impossible while the container forces a full decode, so the container decision (below)
|
|
||||||
is now part of this plan.
|
|
||||||
|
|
||||||
## Deferred bugs (do at the end)
|
|
||||||
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
|
|
||||||
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
|
|
||||||
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
|
|
||||||
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
|
|
||||||
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
|
|
||||||
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
|
|
||||||
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
|
|
||||||
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
|
|
||||||
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
|
|
||||||
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
|
|
||||||
|
|
||||||
## Deferred raster-keyframe-UI bugs (pre-existing; found during Phase 3 testing)
|
|
||||||
Both stem from the **timeline having no model for raster keyframes** — it renders *clip
|
|
||||||
instances* (`layer_clips`/`collect_clip_instances` return `&[ClipInstance]`), but raster layers use
|
|
||||||
`keyframes: Vec<RasterKeyframe>`; every raster arm in timeline.rs is a stub (`Raster => &[]`/`{}`).
|
|
||||||
- **(a) `Timeline > New Keyframe` doesn't refresh the canvas** until you draw. The menu action
|
|
||||||
creates a blank raster keyframe but doesn't trigger a canvas repaint / GPU-cache invalidation for
|
|
||||||
the new (different) `kf.id`, so the stale previous-frame texture stays until a stroke dirties it.
|
|
||||||
- **(b) Raster keyframes never render on the timeline** — no code walks `RasterLayer::keyframes` to
|
|
||||||
draw markers. Needs a raster-keyframe strip (display + click-to-navigate + insert).
|
|
||||||
Confirmed not paging regressions (same before 3a-1). A focused "raster keyframe timeline UI" task.
|
|
||||||
|
|
||||||
## Noted enhancements (later, after the phases)
|
|
||||||
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
|
|
||||||
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
|
|
||||||
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
|
|
||||||
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
|
|
||||||
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
|
|
||||||
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
|
|
||||||
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
|
|
||||||
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
|
|
||||||
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
|
|
||||||
Native multichannel support remains a separate, larger project.
|
|
||||||
- **Export speed:** a 1:14 1080p MP4 took ~9:06 to export (~7.4x slower than realtime). The video
|
|
||||||
export pipeline re-seeks + decodes per output frame (see `[Video Seek]`/`[Video Timing]` logs) and
|
|
||||||
does CPU YUV conversion; likely wins from sequential decode (avoid per-frame seeks), reusing the
|
|
||||||
decode cache, and/or GPU-side color conversion. Profile before optimizing.
|
|
||||||
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
|
|
||||||
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
|
|
||||||
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
|
|
||||||
decode) still worth chasing if it recurs.
|
|
||||||
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
|
|
||||||
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
|
|
||||||
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
|
|
||||||
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
|
|
||||||
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
|
|
||||||
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
|
|
||||||
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
|
|
||||||
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
|
|
||||||
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
|
|
||||||
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
|
|
||||||
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
|
|
||||||
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
|
|
||||||
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
|
|
||||||
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
|
|
||||||
2 h video keeps that work and continues from there on reload.
|
|
||||||
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
|
|
||||||
complete clips; a mid-generation save resumes from where it left off on reload.)*
|
|
||||||
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
|
|
||||||
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
|
|
||||||
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
|
|
||||||
on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap
|
|
||||||
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
|
|
||||||
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
|
|
||||||
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
|
|
||||||
absent). The 5 s interval already bounds count; no extra budget needed.
|
|
||||||
- **Progressive waveform on first import:** generation streams the whole file before the
|
|
||||||
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
|
|
||||||
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
|
|
||||||
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
|
|
||||||
the clip left-to-right instead of appearing all at once. Persistence saves only the final
|
|
||||||
complete pyramid.
|
|
||||||
|
|
||||||
## Guiding principle
|
|
||||||
Three subsystems already have the right streaming primitive; most of the work is wiring,
|
|
||||||
bounding caches, and adding a residency window. The recurring pattern:
|
|
||||||
|
|
||||||
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
|
|
||||||
> stable ID, and evict everything outside a window around the playhead.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Audit summary (where we stand today)
|
|
||||||
|
|
||||||
### Correctly streaming / bounded
|
|
||||||
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` —
|
|
||||||
keyframe-index seek + decode-until-target, one frame resident).
|
|
||||||
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
|
|
||||||
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
|
|
||||||
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
|
|
||||||
frames in flight).
|
|
||||||
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
|
|
||||||
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
|
|
||||||
|
|
||||||
### Fully-loaded, unbounded by file length (the problems)
|
|
||||||
| Site | Issue |
|
|
||||||
|---|---|
|
|
||||||
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
|
|
||||||
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
|
|
||||||
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
|
|
||||||
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
|
|
||||||
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
|
|
||||||
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
|
|
||||||
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
|
|
||||||
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
|
|
||||||
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Container format decision: `.beam` → SQLite *(DECIDED)*
|
|
||||||
|
|
||||||
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
|
|
||||||
`.beam` extension). This is the foundation the rest of the plan builds on.
|
|
||||||
|
|
||||||
### Why
|
|
||||||
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
|
|
||||||
mutation** — every save and every raster frame write-back rewrites the whole archive — and
|
|
||||||
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
|
|
||||||
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
|
|
||||||
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
|
|
||||||
|
|
||||||
SQLite dissolves the single-file-vs-performance tension:
|
|
||||||
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
|
|
||||||
confusion; we have no bundle magic on Linux/Windows).
|
|
||||||
- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
|
|
||||||
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
|
|
||||||
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
|
|
||||||
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
|
|
||||||
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
|
|
||||||
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
|
|
||||||
atomicity).
|
|
||||||
- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly.
|
|
||||||
|
|
||||||
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
|
|
||||||
reads (or external paths); raster frames live in blob rows and write back transactionally.
|
|
||||||
|
|
||||||
### Large-media policy: packed OR referenced
|
|
||||||
Two storage modes per media item, both supported:
|
|
||||||
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
|
|
||||||
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
|
|
||||||
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
|
|
||||||
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
|
|
||||||
on a network drive, or media too large/volatile to pack).
|
|
||||||
|
|
||||||
**Default-mode preference for files over the per-blob limit (~2GB):**
|
|
||||||
- A user preference `large_media_default: Pack | Reference` controls what happens to
|
|
||||||
imports above the threshold.
|
|
||||||
- The **first time** the user imports a media file over the limit, **prompt** them
|
|
||||||
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
|
|
||||||
large imports (changeable later in settings).
|
|
||||||
- Files under the limit are packed by default (chunked only if needed).
|
|
||||||
|
|
||||||
### Schema sketch
|
|
||||||
```
|
|
||||||
media(
|
|
||||||
id BLOB PRIMARY KEY, -- stable Uuid
|
|
||||||
kind INTEGER, -- audio | video | raster | image-asset
|
|
||||||
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
|
|
||||||
storage INTEGER, -- 0 = packed, 1 = referenced
|
|
||||||
ext_path TEXT, -- set when storage = referenced
|
|
||||||
total_len INTEGER, -- bytes (packed) for chunk math
|
|
||||||
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
|
|
||||||
)
|
|
||||||
media_chunk(
|
|
||||||
media_id BLOB, chunk_index INTEGER, bytes BLOB,
|
|
||||||
PRIMARY KEY (media_id, chunk_index)
|
|
||||||
)
|
|
||||||
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
|
|
||||||
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
|
|
||||||
```
|
|
||||||
`project.json` stays the same serialized `BeamProject` for now — only its container and the
|
|
||||||
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
|
|
||||||
first open/save.
|
|
||||||
|
|
||||||
### Streaming reads from packed media
|
|
||||||
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
|
|
||||||
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
|
|
||||||
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
|
|
||||||
plain `File` exactly as `do_import_audio` already does for originals today.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
|
|
||||||
|
|
||||||
### 1a. Turn on the compressed-audio disk reader
|
|
||||||
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
|
|
||||||
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
|
|
||||||
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
|
|
||||||
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
|
|
||||||
assign `AudioClip::read_ahead`.
|
|
||||||
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
|
|
||||||
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
|
|
||||||
from the ring buffer, not RAM.
|
|
||||||
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
|
|
||||||
|
|
||||||
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
|
|
||||||
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
|
|
||||||
already distinguishes.
|
|
||||||
|
|
||||||
### 1b. Stream on project load *(depends on the SQLite container)*
|
|
||||||
Three coupled changes (none works alone):
|
|
||||||
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
|
|
||||||
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
|
|
||||||
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
|
|
||||||
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
|
|
||||||
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
|
|
||||||
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
|
|
||||||
for them. After the engine installs the project, walk all audio clips and
|
|
||||||
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
|
|
||||||
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
|
|
||||||
`BlobReader` instead of a path for packed media.)
|
|
||||||
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
|
|
||||||
table instead of base64 `embedded_data`.
|
|
||||||
|
|
||||||
### 1c. Video's embedded audio track — stream from the video via ffmpeg
|
|
||||||
|
|
||||||
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
|
|
||||||
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
|
|
||||||
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
|
|
||||||
save/reload. **Superseded by the design below.**
|
|
||||||
|
|
||||||
**Proper design — stream the video's audio track on demand, never materialized.**
|
|
||||||
|
|
||||||
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
|
|
||||||
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
|
|
||||||
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
|
|
||||||
interface.
|
|
||||||
|
|
||||||
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
|
|
||||||
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
|
|
||||||
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
|
|
||||||
`sample_rate`/`channels`/`total_frames`.
|
|
||||||
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
|
|
||||||
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
|
|
||||||
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
|
|
||||||
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
|
|
||||||
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
|
|
||||||
entry `path` = the video file.
|
|
||||||
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
|
|
||||||
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
|
|
||||||
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
|
|
||||||
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
|
|
||||||
One ffmpeg context + 3 s buffer per active clip instance.
|
|
||||||
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` →
|
|
||||||
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
|
|
||||||
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
|
|
||||||
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
|
|
||||||
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
|
|
||||||
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
|
|
||||||
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
|
|
||||||
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
|
|
||||||
cleanup.
|
|
||||||
|
|
||||||
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
|
|
||||||
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
|
|
||||||
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
|
|
||||||
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
|
|
||||||
frames before target; for the frame straddling target, drop its leading samples). After
|
|
||||||
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
|
|
||||||
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
|
|
||||||
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
|
|
||||||
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
|
|
||||||
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
|
|
||||||
while here, or at least flag.
|
|
||||||
|
|
||||||
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
|
|
||||||
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
|
|
||||||
|
|
||||||
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
|
|
||||||
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
|
|
||||||
seek (both readers) → waveform-peaks pass.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2 — Video: bound the caches *(small, isolated)*
|
|
||||||
|
|
||||||
### 2a. Bound `VideoManager.frame_cache`
|
|
||||||
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
|
|
||||||
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
|
|
||||||
|
|
||||||
### 2b. Stream the export mux
|
|
||||||
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
|
|
||||||
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
|
|
||||||
|
|
||||||
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
|
|
||||||
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
|
|
||||||
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
|
|
||||||
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
|
|
||||||
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
|
|
||||||
the container is opened only at load/save (no live handle).
|
|
||||||
|
|
||||||
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
|
|
||||||
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
|
|
||||||
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
|
|
||||||
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
|
|
||||||
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
|
|
||||||
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
|
|
||||||
|
|
||||||
### Drive-by (Arc pixels): DROPPED
|
|
||||||
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
|
|
||||||
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
|
|
||||||
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
|
|
||||||
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
|
|
||||||
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
|
|
||||||
clone). Not worth 64 edits. Start at 3a.
|
|
||||||
|
|
||||||
### 3a. Lazy async fault-in + image proxy
|
|
||||||
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
|
|
||||||
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
|
|
||||||
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
|
|
||||||
results applied at top of `update()`. No UI block on cold scrub.
|
|
||||||
- **[DONE 3a-3]** Image proxy: `MediaKind::RasterProxy` (≤192px PNG, derived id), written
|
|
||||||
beside each resident full PNG on save + eager-decoded on load into `RasterKeyframe::proxy`.
|
|
||||||
Separate `proxy_layer_cache` (own LRU, budget 64); the raster render blits the proxy mapped to
|
|
||||||
the keyframe's FULL logical dims (upscales via sampler) when the full texture isn't resident.
|
|
||||||
*(Proxies exist only after a save+reload; eager decode → lazy/paged is a refinement for huge
|
|
||||||
paint projects.)*
|
|
||||||
|
|
||||||
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
|
|
||||||
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
|
|
||||||
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
|
|
||||||
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
|
|
||||||
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
|
|
||||||
leave full `raw_pixels` empty.
|
|
||||||
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
|
|
||||||
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
|
|
||||||
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
|
|
||||||
upscaled proxy. Reused by the exporter (already frame-by-frame).
|
|
||||||
|
|
||||||
### 3b. Residency window + eviction **[DONE]**
|
|
||||||
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
|
|
||||||
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
|
|
||||||
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
|
|
||||||
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
|
|
||||||
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
|
|
||||||
rows via `media_exists` (no data loss) and walks all layers to match load.
|
|
||||||
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
|
|
||||||
|
|
||||||
### 3c. Bound the GPU cache **[DONE for raster_layer_cache]**
|
|
||||||
`raster_layer_cache` (`gpu_brush.rs`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
|
|
||||||
≈ `w·h·16`/entry, was **unbounded**) → recency LRU (`RASTER_LAYER_CACHE_MAX = 12`) in
|
|
||||||
`ensure_layer_texture`: bump-to-most-recent + evict oldest; shown frames protected. F3 overlay
|
|
||||||
now shows tracked VRAM (raster cache MB + count). *(Refinements: count→byte budget; raise/headroom
|
|
||||||
if >12 raster layers are visible at once. Export `raster_cache` lives one export — fine. Vello
|
|
||||||
`ImageCache` is image *assets* → Phase 4.)*
|
|
||||||
|
|
||||||
### 3d. Undo memory **[DONE]**
|
|
||||||
`RasterStrokeAction`/`RasterFillAction` stored `buffer_before`+`buffer_after` full frames.
|
|
||||||
Now store a `RasterDiff` (`actions/raster_diff.rs`) — changed bbox before/after only, computed in
|
|
||||||
`new()`, full buffers dropped. Undo/redo apply onto the keyframe's resident pixels; the editor
|
|
||||||
faults the target frame in first (`Action::raster_resident_hint` + `peek_undo/redo_raster_hint`),
|
|
||||||
correct because a clean evicted frame's container bytes == its logical state. Non-resident base ⇒
|
|
||||||
skip (no corruption). Unit-tested round-trip. *(Refinement: compress full-canvas-fill diffs, whose
|
|
||||||
bbox is the whole frame.)*
|
|
||||||
|
|
||||||
### 3e. Prefetch frames **[DONE for playback]**
|
|
||||||
Implemented for playback: each update during playback, page in the next `PREFETCH_AHEAD=4`
|
|
||||||
upcoming keyframes per raster layer (reusing the async worker + `raster_loads_inflight` dedup), so
|
|
||||||
full frames are resident before the playhead arrives — fixes "proxy on every frame"/flicker during
|
|
||||||
playback. *(Caveat: with many simultaneous raster layers the 12-frame resident budget may evict a
|
|
||||||
prefetched frame before it's shown — raise budget or scale prefetch if that surfaces. Scrub-direction
|
|
||||||
prefetch still TODO.)*
|
|
||||||
|
|
||||||
Original note: *(future, after 3d — pure latency win, no correctness need)*
|
|
||||||
Fault-in is reactive (page in only on a render miss), so a never-visited frame still shows the
|
|
||||||
proxy for a beat before the full lands. **Prefetch the full pixels for frames about to be shown**:
|
|
||||||
on scrub/playback, dispatch background page-ins for the active keyframe ±N in the direction of
|
|
||||||
playhead motion (and during playback, the next K keyframes), reusing the 3a-2 async worker +
|
|
||||||
`raster_loads_inflight` dedup. Keep prefetched frames in the 3b LRU so they're still bounded; cap
|
|
||||||
concurrent prefetch loads so scrubbing fast doesn't thrash the disk. Optional: also prewarm the GPU
|
|
||||||
texture (3c cache) for the immediate next frame. Net effect: cold scrubbing/playback shows full-res
|
|
||||||
frames with no proxy flicker. Proxy stays as the instant fallback when prefetch can't keep up.
|
|
||||||
|
|
||||||
### Build order & tests
|
|
||||||
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
|
|
||||||
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
|
|
||||||
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
|
|
||||||
|
|
||||||
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
|
|
||||||
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
|
|
||||||
|
|
||||||
**Data model.**
|
|
||||||
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
|
|
||||||
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
|
|
||||||
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
|
|
||||||
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
|
|
||||||
`Pixmap` map, keyed by asset id, **unbounded**.
|
|
||||||
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
|
|
||||||
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
|
|
||||||
**No asset→frame or frame→asset index exists today.**
|
|
||||||
|
|
||||||
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
|
|
||||||
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
|
|
||||||
the heavy one).
|
|
||||||
|
|
||||||
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
|
|
||||||
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
|
|
||||||
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
|
|
||||||
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
|
|
||||||
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
|
|
||||||
|
|
||||||
### 4b. Usage bookkeeping (the multi-frame problem)
|
|
||||||
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
|
|
||||||
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
|
|
||||||
paths in `vector_graph` and the relevant actions).
|
|
||||||
- count 0 → dead, fully evictable / GC candidate.
|
|
||||||
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
|
|
||||||
playhead**, not by count (a high-count asset far from the playhead is still evicted).
|
|
||||||
|
|
||||||
Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte
|
|
||||||
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
|
|
||||||
Eviction never touches an asset in needed-now.
|
|
||||||
|
|
||||||
### 4c. Bound the decoded tier
|
|
||||||
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
|
|
||||||
image-texture cache the same way, keyed to the residency window.
|
|
||||||
|
|
||||||
### Nested-clip prefetch (important)
|
|
||||||
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
|
|
||||||
each of which can reference its own image assets. Prefetch must therefore:
|
|
||||||
- Recurse through clip instances when computing both needed-now and needed-soon.
|
|
||||||
- Map outer playhead time → each nested clip's local time, and look ahead along the
|
|
||||||
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
|
|
||||||
keyframe are loaded before the nested clip reaches it.
|
|
||||||
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
|
|
||||||
once); the usage index handles refcounting.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Cross-cutting: a shared residency abstraction
|
|
||||||
|
|
||||||
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
|
|
||||||
disk backing, residency = window/needed-set around playhead + LRU byte budget:
|
|
||||||
|
|
||||||
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
|
|
||||||
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
|
|
||||||
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
|
|
||||||
|
|
||||||
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
|
|
||||||
enumeration + usage index is unique to Phase 4.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Sequencing
|
|
||||||
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
|
|
||||||
2. **Phase 2** — small, isolated, independently shippable; container-independent.
|
|
||||||
3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy +
|
|
||||||
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
|
|
||||||
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
|
|
||||||
5. **Phase 1c** — depends on 1b's pool path.
|
|
||||||
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
|
|
||||||
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
|
|
||||||
|
|
||||||
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Status
|
|
||||||
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
|
|
||||||
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
|
|
||||||
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
|
|
||||||
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
|
|
||||||
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
|
|
||||||
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
|
|
||||||
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
|
|
||||||
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
|
|
||||||
buffer (was effectively silent before, since `read_ahead` was always `None`).
|
|
||||||
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
|
|
||||||
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
|
|
||||||
(bundled) — `lightningbeam-core/src/beam_archive.rs`
|
|
||||||
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
|
|
||||||
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
|
|
||||||
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
|
|
||||||
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
|
|
||||||
referenced-path, overwrite-replaces-chunks.
|
|
||||||
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
|
|
||||||
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
|
|
||||||
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
|
|
||||||
changed rows written; unchanged large media never rewritten); orphan cleanup via
|
|
||||||
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
|
|
||||||
must NOT copy+rename for existing SQLite files.
|
|
||||||
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
|
|
||||||
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
|
|
||||||
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
|
|
||||||
deleted (now packs original bytes with their codec).
|
|
||||||
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
|
|
||||||
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
|
|
||||||
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
|
|
||||||
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
|
|
||||||
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
|
|
||||||
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
|
|
||||||
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
|
|
||||||
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
|
|
||||||
`Ask` behaves as `Reference` at save time.
|
|
||||||
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
|
|
||||||
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
|
|
||||||
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
|
|
||||||
import_audio/video/image) queues a one-time modal; choice persists to config.
|
|
||||||
Threshold shown in the modal is derived from the constant.
|
|
||||||
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
|
|
||||||
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
|
|
||||||
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
|
|
||||||
skip unchanged frames on in-place save (Phase 3).
|
|
||||||
|
|
||||||
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
|
|
||||||
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
|
|
||||||
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
|
|
||||||
> build the lib in normal mode. Worth fixing separately.
|
|
||||||
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
|
|
||||||
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
|
|
||||||
+ proper video-audio reload remain as noted follow-ups.
|
|
||||||
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** —
|
|
||||||
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
|
|
||||||
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
|
|
||||||
violation; referenced-only-first = leaves packed <2GB in RAM.)
|
|
||||||
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
|
|
||||||
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
|
|
||||||
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
|
|
||||||
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
|
|
||||||
never fires for loaded clips (they bypass `AddAudioClip`).
|
|
||||||
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
|
|
||||||
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
|
|
||||||
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
|
|
||||||
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
|
|
||||||
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
|
|
||||||
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
|
|
||||||
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
|
|
||||||
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
|
|
||||||
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
|
|
||||||
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
|
|
||||||
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
|
|
||||||
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
|
|
||||||
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
|
|
||||||
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
|
|
||||||
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory`
|
|
||||||
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
|
|
||||||
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
|
|
||||||
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
|
|
||||||
by the editor between `load_audio_pool` and `set_project`.
|
|
||||||
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
|
|
||||||
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
|
|
||||||
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
|
|
||||||
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
|
|
||||||
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
|
|
||||||
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` —
|
|
||||||
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
|
|
||||||
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
|
|
||||||
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
|
|
||||||
from the full in-RAM decode, which no longer happens). Added
|
|
||||||
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
|
|
||||||
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
|
|
||||||
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
|
|
||||||
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
|
|
||||||
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
|
|
||||||
overview now fills in shortly after load.
|
|
||||||
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
|
|
||||||
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
|
|
||||||
then instantly on subsequent loads once saved; RAM stays flat on a big project.
|
|
||||||
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
|
|
||||||
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
|
|
||||||
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
|
|
||||||
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
|
|
||||||
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
|
|
||||||
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
|
|
||||||
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
|
|
||||||
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
|
|
||||||
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
|
|
||||||
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
|
|
||||||
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
|
|
||||||
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
|
|
||||||
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
|
|
||||||
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
|
|
||||||
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
|
|
||||||
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
|
|
||||||
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
|
|
||||||
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
|
|
||||||
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
|
|
||||||
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
|
|
||||||
→ `from_compressed` placeholder). Higher risk (touches the working referenced path); packed
|
|
||||||
covers the common <2GB case first.
|
|
||||||
- [ ] **Deferred (follow-up): packed video streaming.** Let small videos be packed into the `.beam`
|
|
||||||
(a `MediaKind::Video` blob, `VideoClip` referencing it by id) and stream **both frames and audio**
|
|
||||||
from the DB blob via FFmpeg. ffmpeg-next has no custom-I/O wrapper, so this needs an
|
|
||||||
`AVIOContext`-over-`BlobReader` shim via raw FFI. **Decision (user):** that FFI wrapper lives in
|
|
||||||
its **own crate, version-pinned to the ffmpeg version**, isolating the unsafe + the ABI coupling.
|
|
||||||
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
|
|
||||||
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
|
|
||||||
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
|
|
||||||
small temp partitions) and the temp path doesn't survive reload.
|
|
||||||
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
|
|
||||||
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
|
|
||||||
reference survives save/load.
|
|
||||||
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
|
|
||||||
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
|
|
||||||
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
|
|
||||||
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
|
|
||||||
empty channel layout → must `set_channel_layout` before resampling, else swr returns
|
|
||||||
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
|
|
||||||
until wired. `VideoAudioReader` made `pub` for the integration test.
|
|
||||||
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
|
|
||||||
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
|
|
||||||
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
|
|
||||||
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
|
|
||||||
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
|
|
||||||
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
|
|
||||||
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
|
|
||||||
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
|
|
||||||
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
|
|
||||||
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
|
|
||||||
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
|
|
||||||
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
|
|
||||||
daw-backend + editor compile clean; VideoAudioReader tests still pass.
|
|
||||||
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
|
|
||||||
regression) and that an activated VideoAudio clip produces sound.
|
|
||||||
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` →
|
|
||||||
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
|
|
||||||
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
|
|
||||||
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
|
|
||||||
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
|
|
||||||
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
|
|
||||||
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
|
|
||||||
**→ Feature is live end-to-end; ready for in-app testing.**
|
|
||||||
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
|
|
||||||
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
|
|
||||||
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
|
|
||||||
whole packet is discarded). So compressed clips no longer drift vs video audio after
|
|
||||||
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
|
|
||||||
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
|
|
||||||
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
|
|
||||||
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
|
|
||||||
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
|
|
||||||
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
|
|
||||||
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
|
|
||||||
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` —
|
|
||||||
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
|
|
||||||
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
|
|
||||||
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
|
|
||||||
Ragged last buckets reduce over available children (no value padding). Bounded
|
|
||||||
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
|
|
||||||
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
|
|
||||||
ordering, stereo channels, size bound, chunk-agnostic.
|
|
||||||
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
|
|
||||||
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
|
|
||||||
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
|
|
||||||
memory (one chunk + the pyramid). Test: envelope matches the signal through
|
|
||||||
both backends.
|
|
||||||
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
|
|
||||||
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
|
|
||||||
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
|
|
||||||
- [ ] Orchestration (with 1c).
|
|
||||||
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
|
|
||||||
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
|
|
||||||
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
|
|
||||||
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
|
|
||||||
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
|
|
||||||
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
|
|
||||||
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
|
|
||||||
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
|
|
||||||
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
|
|
||||||
drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
|
|
||||||
- [x] Generation: on video-audio import Success, the same bg thread streams
|
|
||||||
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
|
|
||||||
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
|
|
||||||
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
|
|
||||||
main.rs construction) → `render_layers` → **both** render sites (collapsed-group
|
|
||||||
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
|
|
||||||
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
|
|
||||||
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
|
|
||||||
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
|
|
||||||
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
|
|
||||||
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
|
|
||||||
features hold position at every zoom and line up with playback.
|
|
||||||
See memory `waveform-shader-fractional-mip-offset`.
|
|
||||||
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
|
|
||||||
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
|
|
||||||
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
|
|
||||||
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
|
|
||||||
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
|
|
||||||
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
|
|
||||||
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
|
|
||||||
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
|
|
||||||
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
|
|
||||||
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
|
|
||||||
`register_loaded_videos` only loads frames (not audio), so there is no redundant
|
|
||||||
regeneration to suppress. Compiles clean across all three crates.
|
|
||||||
|
|
||||||
### Waveform LOD pyramid design (step 6)
|
|
||||||
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
|
|
||||||
in → per-sample; seamless between.
|
|
||||||
|
|
||||||
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
|
|
||||||
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
|
|
||||||
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
|
|
||||||
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
|
|
||||||
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
|
|
||||||
stored.
|
|
||||||
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
|
|
||||||
temp before first save). `B` is stored with it (preference is just the default for new gen).
|
|
||||||
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
|
|
||||||
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
|
|
||||||
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
|
|
||||||
cleared only after its children") — so rendering can always walk up to a resident ancestor;
|
|
||||||
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
|
|
||||||
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
|
|
||||||
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 —
|
|
||||||
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
|
|
||||||
above the floor it's a disk read; below it the span is already small.
|
|
||||||
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
|
|
||||||
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
|
|
||||||
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
|
|
||||||
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
|
|
||||||
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
|
|
||||||
|
|
||||||
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
|
|
||||||
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
|
|
||||||
viewports of fine tiles; persist pyramid in `.beam`.
|
|
||||||
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
|
|
||||||
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
|
|
||||||
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
|
|
||||||
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
|
|
||||||
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
|
|
||||||
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
|
|
||||||
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
|
|
||||||
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
|
|
||||||
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
|
|
||||||
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
|
|
||||||
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
|
|
||||||
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
|
|
||||||
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
|
|
||||||
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
|
|
||||||
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
|
|
||||||
in-app export to confirm A/V sync.)*
|
|
||||||
- [ ] Phase 3a — lazy raster fault-in from blob store
|
|
||||||
- [ ] Phase 3b — raster residency window + eviction
|
|
||||||
- [ ] Phase 3c — bound raster GPU/CPU caches
|
|
||||||
- [ ] Phase 3d — spill undo snapshots
|
|
||||||
- [ ] Phase 4a — frame→asset enumeration (recursive)
|
|
||||||
- [ ] Phase 4b — usage bookkeeping + LRU residency
|
|
||||||
- [ ] Phase 4c — bound decoded image tier
|
|
||||||
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
|
|
||||||
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
|
|
||||||
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
|
|
||||||
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
|
|
||||||
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
|
|
||||||
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
|
|
||||||
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.
|
|
||||||
|
|
@ -189,22 +189,22 @@ mod tests {
|
||||||
fn test_add_points_sorted() {
|
fn test_add_points_sorted() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(2.0), 0.5, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(2.0, 0.5, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(3.0), 0.8, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(3.0, 0.8, CurveType::Linear));
|
||||||
|
|
||||||
assert_eq!(lane.points().len(), 3);
|
assert_eq!(lane.points().len(), 3);
|
||||||
assert_eq!(lane.points()[0].time, Beats(1.0));
|
assert_eq!(lane.points()[0].time, 1.0);
|
||||||
assert_eq!(lane.points()[1].time, Beats(2.0));
|
assert_eq!(lane.points()[1].time, 2.0);
|
||||||
assert_eq!(lane.points()[2].time, Beats(3.0));
|
assert_eq!(lane.points()[2].time, 3.0);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_replace_point_at_same_time() {
|
fn test_replace_point_at_same_time() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 0.3, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 0.3, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
|
||||||
|
|
||||||
assert_eq!(lane.points().len(), 1);
|
assert_eq!(lane.points().len(), 1);
|
||||||
assert_eq!(lane.points()[0].value, 0.5);
|
assert_eq!(lane.points()[0].value, 0.5);
|
||||||
|
|
@ -214,59 +214,59 @@ mod tests {
|
||||||
fn test_linear_interpolation() {
|
fn test_linear_interpolation() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(0.0), 0.0, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(0.0, 0.0, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Linear));
|
||||||
|
|
||||||
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.0));
|
assert_eq!(lane.evaluate(0.0), Some(0.0));
|
||||||
assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
|
assert_eq!(lane.evaluate(0.5), Some(0.5));
|
||||||
assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
|
assert_eq!(lane.evaluate(1.0), Some(1.0));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_step_interpolation() {
|
fn test_step_interpolation() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Step));
|
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Step));
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 1.0, CurveType::Step));
|
lane.add_point(AutomationPoint::new(1.0, 1.0, CurveType::Step));
|
||||||
|
|
||||||
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
|
assert_eq!(lane.evaluate(0.0), Some(0.5));
|
||||||
assert_eq!(lane.evaluate(Beats(0.5)), Some(0.5));
|
assert_eq!(lane.evaluate(0.5), Some(0.5));
|
||||||
assert_eq!(lane.evaluate(Beats(0.99)), Some(0.5));
|
assert_eq!(lane.evaluate(0.99), Some(0.5));
|
||||||
assert_eq!(lane.evaluate(Beats(1.0)), Some(1.0));
|
assert_eq!(lane.evaluate(1.0), Some(1.0));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_evaluate_outside_range() {
|
fn test_evaluate_outside_range() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(2.0), 1.0, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(2.0, 1.0, CurveType::Linear));
|
||||||
|
|
||||||
// Before first point
|
// Before first point
|
||||||
assert_eq!(lane.evaluate(Beats(0.0)), Some(0.5));
|
assert_eq!(lane.evaluate(0.0), Some(0.5));
|
||||||
// After last point
|
// After last point
|
||||||
assert_eq!(lane.evaluate(Beats(3.0)), Some(1.0));
|
assert_eq!(lane.evaluate(3.0), Some(1.0));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_disabled_lane() {
|
fn test_disabled_lane() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(0.0), 0.5, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(0.0, 0.5, CurveType::Linear));
|
||||||
lane.enabled = false;
|
lane.enabled = false;
|
||||||
|
|
||||||
assert_eq!(lane.evaluate(Beats(0.0)), None);
|
assert_eq!(lane.evaluate(0.0), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_remove_point() {
|
fn test_remove_point() {
|
||||||
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
let mut lane = AutomationLane::new(0, ParameterId::TrackVolume);
|
||||||
|
|
||||||
lane.add_point(AutomationPoint::new(Beats(1.0), 0.5, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(1.0, 0.5, CurveType::Linear));
|
||||||
lane.add_point(AutomationPoint::new(Beats(2.0), 0.8, CurveType::Linear));
|
lane.add_point(AutomationPoint::new(2.0, 0.8, CurveType::Linear));
|
||||||
|
|
||||||
assert!(lane.remove_point_at_time(Beats(1.0), Beats(0.001)));
|
assert!(lane.remove_point_at_time(1.0, 0.001));
|
||||||
assert_eq!(lane.points().len(), 1);
|
assert_eq!(lane.points().len(), 1);
|
||||||
assert_eq!(lane.points()[0].time, Beats(2.0));
|
assert_eq!(lane.points()[0].time, 2.0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -307,96 +307,23 @@ impl ReadAheadBuffer {
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
/// Wraps a Symphonia decoder for streaming a single compressed audio file.
|
/// Wraps a Symphonia decoder for streaming a single compressed audio file.
|
||||||
///
|
struct CompressedReader {
|
||||||
/// Public (like [`VideoAudioReader`]) only so integration tests can exercise it
|
|
||||||
/// directly; treat it as crate-internal.
|
|
||||||
pub struct CompressedReader {
|
|
||||||
format_reader: Box<dyn symphonia::core::formats::FormatReader>,
|
format_reader: Box<dyn symphonia::core::formats::FormatReader>,
|
||||||
decoder: Box<dyn symphonia::core::codecs::Decoder>,
|
decoder: Box<dyn symphonia::core::codecs::Decoder>,
|
||||||
track_id: u32,
|
track_id: u32,
|
||||||
/// Current decoder position in frames.
|
/// Current decoder position in frames.
|
||||||
current_frame: u64,
|
current_frame: u64,
|
||||||
/// Frames still to drop from the front of decoded output, so that after a
|
|
||||||
/// (coarse) seek the next emitted sample lands exactly on the target frame.
|
|
||||||
pending_discard: u64,
|
|
||||||
sample_rate: u32,
|
sample_rate: u32,
|
||||||
channels: u32,
|
channels: u32,
|
||||||
|
#[allow(dead_code)]
|
||||||
total_frames: u64,
|
total_frames: u64,
|
||||||
/// Temporary decode buffer.
|
/// Temporary decode buffer.
|
||||||
sample_buf: Option<SampleBuffer<f32>>,
|
sample_buf: Option<SampleBuffer<f32>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A seekable byte stream for packed media held in the host's project container.
|
|
||||||
///
|
|
||||||
/// `daw-backend` stays container-agnostic: it never references the `.beam` SQLite
|
|
||||||
/// store directly. Instead the host (lightningbeam-core) implements this trait over
|
|
||||||
/// its incremental blob reader and installs a factory ([`AudioBlobSourceFactory`])
|
|
||||||
/// into the engine, so packed compressed audio can be stream-decoded without ever
|
|
||||||
/// being fully loaded into RAM.
|
|
||||||
pub trait MediaByteSource: std::io::Read + std::io::Seek + Send + Sync {
|
|
||||||
/// Total length of the stream in bytes (Symphonia needs this for seeking).
|
|
||||||
fn byte_len(&self) -> u64;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Opens fresh byte streams for packed media by id. Installed into the engine by
|
|
||||||
/// the host; invoked when activating a clip backed by container-packed audio.
|
|
||||||
/// (`Debug` so it can ride in the `Query` enum, which derives `Debug`.)
|
|
||||||
pub trait AudioBlobSourceFactory: Send + Sync + std::fmt::Debug {
|
|
||||||
/// Open a new independent reader for the packed media item `media_id`
|
|
||||||
/// (the UUID string stored on the audio pool entry).
|
|
||||||
fn open(&self, media_id: &str) -> Result<Box<dyn MediaByteSource>, String>;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Adapts a [`MediaByteSource`] to Symphonia's `MediaSource` (adds the seekable +
|
|
||||||
/// byte-length metadata Symphonia's probe/seek require).
|
|
||||||
struct SymphoniaByteSource(Box<dyn MediaByteSource>);
|
|
||||||
|
|
||||||
impl std::io::Read for SymphoniaByteSource {
|
|
||||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
|
||||||
self.0.read(buf)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl std::io::Seek for SymphoniaByteSource {
|
|
||||||
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
|
|
||||||
self.0.seek(pos)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl symphonia::core::io::MediaSource for SymphoniaByteSource {
|
|
||||||
fn is_seekable(&self) -> bool {
|
|
||||||
true
|
|
||||||
}
|
|
||||||
fn byte_len(&self) -> Option<u64> {
|
|
||||||
Some(self.0.byte_len())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// How to open a streaming audio source: a filesystem path (referenced media or a
|
|
||||||
/// video file) or a host-provided byte stream (container-packed media).
|
|
||||||
pub enum StreamOpen {
|
|
||||||
Path(PathBuf),
|
|
||||||
Source {
|
|
||||||
src: Box<dyn MediaByteSource>,
|
|
||||||
/// Codec/extension hint for the Symphonia probe (e.g. `"mp3"`, `"flac"`).
|
|
||||||
ext: Option<String>,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
impl CompressedReader {
|
impl CompressedReader {
|
||||||
pub fn sample_rate(&self) -> u32 {
|
|
||||||
self.sample_rate
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn channels(&self) -> u32 {
|
|
||||||
self.channels
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Total frames from the codec header (0 if the format doesn't report it).
|
|
||||||
pub fn total_frames(&self) -> u64 {
|
|
||||||
self.total_frames
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Open a compressed audio file and prepare for streaming decode.
|
/// Open a compressed audio file and prepare for streaming decode.
|
||||||
pub fn open(path: &Path) -> Result<Self, String> {
|
fn open(path: &Path) -> Result<Self, String> {
|
||||||
let file =
|
let file =
|
||||||
std::fs::File::open(path).map_err(|e| format!("Failed to open file: {}", e))?;
|
std::fs::File::open(path).map_err(|e| format!("Failed to open file: {}", e))?;
|
||||||
let mss = MediaSourceStream::new(Box::new(file), Default::default());
|
let mss = MediaSourceStream::new(Box::new(file), Default::default());
|
||||||
|
|
@ -405,21 +332,7 @@ impl CompressedReader {
|
||||||
if let Some(ext) = path.extension().and_then(|e| e.to_str()) {
|
if let Some(ext) = path.extension().and_then(|e| e.to_str()) {
|
||||||
hint.with_extension(ext);
|
hint.with_extension(ext);
|
||||||
}
|
}
|
||||||
Self::from_mss(mss, hint)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Open a compressed stream from a host-provided byte source (packed media).
|
|
||||||
pub fn open_source(src: Box<dyn MediaByteSource>, ext: Option<&str>) -> Result<Self, String> {
|
|
||||||
let mss = MediaSourceStream::new(Box::new(SymphoniaByteSource(src)), Default::default());
|
|
||||||
let mut hint = Hint::new();
|
|
||||||
if let Some(ext) = ext {
|
|
||||||
hint.with_extension(ext);
|
|
||||||
}
|
|
||||||
Self::from_mss(mss, hint)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Shared probe + decoder setup over an already-built media stream.
|
|
||||||
fn from_mss(mss: MediaSourceStream, hint: Hint) -> Result<Self, String> {
|
|
||||||
let probed = symphonia::default::get_probe()
|
let probed = symphonia::default::get_probe()
|
||||||
.format(
|
.format(
|
||||||
&hint,
|
&hint,
|
||||||
|
|
@ -455,7 +368,6 @@ impl CompressedReader {
|
||||||
decoder,
|
decoder,
|
||||||
track_id,
|
track_id,
|
||||||
current_frame: 0,
|
current_frame: 0,
|
||||||
pending_discard: 0,
|
|
||||||
sample_rate,
|
sample_rate,
|
||||||
channels,
|
channels,
|
||||||
total_frames,
|
total_frames,
|
||||||
|
|
@ -463,17 +375,9 @@ impl CompressedReader {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Seek to `target_frame`, **sample-accurately**. Uses `SeekMode::Accurate`:
|
/// Seek to a specific frame. Returns the actual frame reached (may differ
|
||||||
/// for an elementary stream like MP3 a *coarse* seek byte-estimates the
|
/// for compressed formats that can only seek to keyframes).
|
||||||
/// position and seeds the timestamp from that estimate — which for VBR (or a
|
fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
|
||||||
/// file whose header padding the estimate ignores) lands off by up to ~1s.
|
|
||||||
/// Accurate mode instead counts frame *headers* (no decode) from a true anchor
|
|
||||||
/// (the current position, or a rewind to the start for backward seeks), so the
|
|
||||||
/// returned `actual_ts` is exact; the small residual to `target_frame` is then
|
|
||||||
/// dropped in `decode_next`. Container formats with seek tables (FLAC/OGG) seek
|
|
||||||
/// cheaply; a long MP3 walks headers from the anchor (I/O, not decode) — a
|
|
||||||
/// per-file seek index would make that O(1) (future work).
|
|
||||||
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
|
|
||||||
let seek_to = SeekTo::TimeStamp {
|
let seek_to = SeekTo::TimeStamp {
|
||||||
ts: target_frame,
|
ts: target_frame,
|
||||||
track_id: self.track_id,
|
track_id: self.track_id,
|
||||||
|
|
@ -481,23 +385,21 @@ impl CompressedReader {
|
||||||
|
|
||||||
let seeked = self
|
let seeked = self
|
||||||
.format_reader
|
.format_reader
|
||||||
.seek(SeekMode::Accurate, seek_to)
|
.seek(SeekMode::Coarse, seek_to)
|
||||||
.map_err(|e| format!("Seek failed: {}", e))?;
|
.map_err(|e| format!("Seek failed: {}", e))?;
|
||||||
|
|
||||||
let actual_frame = seeked.actual_ts;
|
let actual_frame = seeked.actual_ts;
|
||||||
self.current_frame = actual_frame;
|
self.current_frame = actual_frame;
|
||||||
// Drop the frames between where the coarse seek landed and the target.
|
|
||||||
self.pending_discard = target_frame.saturating_sub(actual_frame);
|
|
||||||
|
|
||||||
// Reset the decoder after seeking.
|
// Reset the decoder after seeking.
|
||||||
self.decoder.reset();
|
self.decoder.reset();
|
||||||
|
|
||||||
Ok(target_frame)
|
Ok(actual_frame)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Decode the next chunk of audio into `out`. Returns the number of frames
|
/// Decode the next chunk of audio into `out`. Returns the number of frames
|
||||||
/// decoded. Returns `Ok(0)` at end-of-file.
|
/// decoded. Returns `Ok(0)` at end-of-file.
|
||||||
pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
|
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
|
||||||
out.clear();
|
out.clear();
|
||||||
|
|
||||||
loop {
|
loop {
|
||||||
|
|
@ -526,22 +428,10 @@ impl CompressedReader {
|
||||||
if let Some(ref mut buf) = self.sample_buf {
|
if let Some(ref mut buf) = self.sample_buf {
|
||||||
buf.copy_interleaved_ref(decoded);
|
buf.copy_interleaved_ref(decoded);
|
||||||
let samples = buf.samples();
|
let samples = buf.samples();
|
||||||
let ch = self.channels as usize;
|
out.extend_from_slice(samples);
|
||||||
let frames = samples.len() / ch;
|
let frames = samples.len() / self.channels as usize;
|
||||||
|
self.current_frame += frames as u64;
|
||||||
// Drop leading frames for sample-accurate seek alignment.
|
return Ok(frames);
|
||||||
let discard = self.pending_discard.min(frames as u64) as usize;
|
|
||||||
self.pending_discard -= discard as u64;
|
|
||||||
out.extend_from_slice(&samples[discard * ch..]);
|
|
||||||
let emitted = frames - discard;
|
|
||||||
self.current_frame += emitted as u64;
|
|
||||||
|
|
||||||
if emitted > 0 {
|
|
||||||
return Ok(emitted);
|
|
||||||
}
|
|
||||||
// Whole packet discarded for alignment — keep decoding so
|
|
||||||
// we never falsely report EOF (Ok(0)).
|
|
||||||
continue;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
|
|
@ -555,383 +445,16 @@ impl CompressedReader {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// VideoAudioReader
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/// Streams the audio track out of a media file (a video container, or any audio
|
|
||||||
/// file) using FFmpeg, decoding on demand. Mirrors [`CompressedReader`]'s
|
|
||||||
/// interface so the disk reader can drive either through [`StreamSource`].
|
|
||||||
///
|
|
||||||
/// Seeking is **sample-accurate**: after `seek(target)`, the next `decode_next`
|
|
||||||
/// yields samples beginning at exactly `target`. FFmpeg's container seek only
|
|
||||||
/// lands at-or-before the target, so we decode forward and discard the leading
|
|
||||||
/// samples to hit the frame precisely — this keeps video audio frame-synced with
|
|
||||||
/// other (mmap/in-memory) clips.
|
|
||||||
///
|
|
||||||
/// Public (vs. the private `CompressedReader`) only so integration tests can
|
|
||||||
/// exercise it directly; treat it as crate-internal.
|
|
||||||
pub struct VideoAudioReader {
|
|
||||||
input: ffmpeg_next::format::context::Input,
|
|
||||||
decoder: ffmpeg_next::decoder::Audio,
|
|
||||||
/// Built lazily from the first decoded frame's format/layout → interleaved f32.
|
|
||||||
resampler: Option<ffmpeg_next::software::resampling::Context>,
|
|
||||||
stream_index: usize,
|
|
||||||
/// Seconds per stream-timestamp unit.
|
|
||||||
time_base: f64,
|
|
||||||
sample_rate: u32,
|
|
||||||
channels: u32,
|
|
||||||
total_frames: u64,
|
|
||||||
/// Absolute frame index of the next sample `decode_next` will output.
|
|
||||||
current_frame: u64,
|
|
||||||
/// Frames still to drop from the front of decoded output (seek alignment).
|
|
||||||
pending_discard: u64,
|
|
||||||
/// When set, the next decoded frame establishes the discard needed to land on
|
|
||||||
/// this absolute target frame (sample-accurate seek).
|
|
||||||
align_to: Option<u64>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl VideoAudioReader {
|
|
||||||
pub fn sample_rate(&self) -> u32 {
|
|
||||||
self.sample_rate
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn channels(&self) -> u32 {
|
|
||||||
self.channels
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Estimated total audio frames (from the stream/container duration).
|
|
||||||
pub fn total_frames(&self) -> u64 {
|
|
||||||
self.total_frames
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn open(path: &Path) -> Result<Self, String> {
|
|
||||||
ffmpeg_next::init().map_err(|e| e.to_string())?;
|
|
||||||
let input = ffmpeg_next::format::input(&path)
|
|
||||||
.map_err(|e| format!("Failed to open media: {}", e))?;
|
|
||||||
|
|
||||||
// Pull stream scalars + build the decoder inside a scope so the stream
|
|
||||||
// borrow of `input` ends before we use `input` again.
|
|
||||||
let (stream_index, time_base, stream_duration, decoder) = {
|
|
||||||
let stream = input
|
|
||||||
.streams()
|
|
||||||
.best(ffmpeg_next::media::Type::Audio)
|
|
||||||
.ok_or_else(|| "No audio stream found".to_string())?;
|
|
||||||
let stream_index = stream.index();
|
|
||||||
let time_base = f64::from(stream.time_base());
|
|
||||||
let stream_duration = stream.duration();
|
|
||||||
let ctx = ffmpeg_next::codec::context::Context::from_parameters(stream.parameters())
|
|
||||||
.map_err(|e| e.to_string())?;
|
|
||||||
let decoder = ctx.decoder().audio().map_err(|e| e.to_string())?;
|
|
||||||
(stream_index, time_base, stream_duration, decoder)
|
|
||||||
};
|
|
||||||
|
|
||||||
let sample_rate = decoder.rate();
|
|
||||||
let channels = decoder.channels() as u32;
|
|
||||||
|
|
||||||
let duration_secs = if stream_duration > 0 {
|
|
||||||
stream_duration as f64 * time_base
|
|
||||||
} else if input.duration() > 0 {
|
|
||||||
input.duration() as f64 / f64::from(ffmpeg_next::ffi::AV_TIME_BASE)
|
|
||||||
} else {
|
|
||||||
0.0
|
|
||||||
};
|
|
||||||
let total_frames = (duration_secs * sample_rate as f64).max(0.0) as u64;
|
|
||||||
|
|
||||||
Ok(Self {
|
|
||||||
input,
|
|
||||||
decoder,
|
|
||||||
resampler: None,
|
|
||||||
stream_index,
|
|
||||||
time_base,
|
|
||||||
sample_rate,
|
|
||||||
channels,
|
|
||||||
total_frames,
|
|
||||||
current_frame: 0,
|
|
||||||
pending_discard: 0,
|
|
||||||
align_to: None,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
|
|
||||||
let seconds = target_frame as f64 / self.sample_rate.max(1) as f64;
|
|
||||||
let ts_av = (seconds * f64::from(ffmpeg_next::ffi::AV_TIME_BASE)) as i64;
|
|
||||||
// Seek to at-or-before the target (max_ts = ts_av) so we can decode
|
|
||||||
// forward to it exactly. ffmpeg-next's `seek` wants a bounded range.
|
|
||||||
self.input
|
|
||||||
.seek(ts_av, 0..(ts_av + 1))
|
|
||||||
.map_err(|e| format!("Seek failed: {}", e))?;
|
|
||||||
self.decoder.flush();
|
|
||||||
self.pending_discard = 0;
|
|
||||||
self.align_to = Some(target_frame);
|
|
||||||
self.current_frame = target_frame;
|
|
||||||
// We align to the exact frame below, so the effective position IS target.
|
|
||||||
Ok(target_frame)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
|
|
||||||
out.clear();
|
|
||||||
loop {
|
|
||||||
// Drain a decoded frame if one is ready.
|
|
||||||
let mut decoded = ffmpeg_next::frame::Audio::empty();
|
|
||||||
if self.decoder.receive_frame(&mut decoded).is_ok() {
|
|
||||||
self.ensure_layout(&mut decoded);
|
|
||||||
let n = self.emit(&decoded, out);
|
|
||||||
if n > 0 {
|
|
||||||
return Ok(n);
|
|
||||||
}
|
|
||||||
continue; // frame fully discarded by seek-alignment; keep going
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read one packet (owned), releasing the `input` borrow before decoding.
|
|
||||||
let packet = self.input.packets().next().map(|(_, p)| p);
|
|
||||||
match packet {
|
|
||||||
Some(packet) => {
|
|
||||||
if packet.stream() == self.stream_index {
|
|
||||||
self.decoder
|
|
||||||
.send_packet(&packet)
|
|
||||||
.map_err(|e| e.to_string())?;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => {
|
|
||||||
// EOF: flush and drain whatever remains.
|
|
||||||
let _ = self.decoder.send_eof();
|
|
||||||
let mut decoded = ffmpeg_next::frame::Audio::empty();
|
|
||||||
if self.decoder.receive_frame(&mut decoded).is_ok() {
|
|
||||||
self.ensure_layout(&mut decoded);
|
|
||||||
return Ok(self.emit(&decoded, out));
|
|
||||||
}
|
|
||||||
return Ok(0);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Decoders for some formats (e.g. raw mono WAV) leave the frame's channel
|
|
||||||
/// layout unset. The resampler needs a concrete layout that matches the
|
|
||||||
/// frame, so fill one in from the channel count when it's missing.
|
|
||||||
fn ensure_layout(&self, frame: &mut ffmpeg_next::frame::Audio) {
|
|
||||||
if frame.channel_layout().is_empty() {
|
|
||||||
frame.set_channel_layout(
|
|
||||||
ffmpeg_next::channel_layout::ChannelLayout::default(self.channels as i32),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Resample one decoded frame to interleaved f32, apply any pending
|
|
||||||
/// seek-alignment discard, append to `out`, return frames emitted.
|
|
||||||
fn emit(&mut self, frame: &ffmpeg_next::frame::Audio, out: &mut Vec<f32>) -> usize {
|
|
||||||
// `frame` already has a non-empty channel layout (set by `ensure_layout`
|
|
||||||
// before this call), so the resampler config and the actual frame agree
|
|
||||||
// — otherwise swr fails with AVERROR_INPUT_CHANGED.
|
|
||||||
if self.resampler.is_none() {
|
|
||||||
match ffmpeg_next::software::resampling::Context::get(
|
|
||||||
frame.format(),
|
|
||||||
frame.channel_layout(),
|
|
||||||
self.sample_rate,
|
|
||||||
ffmpeg_next::format::Sample::F32(ffmpeg_next::format::sample::Type::Packed),
|
|
||||||
frame.channel_layout(),
|
|
||||||
self.sample_rate,
|
|
||||||
) {
|
|
||||||
Ok(r) => self.resampler = Some(r),
|
|
||||||
Err(_) => return 0,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut resampled = ffmpeg_next::frame::Audio::empty();
|
|
||||||
if self
|
|
||||||
.resampler
|
|
||||||
.as_mut()
|
|
||||||
.unwrap()
|
|
||||||
.run(frame, &mut resampled)
|
|
||||||
.is_err()
|
|
||||||
{
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// The output is packed (interleaved) f32. Read it from the raw byte plane
|
|
||||||
// `data(0)` — its length is correct (`frames * channels * 4`), whereas
|
|
||||||
// `plane::<f32>(0)` is a known ffmpeg-next footgun that reports only
|
|
||||||
// `samples()` elements (ignoring channels) and would slice out of range
|
|
||||||
// for multi-channel audio.
|
|
||||||
let ch = self.channels.max(1) as usize;
|
|
||||||
let bytes = resampled.data(0);
|
|
||||||
let n_frames = (bytes.len() / 4) / ch;
|
|
||||||
if n_frames == 0 {
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// On the first frame after a seek, compute how many leading frames to
|
|
||||||
// drop so output begins exactly at the seek target.
|
|
||||||
if let Some(target) = self.align_to.take() {
|
|
||||||
let frame_start = self.pts_to_frame(frame.pts());
|
|
||||||
self.pending_discard = target.saturating_sub(frame_start);
|
|
||||||
}
|
|
||||||
|
|
||||||
let discard = (self.pending_discard.min(n_frames as u64)) as usize;
|
|
||||||
self.pending_discard -= discard as u64;
|
|
||||||
|
|
||||||
let start_byte = discard * ch * 4;
|
|
||||||
let end_byte = n_frames * ch * 4;
|
|
||||||
out.extend(
|
|
||||||
bytes[start_byte..end_byte]
|
|
||||||
.chunks_exact(4)
|
|
||||||
.map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]])),
|
|
||||||
);
|
|
||||||
let emitted = n_frames - discard;
|
|
||||||
self.current_frame += emitted as u64;
|
|
||||||
emitted
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Convert a stream PTS to an absolute audio frame index.
|
|
||||||
fn pts_to_frame(&self, pts: Option<i64>) -> u64 {
|
|
||||||
match pts {
|
|
||||||
Some(p) if p >= 0 => {
|
|
||||||
((p as f64 * self.time_base) * self.sample_rate as f64).round() as u64
|
|
||||||
}
|
|
||||||
_ => self.current_frame,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// StreamSource — dispatches the disk reader over either decoder backend.
|
|
||||||
// (Wired into the reader thread in a later step.)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/// Which decoder backend a streaming source uses.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
||||||
pub enum SourceKind {
|
|
||||||
/// Symphonia, for compressed audio files (MP3, FLAC, OGG, …).
|
|
||||||
CompressedAudio,
|
|
||||||
/// FFmpeg, for the audio track of a video container.
|
|
||||||
VideoAudio,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A streaming audio source backing one active clip: either Symphonia
|
|
||||||
/// ([`CompressedReader`]) or FFmpeg ([`VideoAudioReader`]).
|
|
||||||
enum StreamSource {
|
|
||||||
Compressed(CompressedReader),
|
|
||||||
Video(VideoAudioReader),
|
|
||||||
}
|
|
||||||
|
|
||||||
impl StreamSource {
|
|
||||||
fn open(open: StreamOpen, kind: SourceKind) -> Result<Self, String> {
|
|
||||||
match (kind, open) {
|
|
||||||
(SourceKind::CompressedAudio, StreamOpen::Path(p)) => {
|
|
||||||
Ok(StreamSource::Compressed(CompressedReader::open(&p)?))
|
|
||||||
}
|
|
||||||
(SourceKind::CompressedAudio, StreamOpen::Source { src, ext }) => {
|
|
||||||
Ok(StreamSource::Compressed(CompressedReader::open_source(src, ext.as_deref())?))
|
|
||||||
}
|
|
||||||
(SourceKind::VideoAudio, StreamOpen::Path(p)) => {
|
|
||||||
Ok(StreamSource::Video(VideoAudioReader::open(&p)?))
|
|
||||||
}
|
|
||||||
(SourceKind::VideoAudio, StreamOpen::Source { .. }) => {
|
|
||||||
Err("VideoAudio cannot be opened from a packed byte source".to_string())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn sample_rate(&self) -> u32 {
|
|
||||||
match self {
|
|
||||||
StreamSource::Compressed(r) => r.sample_rate,
|
|
||||||
StreamSource::Video(r) => r.sample_rate,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn channels(&self) -> u32 {
|
|
||||||
match self {
|
|
||||||
StreamSource::Compressed(r) => r.channels,
|
|
||||||
StreamSource::Video(r) => r.channels,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn seek(&mut self, target_frame: u64) -> Result<u64, String> {
|
|
||||||
match self {
|
|
||||||
StreamSource::Compressed(r) => r.seek(target_frame),
|
|
||||||
StreamSource::Video(r) => r.seek(target_frame),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn decode_next(&mut self, out: &mut Vec<f32>) -> Result<usize, String> {
|
|
||||||
match self {
|
|
||||||
StreamSource::Compressed(r) => r.decode_next(out),
|
|
||||||
StreamSource::Video(r) => r.decode_next(out),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn total_frames(&self) -> u64 {
|
|
||||||
match self {
|
|
||||||
StreamSource::Compressed(r) => r.total_frames(),
|
|
||||||
StreamSource::Video(r) => r.total_frames(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Decode a media source end-to-end and build its [`WaveformPyramid`] overview,
|
|
||||||
/// streaming — only one decode chunk plus the (bounded) pyramid are ever in
|
|
||||||
/// memory, never the full sample buffer. `floor_samples_per_texel` is the
|
|
||||||
/// finest-level resolution (see [`crate::audio::waveform_pyramid`]).
|
|
||||||
pub fn build_waveform_pyramid(
|
|
||||||
path: &Path,
|
|
||||||
kind: SourceKind,
|
|
||||||
floor_samples_per_texel: u32,
|
|
||||||
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
|
|
||||||
let src = StreamSource::open(StreamOpen::Path(path.to_path_buf()), kind)?;
|
|
||||||
build_pyramid_from_streamsource(src, floor_samples_per_texel)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Build a waveform pyramid from a host-provided byte source (container-packed
|
|
||||||
/// compressed audio) — the load-time counterpart of [`build_waveform_pyramid`]
|
|
||||||
/// for media that has no filesystem path.
|
|
||||||
pub fn build_waveform_pyramid_from_source(
|
|
||||||
src: Box<dyn MediaByteSource>,
|
|
||||||
ext: Option<&str>,
|
|
||||||
floor_samples_per_texel: u32,
|
|
||||||
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
|
|
||||||
let src = StreamSource::open(
|
|
||||||
StreamOpen::Source { src, ext: ext.map(|s| s.to_string()) },
|
|
||||||
SourceKind::CompressedAudio,
|
|
||||||
)?;
|
|
||||||
build_pyramid_from_streamsource(src, floor_samples_per_texel)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn build_pyramid_from_streamsource(
|
|
||||||
mut src: StreamSource,
|
|
||||||
floor_samples_per_texel: u32,
|
|
||||||
) -> Result<crate::audio::waveform_pyramid::WaveformPyramid, String> {
|
|
||||||
use crate::audio::waveform_pyramid::WaveformPyramidBuilder;
|
|
||||||
|
|
||||||
let channels = src.channels();
|
|
||||||
let mut builder = WaveformPyramidBuilder::new(channels, floor_samples_per_texel);
|
|
||||||
builder.reserve_for_frames(src.total_frames());
|
|
||||||
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
loop {
|
|
||||||
let frames = src.decode_next(&mut buf)?;
|
|
||||||
if frames == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
builder.push_interleaved(&buf);
|
|
||||||
}
|
|
||||||
Ok(builder.finish())
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// DiskReaderCommand
|
// DiskReaderCommand
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
/// Commands sent from the engine to the disk reader thread.
|
/// Commands sent from the engine to the disk reader thread.
|
||||||
pub enum DiskReaderCommand {
|
pub enum DiskReaderCommand {
|
||||||
/// Start streaming a file for a clip instance, using the decoder backend
|
/// Start streaming a compressed file for a clip instance.
|
||||||
/// selected by `kind` (compressed audio vs. a video's audio track). `open`
|
|
||||||
/// is either a filesystem path (referenced media / video) or a host-provided
|
|
||||||
/// byte stream (container-packed media).
|
|
||||||
ActivateFile {
|
ActivateFile {
|
||||||
reader_id: u64,
|
reader_id: u64,
|
||||||
open: StreamOpen,
|
path: PathBuf,
|
||||||
kind: SourceKind,
|
|
||||||
buffer: Arc<ReadAheadBuffer>,
|
buffer: Arc<ReadAheadBuffer>,
|
||||||
},
|
},
|
||||||
/// Stop streaming for a clip instance.
|
/// Stop streaming for a clip instance.
|
||||||
|
|
@ -1006,7 +529,7 @@ impl DiskReader {
|
||||||
mut command_rx: rtrb::Consumer<DiskReaderCommand>,
|
mut command_rx: rtrb::Consumer<DiskReaderCommand>,
|
||||||
running: Arc<AtomicBool>,
|
running: Arc<AtomicBool>,
|
||||||
) {
|
) {
|
||||||
let mut active_files: HashMap<u64, (StreamSource, Arc<ReadAheadBuffer>)> =
|
let mut active_files: HashMap<u64, (CompressedReader, Arc<ReadAheadBuffer>)> =
|
||||||
HashMap::new();
|
HashMap::new();
|
||||||
let mut decode_buf = Vec::with_capacity(8192);
|
let mut decode_buf = Vec::with_capacity(8192);
|
||||||
|
|
||||||
|
|
@ -1016,19 +539,18 @@ impl DiskReader {
|
||||||
match cmd {
|
match cmd {
|
||||||
DiskReaderCommand::ActivateFile {
|
DiskReaderCommand::ActivateFile {
|
||||||
reader_id,
|
reader_id,
|
||||||
open,
|
path,
|
||||||
kind,
|
|
||||||
buffer,
|
buffer,
|
||||||
} => match StreamSource::open(open, kind) {
|
} => match CompressedReader::open(&path) {
|
||||||
Ok(reader) => {
|
Ok(reader) => {
|
||||||
eprintln!("[DiskReader] Activated reader={}, kind={:?}, ch={}, sr={}",
|
eprintln!("[DiskReader] Activated reader={}, ch={}, sr={}, path={:?}",
|
||||||
reader_id, kind, reader.channels(), reader.sample_rate());
|
reader_id, reader.channels, reader.sample_rate, path);
|
||||||
active_files.insert(reader_id, (reader, buffer));
|
active_files.insert(reader_id, (reader, buffer));
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
eprintln!(
|
eprintln!(
|
||||||
"[DiskReader] Failed to open reader={} ({:?}): {}",
|
"[DiskReader] Failed to open compressed file {:?}: {}",
|
||||||
reader_id, kind, e
|
path, e
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
@ -1066,7 +588,7 @@ impl DiskReader {
|
||||||
|
|
||||||
// If the target has jumped behind or far ahead of the buffer,
|
// If the target has jumped behind or far ahead of the buffer,
|
||||||
// seek the decoder and reset.
|
// seek the decoder and reset.
|
||||||
if target < buf_start || target > buf_end + reader.sample_rate() as u64 {
|
if target < buf_start || target > buf_end + reader.sample_rate as u64 {
|
||||||
buffer.reset(target);
|
buffer.reset(target);
|
||||||
let _ = reader.seek(target);
|
let _ = reader.seek(target);
|
||||||
continue;
|
continue;
|
||||||
|
|
@ -1085,7 +607,7 @@ impl DiskReader {
|
||||||
let buf_valid = buffer.valid_frames_count();
|
let buf_valid = buffer.valid_frames_count();
|
||||||
let buf_end = buf_start + buf_valid;
|
let buf_end = buf_start + buf_valid;
|
||||||
let prefetch_target =
|
let prefetch_target =
|
||||||
target + (PREFETCH_SECONDS * reader.sample_rate() as f64) as u64;
|
target + (PREFETCH_SECONDS * reader.sample_rate as f64) as u64;
|
||||||
|
|
||||||
if buf_end >= prefetch_target {
|
if buf_end >= prefetch_target {
|
||||||
continue; // Already filled far enough ahead.
|
continue; // Already filled far enough ahead.
|
||||||
|
|
@ -1127,7 +649,3 @@ impl Drop for DiskReader {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Tests for VideoAudioReader live in `daw-backend/tests/video_audio_stream.rs`
|
|
||||||
// (integration tests) so they build the lib in normal mode, independent of
|
|
||||||
// pre-existing breakage in the crate's `#[cfg(test)]` unit tests (automation.rs).
|
|
||||||
|
|
|
||||||
|
|
@ -91,10 +91,6 @@ pub struct Engine {
|
||||||
// Disk reader for streaming playback of compressed files
|
// Disk reader for streaming playback of compressed files
|
||||||
disk_reader: Option<crate::audio::disk_reader::DiskReader>,
|
disk_reader: Option<crate::audio::disk_reader::DiskReader>,
|
||||||
|
|
||||||
// Host-installed factory for opening container-packed audio as a byte stream
|
|
||||||
// (set on load, before the project's clips are bulk-activated for streaming).
|
|
||||||
blob_source_factory: Option<Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>>,
|
|
||||||
|
|
||||||
// Input monitoring and metering
|
// Input monitoring and metering
|
||||||
input_monitoring: bool,
|
input_monitoring: bool,
|
||||||
input_gain: f32,
|
input_gain: f32,
|
||||||
|
|
@ -180,7 +176,6 @@ impl Engine {
|
||||||
metronome: Metronome::new(sample_rate),
|
metronome: Metronome::new(sample_rate),
|
||||||
recording_sample_buffer: Vec::with_capacity(4096),
|
recording_sample_buffer: Vec::with_capacity(4096),
|
||||||
disk_reader: Some(disk_reader),
|
disk_reader: Some(disk_reader),
|
||||||
blob_source_factory: None,
|
|
||||||
input_monitoring: false,
|
input_monitoring: false,
|
||||||
input_gain: 1.0,
|
input_gain: 1.0,
|
||||||
input_level_peak: 0.0,
|
input_level_peak: 0.0,
|
||||||
|
|
@ -278,97 +273,6 @@ impl Engine {
|
||||||
|
|
||||||
/// Rebuild the clip snapshot from the current project state.
|
/// Rebuild the clip snapshot from the current project state.
|
||||||
/// Call this after any command that adds, removes, or modifies clip instances.
|
/// Call this after any command that adds, removes, or modifies clip instances.
|
||||||
/// Set up disk streaming for a clip backed by a streaming pool entry
|
|
||||||
/// (Compressed or VideoAudio). Sends `ActivateFile` to the disk reader and
|
|
||||||
/// returns the read-ahead buffer to attach to the clip. Returns `None` if the
|
|
||||||
/// entry isn't streamed, or a packed source can't be opened.
|
|
||||||
fn activate_streaming_for(
|
|
||||||
&mut self,
|
|
||||||
reader_id: u64,
|
|
||||||
pool_index: usize,
|
|
||||||
) -> Option<Arc<crate::audio::disk_reader::ReadAheadBuffer>> {
|
|
||||||
use crate::audio::pool::AudioStorage;
|
|
||||||
use crate::audio::disk_reader::{DiskReader, DiskReaderCommand, SourceKind, StreamOpen};
|
|
||||||
|
|
||||||
// Decide how to open the source. `Packed` ⇒ via the host factory (bytes in
|
|
||||||
// the container); otherwise stream from the file path. Extract owned values
|
|
||||||
// first so the immutable pool borrow ends before we touch the disk reader.
|
|
||||||
enum OpenDesc {
|
|
||||||
Path(std::path::PathBuf),
|
|
||||||
Packed { media_id: String, ext: Option<String> },
|
|
||||||
}
|
|
||||||
let (kind, sample_rate, channels, desc) = {
|
|
||||||
let file = self.audio_pool.get_file(pool_index)?;
|
|
||||||
let kind = match file.storage {
|
|
||||||
AudioStorage::Compressed { .. } => SourceKind::CompressedAudio,
|
|
||||||
AudioStorage::VideoAudio { .. } => SourceKind::VideoAudio,
|
|
||||||
_ => return None,
|
|
||||||
};
|
|
||||||
let desc = match &file.packed_media_id {
|
|
||||||
Some(id) => OpenDesc::Packed { media_id: id.clone(), ext: file.original_format.clone() },
|
|
||||||
None => OpenDesc::Path(file.path.clone()),
|
|
||||||
};
|
|
||||||
(kind, file.sample_rate, file.channels, desc)
|
|
||||||
};
|
|
||||||
|
|
||||||
let open = match desc {
|
|
||||||
OpenDesc::Path(p) => StreamOpen::Path(p),
|
|
||||||
OpenDesc::Packed { media_id, ext } => {
|
|
||||||
let factory = match self.blob_source_factory.as_ref() {
|
|
||||||
Some(f) => f,
|
|
||||||
None => {
|
|
||||||
eprintln!("[Engine] packed audio (pool {}) but no blob factory installed", pool_index);
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
match factory.open(&media_id) {
|
|
||||||
Ok(src) => StreamOpen::Source { src, ext },
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[Engine] blob factory open({}) failed: {}", media_id, e);
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let buffer = DiskReader::create_buffer(sample_rate, channels);
|
|
||||||
if let Some(ref mut dr) = self.disk_reader {
|
|
||||||
dr.send(DiskReaderCommand::ActivateFile { reader_id, open, kind, buffer: buffer.clone() });
|
|
||||||
}
|
|
||||||
Some(buffer)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Activate disk streaming for every loaded clip backed by a streaming pool
|
|
||||||
/// entry. Called after `SetProject` since loaded clips bypass `AddAudioClip`.
|
|
||||||
fn activate_all_streaming_clips(&mut self) {
|
|
||||||
use crate::audio::track::TrackNode;
|
|
||||||
// Collect (track, clip, pool) first via an immutable walk, then activate
|
|
||||||
// (needs &mut self) and attach the buffer back to the clip.
|
|
||||||
let targets: Vec<(TrackId, u64, usize)> = self
|
|
||||||
.project
|
|
||||||
.track_iter()
|
|
||||||
.filter_map(|(track_id, node)| match node {
|
|
||||||
TrackNode::Audio(t) => Some((track_id, t)),
|
|
||||||
_ => None,
|
|
||||||
})
|
|
||||||
.flat_map(|(track_id, t)| {
|
|
||||||
t.clips
|
|
||||||
.iter()
|
|
||||||
.map(move |c| (track_id, c.id as u64, c.audio_pool_index))
|
|
||||||
})
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
for (track_id, clip_id, pool_index) in targets {
|
|
||||||
if let Some(buffer) = self.activate_streaming_for(clip_id, pool_index) {
|
|
||||||
if let Some(TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
|
|
||||||
if let Some(clip) = track.clips.iter_mut().find(|c| c.id as u64 == clip_id) {
|
|
||||||
clip.read_ahead = Some(buffer);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn refresh_clip_snapshot(&self) {
|
fn refresh_clip_snapshot(&self) {
|
||||||
let mut snap = self.clip_snapshot.write().unwrap();
|
let mut snap = self.clip_snapshot.write().unwrap();
|
||||||
snap.audio.clear();
|
snap.audio.clear();
|
||||||
|
|
@ -1010,7 +914,7 @@ impl Engine {
|
||||||
let start_secs = self.tempo_map.beats_to_seconds(start_beats);
|
let start_secs = self.tempo_map.beats_to_seconds(start_beats);
|
||||||
let end_secs = self.tempo_map.beats_to_seconds(end_beats);
|
let end_secs = self.tempo_map.beats_to_seconds(end_beats);
|
||||||
let content_dur_secs = (end_secs - start_secs).seconds_to_f64();
|
let content_dur_secs = (end_secs - start_secs).seconds_to_f64();
|
||||||
let mut clip = AudioClipInstance::new(
|
let clip = AudioClipInstance::new(
|
||||||
clip_id,
|
clip_id,
|
||||||
pool_index,
|
pool_index,
|
||||||
Seconds(offset),
|
Seconds(offset),
|
||||||
|
|
@ -1019,13 +923,6 @@ impl Engine {
|
||||||
Beats(duration),
|
Beats(duration),
|
||||||
);
|
);
|
||||||
|
|
||||||
// If the source is streamed (a compressed audio file, or a video's
|
|
||||||
// audio track), set up disk streaming instead of an in-memory decode.
|
|
||||||
// Each clip instance gets its own read-ahead buffer keyed by clip_id.
|
|
||||||
if let Some(buffer) = self.activate_streaming_for(clip_id as u64, pool_index) {
|
|
||||||
clip.read_ahead = Some(buffer);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Add clip to track
|
// Add clip to track
|
||||||
if let Some(crate::audio::track::TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
|
if let Some(crate::audio::track::TrackNode::Audio(track)) = self.project.get_track_mut(track_id) {
|
||||||
track.clips.push(clip);
|
track.clips.push(clip);
|
||||||
|
|
@ -2416,34 +2313,6 @@ impl Engine {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Add a video file's audio track as a streaming pool entry (FFmpeg, decoded
|
|
||||||
/// on demand — no extraction). Probes the audio track for channels/rate/frames
|
|
||||||
/// without decoding, and returns the pool index. Playback activation
|
|
||||||
/// (per-clip read-ahead) is wired separately when a clip references it.
|
|
||||||
fn do_add_video_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
|
|
||||||
use crate::audio::disk_reader::VideoAudioReader;
|
|
||||||
|
|
||||||
let reader = VideoAudioReader::open(path)?;
|
|
||||||
let channels = reader.channels();
|
|
||||||
let sample_rate = reader.sample_rate();
|
|
||||||
let total_frames = reader.total_frames();
|
|
||||||
drop(reader);
|
|
||||||
|
|
||||||
let audio_file = crate::audio::pool::AudioFile::from_video_audio(
|
|
||||||
path.to_path_buf(),
|
|
||||||
channels,
|
|
||||||
sample_rate,
|
|
||||||
total_frames,
|
|
||||||
);
|
|
||||||
let pool_index = self.audio_pool.add_file(audio_file);
|
|
||||||
|
|
||||||
eprintln!(
|
|
||||||
"[ENGINE] AddVideoAudio: ch={}, sr={}, total_frames={}, pool_index={}, path={:?}",
|
|
||||||
channels, sample_rate, total_frames, pool_index, path
|
|
||||||
);
|
|
||||||
Ok(pool_index)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Import an audio file into the pool: mmap for PCM, streaming for compressed.
|
/// Import an audio file into the pool: mmap for PCM, streaming for compressed.
|
||||||
/// Returns the pool index on success. Emits AudioFileReady event.
|
/// Returns the pool index on success. Emits AudioFileReady event.
|
||||||
fn do_import_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
|
fn do_import_audio(&mut self, path: &std::path::Path) -> Result<usize, String> {
|
||||||
|
|
@ -2869,14 +2738,10 @@ impl Engine {
|
||||||
Query::GetPoolAudioSamples(pool_index) => {
|
Query::GetPoolAudioSamples(pool_index) => {
|
||||||
match self.audio_pool.get_file(pool_index) {
|
match self.audio_pool.get_file(pool_index) {
|
||||||
Some(file) => {
|
Some(file) => {
|
||||||
// For streamed (Compressed/VideoAudio) storage, return the
|
// For Compressed storage, return decoded_for_waveform if available
|
||||||
// progressively-decoded waveform overview if available.
|
|
||||||
let samples = match &file.storage {
|
let samples = match &file.storage {
|
||||||
crate::audio::pool::AudioStorage::Compressed {
|
crate::audio::pool::AudioStorage::Compressed {
|
||||||
decoded_for_waveform, decoded_frames, ..
|
decoded_for_waveform, decoded_frames, ..
|
||||||
}
|
|
||||||
| crate::audio::pool::AudioStorage::VideoAudio {
|
|
||||||
decoded_for_waveform, decoded_frames, ..
|
|
||||||
} if *decoded_frames > 0 => {
|
} if *decoded_frames > 0 => {
|
||||||
decoded_for_waveform.clone()
|
decoded_for_waveform.clone()
|
||||||
}
|
}
|
||||||
|
|
@ -2929,12 +2794,77 @@ impl Engine {
|
||||||
self.refresh_clip_snapshot();
|
self.refresh_clip_snapshot();
|
||||||
result
|
result
|
||||||
}
|
}
|
||||||
|
Query::AddAudioFileSync(path, data, channels, sample_rate) => {
|
||||||
|
// Add audio file to pool and return the pool index
|
||||||
|
// Detect original format from file extension
|
||||||
|
let path_buf = std::path::PathBuf::from(&path);
|
||||||
|
let original_format = path_buf.extension()
|
||||||
|
.and_then(|ext| ext.to_str())
|
||||||
|
.map(|s| s.to_lowercase());
|
||||||
|
|
||||||
|
// Create AudioFile and add to pool
|
||||||
|
let audio_file = crate::audio::pool::AudioFile::with_format(
|
||||||
|
path_buf.clone(),
|
||||||
|
data.clone(), // Clone data for background thread
|
||||||
|
channels,
|
||||||
|
sample_rate,
|
||||||
|
original_format,
|
||||||
|
);
|
||||||
|
let pool_index = self.audio_pool.add_file(audio_file);
|
||||||
|
|
||||||
|
// Generate Level 0 (overview) waveform chunks asynchronously in background thread
|
||||||
|
let chunk_tx = self.chunk_generation_tx.clone();
|
||||||
|
let duration = data.len() as f64 / (sample_rate as f64 * channels as f64);
|
||||||
|
println!("🔄 [ENGINE] Spawning background thread to generate Level 0 chunks for pool {}", pool_index);
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
// Create temporary AudioFile for chunk generation
|
||||||
|
let temp_audio_file = crate::audio::pool::AudioFile::with_format(
|
||||||
|
path_buf,
|
||||||
|
data,
|
||||||
|
channels,
|
||||||
|
sample_rate,
|
||||||
|
None,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Generate Level 0 chunks
|
||||||
|
let chunk_count = crate::audio::waveform_cache::WaveformCache::calculate_chunk_count(duration, 0);
|
||||||
|
println!("🔄 [BACKGROUND] Generating {} Level 0 chunks for pool {}", chunk_count, pool_index);
|
||||||
|
let chunks = crate::audio::waveform_cache::WaveformCache::generate_chunks(
|
||||||
|
&temp_audio_file,
|
||||||
|
pool_index,
|
||||||
|
0, // Level 0 (overview)
|
||||||
|
&(0..chunk_count).collect::<Vec<_>>(),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Send chunks via MPSC channel (will be forwarded by audio thread)
|
||||||
|
if !chunks.is_empty() {
|
||||||
|
println!("📤 [BACKGROUND] Generated {} chunks, sending to audio thread (pool {})", chunks.len(), pool_index);
|
||||||
|
let event_chunks: Vec<(u32, (f64, f64), Vec<crate::io::WaveformPeak>)> = chunks
|
||||||
|
.into_iter()
|
||||||
|
.map(|chunk| (chunk.chunk_index, chunk.time_range, chunk.peaks))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
match chunk_tx.send(AudioEvent::WaveformChunksReady {
|
||||||
|
pool_index,
|
||||||
|
detail_level: 0,
|
||||||
|
chunks: event_chunks,
|
||||||
|
}) {
|
||||||
|
Ok(_) => println!("✅ [BACKGROUND] Chunks sent successfully for pool {}", pool_index),
|
||||||
|
Err(e) => eprintln!("❌ [BACKGROUND] Failed to send chunks: {}", e),
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
eprintln!("⚠️ [BACKGROUND] No chunks generated for pool {}", pool_index);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Notify UI about the new audio file (for event listeners)
|
||||||
|
let _ = self.event_tx.push(AudioEvent::AudioFileAdded(pool_index, path));
|
||||||
|
|
||||||
|
QueryResponse::AudioFileAddedSync(Ok(pool_index))
|
||||||
|
}
|
||||||
Query::ImportAudioSync(path) => {
|
Query::ImportAudioSync(path) => {
|
||||||
QueryResponse::AudioImportedSync(self.do_import_audio(&path))
|
QueryResponse::AudioImportedSync(self.do_import_audio(&path))
|
||||||
}
|
}
|
||||||
Query::AddVideoAudioSync(path) => {
|
|
||||||
QueryResponse::AudioImportedSync(self.do_add_video_audio(&path))
|
|
||||||
}
|
|
||||||
Query::GetProject => {
|
Query::GetProject => {
|
||||||
// Save graph presets before cloning — AudioTrack::clone() creates
|
// Save graph presets before cloning — AudioTrack::clone() creates
|
||||||
// a fresh default graph (not a copy), so the preset must be populated
|
// a fresh default graph (not a copy), so the preset must be populated
|
||||||
|
|
@ -2949,19 +2879,11 @@ impl Engine {
|
||||||
match project.rebuild_audio_graphs(self.buffer_pool.buffer_size()) {
|
match project.rebuild_audio_graphs(self.buffer_pool.buffer_size()) {
|
||||||
Ok(()) => {
|
Ok(()) => {
|
||||||
self.project = project;
|
self.project = project;
|
||||||
// Loaded clips bypass AddAudioClip, so their disk streaming was
|
|
||||||
// never activated — do it now for every streaming-backed clip.
|
|
||||||
self.activate_all_streaming_clips();
|
|
||||||
self.refresh_clip_snapshot();
|
|
||||||
QueryResponse::ProjectSet(Ok(()))
|
QueryResponse::ProjectSet(Ok(()))
|
||||||
}
|
}
|
||||||
Err(e) => QueryResponse::ProjectSet(Err(format!("Failed to rebuild audio graphs: {}", e))),
|
Err(e) => QueryResponse::ProjectSet(Err(format!("Failed to rebuild audio graphs: {}", e))),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Query::SetBlobSourceFactory(factory) => {
|
|
||||||
self.blob_source_factory = Some(factory);
|
|
||||||
QueryResponse::BlobSourceFactorySet(Ok(()))
|
|
||||||
}
|
|
||||||
Query::DuplicateMidiClipSync(clip_id) => {
|
Query::DuplicateMidiClipSync(clip_id) => {
|
||||||
match self.project.midi_clip_pool.duplicate_clip(clip_id) {
|
match self.project.midi_clip_pool.duplicate_clip(clip_id) {
|
||||||
Some(new_id) => QueryResponse::MidiClipDuplicated(Ok(new_id)),
|
Some(new_id) => QueryResponse::MidiClipDuplicated(Ok(new_id)),
|
||||||
|
|
@ -3473,6 +3395,16 @@ impl EngineController {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Add an audio file to the pool synchronously and get the pool index
|
||||||
|
/// Returns the pool index where the audio file was added
|
||||||
|
pub fn add_audio_file_sync(&mut self, path: String, data: Vec<f32>, channels: u32, sample_rate: u32) -> Result<usize, String> {
|
||||||
|
let query = Query::AddAudioFileSync(path, data, channels, sample_rate);
|
||||||
|
match self.send_query(query)? {
|
||||||
|
QueryResponse::AudioFileAddedSync(result) => result,
|
||||||
|
_ => Err("Unexpected query response".to_string()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Import an audio file asynchronously. The engine will memory-map WAV/AIFF
|
/// Import an audio file asynchronously. The engine will memory-map WAV/AIFF
|
||||||
/// files for instant availability, or set up stream decoding for compressed
|
/// files for instant availability, or set up stream decoding for compressed
|
||||||
/// formats. Listen for `AudioEvent::AudioFileReady` to get the pool index.
|
/// formats. Listen for `AudioEvent::AudioFileReady` to get the pool index.
|
||||||
|
|
@ -3495,30 +3427,6 @@ impl EngineController {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Add a video file's audio track as a streaming pool entry (decoded on
|
|
||||||
/// demand via FFmpeg — no extraction to disk or RAM). Probes the audio track
|
|
||||||
/// and returns the pool index. Use this for a video clip's embedded audio.
|
|
||||||
pub fn add_video_audio_sync(&mut self, path: std::path::PathBuf) -> Result<usize, String> {
|
|
||||||
let query = Query::AddVideoAudioSync(path);
|
|
||||||
match self.send_query(query)? {
|
|
||||||
QueryResponse::AudioImportedSync(result) => result,
|
|
||||||
_ => Err("Unexpected query response".to_string()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Install the host's packed-media byte-source factory. Must be called before
|
|
||||||
/// loading a project so its container-packed audio can be streamed (the disk
|
|
||||||
/// reader opens packed entries through this factory).
|
|
||||||
pub fn set_blob_source_factory(
|
|
||||||
&mut self,
|
|
||||||
factory: std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
match self.send_query(Query::SetBlobSourceFactory(factory))? {
|
|
||||||
QueryResponse::BlobSourceFactorySet(result) => result,
|
|
||||||
_ => Err("Unexpected query response".to_string()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Generate the next unique audio clip instance ID (atomic, thread-safe)
|
/// Generate the next unique audio clip instance ID (atomic, thread-safe)
|
||||||
pub fn next_audio_clip_id(&self) -> AudioClipInstanceId {
|
pub fn next_audio_clip_id(&self) -> AudioClipInstanceId {
|
||||||
self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed)
|
self.next_audio_clip_id.fetch_add(1, Ordering::Relaxed)
|
||||||
|
|
|
||||||
|
|
@ -671,39 +671,14 @@ fn convert_chunk_to_planar_i16(interleaved: &[f32], channels: u32) -> Vec<Vec<i1
|
||||||
planar
|
planar
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Convert a chunk of interleaved f32 samples to planar f32 format.
|
/// Convert a chunk of interleaved f32 samples to planar f32 format
|
||||||
///
|
|
||||||
/// Non-finite samples (NaN/±Inf) are replaced with `0.0` and finite samples are
|
|
||||||
/// clamped to `[-1.0, 1.0]`: the float encoders (e.g. AAC, which takes `fltp`)
|
|
||||||
/// reject a frame outright on "(near) NaN/+-Inf", failing the whole export, so we
|
|
||||||
/// sanitize here exactly as the integer paths already clamp.
|
|
||||||
fn convert_chunk_to_planar_f32(interleaved: &[f32], channels: u32) -> Vec<Vec<f32>> {
|
fn convert_chunk_to_planar_f32(interleaved: &[f32], channels: u32) -> Vec<Vec<f32>> {
|
||||||
let num_frames = interleaved.len() / channels as usize;
|
let num_frames = interleaved.len() / channels as usize;
|
||||||
let mut planar = vec![vec![0.0f32; num_frames]; channels as usize];
|
let mut planar = vec![vec![0.0f32; num_frames]; channels as usize];
|
||||||
|
|
||||||
let mut non_finite = 0u64;
|
|
||||||
for (i, chunk) in interleaved.chunks(channels as usize).enumerate() {
|
for (i, chunk) in interleaved.chunks(channels as usize).enumerate() {
|
||||||
for (ch, &sample) in chunk.iter().enumerate() {
|
for (ch, &sample) in chunk.iter().enumerate() {
|
||||||
planar[ch][i] = if sample.is_finite() {
|
planar[ch][i] = sample;
|
||||||
sample.clamp(-1.0, 1.0)
|
|
||||||
} else {
|
|
||||||
non_finite += 1;
|
|
||||||
0.0
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if non_finite > 0 {
|
|
||||||
// One-time warning: we sanitized rather than failed, but a non-finite
|
|
||||||
// sample reaching here means something upstream (an effect, automation,
|
|
||||||
// or a source decode) produced NaN/Inf — worth chasing if audio is wrong.
|
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
|
||||||
static WARNED: AtomicBool = AtomicBool::new(false);
|
|
||||||
if !WARNED.swap(true, Ordering::Relaxed) {
|
|
||||||
eprintln!(
|
|
||||||
"⚠️ [EXPORT] sanitized {} non-finite (NaN/Inf) audio sample(s) in a chunk — \
|
|
||||||
check effects/automation/source decode",
|
|
||||||
non_finite
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,12 +15,10 @@ pub mod recording;
|
||||||
pub mod sample_loader;
|
pub mod sample_loader;
|
||||||
pub mod track;
|
pub mod track;
|
||||||
pub mod waveform_cache;
|
pub mod waveform_cache;
|
||||||
pub mod waveform_pyramid;
|
|
||||||
|
|
||||||
pub use automation::{AutomationLane, AutomationLaneId, AutomationPoint, CurveType, ParameterId};
|
pub use automation::{AutomationLane, AutomationLaneId, AutomationPoint, CurveType, ParameterId};
|
||||||
pub use buffer_pool::BufferPool;
|
pub use buffer_pool::BufferPool;
|
||||||
pub use clip::{AudioClipInstance, AudioClipInstanceId, Clip, ClipId};
|
pub use clip::{AudioClipInstance, AudioClipInstanceId, Clip, ClipId};
|
||||||
pub use disk_reader::{AudioBlobSourceFactory, MediaByteSource};
|
|
||||||
pub use engine::{AudioClipSnapshot, Engine, EngineController};
|
pub use engine::{AudioClipSnapshot, Engine, EngineController};
|
||||||
pub use export::{export_audio, ExportFormat, ExportSettings};
|
pub use export::{export_audio, ExportFormat, ExportSettings};
|
||||||
pub use metronome::Metronome;
|
pub use metronome::Metronome;
|
||||||
|
|
|
||||||
|
|
@ -4,46 +4,6 @@ use std::f32::consts::PI;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use crate::time::Seconds;
|
use crate::time::Seconds;
|
||||||
|
|
||||||
/// Per-output-channel mix coefficients to fold a multichannel source down to
|
|
||||||
/// stereo, indexed `[out_channel(0=L,1=R)][src_channel]`.
|
|
||||||
///
|
|
||||||
/// Assumes the conventional interleave order for each channel count (FL, FR, FC,
|
|
||||||
/// LFE, BL, BR, SL, SR …). Uses standard ITU/AC-3-style coefficients: full level
|
|
||||||
/// for the matching front channel, `1/√2` (≈ −3 dB) for centre and each surround,
|
|
||||||
/// LFE dropped. Each row is then normalized so its absolute-coefficient sum ≤ 1,
|
|
||||||
/// which prevents clipping (matching FFmpeg's default `normalize` behaviour).
|
|
||||||
///
|
|
||||||
/// Returns `None` for layouts we don't special-case (caller falls back to taking
|
|
||||||
/// the front L/R pair).
|
|
||||||
fn stereo_downmix_matrix(src_channels: usize) -> Option<[Vec<f32>; 2]> {
|
|
||||||
const C: f32 = std::f32::consts::FRAC_1_SQRT_2; // ≈ 0.7071
|
|
||||||
// (L row, R row); each entry is the gain applied to that source channel.
|
|
||||||
let (l, r): (Vec<f32>, Vec<f32>) = match src_channels {
|
|
||||||
3 => (vec![1.0, 0.0, C], vec![0.0, 1.0, C]), // FL FR FC
|
|
||||||
4 => (vec![1.0, 0.0, C, 0.0], vec![0.0, 1.0, 0.0, C]), // quad: FL FR BL BR
|
|
||||||
5 => (vec![1.0, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C]), // FL FR FC BL BR
|
|
||||||
// 5.1: FL FR FC LFE BL BR (LFE dropped)
|
|
||||||
6 => (vec![1.0, 0.0, C, 0.0, C, 0.0], vec![0.0, 1.0, C, 0.0, 0.0, C]),
|
|
||||||
// 6.1: FL FR FC LFE BC SL SR (BC → both)
|
|
||||||
7 => (vec![1.0, 0.0, C, 0.0, C, C, 0.0], vec![0.0, 1.0, C, 0.0, C, 0.0, C]),
|
|
||||||
// 7.1: FL FR FC LFE BL BR SL SR
|
|
||||||
8 => (
|
|
||||||
vec![1.0, 0.0, C, 0.0, C, 0.0, C, 0.0],
|
|
||||||
vec![0.0, 1.0, C, 0.0, 0.0, C, 0.0, C],
|
|
||||||
),
|
|
||||||
_ => return None,
|
|
||||||
};
|
|
||||||
let normalize = |row: Vec<f32>| -> Vec<f32> {
|
|
||||||
let sum: f32 = row.iter().map(|c| c.abs()).sum();
|
|
||||||
if sum > 1.0 {
|
|
||||||
row.into_iter().map(|c| c / sum).collect()
|
|
||||||
} else {
|
|
||||||
row
|
|
||||||
}
|
|
||||||
};
|
|
||||||
Some([normalize(l), normalize(r)])
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Windowed sinc interpolation for high-quality time stretching
|
/// Windowed sinc interpolation for high-quality time stretching
|
||||||
/// This is stateless and can handle arbitrary fractional positions
|
/// This is stateless and can handle arbitrary fractional positions
|
||||||
#[inline]
|
#[inline]
|
||||||
|
|
@ -123,16 +83,6 @@ pub enum AudioStorage {
|
||||||
decoded_frames: u64,
|
decoded_frames: u64,
|
||||||
total_frames: u64,
|
total_frames: u64,
|
||||||
},
|
},
|
||||||
|
|
||||||
/// Audio track of a video container, decoded on demand via FFmpeg
|
|
||||||
/// (`VideoAudioReader`). The source video file is `AudioFile::path`. Like
|
|
||||||
/// `Compressed`, playback is streamed through the disk reader and
|
|
||||||
/// `decoded_for_waveform` is filled progressively for the overview.
|
|
||||||
VideoAudio {
|
|
||||||
decoded_for_waveform: Vec<f32>,
|
|
||||||
decoded_frames: u64,
|
|
||||||
total_frames: u64,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Audio file stored in the pool
|
/// Audio file stored in the pool
|
||||||
|
|
@ -148,10 +98,6 @@ pub struct AudioFile {
|
||||||
pub original_format: Option<String>,
|
pub original_format: Option<String>,
|
||||||
/// Original compressed file bytes (preserved across save/load to avoid re-encoding)
|
/// Original compressed file bytes (preserved across save/load to avoid re-encoding)
|
||||||
pub original_bytes: Option<Vec<u8>>,
|
pub original_bytes: Option<Vec<u8>>,
|
||||||
/// When `Some`, this entry's bytes are packed in the project container (not on
|
|
||||||
/// disk at `path`); the disk reader opens them via the host's
|
|
||||||
/// `AudioBlobSourceFactory` using this media id. `None` ⇒ stream from `path`.
|
|
||||||
pub packed_media_id: Option<String>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AudioFile {
|
impl AudioFile {
|
||||||
|
|
@ -166,7 +112,6 @@ impl AudioFile {
|
||||||
frames,
|
frames,
|
||||||
original_format: None,
|
original_format: None,
|
||||||
original_bytes: None,
|
original_bytes: None,
|
||||||
packed_media_id: None,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -181,7 +126,6 @@ impl AudioFile {
|
||||||
frames,
|
frames,
|
||||||
original_format,
|
original_format,
|
||||||
original_bytes: None,
|
original_bytes: None,
|
||||||
packed_media_id: None,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -214,7 +158,6 @@ impl AudioFile {
|
||||||
frames: total_frames,
|
frames: total_frames,
|
||||||
original_format: Some("wav".to_string()),
|
original_format: Some("wav".to_string()),
|
||||||
original_bytes: None,
|
original_bytes: None,
|
||||||
packed_media_id: None,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -238,32 +181,6 @@ impl AudioFile {
|
||||||
frames: total_frames,
|
frames: total_frames,
|
||||||
original_format,
|
original_format,
|
||||||
original_bytes: None,
|
original_bytes: None,
|
||||||
packed_media_id: None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Create a placeholder AudioFile for a video's audio track. `path` is the
|
|
||||||
/// source video file; the audio is streamed on demand by the disk reader's
|
|
||||||
/// FFmpeg-backed `VideoAudioReader`.
|
|
||||||
pub fn from_video_audio(
|
|
||||||
path: PathBuf,
|
|
||||||
channels: u32,
|
|
||||||
sample_rate: u32,
|
|
||||||
total_frames: u64,
|
|
||||||
) -> Self {
|
|
||||||
Self {
|
|
||||||
path,
|
|
||||||
storage: AudioStorage::VideoAudio {
|
|
||||||
decoded_for_waveform: Vec::new(),
|
|
||||||
decoded_frames: 0,
|
|
||||||
total_frames,
|
|
||||||
},
|
|
||||||
channels,
|
|
||||||
sample_rate,
|
|
||||||
frames: total_frames,
|
|
||||||
original_format: None,
|
|
||||||
original_bytes: None,
|
|
||||||
packed_media_id: None,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -357,8 +274,8 @@ impl AudioFile {
|
||||||
}
|
}
|
||||||
written
|
written
|
||||||
}
|
}
|
||||||
AudioStorage::Compressed { .. } | AudioStorage::VideoAudio { .. } => {
|
AudioStorage::Compressed { .. } => {
|
||||||
// Streamed through the disk reader, not via read_samples().
|
// Compressed files are read through the disk reader
|
||||||
0
|
0
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -620,15 +537,6 @@ impl AudioClipPool {
|
||||||
let dst_channels = engine_channels as usize;
|
let dst_channels = engine_channels as usize;
|
||||||
let output_frames = output.len() / dst_channels;
|
let output_frames = output.len() / dst_channels;
|
||||||
|
|
||||||
// Fold a multichannel source (5.1, 7.1, …) down to stereo with proper
|
|
||||||
// coefficients (centre + surrounds mixed in, LFE dropped) instead of just
|
|
||||||
// taking the front L/R pair. `None` ⇒ no downmix needed / unknown layout.
|
|
||||||
let downmix = if dst_channels == 2 && src_channels > 2 {
|
|
||||||
stereo_downmix_matrix(src_channels)
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
|
|
||||||
let src_start_position = start_time_seconds * audio_file.sample_rate as f64;
|
let src_start_position = start_time_seconds * audio_file.sample_rate as f64;
|
||||||
|
|
||||||
// Tell the disk reader where we're reading so it buffers the right region.
|
// Tell the disk reader where we're reading so it buffers the right region.
|
||||||
|
|
@ -674,15 +582,6 @@ impl AudioClipPool {
|
||||||
sum += get_sample!(sf, src_ch);
|
sum += get_sample!(sf, src_ch);
|
||||||
}
|
}
|
||||||
sum / src_channels as f32
|
sum / src_channels as f32
|
||||||
} else if let Some(ref mat) = downmix {
|
|
||||||
// Surround → stereo with proper coefficients.
|
|
||||||
let mut s = 0.0f32;
|
|
||||||
for (src_ch, &c) in mat[dst_ch].iter().enumerate() {
|
|
||||||
if c != 0.0 {
|
|
||||||
s += c * get_sample!(sf, src_ch);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
s
|
|
||||||
} else {
|
} else {
|
||||||
get_sample!(sf, dst_ch % src_channels)
|
get_sample!(sf, dst_ch % src_channels)
|
||||||
};
|
};
|
||||||
|
|
@ -707,45 +606,39 @@ impl AudioClipPool {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Sinc-interpolate a single source channel at the current position.
|
|
||||||
macro_rules! sinc_ch {
|
|
||||||
($ch:expr) => {{
|
|
||||||
let mut channel_samples = [0.0f32; KERNEL_SIZE];
|
|
||||||
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
|
|
||||||
let idx = src_frame + i;
|
|
||||||
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
|
|
||||||
channel_samples[j] = get_sample!(idx as usize, $ch);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
windowed_sinc_interpolate(&channel_samples, frac)
|
|
||||||
}};
|
|
||||||
}
|
|
||||||
|
|
||||||
for dst_ch in 0..dst_channels {
|
for dst_ch in 0..dst_channels {
|
||||||
let sample = if let Some(ref mat) = downmix {
|
|
||||||
// Surround → stereo: interpolate each contributing channel.
|
|
||||||
let mut s = 0.0f32;
|
|
||||||
for (ch, &c) in mat[dst_ch].iter().enumerate() {
|
|
||||||
if c != 0.0 {
|
|
||||||
s += c * sinc_ch!(ch);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
s
|
|
||||||
} else if dst_channels == 1 {
|
|
||||||
let mut sum = 0.0;
|
|
||||||
for ch in 0..src_channels {
|
|
||||||
sum += sinc_ch!(ch);
|
|
||||||
}
|
|
||||||
sum / src_channels as f32
|
|
||||||
} else {
|
|
||||||
let src_ch = if src_channels == dst_channels {
|
let src_ch = if src_channels == dst_channels {
|
||||||
dst_ch
|
dst_ch
|
||||||
} else if src_channels == 1 {
|
} else if src_channels == 1 {
|
||||||
0
|
0
|
||||||
|
} else if dst_channels == 1 {
|
||||||
|
usize::MAX // sentinel: average all channels below
|
||||||
} else {
|
} else {
|
||||||
dst_ch % src_channels
|
dst_ch % src_channels
|
||||||
};
|
};
|
||||||
sinc_ch!(src_ch)
|
|
||||||
|
let sample = if src_ch == usize::MAX {
|
||||||
|
let mut sum = 0.0;
|
||||||
|
for ch in 0..src_channels {
|
||||||
|
let mut channel_samples = [0.0f32; KERNEL_SIZE];
|
||||||
|
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
|
||||||
|
let idx = src_frame + i;
|
||||||
|
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
|
||||||
|
channel_samples[j] = get_sample!(idx as usize, ch);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sum += windowed_sinc_interpolate(&channel_samples, frac);
|
||||||
|
}
|
||||||
|
sum / src_channels as f32
|
||||||
|
} else {
|
||||||
|
let mut channel_samples = [0.0f32; KERNEL_SIZE];
|
||||||
|
for (j, i) in (-(HALF_KERNEL as i32)..(HALF_KERNEL as i32)).enumerate() {
|
||||||
|
let idx = src_frame + i;
|
||||||
|
if idx >= 0 && (idx as usize) < audio_file.frames as usize {
|
||||||
|
channel_samples[j] = get_sample!(idx as usize, src_ch);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
windowed_sinc_interpolate(&channel_samples, frac)
|
||||||
};
|
};
|
||||||
|
|
||||||
output[output_frame * dst_channels + dst_ch] += sample * gain;
|
output[output_frame * dst_channels + dst_ch] += sample * gain;
|
||||||
|
|
@ -893,25 +786,6 @@ pub struct AudioPoolEntry {
|
||||||
pub channels: u32,
|
pub channels: u32,
|
||||||
/// Embedded audio data (for files < 10MB)
|
/// Embedded audio data (for files < 10MB)
|
||||||
pub embedded_data: Option<EmbeddedAudioData>,
|
pub embedded_data: Option<EmbeddedAudioData>,
|
||||||
/// Stable media id (UUID string) for the SQLite `.beam` container. When set,
|
|
||||||
/// the audio bytes live in the container's `media` table keyed by this id
|
|
||||||
/// (packed storage). `None` for referenced entries (use `relative_path`) or
|
|
||||||
/// legacy ZIP-loaded entries. Populated by the file_io save/load layer.
|
|
||||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
|
||||||
pub media_id: Option<String>,
|
|
||||||
/// Transient carrier for this entry's serialized waveform-pyramid blob (LBWF
|
|
||||||
/// bytes). Never serialized into project.json — the bytes live in the
|
|
||||||
/// container's `media` table (kind `Waveform`). Set by the file_io save layer
|
|
||||||
/// (in) and load layer (out); `None` everywhere else.
|
|
||||||
#[serde(skip)]
|
|
||||||
pub waveform_blob: Option<Vec<u8>>,
|
|
||||||
/// This entry is a video container's audio track (`relative_path` points at the
|
|
||||||
/// video file). It is always stored as a path reference (never packed/embedded
|
|
||||||
/// — the `VideoClip` already references the file) and reloaded by re-probing
|
|
||||||
/// the video via FFmpeg, so multichannel (5.1/7.1) audio survives the round-trip
|
|
||||||
/// (Symphonia reconstitution would otherwise collapse it).
|
|
||||||
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
|
|
||||||
pub is_video_audio: bool,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AudioClipPool {
|
impl AudioClipPool {
|
||||||
|
|
@ -926,66 +800,6 @@ impl AudioClipPool {
|
||||||
let mut entries = Vec::new();
|
let mut entries = Vec::new();
|
||||||
|
|
||||||
for (index, file) in self.files.iter().enumerate() {
|
for (index, file) in self.files.iter().enumerate() {
|
||||||
// Skip placeholder pool slots: `load_from_serialized` resizes the pool to
|
|
||||||
// `max_index + 1` filled with empty `AudioFile::new(PathBuf::new(), …)` to
|
|
||||||
// cover index gaps (and creates one even when there are no entries at all).
|
|
||||||
// Such a slot has an empty path and no packed media — there's nothing to
|
|
||||||
// persist, and emitting it yields an entry whose empty `relative_path`
|
|
||||||
// resolves to the project directory itself (unreadable on the next save).
|
|
||||||
if file.path.as_os_str().is_empty() && file.packed_media_id.is_none() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Video's audio track: reference the video file (it's also referenced
|
|
||||||
// by the VideoClip) and re-probe it via FFmpeg on load. Never pack or
|
|
||||||
// embed it as audio media — that both wastes space and loses the 5.1+
|
|
||||||
// layout when Symphonia later decodes it.
|
|
||||||
if matches!(file.storage, AudioStorage::VideoAudio { .. }) {
|
|
||||||
let relative_path = pathdiff::diff_paths(&file.path, project_dir)
|
|
||||||
.map(|r| r.to_string_lossy().to_string())
|
|
||||||
.or_else(|| Some(file.path.to_string_lossy().to_string()));
|
|
||||||
entries.push(AudioPoolEntry {
|
|
||||||
pool_index: index,
|
|
||||||
is_video_audio: true,
|
|
||||||
waveform_blob: None,
|
|
||||||
name: file
|
|
||||||
.path
|
|
||||||
.file_name()
|
|
||||||
.map(|n| n.to_string_lossy().to_string())
|
|
||||||
.unwrap_or_else(|| format!("file_{}", index)),
|
|
||||||
relative_path,
|
|
||||||
duration: file.duration_seconds(),
|
|
||||||
sample_rate: file.sample_rate,
|
|
||||||
channels: file.channels,
|
|
||||||
embedded_data: None,
|
|
||||||
media_id: None,
|
|
||||||
});
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Packed-in-container streaming entry: its bytes already live in the
|
|
||||||
// `.beam` media table (kept in place across re-saves). Emit just the
|
|
||||||
// media id — no path, no embedded bytes, nothing to decode.
|
|
||||||
if let Some(media_id) = &file.packed_media_id {
|
|
||||||
entries.push(AudioPoolEntry {
|
|
||||||
pool_index: index,
|
|
||||||
is_video_audio: false,
|
|
||||||
waveform_blob: None,
|
|
||||||
name: file
|
|
||||||
.path
|
|
||||||
.file_name()
|
|
||||||
.map(|n| n.to_string_lossy().to_string())
|
|
||||||
.unwrap_or_else(|| format!("file_{}", index)),
|
|
||||||
relative_path: None,
|
|
||||||
duration: file.duration_seconds(),
|
|
||||||
sample_rate: file.sample_rate,
|
|
||||||
channels: file.channels,
|
|
||||||
embedded_data: None,
|
|
||||||
media_id: Some(media_id.clone()),
|
|
||||||
});
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
let file_path = &file.path;
|
let file_path = &file.path;
|
||||||
let file_path_str = file_path.to_string_lossy();
|
let file_path_str = file_path.to_string_lossy();
|
||||||
|
|
||||||
|
|
@ -1016,8 +830,6 @@ impl AudioClipPool {
|
||||||
|
|
||||||
let entry = AudioPoolEntry {
|
let entry = AudioPoolEntry {
|
||||||
pool_index: index,
|
pool_index: index,
|
||||||
is_video_audio: false,
|
|
||||||
waveform_blob: None,
|
|
||||||
name: file_path
|
name: file_path
|
||||||
.file_name()
|
.file_name()
|
||||||
.map(|n| n.to_string_lossy().to_string())
|
.map(|n| n.to_string_lossy().to_string())
|
||||||
|
|
@ -1027,7 +839,6 @@ impl AudioClipPool {
|
||||||
sample_rate: file.sample_rate,
|
sample_rate: file.sample_rate,
|
||||||
channels: file.channels,
|
channels: file.channels,
|
||||||
embedded_data,
|
embedded_data,
|
||||||
media_id: None,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
entries.push(entry);
|
entries.push(entry);
|
||||||
|
|
@ -1151,86 +962,22 @@ impl AudioClipPool {
|
||||||
self.files.clear();
|
self.files.clear();
|
||||||
eprintln!("📊 [LOAD_SERIALIZED] Clear pool took {:.2}ms", clear_start.elapsed().as_secs_f64() * 1000.0);
|
eprintln!("📊 [LOAD_SERIALIZED] Clear pool took {:.2}ms", clear_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
// Size the pool to hold the highest pool_index (slots are addressed by index,
|
// Find the maximum pool index to determine required size
|
||||||
// so gaps are filled with placeholders). No entries → length 0, NOT 1: the old
|
let max_index = entries.iter()
|
||||||
// `max().unwrap_or(0) + 1` produced a spurious placeholder for an empty pool.
|
.map(|e| e.pool_index)
|
||||||
let pool_size = entries.iter()
|
|
||||||
.map(|e| e.pool_index + 1)
|
|
||||||
.max()
|
.max()
|
||||||
.unwrap_or(0);
|
.unwrap_or(0);
|
||||||
|
|
||||||
// Ensure we have space for all entries
|
// Ensure we have space for all entries
|
||||||
let resize_start = std::time::Instant::now();
|
let resize_start = std::time::Instant::now();
|
||||||
self.files.resize(pool_size, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100));
|
self.files.resize(max_index + 1, AudioFile::new(PathBuf::new(), Vec::new(), 2, 44100));
|
||||||
eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", pool_size, resize_start.elapsed().as_secs_f64() * 1000.0);
|
eprintln!("📊 [LOAD_SERIALIZED] Resize pool to {} took {:.2}ms", max_index + 1, resize_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
for (i, entry) in entries.iter().enumerate() {
|
for (i, entry) in entries.iter().enumerate() {
|
||||||
let entry_start = std::time::Instant::now();
|
let entry_start = std::time::Instant::now();
|
||||||
eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name);
|
eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name);
|
||||||
|
|
||||||
let success = if entry.is_video_audio {
|
let success = if let Some(ref embedded) = entry.embedded_data {
|
||||||
// Re-probe the video's audio track via FFmpeg → a streaming
|
|
||||||
// VideoAudio entry (keeps full 5.1/7.1; no decode-to-RAM).
|
|
||||||
match entry.relative_path.as_ref() {
|
|
||||||
Some(rel) => {
|
|
||||||
let full = if std::path::Path::new(rel).is_absolute() {
|
|
||||||
PathBuf::from(rel)
|
|
||||||
} else {
|
|
||||||
project_dir.join(rel)
|
|
||||||
};
|
|
||||||
if full.exists() {
|
|
||||||
match crate::audio::disk_reader::VideoAudioReader::open(&full) {
|
|
||||||
Ok(reader) => {
|
|
||||||
let file = AudioFile::from_video_audio(
|
|
||||||
full,
|
|
||||||
reader.channels(),
|
|
||||||
reader.sample_rate(),
|
|
||||||
reader.total_frames(),
|
|
||||||
);
|
|
||||||
if entry.pool_index < self.files.len() {
|
|
||||||
self.files[entry.pool_index] = file;
|
|
||||||
true
|
|
||||||
} else {
|
|
||||||
false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[AudioPool] Failed to reopen video audio {:?}: {}", full, e);
|
|
||||||
false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
eprintln!("[AudioPool] Video file not found for audio: {:?}", full);
|
|
||||||
false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => false,
|
|
||||||
}
|
|
||||||
} else if entry.media_id.is_some() && entry.embedded_data.is_none() {
|
|
||||||
// Packed-in-container streaming entry: build a Compressed placeholder
|
|
||||||
// backed by the host blob factory (opened at clip-activation time).
|
|
||||||
// No decode here — playback streams through the disk reader.
|
|
||||||
let media_id = entry.media_id.clone().unwrap();
|
|
||||||
let ext = std::path::Path::new(&entry.name)
|
|
||||||
.extension()
|
|
||||||
.and_then(|e| e.to_str())
|
|
||||||
.map(|s| s.to_lowercase());
|
|
||||||
let total_frames = (entry.duration * entry.sample_rate as f64).ceil() as u64;
|
|
||||||
let mut file = AudioFile::from_compressed(
|
|
||||||
PathBuf::from(&entry.name),
|
|
||||||
entry.channels,
|
|
||||||
entry.sample_rate,
|
|
||||||
total_frames,
|
|
||||||
ext,
|
|
||||||
);
|
|
||||||
file.packed_media_id = Some(media_id);
|
|
||||||
if entry.pool_index < self.files.len() {
|
|
||||||
self.files[entry.pool_index] = file;
|
|
||||||
true
|
|
||||||
} else {
|
|
||||||
false
|
|
||||||
}
|
|
||||||
} else if let Some(ref embedded) = entry.embedded_data {
|
|
||||||
// Load from embedded data
|
// Load from embedded data
|
||||||
eprintln!("📊 [LOAD_SERIALIZED] Entry has embedded data (format: {})", embedded.format);
|
eprintln!("📊 [LOAD_SERIALIZED] Entry has embedded data (format: {})", embedded.format);
|
||||||
match Self::load_from_embedded_into_pool(self, entry.pool_index, embedded.clone(), &entry.name) {
|
match Self::load_from_embedded_into_pool(self, entry.pool_index, embedded.clone(), &entry.name) {
|
||||||
|
|
|
||||||
|
|
@ -1,292 +0,0 @@
|
||||||
//! Streaming min/max waveform LOD pyramid.
|
|
||||||
//!
|
|
||||||
//! A waveform pyramid is a tree of zoom levels. **Index = tree depth:**
|
|
||||||
//! `levels[0]` is the **root** (a single texel — the min/max envelope of the
|
|
||||||
//! whole file, lowest resolution); each deeper level is `BRANCH`× finer, and
|
|
||||||
//! `levels.last()` is the **floor** (one texel per `floor_samples_per_texel`
|
|
||||||
//! source frames — the finest *persisted* level). A node's children live at
|
|
||||||
//! `index + 1`, so the residency invariant ("a node is cleared only after its
|
|
||||||
//! children") reads straight off the index.
|
|
||||||
//!
|
|
||||||
//! Below the floor (finer than the floor bucket) is *not* stored; the caller
|
|
||||||
//! re-decodes the source window on demand for true per-sample detail.
|
|
||||||
//!
|
|
||||||
//! The builder is **streaming**: samples are pushed once, in order, and only the
|
|
||||||
//! finest level is accumulated (~`total_frames / floor` texels); the coarser
|
|
||||||
//! levels are derived by repeated `BRANCH:1` min/max reduction in [`finish`].
|
|
||||||
//! This yields the identical pyramid to an in-stream cascade (each parent = the
|
|
||||||
//! min/max of its children) without ever holding the full sample buffer.
|
|
||||||
//!
|
|
||||||
//! **Ragged edges are handled by reducing over available children:** a bucket
|
|
||||||
//! whose group is partial (1..BRANCH children, or `< floor` samples at the floor)
|
|
||||||
//! simply takes the min/max of what's there — no value padding. Padding to a
|
|
||||||
//! regular shape, if ever needed, is a GPU-texture/tile concern, not the data's.
|
|
||||||
//!
|
|
||||||
//! Each texel carries per-channel min/max for up to two channels
|
|
||||||
//! (`Lmin,Lmax,Rmin,Rmax`), matching the GPU waveform texture; mono mirrors the
|
|
||||||
//! left channel into the right.
|
|
||||||
//!
|
|
||||||
//! [`finish`]: WaveformPyramidBuilder::finish
|
|
||||||
|
|
||||||
/// Reduction factor between adjacent pyramid levels.
|
|
||||||
pub const BRANCH: u32 = 4;
|
|
||||||
|
|
||||||
/// Default finest-level resolution (source frames per floor texel). Trades
|
|
||||||
/// on-disk pyramid size against how soon zoom-in must re-decode the source.
|
|
||||||
pub const DEFAULT_FLOOR_SAMPLES_PER_TEXEL: u32 = 256;
|
|
||||||
|
|
||||||
/// One waveform texel: per-channel min/max (stereo; mono duplicates left→right).
|
|
||||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
|
||||||
pub struct Texel {
|
|
||||||
pub l_min: f32,
|
|
||||||
pub l_max: f32,
|
|
||||||
pub r_min: f32,
|
|
||||||
pub r_max: f32,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Texel {
|
|
||||||
const EMPTY: Texel = Texel {
|
|
||||||
l_min: f32::INFINITY,
|
|
||||||
l_max: f32::NEG_INFINITY,
|
|
||||||
r_min: f32::INFINITY,
|
|
||||||
r_max: f32::NEG_INFINITY,
|
|
||||||
};
|
|
||||||
|
|
||||||
#[inline]
|
|
||||||
fn include_sample(&mut self, l: f32, r: f32) {
|
|
||||||
self.l_min = self.l_min.min(l);
|
|
||||||
self.l_max = self.l_max.max(l);
|
|
||||||
self.r_min = self.r_min.min(r);
|
|
||||||
self.r_max = self.r_max.max(r);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[inline]
|
|
||||||
fn include_texel(&mut self, c: &Texel) {
|
|
||||||
self.l_min = self.l_min.min(c.l_min);
|
|
||||||
self.l_max = self.l_max.max(c.l_max);
|
|
||||||
self.r_min = self.r_min.min(c.r_min);
|
|
||||||
self.r_max = self.r_max.max(c.r_max);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A built min/max LOD pyramid, **root-first**: `levels[0]` is the coarsest
|
|
||||||
/// (whole-file envelope), `levels.last()` is the finest persisted (floor).
|
|
||||||
#[derive(Clone, Debug)]
|
|
||||||
pub struct WaveformPyramid {
|
|
||||||
pub floor_samples_per_texel: u32,
|
|
||||||
pub branch: u32,
|
|
||||||
pub channels: u32,
|
|
||||||
pub total_frames: u64,
|
|
||||||
pub levels: Vec<Vec<Texel>>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl WaveformPyramid {
|
|
||||||
/// Coarsest level — a single texel (whole-file envelope), or empty if no
|
|
||||||
/// samples were pushed.
|
|
||||||
pub fn root(&self) -> &[Texel] {
|
|
||||||
self.levels.first().map_or(&[][..], |v| v)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Finest persisted level (`floor_samples_per_texel` frames per texel).
|
|
||||||
pub fn floor(&self) -> &[Texel] {
|
|
||||||
self.levels.last().map_or(&[][..], |v| v)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Number of levels (tree depth + 1).
|
|
||||||
pub fn depth(&self) -> usize {
|
|
||||||
self.levels.len()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Serialize to a compact binary blob (for persisting in the `.beam`
|
|
||||||
/// container). Header carries `B`/branch/channels/total_frames + per-level
|
|
||||||
/// lengths, then root-first texel data (`f32` min/max).
|
|
||||||
pub fn to_bytes(&self) -> Vec<u8> {
|
|
||||||
let total_texels: usize = self.levels.iter().map(|l| l.len()).sum();
|
|
||||||
let mut out = Vec::with_capacity(32 + self.levels.len() * 4 + total_texels * 16);
|
|
||||||
out.extend_from_slice(b"LBWF");
|
|
||||||
out.extend_from_slice(&1u32.to_le_bytes()); // format version
|
|
||||||
out.extend_from_slice(&self.floor_samples_per_texel.to_le_bytes());
|
|
||||||
out.extend_from_slice(&self.branch.to_le_bytes());
|
|
||||||
out.extend_from_slice(&self.channels.to_le_bytes());
|
|
||||||
out.extend_from_slice(&self.total_frames.to_le_bytes());
|
|
||||||
out.extend_from_slice(&(self.levels.len() as u32).to_le_bytes());
|
|
||||||
for level in &self.levels {
|
|
||||||
out.extend_from_slice(&(level.len() as u32).to_le_bytes());
|
|
||||||
}
|
|
||||||
for level in &self.levels {
|
|
||||||
for t in level {
|
|
||||||
out.extend_from_slice(&t.l_min.to_le_bytes());
|
|
||||||
out.extend_from_slice(&t.l_max.to_le_bytes());
|
|
||||||
out.extend_from_slice(&t.r_min.to_le_bytes());
|
|
||||||
out.extend_from_slice(&t.r_max.to_le_bytes());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
out
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Reconstruct from [`WaveformPyramid::to_bytes`].
|
|
||||||
pub fn from_bytes(data: &[u8]) -> Result<WaveformPyramid, String> {
|
|
||||||
let mut r = ByteReader::new(data);
|
|
||||||
if r.take(4)? != b"LBWF" {
|
|
||||||
return Err("Not a waveform pyramid blob".to_string());
|
|
||||||
}
|
|
||||||
let version = r.u32()?;
|
|
||||||
if version != 1 {
|
|
||||||
return Err(format!("Unsupported waveform pyramid version {}", version));
|
|
||||||
}
|
|
||||||
let floor_samples_per_texel = r.u32()?;
|
|
||||||
let branch = r.u32()?;
|
|
||||||
let channels = r.u32()?;
|
|
||||||
let total_frames = r.u64()?;
|
|
||||||
let num_levels = r.u32()? as usize;
|
|
||||||
let mut level_lens = Vec::with_capacity(num_levels);
|
|
||||||
for _ in 0..num_levels {
|
|
||||||
level_lens.push(r.u32()? as usize);
|
|
||||||
}
|
|
||||||
let mut levels = Vec::with_capacity(num_levels);
|
|
||||||
for &len in &level_lens {
|
|
||||||
let mut level = Vec::with_capacity(len);
|
|
||||||
for _ in 0..len {
|
|
||||||
level.push(Texel {
|
|
||||||
l_min: r.f32()?,
|
|
||||||
l_max: r.f32()?,
|
|
||||||
r_min: r.f32()?,
|
|
||||||
r_max: r.f32()?,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
levels.push(level);
|
|
||||||
}
|
|
||||||
Ok(WaveformPyramid {
|
|
||||||
floor_samples_per_texel,
|
|
||||||
branch,
|
|
||||||
channels,
|
|
||||||
total_frames,
|
|
||||||
levels,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Minimal little-endian byte cursor for [`WaveformPyramid::from_bytes`].
|
|
||||||
struct ByteReader<'a> {
|
|
||||||
data: &'a [u8],
|
|
||||||
pos: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl<'a> ByteReader<'a> {
|
|
||||||
fn new(data: &'a [u8]) -> Self {
|
|
||||||
Self { data, pos: 0 }
|
|
||||||
}
|
|
||||||
fn take(&mut self, n: usize) -> Result<&'a [u8], String> {
|
|
||||||
let end = self.pos.checked_add(n).ok_or("overflow")?;
|
|
||||||
if end > self.data.len() {
|
|
||||||
return Err("Truncated waveform pyramid blob".to_string());
|
|
||||||
}
|
|
||||||
let s = &self.data[self.pos..end];
|
|
||||||
self.pos = end;
|
|
||||||
Ok(s)
|
|
||||||
}
|
|
||||||
fn u32(&mut self) -> Result<u32, String> {
|
|
||||||
Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap()))
|
|
||||||
}
|
|
||||||
fn u64(&mut self) -> Result<u64, String> {
|
|
||||||
Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap()))
|
|
||||||
}
|
|
||||||
fn f32(&mut self) -> Result<f32, String> {
|
|
||||||
Ok(f32::from_le_bytes(self.take(4)?.try_into().unwrap()))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Streaming builder for a [`WaveformPyramid`]. See the module docs.
|
|
||||||
pub struct WaveformPyramidBuilder {
|
|
||||||
floor: u32,
|
|
||||||
branch: u32,
|
|
||||||
channels: u32,
|
|
||||||
total_frames: u64,
|
|
||||||
floor_level: Vec<Texel>,
|
|
||||||
acc: Texel,
|
|
||||||
acc_count: u32,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl WaveformPyramidBuilder {
|
|
||||||
pub fn new(channels: u32, floor_samples_per_texel: u32) -> Self {
|
|
||||||
Self {
|
|
||||||
floor: floor_samples_per_texel.max(1),
|
|
||||||
branch: BRANCH,
|
|
||||||
channels: channels.max(1),
|
|
||||||
total_frames: 0,
|
|
||||||
floor_level: Vec::new(),
|
|
||||||
acc: Texel::EMPTY,
|
|
||||||
acc_count: 0,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Pre-reserve the floor `Vec` from an estimated total frame count (e.g. the
|
|
||||||
/// probe's `total_frames`), to avoid reallocations during streaming. Purely a
|
|
||||||
/// hint — the final size is set by the actual number of frames pushed.
|
|
||||||
pub fn reserve_for_frames(&mut self, estimated_frames: u64) {
|
|
||||||
let est_texels = (estimated_frames / self.floor as u64).saturating_add(1);
|
|
||||||
self.floor_level.reserve(est_texels.min(usize::MAX as u64) as usize);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Push a block of interleaved samples (`channels` per frame). Partial
|
|
||||||
/// trailing frames (fewer than `channels`) are ignored.
|
|
||||||
pub fn push_interleaved(&mut self, samples: &[f32]) {
|
|
||||||
let ch = self.channels as usize;
|
|
||||||
for frame in samples.chunks_exact(ch) {
|
|
||||||
let l = frame[0];
|
|
||||||
let r = if ch >= 2 { frame[1] } else { l };
|
|
||||||
self.push_frame(l, r);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[inline]
|
|
||||||
fn push_frame(&mut self, l: f32, r: f32) {
|
|
||||||
self.total_frames += 1;
|
|
||||||
self.acc.include_sample(l, r);
|
|
||||||
self.acc_count += 1;
|
|
||||||
if self.acc_count >= self.floor {
|
|
||||||
self.floor_level.push(std::mem::replace(&mut self.acc, Texel::EMPTY));
|
|
||||||
self.acc_count = 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Flush the trailing partial bucket and reduce up to the root.
|
|
||||||
pub fn finish(mut self) -> WaveformPyramid {
|
|
||||||
if self.acc_count > 0 {
|
|
||||||
self.floor_level.push(self.acc);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build finest-first by repeated BRANCH:1 reduction until one texel.
|
|
||||||
// The shape is fully determined by the floor texel count; the last group
|
|
||||||
// at each level may be ragged (1..BRANCH children) and reduces over what
|
|
||||||
// it has.
|
|
||||||
let mut levels = vec![std::mem::take(&mut self.floor_level)];
|
|
||||||
let branch = self.branch as usize;
|
|
||||||
while levels.last().map_or(0, |l| l.len()) > 1 {
|
|
||||||
let prev = levels.last().unwrap();
|
|
||||||
let mut next = Vec::with_capacity(prev.len().div_ceil(branch));
|
|
||||||
for chunk in prev.chunks(branch) {
|
|
||||||
let mut t = Texel::EMPTY;
|
|
||||||
for c in chunk {
|
|
||||||
t.include_texel(c);
|
|
||||||
}
|
|
||||||
next.push(t);
|
|
||||||
}
|
|
||||||
levels.push(next);
|
|
||||||
}
|
|
||||||
// Output is root-first (convention B): levels[0] = root, last = floor.
|
|
||||||
levels.reverse();
|
|
||||||
|
|
||||||
WaveformPyramid {
|
|
||||||
floor_samples_per_texel: self.floor,
|
|
||||||
branch: self.branch,
|
|
||||||
channels: self.channels,
|
|
||||||
total_frames: self.total_frames,
|
|
||||||
levels,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tests live in `daw-backend/tests/waveform_pyramid.rs` (integration tests) so
|
|
||||||
// they build the lib in normal mode, independent of the crate's pre-existing
|
|
||||||
// broken `#[cfg(test)]` unit tests (automation.rs).
|
|
||||||
|
|
@ -430,6 +430,8 @@ pub enum Query {
|
||||||
/// Add a MIDI clip instance to a track synchronously (track_id, instance) - returns instance ID
|
/// Add a MIDI clip instance to a track synchronously (track_id, instance) - returns instance ID
|
||||||
/// The clip must already exist in the MidiClipPool
|
/// The clip must already exist in the MidiClipPool
|
||||||
AddMidiClipInstanceSync(TrackId, crate::audio::midi::MidiClipInstance),
|
AddMidiClipInstanceSync(TrackId, crate::audio::midi::MidiClipInstance),
|
||||||
|
/// Add an audio file to the pool synchronously (path, data, channels, sample_rate) - returns pool index
|
||||||
|
AddAudioFileSync(String, Vec<f32>, u32, u32),
|
||||||
/// Import an audio file synchronously (path) - returns pool index.
|
/// Import an audio file synchronously (path) - returns pool index.
|
||||||
/// Does the same work as Command::ImportAudio (mmap for PCM, streaming
|
/// Does the same work as Command::ImportAudio (mmap for PCM, streaming
|
||||||
/// setup for compressed) but returns the real pool index in the response.
|
/// setup for compressed) but returns the real pool index in the response.
|
||||||
|
|
@ -438,20 +440,12 @@ pub enum Query {
|
||||||
/// problem for very large files, switch to async import with event-based
|
/// problem for very large files, switch to async import with event-based
|
||||||
/// pool index reconciliation.
|
/// pool index reconciliation.
|
||||||
ImportAudioSync(std::path::PathBuf),
|
ImportAudioSync(std::path::PathBuf),
|
||||||
/// Add the audio track of a video file as a streaming pool entry (FFmpeg,
|
|
||||||
/// decoded on demand — no extraction). Probes the audio track and returns
|
|
||||||
/// the pool index. Response: `AudioImportedSync`.
|
|
||||||
AddVideoAudioSync(std::path::PathBuf),
|
|
||||||
/// Get raw audio samples from pool (pool_index) - returns (samples, sample_rate, channels)
|
/// Get raw audio samples from pool (pool_index) - returns (samples, sample_rate, channels)
|
||||||
GetPoolAudioSamples(usize),
|
GetPoolAudioSamples(usize),
|
||||||
/// Get a clone of the current project for serialization
|
/// Get a clone of the current project for serialization
|
||||||
GetProject,
|
GetProject,
|
||||||
/// Set the project (replaces current project state)
|
/// Set the project (replaces current project state)
|
||||||
SetProject(Box<crate::audio::project::Project>),
|
SetProject(Box<crate::audio::project::Project>),
|
||||||
/// Install the host's packed-media byte-source factory (for streaming
|
|
||||||
/// container-packed audio on load). Sent before `SetProject` so bulk
|
|
||||||
/// activation can open packed sources.
|
|
||||||
SetBlobSourceFactory(std::sync::Arc<dyn crate::audio::disk_reader::AudioBlobSourceFactory>),
|
|
||||||
/// Duplicate a MIDI clip in the pool, returning the new clip's ID
|
/// Duplicate a MIDI clip in the pool, returning the new clip's ID
|
||||||
DuplicateMidiClipSync(MidiClipId),
|
DuplicateMidiClipSync(MidiClipId),
|
||||||
/// Get whether a track's graph is still the auto-generated default
|
/// Get whether a track's graph is still the auto-generated default
|
||||||
|
|
@ -522,10 +516,10 @@ pub enum QueryResponse {
|
||||||
AudioExported(Result<(), String>),
|
AudioExported(Result<(), String>),
|
||||||
/// MIDI clip instance added (returns instance ID)
|
/// MIDI clip instance added (returns instance ID)
|
||||||
MidiClipInstanceAdded(Result<MidiClipInstanceId, String>),
|
MidiClipInstanceAdded(Result<MidiClipInstanceId, String>),
|
||||||
|
/// Audio file added to pool (returns pool index)
|
||||||
|
AudioFileAddedSync(Result<usize, String>),
|
||||||
/// Audio file imported to pool (returns pool index)
|
/// Audio file imported to pool (returns pool index)
|
||||||
AudioImportedSync(Result<usize, String>),
|
AudioImportedSync(Result<usize, String>),
|
||||||
/// Packed-media byte-source factory installed
|
|
||||||
BlobSourceFactorySet(Result<(), String>),
|
|
||||||
/// Raw audio samples from pool (samples, sample_rate, channels)
|
/// Raw audio samples from pool (samples, sample_rate, channels)
|
||||||
PoolAudioSamples(Result<(Vec<f32>, u32, u32), String>),
|
PoolAudioSamples(Result<(Vec<f32>, u32, u32), String>),
|
||||||
/// Project retrieved
|
/// Project retrieved
|
||||||
|
|
|
||||||
|
|
@ -1,89 +0,0 @@
|
||||||
//! Integration test for `CompressedReader::open_source` — decoding a streaming
|
|
||||||
//! audio source from an in-memory byte stream (the packed-in-container path)
|
|
||||||
//! rather than a filesystem path. Proves the `MediaByteSource` adapter feeds
|
|
||||||
//! Symphonia correctly (probe + decode + seekable byte length).
|
|
||||||
|
|
||||||
use std::io::{Cursor, Read, Seek, SeekFrom};
|
|
||||||
|
|
||||||
use daw_backend::audio::disk_reader::{CompressedReader, MediaByteSource};
|
|
||||||
|
|
||||||
/// A `MediaByteSource` over an in-memory buffer (stands in for core's BlobReader).
|
|
||||||
struct VecSource(Cursor<Vec<u8>>);
|
|
||||||
|
|
||||||
impl Read for VecSource {
|
|
||||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
|
||||||
self.0.read(buf)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl Seek for VecSource {
|
|
||||||
fn seek(&mut self, pos: SeekFrom) -> std::io::Result<u64> {
|
|
||||||
self.0.seek(pos)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl MediaByteSource for VecSource {
|
|
||||||
fn byte_len(&self) -> u64 {
|
|
||||||
self.0.get_ref().len() as u64
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Build a minimal PCM16 mono WAV byte buffer holding `frames` samples of a ramp.
|
|
||||||
fn make_wav(sample_rate: u32, frames: u32) -> Vec<u8> {
|
|
||||||
let channels: u16 = 1;
|
|
||||||
let bits: u16 = 16;
|
|
||||||
let block_align: u16 = channels * bits / 8;
|
|
||||||
let byte_rate: u32 = sample_rate * block_align as u32;
|
|
||||||
let data_len: u32 = frames * block_align as u32;
|
|
||||||
|
|
||||||
let mut v = Vec::new();
|
|
||||||
v.extend_from_slice(b"RIFF");
|
|
||||||
v.extend_from_slice(&(36 + data_len).to_le_bytes());
|
|
||||||
v.extend_from_slice(b"WAVE");
|
|
||||||
v.extend_from_slice(b"fmt ");
|
|
||||||
v.extend_from_slice(&16u32.to_le_bytes());
|
|
||||||
v.extend_from_slice(&1u16.to_le_bytes()); // PCM
|
|
||||||
v.extend_from_slice(&channels.to_le_bytes());
|
|
||||||
v.extend_from_slice(&sample_rate.to_le_bytes());
|
|
||||||
v.extend_from_slice(&byte_rate.to_le_bytes());
|
|
||||||
v.extend_from_slice(&block_align.to_le_bytes());
|
|
||||||
v.extend_from_slice(&bits.to_le_bytes());
|
|
||||||
v.extend_from_slice(b"data");
|
|
||||||
v.extend_from_slice(&data_len.to_le_bytes());
|
|
||||||
for i in 0..frames {
|
|
||||||
// A ramp from -16000..16000 so values are recognizable.
|
|
||||||
let s = (((i % 1000) as i32 - 500) * 32) as i16;
|
|
||||||
v.extend_from_slice(&s.to_le_bytes());
|
|
||||||
}
|
|
||||||
v
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn open_source_decodes_in_memory_wav() {
|
|
||||||
let sample_rate = 8000;
|
|
||||||
let frames = 4096;
|
|
||||||
let bytes = make_wav(sample_rate, frames);
|
|
||||||
|
|
||||||
let src = Box::new(VecSource(Cursor::new(bytes)));
|
|
||||||
let mut reader = CompressedReader::open_source(src, Some("wav"))
|
|
||||||
.expect("open_source should probe the in-memory WAV");
|
|
||||||
|
|
||||||
assert_eq!(reader.sample_rate(), sample_rate);
|
|
||||||
assert_eq!(reader.channels(), 1);
|
|
||||||
|
|
||||||
// Decode the whole stream and count emitted frames.
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut decoded = 0usize;
|
|
||||||
loop {
|
|
||||||
let n = reader.decode_next(&mut buf).expect("decode_next");
|
|
||||||
if n == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
decoded += n;
|
|
||||||
}
|
|
||||||
// Should recover (approximately) all frames — codec frame counts can round.
|
|
||||||
assert!(
|
|
||||||
(decoded as i64 - frames as i64).abs() < 64,
|
|
||||||
"decoded {} vs expected {}",
|
|
||||||
decoded,
|
|
||||||
frames
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
@ -1,253 +0,0 @@
|
||||||
//! Integration tests for `VideoAudioReader` (FFmpeg streaming audio source).
|
|
||||||
//!
|
|
||||||
//! These build the daw-backend lib in normal mode, so they're independent of
|
|
||||||
//! the crate's pre-existing broken `#[cfg(test)]` unit tests (automation.rs).
|
|
||||||
//! They synthesize a mono 32-bit-float WAV whose sample `i` has value `i/n`, so
|
|
||||||
//! a decoded sample's value identifies its frame index — letting us assert both
|
|
||||||
//! in-order decoding and **sample-accurate seeking** (the property video audio
|
|
||||||
//! needs to stay synced with other clips).
|
|
||||||
|
|
||||||
use daw_backend::audio::disk_reader::{
|
|
||||||
build_waveform_pyramid, CompressedReader, SourceKind, VideoAudioReader,
|
|
||||||
};
|
|
||||||
use std::io::Write;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
fn write_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
|
|
||||||
let channels = 1u16;
|
|
||||||
let bytes_per_sample = 4u32;
|
|
||||||
let data_size = n * bytes_per_sample;
|
|
||||||
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
|
|
||||||
buf.extend_from_slice(b"RIFF");
|
|
||||||
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
|
|
||||||
buf.extend_from_slice(b"WAVE");
|
|
||||||
buf.extend_from_slice(b"fmt ");
|
|
||||||
buf.extend_from_slice(&16u32.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
|
|
||||||
buf.extend_from_slice(&channels.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&sample_rate.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
|
|
||||||
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
|
|
||||||
buf.extend_from_slice(&32u16.to_le_bytes());
|
|
||||||
buf.extend_from_slice(b"data");
|
|
||||||
buf.extend_from_slice(&data_size.to_le_bytes());
|
|
||||||
for i in 0..n {
|
|
||||||
buf.extend_from_slice(&((i as f32) / (n as f32)).to_le_bytes());
|
|
||||||
}
|
|
||||||
let mut f = std::fs::File::create(path).unwrap();
|
|
||||||
f.write_all(&buf).unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Stereo ramp: frame `i` has left = `i/n`, right = `0.5 - i/n` (distinct per
|
|
||||||
/// channel), interleaved `[L0,R0,L1,R1,…]`. Exercises the channels>1 path.
|
|
||||||
fn write_stereo_ramp_wav(path: &Path, n: u32, sample_rate: u32) {
|
|
||||||
let channels = 2u16;
|
|
||||||
let bytes_per_sample = 4u32;
|
|
||||||
let data_size = n * channels as u32 * bytes_per_sample;
|
|
||||||
let mut buf: Vec<u8> = Vec::with_capacity(44 + data_size as usize);
|
|
||||||
buf.extend_from_slice(b"RIFF");
|
|
||||||
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
|
|
||||||
buf.extend_from_slice(b"WAVE");
|
|
||||||
buf.extend_from_slice(b"fmt ");
|
|
||||||
buf.extend_from_slice(&16u32.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
|
|
||||||
buf.extend_from_slice(&channels.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&sample_rate.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&(sample_rate * channels as u32 * bytes_per_sample).to_le_bytes());
|
|
||||||
buf.extend_from_slice(&((channels as u32 * bytes_per_sample) as u16).to_le_bytes());
|
|
||||||
buf.extend_from_slice(&32u16.to_le_bytes());
|
|
||||||
buf.extend_from_slice(b"data");
|
|
||||||
buf.extend_from_slice(&data_size.to_le_bytes());
|
|
||||||
for i in 0..n {
|
|
||||||
let l = i as f32 / n as f32;
|
|
||||||
let r = 0.5 - i as f32 / n as f32;
|
|
||||||
buf.extend_from_slice(&l.to_le_bytes());
|
|
||||||
buf.extend_from_slice(&r.to_le_bytes());
|
|
||||||
}
|
|
||||||
let mut f = std::fs::File::create(path).unwrap();
|
|
||||||
f.write_all(&buf).unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
fn temp_path(tag: &str) -> std::path::PathBuf {
|
|
||||||
let mut p = std::env::temp_dir();
|
|
||||||
p.push(format!("lb_videoaudio_test_{}_{}.wav", std::process::id(), tag));
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn decodes_samples_in_order() {
|
|
||||||
let n = 4000u32;
|
|
||||||
let sr = 8000u32;
|
|
||||||
let path = temp_path("seq");
|
|
||||||
write_ramp_wav(&path, n, sr);
|
|
||||||
|
|
||||||
let mut reader = VideoAudioReader::open(&path).unwrap();
|
|
||||||
assert_eq!(reader.channels(), 1);
|
|
||||||
assert_eq!(reader.sample_rate(), sr);
|
|
||||||
// Probe estimate (used by add_video_audio_sync) should be ~n frames.
|
|
||||||
let tf = reader.total_frames() as f64;
|
|
||||||
assert!(
|
|
||||||
(tf - n as f64).abs() < n as f64 * 0.1,
|
|
||||||
"total_frames {} not ~{}",
|
|
||||||
tf, n
|
|
||||||
);
|
|
||||||
|
|
||||||
let mut all = Vec::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
loop {
|
|
||||||
let frames = reader.decode_next(&mut buf).unwrap();
|
|
||||||
if frames == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
all.extend_from_slice(&buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Allow a couple of priming/flush samples of slack at the very end.
|
|
||||||
assert!(all.len() + 4 >= n as usize, "decoded too few samples: {}", all.len());
|
|
||||||
for (i, &v) in all.iter().enumerate().take(n as usize) {
|
|
||||||
let expected = i as f32 / n as f32;
|
|
||||||
assert!((v - expected).abs() < 1e-3, "sample {} = {}, expected {}", i, v, expected);
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// CompressedReader (symphonia) must seek **sample-accurately** too, so compressed
|
|
||||||
/// audio stays frame-synced with video audio. Symphonia decodes WAV via the same
|
|
||||||
/// path; its coarse seek lands on packet boundaries, exercising the decode-discard.
|
|
||||||
#[test]
|
|
||||||
fn compressed_reader_seek_is_sample_accurate() {
|
|
||||||
let n = 4000u32;
|
|
||||||
let sr = 8000u32;
|
|
||||||
let path = temp_path("comp_seek");
|
|
||||||
write_ramp_wav(&path, n, sr);
|
|
||||||
|
|
||||||
let mut reader = CompressedReader::open(&path).unwrap();
|
|
||||||
assert_eq!(reader.channels(), 1);
|
|
||||||
assert_eq!(reader.sample_rate(), sr);
|
|
||||||
|
|
||||||
for &target in &[2000u64, 137, 3500, 0] {
|
|
||||||
let actual = reader.seek(target).unwrap();
|
|
||||||
assert_eq!(actual, target, "seek should report the exact target");
|
|
||||||
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut frames = 0;
|
|
||||||
for _ in 0..128 {
|
|
||||||
frames = reader.decode_next(&mut buf).unwrap();
|
|
||||||
if frames > 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
assert!(frames > 0, "no samples after seek to {}", target);
|
|
||||||
let expected = target as f32 / n as f32;
|
|
||||||
assert!(
|
|
||||||
(buf[0] - expected).abs() < 1e-3,
|
|
||||||
"compressed seek to {}: first sample = {}, expected {}",
|
|
||||||
target, buf[0], expected
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The decode→pyramid bridge should produce an envelope matching the signal,
|
|
||||||
/// through both reader backends (symphonia + ffmpeg), with bounded memory.
|
|
||||||
#[test]
|
|
||||||
fn waveform_pyramid_from_decode_matches_signal() {
|
|
||||||
let n = 5000u32;
|
|
||||||
let sr = 8000u32;
|
|
||||||
let path = temp_path("pyr");
|
|
||||||
write_ramp_wav(&path, n, sr); // ramp 0 .. (n-1)/n, all positive
|
|
||||||
|
|
||||||
for kind in [SourceKind::CompressedAudio, SourceKind::VideoAudio] {
|
|
||||||
let p = build_waveform_pyramid(&path, kind, 256).unwrap();
|
|
||||||
assert_eq!(p.channels, 1);
|
|
||||||
assert_eq!(p.root().len(), 1, "{:?}: root should be one texel", kind);
|
|
||||||
let root = p.root()[0];
|
|
||||||
assert!(root.l_min.abs() < 1e-2, "{:?}: root min {} ~ 0", kind, root.l_min);
|
|
||||||
let expected_max = (n - 1) as f32 / n as f32;
|
|
||||||
assert!(
|
|
||||||
(root.l_max - expected_max).abs() < 1e-2,
|
|
||||||
"{:?}: root max {} ~ {}", kind, root.l_max, expected_max
|
|
||||||
);
|
|
||||||
// Frame count is approximate across decoders (priming/resampler overhead);
|
|
||||||
// the envelope above is the real check. Just confirm it's about right.
|
|
||||||
assert!((p.total_frames as i64 - n as i64).abs() < 128, "{:?}: frames {}", kind, p.total_frames);
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn decodes_stereo_interleaved() {
|
|
||||||
let n = 2000u32;
|
|
||||||
let sr = 8000u32;
|
|
||||||
let path = temp_path("stereo");
|
|
||||||
write_stereo_ramp_wav(&path, n, sr);
|
|
||||||
|
|
||||||
let mut reader = VideoAudioReader::open(&path).unwrap();
|
|
||||||
assert_eq!(reader.channels(), 2);
|
|
||||||
|
|
||||||
let mut all = Vec::new();
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
loop {
|
|
||||||
let frames = reader.decode_next(&mut buf).unwrap();
|
|
||||||
if frames == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
// Each decode_next returns whole interleaved frames.
|
|
||||||
assert_eq!(buf.len() % 2, 0, "stereo decode returned a partial frame");
|
|
||||||
all.extend_from_slice(&buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Interleaved L/R, ~n frames.
|
|
||||||
assert!(all.len() + 8 >= (n * 2) as usize, "decoded too few samples: {}", all.len());
|
|
||||||
for i in 0..n as usize {
|
|
||||||
let l = all[2 * i];
|
|
||||||
let r = all[2 * i + 1];
|
|
||||||
assert!((l - i as f32 / n as f32).abs() < 1e-3, "L[{}]={} expected {}", i, l, i as f32 / n as f32);
|
|
||||||
assert!(
|
|
||||||
(r - (0.5 - i as f32 / n as f32)).abs() < 1e-3,
|
|
||||||
"R[{}]={} expected {}", i, r, 0.5 - i as f32 / n as f32
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn seek_is_sample_accurate() {
|
|
||||||
let n = 4000u32;
|
|
||||||
let sr = 8000u32;
|
|
||||||
let path = temp_path("seek");
|
|
||||||
write_ramp_wav(&path, n, sr);
|
|
||||||
|
|
||||||
let mut reader = VideoAudioReader::open(&path).unwrap();
|
|
||||||
|
|
||||||
for &target in &[2000u64, 137, 3500, 0] {
|
|
||||||
let actual = reader.seek(target).unwrap();
|
|
||||||
assert_eq!(actual, target);
|
|
||||||
|
|
||||||
// Pull the first non-empty decode after the seek.
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
let mut frames = 0;
|
|
||||||
for _ in 0..64 {
|
|
||||||
frames = reader.decode_next(&mut buf).unwrap();
|
|
||||||
if frames > 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
assert!(frames > 0, "no samples after seek to {}", target);
|
|
||||||
|
|
||||||
let expected = target as f32 / n as f32;
|
|
||||||
assert!(
|
|
||||||
(buf[0] - expected).abs() < 1e-3,
|
|
||||||
"after seek to {}: first sample = {}, expected {}",
|
|
||||||
target,
|
|
||||||
buf[0],
|
|
||||||
expected
|
|
||||||
);
|
|
||||||
// And the next few advance in order.
|
|
||||||
for k in 0..frames.min(8) {
|
|
||||||
let exp = (target as usize + k) as f32 / n as f32;
|
|
||||||
assert!((buf[k] - exp).abs() < 1e-3, "seek {}+{}: {} vs {}", target, k, buf[k], exp);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
@ -1,135 +0,0 @@
|
||||||
//! Integration tests for the streaming waveform LOD pyramid builder.
|
|
||||||
//!
|
|
||||||
//! Convention B: `levels[0]` is the root (coarsest), `levels.last()` the floor
|
|
||||||
//! (finest). Tests use the `.root()` / `.floor()` accessors so they don't depend
|
|
||||||
//! on the raw index ordering.
|
|
||||||
|
|
||||||
use daw_backend::audio::waveform_pyramid::{Texel, WaveformPyramid, WaveformPyramidBuilder};
|
|
||||||
|
|
||||||
fn build_mono(samples: &[f32], floor: u32) -> WaveformPyramid {
|
|
||||||
let mut b = WaveformPyramidBuilder::new(1, floor);
|
|
||||||
b.push_interleaved(samples);
|
|
||||||
b.finish()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn floor_level_min_max_per_bucket() {
|
|
||||||
// 8 samples, floor 4 → two floor texels covering [0..4) and [4..8).
|
|
||||||
let s: Vec<f32> = (0..8).map(|i| i as f32).collect();
|
|
||||||
let p = build_mono(&s, 4);
|
|
||||||
assert_eq!(p.floor().len(), 2);
|
|
||||||
assert_eq!(p.floor()[0], Texel { l_min: 0.0, l_max: 3.0, r_min: 0.0, r_max: 3.0 });
|
|
||||||
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 7.0, r_min: 4.0, r_max: 7.0 });
|
|
||||||
// Root reduces the two floor texels into the envelope [0..8).
|
|
||||||
assert_eq!(p.root().len(), 1);
|
|
||||||
assert_eq!(p.root()[0], Texel { l_min: 0.0, l_max: 7.0, r_min: 0.0, r_max: 7.0 });
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn partial_trailing_bucket_is_flushed() {
|
|
||||||
// 6 samples, floor 4 → texels [0..4) and a ragged [4..6).
|
|
||||||
let s: Vec<f32> = (0..6).map(|i| i as f32).collect();
|
|
||||||
let p = build_mono(&s, 4);
|
|
||||||
assert_eq!(p.floor().len(), 2);
|
|
||||||
assert_eq!(p.floor()[1], Texel { l_min: 4.0, l_max: 5.0, r_min: 4.0, r_max: 5.0 });
|
|
||||||
assert_eq!(p.total_frames, 6);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn multi_level_envelope_matches_global_min_max() {
|
|
||||||
let s: Vec<f32> = (0..1000).map(|i| ((i as f32) * 0.01).sin()).collect();
|
|
||||||
let g_min = s.iter().cloned().fold(f32::INFINITY, f32::min);
|
|
||||||
let g_max = s.iter().cloned().fold(f32::NEG_INFINITY, f32::max);
|
|
||||||
let p = build_mono(&s, 16);
|
|
||||||
assert_eq!(p.root().len(), 1);
|
|
||||||
assert!((p.root()[0].l_min - g_min).abs() < 1e-6);
|
|
||||||
assert!((p.root()[0].l_max - g_max).abs() < 1e-6);
|
|
||||||
// Every level's overall min/max equals the global (extremes are lossless).
|
|
||||||
for level in &p.levels {
|
|
||||||
let lmin = level.iter().map(|t| t.l_min).fold(f32::INFINITY, f32::min);
|
|
||||||
let lmax = level.iter().map(|t| t.l_max).fold(f32::NEG_INFINITY, f32::max);
|
|
||||||
assert!((lmin - g_min).abs() < 1e-6);
|
|
||||||
assert!((lmax - g_max).abs() < 1e-6);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn levels_are_root_first_and_get_finer() {
|
|
||||||
let s: Vec<f32> = (0..1000).map(|i| i as f32).collect();
|
|
||||||
let p = build_mono(&s, 16);
|
|
||||||
// Root first, floor last; strictly finer (more texels) as depth increases.
|
|
||||||
assert_eq!(p.root().len(), 1);
|
|
||||||
assert!(p.depth() >= 3);
|
|
||||||
for w in p.levels.windows(2) {
|
|
||||||
assert!(w[1].len() >= w[0].len(), "deeper level should be finer");
|
|
||||||
}
|
|
||||||
// Floor has ceil(1000/16) = 63 texels.
|
|
||||||
assert_eq!(p.floor().len(), 63);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn stereo_channels_tracked_separately() {
|
|
||||||
// L ramps up, R ramps down; interleaved.
|
|
||||||
let n = 64;
|
|
||||||
let mut s = Vec::new();
|
|
||||||
for i in 0..n {
|
|
||||||
s.push(i as f32); // L
|
|
||||||
s.push(-(i as f32)); // R
|
|
||||||
}
|
|
||||||
let mut b = WaveformPyramidBuilder::new(2, 16);
|
|
||||||
b.push_interleaved(&s);
|
|
||||||
let p = b.finish();
|
|
||||||
assert_eq!(p.root().len(), 1);
|
|
||||||
assert_eq!(p.root()[0].l_min, 0.0);
|
|
||||||
assert_eq!(p.root()[0].l_max, (n - 1) as f32);
|
|
||||||
assert_eq!(p.root()[0].r_min, -((n - 1) as f32));
|
|
||||||
assert_eq!(p.root()[0].r_max, 0.0);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn pyramid_size_is_bounded() {
|
|
||||||
let n = 100_000usize;
|
|
||||||
let s: Vec<f32> = (0..n).map(|i| (i % 7) as f32).collect();
|
|
||||||
let floor = 256u32;
|
|
||||||
let p = build_mono(&s, floor);
|
|
||||||
let total: usize = p.levels.iter().map(|l| l.len()).sum();
|
|
||||||
let floor_texels = (n as u32).div_ceil(floor) as usize;
|
|
||||||
// Geometric bound: < floor_texels * branch/(branch-1) + small per-level slack.
|
|
||||||
let bound = floor_texels * 4 / 3 + p.depth() + 2;
|
|
||||||
assert!(total <= bound, "pyramid too big: {} > {}", total, bound);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn bytes_round_trip() {
|
|
||||||
let s: Vec<f32> = (0..3333).map(|i| ((i as f32) * 0.013).sin()).collect();
|
|
||||||
let p = build_mono(&s, 64);
|
|
||||||
let bytes = p.to_bytes();
|
|
||||||
let q = WaveformPyramid::from_bytes(&bytes).unwrap();
|
|
||||||
assert_eq!(p.floor_samples_per_texel, q.floor_samples_per_texel);
|
|
||||||
assert_eq!(p.branch, q.branch);
|
|
||||||
assert_eq!(p.channels, q.channels);
|
|
||||||
assert_eq!(p.total_frames, q.total_frames);
|
|
||||||
assert_eq!(p.levels, q.levels);
|
|
||||||
// Truncated/garbage input is rejected, not panicking.
|
|
||||||
assert!(WaveformPyramid::from_bytes(&bytes[..bytes.len() - 4]).is_err());
|
|
||||||
assert!(WaveformPyramid::from_bytes(b"nope").is_err());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn pushing_in_arbitrary_chunks_matches() {
|
|
||||||
// The streaming builder must be agnostic to how samples are chunked.
|
|
||||||
let s: Vec<f32> = (0..5000).map(|i| ((i * 13) % 97) as f32 - 48.0).collect();
|
|
||||||
let whole = build_mono(&s, 32);
|
|
||||||
|
|
||||||
let mut b = WaveformPyramidBuilder::new(1, 32);
|
|
||||||
b.reserve_for_frames(5000);
|
|
||||||
for chunk in s.chunks(37) {
|
|
||||||
b.push_interleaved(chunk);
|
|
||||||
}
|
|
||||||
let chunked = b.finish();
|
|
||||||
|
|
||||||
assert_eq!(whole.depth(), chunked.depth());
|
|
||||||
for (a, c) in whole.levels.iter().zip(chunked.levels.iter()) {
|
|
||||||
assert_eq!(a, c);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -2160,18 +2160,6 @@ version = "0.1.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
|
checksum = "af9673d8203fcb076b19dfd17e38b3d4ae9f44959416ea532ce72415a6020365"
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "fallible-iterator"
|
|
||||||
version = "0.3.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "fallible-streaming-iterator"
|
|
||||||
version = "0.1.9"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fancy-regex"
|
name = "fancy-regex"
|
||||||
version = "0.16.2"
|
version = "0.16.2"
|
||||||
|
|
@ -2937,9 +2925,6 @@ name = "hashbrown"
|
||||||
version = "0.14.5"
|
version = "0.14.5"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
|
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
|
||||||
dependencies = [
|
|
||||||
"ahash 0.8.12",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "hashbrown"
|
name = "hashbrown"
|
||||||
|
|
@ -2961,15 +2946,6 @@ dependencies = [
|
||||||
"foldhash 0.2.0",
|
"foldhash 0.2.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "hashlink"
|
|
||||||
version = "0.9.1"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
|
|
||||||
dependencies = [
|
|
||||||
"hashbrown 0.14.5",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "heapless"
|
name = "heapless"
|
||||||
version = "0.8.0"
|
version = "0.8.0"
|
||||||
|
|
@ -3439,17 +3415,6 @@ dependencies = [
|
||||||
"redox_syscall 0.5.18",
|
"redox_syscall 0.5.18",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "libsqlite3-sys"
|
|
||||||
version = "0.28.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f"
|
|
||||||
dependencies = [
|
|
||||||
"cc",
|
|
||||||
"pkg-config",
|
|
||||||
"vcpkg",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "libxdo"
|
name = "libxdo"
|
||||||
version = "0.6.0"
|
version = "0.6.0"
|
||||||
|
|
@ -3490,7 +3455,6 @@ dependencies = [
|
||||||
"objc2-foundation 0.3.2",
|
"objc2-foundation 0.3.2",
|
||||||
"pathdiff",
|
"pathdiff",
|
||||||
"rstar",
|
"rstar",
|
||||||
"rusqlite",
|
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"tiny-skia",
|
"tiny-skia",
|
||||||
|
|
@ -3505,7 +3469,7 @@ dependencies = [
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "lightningbeam-editor"
|
name = "lightningbeam-editor"
|
||||||
version = "1.0.4-alpha"
|
version = "1.0.3-alpha"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"beamdsp",
|
"beamdsp",
|
||||||
"bytemuck",
|
"bytemuck",
|
||||||
|
|
@ -5514,20 +5478,6 @@ version = "0.3.2"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba"
|
checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba"
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "rusqlite"
|
|
||||||
version = "0.31.0"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae"
|
|
||||||
dependencies = [
|
|
||||||
"bitflags 2.10.0",
|
|
||||||
"fallible-iterator",
|
|
||||||
"fallible-streaming-iterator",
|
|
||||||
"hashlink",
|
|
||||||
"libsqlite3-sys",
|
|
||||||
"smallvec",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "rustc-hash"
|
name = "rustc-hash"
|
||||||
version = "1.1.0"
|
version = "1.1.0"
|
||||||
|
|
|
||||||
|
|
@ -36,9 +36,6 @@ zip = "0.6"
|
||||||
chrono = "0.4"
|
chrono = "0.4"
|
||||||
base64 = "0.21"
|
base64 = "0.21"
|
||||||
pathdiff = "0.2"
|
pathdiff = "0.2"
|
||||||
# .beam container: SQLite database file. `bundled` compiles SQLite from source
|
|
||||||
# (no system dependency); `blob` enables incremental blob I/O for streaming.
|
|
||||||
rusqlite = { version = "0.31", features = ["bundled", "blob"] }
|
|
||||||
|
|
||||||
# Audio encoding for embedded files
|
# Audio encoding for embedded files
|
||||||
flacenc = "0.4" # For FLAC encoding (lossless)
|
flacenc = "0.4" # For FLAC encoding (lossless)
|
||||||
|
|
|
||||||
|
|
@ -71,15 +71,6 @@ pub trait Action: Send {
|
||||||
/// Get a human-readable description of this action (for UI display)
|
/// Get a human-readable description of this action (for UI display)
|
||||||
fn description(&self) -> String;
|
fn description(&self) -> String;
|
||||||
|
|
||||||
/// For raster actions that store dirty-rect diffs: the `(layer_id, time)` of the
|
|
||||||
/// keyframe whose full pixels must be resident before `execute`/`rollback` can
|
|
||||||
/// apply the diff. The editor faults the frame in (synchronously) before undo/redo
|
|
||||||
/// so a paged-out clean frame is restored to its container state first. Non-raster
|
|
||||||
/// actions (and full-buffer ones) return `None`.
|
|
||||||
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
|
|
||||||
None
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Execute backend operations after document changes
|
/// Execute backend operations after document changes
|
||||||
///
|
///
|
||||||
/// Called AFTER execute() succeeds. If this returns an error, execute()
|
/// Called AFTER execute() succeeds. If this returns an error, execute()
|
||||||
|
|
@ -299,18 +290,6 @@ impl ActionExecutor {
|
||||||
self.undo_stack.last().map(|a| a.description())
|
self.undo_stack.last().map(|a| a.description())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `(layer_id, time)` of the raster keyframe the next undo needs resident, if any.
|
|
||||||
/// The editor faults it in before calling `undo()` so a paged-out clean frame is
|
|
||||||
/// restored to its container state, giving the diff a correct base to apply onto.
|
|
||||||
pub fn peek_undo_raster_hint(&self) -> Option<(Uuid, f64)> {
|
|
||||||
self.undo_stack.last().and_then(|a| a.raster_resident_hint())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `(layer_id, time)` of the raster keyframe the next redo needs resident, if any.
|
|
||||||
pub fn peek_redo_raster_hint(&self) -> Option<(Uuid, f64)> {
|
|
||||||
self.redo_stack.last().and_then(|a| a.raster_resident_hint())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Get MIDI cache data from the last action on the undo stack (after redo).
|
/// Get MIDI cache data from the last action on the undo stack (after redo).
|
||||||
/// Returns the notes reflecting execute state.
|
/// Returns the notes reflecting execute state.
|
||||||
pub fn last_undo_midi_notes(&self) -> Option<(u32, &[(f64, u8, u8, f64)])> {
|
pub fn last_undo_midi_notes(&self) -> Option<(u32, &[(f64, u8, u8, f64)])> {
|
||||||
|
|
|
||||||
|
|
@ -235,22 +235,11 @@ impl Action for AddClipInstanceAction {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
AudioClipType::Sampled { audio_pool_index } => {
|
AudioClipType::Sampled { audio_pool_index } => {
|
||||||
// `trim_*` / `clip.duration` are in SECONDS (audio content time),
|
|
||||||
// while `timeline_*` and the backend's `duration` are in BEATS.
|
|
||||||
let internal_start = self.clip_instance.trim_start;
|
let internal_start = self.clip_instance.trim_start;
|
||||||
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
|
let internal_end = self.clip_instance.trim_end.unwrap_or(clip.duration);
|
||||||
|
let effective_duration = self.clip_instance.timeline_duration
|
||||||
|
.unwrap_or(internal_end - internal_start);
|
||||||
let start_time = self.clip_instance.timeline_start;
|
let start_time = self.clip_instance.timeline_start;
|
||||||
// `effective_duration` is in BEATS. When `timeline_duration` is set
|
|
||||||
// it already is; otherwise the clip occupies its natural content
|
|
||||||
// length, so convert that seconds-span to beats at the clip's start
|
|
||||||
// (NOT `internal_end - internal_start`, which is seconds — that was
|
|
||||||
// the seconds-as-beats bug that made clips stop early off 60 BPM).
|
|
||||||
let effective_duration = self.clip_instance.timeline_duration.unwrap_or_else(|| {
|
|
||||||
let tempo_map = document.tempo_map();
|
|
||||||
let content_secs = internal_end - internal_start;
|
|
||||||
tempo_map.inverse_transform(tempo_map.transform(start_time) + content_secs)
|
|
||||||
- start_time
|
|
||||||
});
|
|
||||||
|
|
||||||
let instance_id = controller.add_audio_clip(
|
let instance_id = controller.add_audio_clip(
|
||||||
*backend_track_id,
|
*backend_track_id,
|
||||||
|
|
@ -316,11 +305,8 @@ mod tests {
|
||||||
let layer_id = layer.layer.id;
|
let layer_id = layer.layer.id;
|
||||||
document.root_mut().add_child(AnyLayer::Vector(layer));
|
document.root_mut().add_child(AnyLayer::Vector(layer));
|
||||||
|
|
||||||
// Register a vector clip so get_clip_duration succeeds
|
// Create a clip instance (using a fake clip_id since we're just testing the action)
|
||||||
let clip_id = Uuid::new_v4();
|
let clip_id = Uuid::new_v4();
|
||||||
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 5.0);
|
|
||||||
document.vector_clips.insert(clip_id, clip);
|
|
||||||
|
|
||||||
let clip_instance = ClipInstance::new(clip_id);
|
let clip_instance = ClipInstance::new(clip_id);
|
||||||
let instance_id = clip_instance.id;
|
let instance_id = clip_instance.id;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
//! Add a blank raster keyframe at the playhead (the explicit "New Keyframe" command
|
|
||||||
//! for raster layers — mirrors `SetKeyframeAction` for vector, but inserts an empty
|
|
||||||
//! cel rather than copying the current graph).
|
|
||||||
|
|
||||||
use crate::action::Action;
|
|
||||||
use crate::document::Document;
|
|
||||||
use crate::layer::AnyLayer;
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
pub struct AddRasterKeyframeAction {
|
|
||||||
layer_id: Uuid,
|
|
||||||
time: f64,
|
|
||||||
width: u32,
|
|
||||||
height: u32,
|
|
||||||
/// Id of the keyframe created by the last `execute` (so `rollback` can remove
|
|
||||||
/// exactly that one). `None` if a keyframe already existed at `time` (no-op).
|
|
||||||
created_id: Option<Uuid>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl AddRasterKeyframeAction {
|
|
||||||
pub fn new(layer_id: Uuid, time: f64, width: u32, height: u32) -> Self {
|
|
||||||
Self { layer_id, time, width, height, created_id: None }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Action for AddRasterKeyframeAction {
|
|
||||||
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
|
|
||||||
let layer = document
|
|
||||||
.get_layer_mut(&self.layer_id)
|
|
||||||
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
|
|
||||||
let rl = match layer {
|
|
||||||
AnyLayer::Raster(rl) => rl,
|
|
||||||
_ => return Err("Not a raster layer".to_string()),
|
|
||||||
};
|
|
||||||
// Inserts a blank cel only if one doesn't already exist at this time.
|
|
||||||
self.created_id = rl.insert_blank_keyframe_at(self.time, self.width, self.height);
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
|
|
||||||
if let Some(id) = self.created_id.take() {
|
|
||||||
if let Some(AnyLayer::Raster(rl)) = document.get_layer_mut(&self.layer_id) {
|
|
||||||
rl.remove_keyframe(id);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn description(&self) -> String {
|
|
||||||
"New keyframe".to_string()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -37,9 +37,6 @@ pub struct DeleteFolderAction {
|
||||||
|
|
||||||
/// Asset IDs that were deleted (for DeleteRecursive strategy)
|
/// Asset IDs that were deleted (for DeleteRecursive strategy)
|
||||||
deleted_asset_ids: Vec<Uuid>,
|
deleted_asset_ids: Vec<Uuid>,
|
||||||
|
|
||||||
/// Subfolder IDs that were reparented to the deleted folder's parent (for MoveToParent strategy)
|
|
||||||
moved_subfolder_ids: Vec<Uuid>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl DeleteFolderAction {
|
impl DeleteFolderAction {
|
||||||
|
|
@ -58,7 +55,6 @@ impl DeleteFolderAction {
|
||||||
removed_folders: Vec::new(),
|
removed_folders: Vec::new(),
|
||||||
moved_asset_ids: Vec::new(),
|
moved_asset_ids: Vec::new(),
|
||||||
deleted_asset_ids: Vec::new(),
|
deleted_asset_ids: Vec::new(),
|
||||||
moved_subfolder_ids: Vec::new(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -134,7 +130,6 @@ impl Action for DeleteFolderAction {
|
||||||
for subfolder_id in subfolder_ids {
|
for subfolder_id in subfolder_ids {
|
||||||
if let Some(subfolder) = tree.folders.get_mut(&subfolder_id) {
|
if let Some(subfolder) = tree.folders.get_mut(&subfolder_id) {
|
||||||
subfolder.parent_id = parent_id;
|
subfolder.parent_id = parent_id;
|
||||||
self.moved_subfolder_ids.push(subfolder_id);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -264,13 +259,6 @@ impl Action for DeleteFolderAction {
|
||||||
tree.add_folder(folder.clone());
|
tree.add_folder(folder.clone());
|
||||||
}
|
}
|
||||||
|
|
||||||
// Restore reparented subfolders back under the deleted folder
|
|
||||||
for subfolder_id in &self.moved_subfolder_ids {
|
|
||||||
if let Some(subfolder) = tree.folders.get_mut(subfolder_id) {
|
|
||||||
subfolder.parent_id = Some(self.folder_id);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
match self.strategy {
|
match self.strategy {
|
||||||
DeleteStrategy::MoveToParent => {
|
DeleteStrategy::MoveToParent => {
|
||||||
// Restore folder_id for moved assets
|
// Restore folder_id for moved assets
|
||||||
|
|
@ -324,7 +312,6 @@ impl Action for DeleteFolderAction {
|
||||||
self.removed_folders.clear();
|
self.removed_folders.clear();
|
||||||
self.moved_asset_ids.clear();
|
self.moved_asset_ids.clear();
|
||||||
self.deleted_asset_ids.clear();
|
self.deleted_asset_ids.clear();
|
||||||
self.moved_subfolder_ids.clear();
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -34,10 +34,8 @@ pub mod convert_to_movie_clip;
|
||||||
pub mod region_split;
|
pub mod region_split;
|
||||||
pub mod toggle_group_expansion;
|
pub mod toggle_group_expansion;
|
||||||
pub mod group_layers;
|
pub mod group_layers;
|
||||||
pub mod raster_diff;
|
|
||||||
pub mod raster_stroke;
|
pub mod raster_stroke;
|
||||||
pub mod raster_fill;
|
pub mod raster_fill;
|
||||||
pub mod add_raster_keyframe;
|
|
||||||
pub mod move_layer;
|
pub mod move_layer;
|
||||||
pub mod set_fill_paint;
|
pub mod set_fill_paint;
|
||||||
|
|
||||||
|
|
@ -72,7 +70,6 @@ pub use toggle_group_expansion::ToggleGroupExpansionAction;
|
||||||
pub use group_layers::GroupLayersAction;
|
pub use group_layers::GroupLayersAction;
|
||||||
pub use raster_stroke::RasterStrokeAction;
|
pub use raster_stroke::RasterStrokeAction;
|
||||||
pub use raster_fill::RasterFillAction;
|
pub use raster_fill::RasterFillAction;
|
||||||
pub use add_raster_keyframe::AddRasterKeyframeAction;
|
|
||||||
pub use move_layer::MoveLayerAction;
|
pub use move_layer::MoveLayerAction;
|
||||||
pub use set_fill_paint::SetFillPaintAction;
|
pub use set_fill_paint::SetFillPaintAction;
|
||||||
pub use change_bpm::ChangeBpmAction;
|
pub use change_bpm::ChangeBpmAction;
|
||||||
|
|
|
||||||
|
|
@ -1,232 +0,0 @@
|
||||||
//! Dirty-rect diff for raster undo/redo.
|
|
||||||
//!
|
|
||||||
//! Brush strokes and fills used to store the *entire* before/after RGBA frame in the
|
|
||||||
//! undo stack (~8 MB each at 1080p → up to ~1.6 GB at the 100-action cap). A
|
|
||||||
//! [`RasterDiff`] instead stores only the changed bounding box's pixels before and
|
|
||||||
//! after, which for a typical brush dab is a few tens of KB.
|
|
||||||
//!
|
|
||||||
//! Applying a diff overwrites just the bbox of the keyframe's `raw_pixels`, so the
|
|
||||||
//! buffer **must be resident** (full length `w*h*4`) when `apply_*` runs. The editor
|
|
||||||
//! guarantees this by faulting the target frame in before undo/redo (a clean evicted
|
|
||||||
//! frame's container bytes equal its current logical state, so the restored base is
|
|
||||||
//! correct). If the base is somehow not resident we skip rather than corrupt.
|
|
||||||
|
|
||||||
/// Normalize a buffer to full length `n`; an empty/short buffer becomes transparent.
|
|
||||||
fn normalize(buf: &[u8], n: usize) -> std::borrow::Cow<[u8]> {
|
|
||||||
if buf.len() == n {
|
|
||||||
std::borrow::Cow::Borrowed(buf)
|
|
||||||
} else {
|
|
||||||
std::borrow::Cow::Owned(vec![0u8; n])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A minimal before/after record of the region a raster edit changed.
|
|
||||||
#[derive(Clone, Debug)]
|
|
||||||
pub struct RasterDiff {
|
|
||||||
full_width: u32,
|
|
||||||
full_height: u32,
|
|
||||||
/// Changed bounding box `(x, y, w, h)`; `None` when before == after (no-op).
|
|
||||||
bbox: Option<(u32, u32, u32, u32)>,
|
|
||||||
/// bbox-sized RGBA (`w*h*4`) of the region before the edit.
|
|
||||||
before_region: Vec<u8>,
|
|
||||||
/// bbox-sized RGBA (`w*h*4`) of the region after the edit.
|
|
||||||
after_region: Vec<u8>,
|
|
||||||
/// The pre-edit buffer was blank (empty/unallocated) — i.e. this was the first
|
|
||||||
/// edit on a fresh keyframe. Lets `apply_after` build from a transparent base
|
|
||||||
/// (the commit/redo path often starts with empty `raw_pixels`) and `apply_before`
|
|
||||||
/// restore to blank, instead of requiring a resident base.
|
|
||||||
before_blank: bool,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl RasterDiff {
|
|
||||||
/// Build a diff from full before/after buffers. `after` is expected to be the
|
|
||||||
/// resident post-edit buffer (`width*height*4`); `before` may be empty (a blank
|
|
||||||
/// keyframe's first stroke), treated as fully transparent.
|
|
||||||
pub fn compute(before: &[u8], after: &[u8], width: u32, height: u32) -> Self {
|
|
||||||
let n = width as usize * height as usize * 4;
|
|
||||||
let before_blank = before.len() != n;
|
|
||||||
// Normalize both sides to full length; empty/short ⇒ transparent.
|
|
||||||
let before_full = normalize(before, n);
|
|
||||||
let after_full = normalize(after, n);
|
|
||||||
|
|
||||||
// Find the tight bbox of differing pixels (compare 4-byte texels).
|
|
||||||
let (w, h) = (width as usize, height as usize);
|
|
||||||
let (mut min_x, mut min_y, mut max_x, mut max_y) = (usize::MAX, usize::MAX, 0usize, 0usize);
|
|
||||||
let mut any = false;
|
|
||||||
for y in 0..h {
|
|
||||||
let row = y * w * 4;
|
|
||||||
for x in 0..w {
|
|
||||||
let i = row + x * 4;
|
|
||||||
if before_full[i..i + 4] != after_full[i..i + 4] {
|
|
||||||
any = true;
|
|
||||||
if x < min_x { min_x = x; }
|
|
||||||
if x > max_x { max_x = x; }
|
|
||||||
if y < min_y { min_y = y; }
|
|
||||||
if y > max_y { max_y = y; }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if !any {
|
|
||||||
return Self { full_width: width, full_height: height, bbox: None,
|
|
||||||
before_region: Vec::new(), after_region: Vec::new(), before_blank };
|
|
||||||
}
|
|
||||||
|
|
||||||
let bw = max_x - min_x + 1;
|
|
||||||
let bh = max_y - min_y + 1;
|
|
||||||
let crop = |full: &[u8]| -> Vec<u8> {
|
|
||||||
let mut out = Vec::with_capacity(bw * bh * 4);
|
|
||||||
for row in 0..bh {
|
|
||||||
let src = ((min_y + row) * w + min_x) * 4;
|
|
||||||
out.extend_from_slice(&full[src..src + bw * 4]);
|
|
||||||
}
|
|
||||||
out
|
|
||||||
};
|
|
||||||
|
|
||||||
Self {
|
|
||||||
full_width: width,
|
|
||||||
full_height: height,
|
|
||||||
bbox: Some((min_x as u32, min_y as u32, bw as u32, bh as u32)),
|
|
||||||
before_region: crop(&before_full),
|
|
||||||
after_region: crop(&after_full),
|
|
||||||
before_blank,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Approximate retained size in bytes (the two cropped regions).
|
|
||||||
pub fn byte_size(&self) -> usize {
|
|
||||||
self.before_region.len() + self.after_region.len()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Restore the pre-edit pixels into `raw` (undo / first-execute rollback).
|
|
||||||
pub fn apply_before(&self, raw: &mut Vec<u8>) {
|
|
||||||
if self.bbox.is_none() {
|
|
||||||
return; // no change
|
|
||||||
}
|
|
||||||
if self.before_blank {
|
|
||||||
// The frame was blank before this edit (it was the first stroke); undoing
|
|
||||||
// it returns to blank regardless of the current buffer.
|
|
||||||
raw.clear();
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
self.stamp_resident(&self.before_region, raw);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Apply the post-edit pixels into `raw` (commit / redo).
|
|
||||||
pub fn apply_after(&self, raw: &mut Vec<u8>) {
|
|
||||||
if self.bbox.is_none() {
|
|
||||||
return; // no change
|
|
||||||
}
|
|
||||||
if self.before_blank {
|
|
||||||
// Base was blank: build a full transparent buffer then stamp the bbox. The
|
|
||||||
// commit/redo path frequently starts from empty `raw_pixels` here.
|
|
||||||
let n = self.full_width as usize * self.full_height as usize * 4;
|
|
||||||
raw.clear();
|
|
||||||
raw.resize(n, 0);
|
|
||||||
}
|
|
||||||
self.stamp_resident(&self.after_region, raw);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Stamp a bbox-sized region into `raw`, which must already be full-size. If it
|
|
||||||
/// isn't (a non-blank base that the editor failed to fault in), skip rather than
|
|
||||||
/// resize-and-corrupt — the frame will re-page to its container state.
|
|
||||||
fn stamp_resident(&self, region: &[u8], raw: &mut [u8]) {
|
|
||||||
let n = self.full_width as usize * self.full_height as usize * 4;
|
|
||||||
let (x, y, bw, bh) = match self.bbox {
|
|
||||||
Some(b) => b,
|
|
||||||
None => return,
|
|
||||||
};
|
|
||||||
if raw.len() != n {
|
|
||||||
eprintln!(
|
|
||||||
"⚠️ [RASTER_DIFF] base not resident ({} != {}); skipping undo/redo apply",
|
|
||||||
raw.len(), n
|
|
||||||
);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
let (x, y, bw, bh) = (x as usize, y as usize, bw as usize, bh as usize);
|
|
||||||
let fw = self.full_width as usize;
|
|
||||||
for row in 0..bh {
|
|
||||||
let dst = ((y + row) * fw + x) * 4;
|
|
||||||
let src = row * bw * 4;
|
|
||||||
raw[dst..dst + bw * 4].copy_from_slice(®ion[src..src + bw * 4]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
fn solid(w: u32, h: u32, px: [u8; 4]) -> Vec<u8> {
|
|
||||||
px.iter().copied().cycle().take((w * h * 4) as usize).collect()
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn roundtrip_reproduces_buffers_exactly() {
|
|
||||||
let (w, h) = (8, 6);
|
|
||||||
let before = solid(w, h, [10, 20, 30, 255]);
|
|
||||||
let mut after = before.clone();
|
|
||||||
// Change a 2x2 region at (3,2).
|
|
||||||
for (dy, dx) in [(0, 0), (0, 1), (1, 0), (1, 1)] {
|
|
||||||
let i = (((2 + dy) * w + (3 + dx)) * 4) as usize;
|
|
||||||
after[i..i + 4].copy_from_slice(&[200, 100, 50, 255]);
|
|
||||||
}
|
|
||||||
let diff = RasterDiff::compute(&before, &after, w, h);
|
|
||||||
assert_eq!(diff.bbox, Some((3, 2, 2, 2)));
|
|
||||||
|
|
||||||
let mut buf = after.clone();
|
|
||||||
diff.apply_before(&mut buf);
|
|
||||||
assert_eq!(buf, before, "undo must reproduce the pre-edit buffer exactly");
|
|
||||||
diff.apply_after(&mut buf);
|
|
||||||
assert_eq!(buf, after, "redo must reproduce the post-edit buffer exactly");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn blank_before_first_stroke() {
|
|
||||||
let (w, h) = (4, 4);
|
|
||||||
let n = (w * h * 4) as usize;
|
|
||||||
let before: Vec<u8> = Vec::new(); // blank keyframe
|
|
||||||
let mut after = vec![0u8; n];
|
|
||||||
let i = ((1 * w + 1) * 4) as usize;
|
|
||||||
after[i..i + 4].copy_from_slice(&[255, 0, 0, 255]);
|
|
||||||
let diff = RasterDiff::compute(&before, &after, w, h);
|
|
||||||
assert_eq!(diff.bbox, Some((1, 1, 1, 1)));
|
|
||||||
|
|
||||||
// First execute / redo from EMPTY raw_pixels (the real commit path): builds
|
|
||||||
// the full buffer from transparent + the stroke.
|
|
||||||
let mut buf: Vec<u8> = Vec::new();
|
|
||||||
diff.apply_after(&mut buf);
|
|
||||||
assert_eq!(buf, after, "commit/redo must build the frame from a blank base");
|
|
||||||
|
|
||||||
// Undo the first stroke → back to blank (empty).
|
|
||||||
diff.apply_before(&mut buf);
|
|
||||||
assert!(buf.is_empty(), "undoing the first stroke restores the blank keyframe");
|
|
||||||
|
|
||||||
// Redo again from the now-empty buffer.
|
|
||||||
diff.apply_after(&mut buf);
|
|
||||||
assert_eq!(buf, after);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn no_change_is_noop() {
|
|
||||||
let (w, h) = (4, 4);
|
|
||||||
let buf = solid(w, h, [1, 2, 3, 4]);
|
|
||||||
let diff = RasterDiff::compute(&buf, &buf, w, h);
|
|
||||||
assert_eq!(diff.bbox, None);
|
|
||||||
assert_eq!(diff.byte_size(), 0);
|
|
||||||
let mut b = buf.clone();
|
|
||||||
diff.apply_before(&mut b);
|
|
||||||
assert_eq!(b, buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn not_resident_base_is_skipped_not_corrupted() {
|
|
||||||
let (w, h) = (4, 4);
|
|
||||||
let before = solid(w, h, [9, 9, 9, 255]);
|
|
||||||
let after = solid(w, h, [1, 2, 3, 255]);
|
|
||||||
let diff = RasterDiff::compute(&before, &after, w, h);
|
|
||||||
let mut empty: Vec<u8> = Vec::new();
|
|
||||||
diff.apply_before(&mut empty); // base not resident
|
|
||||||
assert!(empty.is_empty(), "must not resize/corrupt a non-resident base");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
//! Raster flood-fill action — records and undoes a paint bucket fill on a RasterLayer.
|
//! Raster flood-fill action — records and undoes a paint bucket fill on a RasterLayer.
|
||||||
|
|
||||||
use crate::action::Action;
|
use crate::action::Action;
|
||||||
use crate::actions::raster_diff::RasterDiff;
|
|
||||||
use crate::document::Document;
|
use crate::document::Document;
|
||||||
use crate::layer::AnyLayer;
|
use crate::layer::AnyLayer;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
@ -9,13 +8,11 @@ use uuid::Uuid;
|
||||||
pub struct RasterFillAction {
|
pub struct RasterFillAction {
|
||||||
layer_id: Uuid,
|
layer_id: Uuid,
|
||||||
time: f64,
|
time: f64,
|
||||||
|
buffer_before: Vec<u8>,
|
||||||
|
buffer_after: Vec<u8>,
|
||||||
width: u32,
|
width: u32,
|
||||||
height: u32,
|
height: u32,
|
||||||
name: String,
|
name: String,
|
||||||
diff: RasterDiff,
|
|
||||||
/// Full post-fill buffer, kept only for the first `execute` (commit); see
|
|
||||||
/// `RasterStrokeAction::full_after`.
|
|
||||||
full_after: Option<Vec<u8>>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RasterFillAction {
|
impl RasterFillAction {
|
||||||
|
|
@ -27,9 +24,7 @@ impl RasterFillAction {
|
||||||
width: u32,
|
width: u32,
|
||||||
height: u32,
|
height: u32,
|
||||||
) -> Self {
|
) -> Self {
|
||||||
let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
|
Self { layer_id, time, buffer_before, buffer_after, width, height, name: "Flood fill".to_string() }
|
||||||
Self { layer_id, time, width, height, name: "Flood fill".to_string(),
|
|
||||||
diff, full_after: Some(buffer_after) }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn with_description(mut self, name: &str) -> Self {
|
pub fn with_description(mut self, name: &str) -> Self {
|
||||||
|
|
@ -46,17 +41,9 @@ impl Action for RasterFillAction {
|
||||||
AnyLayer::Raster(rl) => rl,
|
AnyLayer::Raster(rl) => rl,
|
||||||
_ => return Err("Not a raster layer".to_string()),
|
_ => return Err("Not a raster layer".to_string()),
|
||||||
};
|
};
|
||||||
let _ = (self.width, self.height);
|
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height);
|
||||||
let kf = raster
|
kf.raw_pixels = self.buffer_after.clone();
|
||||||
.keyframe_at_mut(self.time)
|
|
||||||
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
|
|
||||||
if let Some(full) = self.full_after.take() {
|
|
||||||
kf.raw_pixels = full;
|
|
||||||
} else {
|
|
||||||
self.diff.apply_after(&mut kf.raw_pixels);
|
|
||||||
}
|
|
||||||
kf.texture_dirty = true;
|
kf.texture_dirty = true;
|
||||||
kf.dirty = true;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -67,21 +54,13 @@ impl Action for RasterFillAction {
|
||||||
AnyLayer::Raster(rl) => rl,
|
AnyLayer::Raster(rl) => rl,
|
||||||
_ => return Err("Not a raster layer".to_string()),
|
_ => return Err("Not a raster layer".to_string()),
|
||||||
};
|
};
|
||||||
let _ = (self.width, self.height);
|
let kf = raster.ensure_keyframe_at(self.time, self.width, self.height);
|
||||||
let kf = raster
|
kf.raw_pixels = self.buffer_before.clone();
|
||||||
.keyframe_at_mut(self.time)
|
|
||||||
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
|
|
||||||
self.diff.apply_before(&mut kf.raw_pixels);
|
|
||||||
kf.texture_dirty = true;
|
kf.texture_dirty = true;
|
||||||
kf.dirty = true;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn description(&self) -> String {
|
fn description(&self) -> String {
|
||||||
self.name.clone()
|
self.name.clone()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
|
|
||||||
Some((self.layer_id, self.time))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,6 @@
|
||||||
//! `rollback` → swap in `buffer_before`
|
//! `rollback` → swap in `buffer_before`
|
||||||
|
|
||||||
use crate::action::Action;
|
use crate::action::Action;
|
||||||
use crate::actions::raster_diff::RasterDiff;
|
|
||||||
use crate::document::Document;
|
use crate::document::Document;
|
||||||
use crate::layer::AnyLayer;
|
use crate::layer::AnyLayer;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
@ -17,20 +16,16 @@ use uuid::Uuid;
|
||||||
/// Action that records a single brush stroke for undo/redo.
|
/// Action that records a single brush stroke for undo/redo.
|
||||||
///
|
///
|
||||||
/// The stroke must already be painted into the document's `raw_pixels` before
|
/// The stroke must already be painted into the document's `raw_pixels` before
|
||||||
/// this action is executed for the first time. Only the changed bounding box is
|
/// this action is executed for the first time.
|
||||||
/// retained (see [`RasterDiff`]) rather than two full frame buffers.
|
|
||||||
pub struct RasterStrokeAction {
|
pub struct RasterStrokeAction {
|
||||||
layer_id: Uuid,
|
layer_id: Uuid,
|
||||||
time: f64,
|
time: f64,
|
||||||
|
/// Raw RGBA pixels *before* the stroke (for rollback / undo)
|
||||||
|
buffer_before: Vec<u8>,
|
||||||
|
/// Raw RGBA pixels *after* the stroke (for execute / redo)
|
||||||
|
buffer_after: Vec<u8>,
|
||||||
width: u32,
|
width: u32,
|
||||||
height: u32,
|
height: u32,
|
||||||
diff: RasterDiff,
|
|
||||||
/// The full post-stroke buffer, kept ONLY for the first `execute` (the commit),
|
|
||||||
/// which establishes `raw_pixels` exactly like the old code did — robust no matter
|
|
||||||
/// what state the working buffer is in (empty new keyframe, GPU-canvas readback,
|
|
||||||
/// etc.). Taken (dropped) on first execute, so the action sitting in the undo stack
|
|
||||||
/// retains only the small `diff`; redo then replays via the diff.
|
|
||||||
full_after: Option<Vec<u8>>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RasterStrokeAction {
|
impl RasterStrokeAction {
|
||||||
|
|
@ -38,8 +33,6 @@ impl RasterStrokeAction {
|
||||||
///
|
///
|
||||||
/// * `buffer_before` – raw RGBA pixels captured just before the stroke began.
|
/// * `buffer_before` – raw RGBA pixels captured just before the stroke began.
|
||||||
/// * `buffer_after` – raw RGBA pixels captured just after the stroke finished.
|
/// * `buffer_after` – raw RGBA pixels captured just after the stroke finished.
|
||||||
///
|
|
||||||
/// The full buffers are diffed down to the changed bbox here and then dropped.
|
|
||||||
pub fn new(
|
pub fn new(
|
||||||
layer_id: Uuid,
|
layer_id: Uuid,
|
||||||
time: f64,
|
time: f64,
|
||||||
|
|
@ -48,41 +41,28 @@ impl RasterStrokeAction {
|
||||||
width: u32,
|
width: u32,
|
||||||
height: u32,
|
height: u32,
|
||||||
) -> Self {
|
) -> Self {
|
||||||
let diff = RasterDiff::compute(&buffer_before, &buffer_after, width, height);
|
Self { layer_id, time, buffer_before, buffer_after, width, height }
|
||||||
Self { layer_id, time, width, height, diff, full_after: Some(buffer_after) }
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Action for RasterStrokeAction {
|
impl Action for RasterStrokeAction {
|
||||||
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
|
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
|
||||||
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
|
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
|
||||||
if let Some(full) = self.full_after.take() {
|
kf.raw_pixels = self.buffer_after.clone();
|
||||||
// First execute (commit): assign the full buffer outright.
|
|
||||||
kf.raw_pixels = full;
|
|
||||||
} else {
|
|
||||||
// Redo: replay via the diff onto the (resident) base.
|
|
||||||
self.diff.apply_after(&mut kf.raw_pixels);
|
|
||||||
}
|
|
||||||
kf.texture_dirty = true;
|
kf.texture_dirty = true;
|
||||||
kf.dirty = true;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
|
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
|
||||||
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
|
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
|
||||||
self.diff.apply_before(&mut kf.raw_pixels);
|
kf.raw_pixels = self.buffer_before.clone();
|
||||||
kf.texture_dirty = true;
|
kf.texture_dirty = true;
|
||||||
kf.dirty = true;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn description(&self) -> String {
|
fn description(&self) -> String {
|
||||||
"Paint stroke".to_string()
|
"Paint stroke".to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn raster_resident_hint(&self) -> Option<(Uuid, f64)> {
|
|
||||||
Some((self.layer_id, self.time))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn get_keyframe_mut<'a>(
|
fn get_keyframe_mut<'a>(
|
||||||
|
|
@ -99,10 +79,5 @@ fn get_keyframe_mut<'a>(
|
||||||
AnyLayer::Raster(rl) => rl,
|
AnyLayer::Raster(rl) => rl,
|
||||||
_ => return Err("Not a raster layer".to_string()),
|
_ => return Err("Not a raster layer".to_string()),
|
||||||
};
|
};
|
||||||
let _ = (width, height);
|
Ok(raster.ensure_keyframe_at(time, width, height))
|
||||||
// Edit the ACTIVE keyframe (at-or-before `time`); never create one — keyframes
|
|
||||||
// are made explicitly via "New Keyframe".
|
|
||||||
raster
|
|
||||||
.keyframe_at_mut(time)
|
|
||||||
.ok_or_else(|| format!("No raster keyframe at/before t={time}"))
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -547,8 +547,6 @@ mod tests {
|
||||||
|
|
||||||
// Create a clip ID (ClipInstance references clip by ID)
|
// Create a clip ID (ClipInstance references clip by ID)
|
||||||
let clip_id = uuid::Uuid::new_v4();
|
let clip_id = uuid::Uuid::new_v4();
|
||||||
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
|
|
||||||
document.vector_clips.insert(clip_id, clip);
|
|
||||||
|
|
||||||
let mut vector_layer = VectorLayer::new("Layer 1");
|
let mut vector_layer = VectorLayer::new("Layer 1");
|
||||||
|
|
||||||
|
|
@ -609,8 +607,6 @@ mod tests {
|
||||||
|
|
||||||
// Create a clip ID (ClipInstance references clip by ID)
|
// Create a clip ID (ClipInstance references clip by ID)
|
||||||
let clip_id = uuid::Uuid::new_v4();
|
let clip_id = uuid::Uuid::new_v4();
|
||||||
let clip = crate::clip::VectorClip::with_id(clip_id, "Test Clip", 100.0, 100.0, 10.0);
|
|
||||||
document.vector_clips.insert(clip_id, clip);
|
|
||||||
|
|
||||||
let mut vector_layer = VectorLayer::new("Layer 1");
|
let mut vector_layer = VectorLayer::new("Layer 1");
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,813 +0,0 @@
|
||||||
//! SQLite-backed `.beam` project container.
|
|
||||||
//!
|
|
||||||
//! The `.beam` format is a single SQLite database file. It replaces the older
|
|
||||||
//! ZIP-archive format. SQLite gives us, in one file:
|
|
||||||
//!
|
|
||||||
//! - **Streaming reads** — packed media is split into chunk rows and read on
|
|
||||||
//! demand through [`BlobReader`] (`Read + Seek`), so arbitrary-length audio /
|
|
||||||
//! video never has to be fully decoded into RAM on load.
|
|
||||||
//! - **In-place, crash-safe mutation** — raster frame write-back and re-save are
|
|
||||||
//! transactional `UPDATE`s rather than rewriting a whole archive.
|
|
||||||
//! - **Single-file UX** — behaves like a file on every platform.
|
|
||||||
//!
|
|
||||||
//! ## Media storage
|
|
||||||
//!
|
|
||||||
//! Each media item is one row in `media` plus, when *packed*, N rows in
|
|
||||||
//! `media_chunk`:
|
|
||||||
//!
|
|
||||||
//! - **Packed** (`MediaStorage::Packed`) — bytes live in the database, split
|
|
||||||
//! into [`CHUNK_SIZE`]-byte chunks. Chunking keeps each blob well under
|
|
||||||
//! SQLite's ~2 GB per-blob ceiling and bounds the working set of a streaming
|
|
||||||
//! reader to a single chunk.
|
|
||||||
//! - **Referenced** (`MediaStorage::Referenced`) — only an external path is
|
|
||||||
//! stored; the bytes stay on disk (useful for shared media on a network drive,
|
|
||||||
//! or media too large/volatile to pack). Callers open the path directly.
|
|
||||||
//!
|
|
||||||
//! `project.json` (the serialized `BeamProject`) is stored verbatim in the
|
|
||||||
//! single-row `project_json` table; only the container and media storage change
|
|
||||||
//! relative to the legacy format.
|
|
||||||
|
|
||||||
use rusqlite::blob::Blob;
|
|
||||||
use rusqlite::{Connection, DatabaseName, OpenFlags, OptionalExtension};
|
|
||||||
use std::io::{self, Read, Seek, SeekFrom};
|
|
||||||
use std::path::Path;
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
/// Default packed-media chunk size: 4 MiB.
|
|
||||||
///
|
|
||||||
/// Small enough to bound a streaming reader's per-chunk work and any
|
|
||||||
/// whole-chunk buffering, large enough to keep row counts modest (a 1 GB file
|
|
||||||
/// is 256 rows). Comfortably under SQLite's ~2 GB per-blob limit.
|
|
||||||
pub const CHUNK_SIZE: u64 = 4 * 1024 * 1024;
|
|
||||||
|
|
||||||
/// Files at or above this size prompt the user to pick packed vs referenced
|
|
||||||
/// (and the choice is then persisted as the default). Matches SQLite's
|
|
||||||
/// practical large-blob threshold.
|
|
||||||
pub const LARGE_MEDIA_THRESHOLD: u64 = 2 * 1024 * 1024 * 1024;
|
|
||||||
|
|
||||||
/// Kind of media stored in the `media` table.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
||||||
pub enum MediaKind {
|
|
||||||
Audio = 0,
|
|
||||||
Video = 1,
|
|
||||||
Raster = 2,
|
|
||||||
ImageAsset = 3,
|
|
||||||
/// A precomputed waveform LOD pyramid blob for an audio item (keyed by the
|
|
||||||
/// same id as the audio it describes). See `daw_backend::audio::waveform_pyramid`.
|
|
||||||
Waveform = 4,
|
|
||||||
/// A pack of precomputed video thumbnails for a video clip (keyed by a
|
|
||||||
/// sentinel-derived id from the clip id). Opaque blob; format owned by the editor.
|
|
||||||
Thumbnail = 5,
|
|
||||||
/// A low-res PNG proxy of a raster keyframe (keyed by a sentinel-derived id from
|
|
||||||
/// the keyframe id). Decoded eagerly on load and shown while the full-res pixels
|
|
||||||
/// page in, so cold scrubs don't flash blank. See `raster_proxy_media_id`.
|
|
||||||
RasterProxy = 6,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MediaKind {
|
|
||||||
fn from_i64(v: i64) -> Option<Self> {
|
|
||||||
match v {
|
|
||||||
0 => Some(Self::Audio),
|
|
||||||
1 => Some(Self::Video),
|
|
||||||
2 => Some(Self::Raster),
|
|
||||||
3 => Some(Self::ImageAsset),
|
|
||||||
4 => Some(Self::Waveform),
|
|
||||||
5 => Some(Self::Thumbnail),
|
|
||||||
6 => Some(Self::RasterProxy),
|
|
||||||
_ => None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// How a media item's bytes are stored.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
||||||
pub enum MediaStorage {
|
|
||||||
/// Bytes are chunked into `media_chunk` rows inside the database.
|
|
||||||
Packed = 0,
|
|
||||||
/// Only an external path is stored; bytes live on disk.
|
|
||||||
Referenced = 1,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MediaStorage {
|
|
||||||
fn from_i64(v: i64) -> Option<Self> {
|
|
||||||
match v {
|
|
||||||
0 => Some(Self::Packed),
|
|
||||||
1 => Some(Self::Referenced),
|
|
||||||
_ => None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Metadata row for a media item (no bytes).
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct MediaInfo {
|
|
||||||
pub id: Uuid,
|
|
||||||
pub kind: MediaKind,
|
|
||||||
/// Original codec / container extension, e.g. `"flac"`, `"mp3"`, `"png"`.
|
|
||||||
pub codec: String,
|
|
||||||
pub storage: MediaStorage,
|
|
||||||
/// Set when `storage == Referenced`.
|
|
||||||
pub ext_path: Option<String>,
|
|
||||||
/// Total byte length of the media payload (packed only; 0 for referenced).
|
|
||||||
pub total_len: u64,
|
|
||||||
// Kind-specific metadata (nullable; meaning depends on `kind`).
|
|
||||||
pub channels: Option<u32>,
|
|
||||||
pub sample_rate: Option<u32>,
|
|
||||||
pub width: Option<u32>,
|
|
||||||
pub height: Option<u32>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Optional kind-specific metadata supplied when writing a media item.
|
|
||||||
#[derive(Debug, Clone, Copy, Default)]
|
|
||||||
pub struct MediaMeta {
|
|
||||||
pub channels: Option<u32>,
|
|
||||||
pub sample_rate: Option<u32>,
|
|
||||||
pub width: Option<u32>,
|
|
||||||
pub height: Option<u32>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A `.beam` project container backed by a SQLite database.
|
|
||||||
pub struct BeamArchive {
|
|
||||||
conn: Connection,
|
|
||||||
chunk_size: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl BeamArchive {
|
|
||||||
/// Schema version stored in `meta` under `"schema_version"`.
|
|
||||||
pub const SCHEMA_VERSION: i64 = 1;
|
|
||||||
|
|
||||||
/// Create a new (empty) archive at `path`, replacing any existing file.
|
|
||||||
pub fn create(path: &Path) -> Result<Self, String> {
|
|
||||||
// Remove any existing file so we start from a clean schema.
|
|
||||||
if path.exists() {
|
|
||||||
std::fs::remove_file(path).map_err(|e| format!("Failed to replace {:?}: {}", path, e))?;
|
|
||||||
}
|
|
||||||
let conn = Connection::open(path).map_err(map_sql)?;
|
|
||||||
let mut archive = Self { conn, chunk_size: CHUNK_SIZE };
|
|
||||||
archive.init_schema()?;
|
|
||||||
Ok(archive)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Open an existing archive for read/write.
|
|
||||||
pub fn open(path: &Path) -> Result<Self, String> {
|
|
||||||
let conn = Connection::open(path).map_err(map_sql)?;
|
|
||||||
let archive = Self { conn, chunk_size: CHUNK_SIZE };
|
|
||||||
archive.verify_schema()?;
|
|
||||||
Ok(archive)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Quick check: does `path` look like a SQLite database (vs. a legacy ZIP)?
|
|
||||||
/// Reads the 16-byte SQLite header magic. Used to route between the SQLite
|
|
||||||
/// loader and the legacy-ZIP migration path.
|
|
||||||
pub fn is_sqlite(path: &Path) -> bool {
|
|
||||||
use std::io::Read as _;
|
|
||||||
let mut f = match std::fs::File::open(path) {
|
|
||||||
Ok(f) => f,
|
|
||||||
Err(_) => return false,
|
|
||||||
};
|
|
||||||
let mut magic = [0u8; 16];
|
|
||||||
if f.read_exact(&mut magic).is_err() {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
&magic == b"SQLite format 3\0"
|
|
||||||
}
|
|
||||||
|
|
||||||
fn init_schema(&mut self) -> Result<(), String> {
|
|
||||||
self.conn
|
|
||||||
.execute_batch(
|
|
||||||
"BEGIN;
|
|
||||||
CREATE TABLE media (
|
|
||||||
id BLOB PRIMARY KEY, -- 16-byte Uuid
|
|
||||||
kind INTEGER NOT NULL,
|
|
||||||
codec TEXT NOT NULL,
|
|
||||||
storage INTEGER NOT NULL,
|
|
||||||
ext_path TEXT,
|
|
||||||
total_len INTEGER NOT NULL DEFAULT 0,
|
|
||||||
channels INTEGER,
|
|
||||||
sample_rate INTEGER,
|
|
||||||
width INTEGER,
|
|
||||||
height INTEGER
|
|
||||||
);
|
|
||||||
CREATE TABLE media_chunk (
|
|
||||||
id INTEGER PRIMARY KEY, -- rowid, for blob_open
|
|
||||||
media_id BLOB NOT NULL,
|
|
||||||
chunk_index INTEGER NOT NULL,
|
|
||||||
bytes BLOB NOT NULL,
|
|
||||||
UNIQUE (media_id, chunk_index)
|
|
||||||
);
|
|
||||||
CREATE TABLE project_json (
|
|
||||||
id INTEGER PRIMARY KEY CHECK (id = 0),
|
|
||||||
data TEXT NOT NULL
|
|
||||||
);
|
|
||||||
CREATE TABLE meta (
|
|
||||||
key TEXT PRIMARY KEY,
|
|
||||||
value TEXT NOT NULL
|
|
||||||
);
|
|
||||||
COMMIT;",
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
self.set_meta("schema_version", &Self::SCHEMA_VERSION.to_string())?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn verify_schema(&self) -> Result<(), String> {
|
|
||||||
let v: Option<String> = self.get_meta("schema_version")?;
|
|
||||||
match v.as_deref().and_then(|s| s.parse::<i64>().ok()) {
|
|
||||||
Some(n) if n <= Self::SCHEMA_VERSION => Ok(()),
|
|
||||||
Some(n) => Err(format!(
|
|
||||||
"Unsupported .beam schema version {} (this build supports up to {})",
|
|
||||||
n,
|
|
||||||
Self::SCHEMA_VERSION
|
|
||||||
)),
|
|
||||||
None => Err("Not a valid .beam archive (missing schema_version)".to_string()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Begin a write transaction grouping multiple media/json writes into one
|
|
||||||
/// atomic, crash-safe commit. Used by saves so unchanged (large) media is
|
|
||||||
/// never rewritten — only dirty rows are touched, in place.
|
|
||||||
pub fn transaction(&mut self) -> Result<BeamTxn<'_>, String> {
|
|
||||||
let tx = self.conn.transaction().map_err(map_sql)?;
|
|
||||||
Ok(BeamTxn { tx, chunk_size: self.chunk_size })
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- meta key/value --------------------------------------------------
|
|
||||||
|
|
||||||
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
|
|
||||||
set_meta_conn(&self.conn, key, value)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn get_meta(&self, key: &str) -> Result<Option<String>, String> {
|
|
||||||
self.conn
|
|
||||||
.query_row("SELECT value FROM meta WHERE key = ?1", [key], |r| r.get(0))
|
|
||||||
.optional()
|
|
||||||
.map_err(map_sql)
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- project.json ----------------------------------------------------
|
|
||||||
|
|
||||||
/// Store the serialized `project.json` (single row).
|
|
||||||
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
|
|
||||||
set_project_json_conn(&self.conn, json)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read the serialized `project.json`.
|
|
||||||
pub fn get_project_json(&self) -> Result<String, String> {
|
|
||||||
self.conn
|
|
||||||
.query_row("SELECT data FROM project_json WHERE id = 0", [], |r| r.get(0))
|
|
||||||
.optional()
|
|
||||||
.map_err(map_sql)?
|
|
||||||
.ok_or_else(|| "Archive has no project.json".to_string())
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- media write -----------------------------------------------------
|
|
||||||
|
|
||||||
/// Write a media item whose bytes are packed (chunked) into the database.
|
|
||||||
/// Replaces any existing rows for `id`.
|
|
||||||
pub fn put_media_packed(
|
|
||||||
&mut self,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
bytes: &[u8],
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let tx = self.conn.transaction().map_err(map_sql)?;
|
|
||||||
write_media_packed(&tx, self.chunk_size, id, kind, codec, bytes, meta)?;
|
|
||||||
tx.commit().map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Write a media item that references an external file by path (no bytes
|
|
||||||
/// stored). Replaces any existing rows for `id`.
|
|
||||||
pub fn put_media_referenced(
|
|
||||||
&mut self,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
ext_path: &str,
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let tx = self.conn.transaction().map_err(map_sql)?;
|
|
||||||
write_media_referenced(&tx, id, kind, codec, ext_path, meta)?;
|
|
||||||
tx.commit().map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- media read ------------------------------------------------------
|
|
||||||
|
|
||||||
/// Look up a media item's metadata.
|
|
||||||
pub fn media_info(&self, id: Uuid) -> Result<Option<MediaInfo>, String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
self.conn
|
|
||||||
.query_row(
|
|
||||||
"SELECT kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height
|
|
||||||
FROM media WHERE id = ?1",
|
|
||||||
[&id_bytes],
|
|
||||||
|r| {
|
|
||||||
Ok((
|
|
||||||
r.get::<_, i64>(0)?,
|
|
||||||
r.get::<_, String>(1)?,
|
|
||||||
r.get::<_, i64>(2)?,
|
|
||||||
r.get::<_, Option<String>>(3)?,
|
|
||||||
r.get::<_, i64>(4)?,
|
|
||||||
r.get::<_, Option<u32>>(5)?,
|
|
||||||
r.get::<_, Option<u32>>(6)?,
|
|
||||||
r.get::<_, Option<u32>>(7)?,
|
|
||||||
r.get::<_, Option<u32>>(8)?,
|
|
||||||
))
|
|
||||||
},
|
|
||||||
)
|
|
||||||
.optional()
|
|
||||||
.map_err(map_sql)?
|
|
||||||
.map(|(kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)| {
|
|
||||||
Ok(MediaInfo {
|
|
||||||
id,
|
|
||||||
kind: MediaKind::from_i64(kind)
|
|
||||||
.ok_or_else(|| format!("Unknown media kind {}", kind))?,
|
|
||||||
codec,
|
|
||||||
storage: MediaStorage::from_i64(storage)
|
|
||||||
.ok_or_else(|| format!("Unknown media storage {}", storage))?,
|
|
||||||
ext_path,
|
|
||||||
total_len: total_len.max(0) as u64,
|
|
||||||
channels,
|
|
||||||
sample_rate,
|
|
||||||
width,
|
|
||||||
height,
|
|
||||||
})
|
|
||||||
})
|
|
||||||
.transpose()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// List every media item of a given kind.
|
|
||||||
pub fn media_ids_of_kind(&self, kind: MediaKind) -> Result<Vec<Uuid>, String> {
|
|
||||||
let mut stmt = self
|
|
||||||
.conn
|
|
||||||
.prepare("SELECT id FROM media WHERE kind = ?1")
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let rows = stmt
|
|
||||||
.query_map([kind as i64], |r| r.get::<_, Vec<u8>>(0))
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
for row in rows {
|
|
||||||
let bytes = row.map_err(map_sql)?;
|
|
||||||
out.push(uuid_from_bytes(&bytes)?);
|
|
||||||
}
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read an entire packed media item into memory. Convenience for small
|
|
||||||
/// media (raster frames, image assets); large media should stream via
|
|
||||||
/// [`BeamArchive::open_blob_reader`] instead.
|
|
||||||
pub fn read_media_full(&self, id: Uuid) -> Result<Vec<u8>, String> {
|
|
||||||
let info = self
|
|
||||||
.media_info(id)?
|
|
||||||
.ok_or_else(|| format!("Media {} not found", id))?;
|
|
||||||
if info.storage != MediaStorage::Packed {
|
|
||||||
return Err(format!("Media {} is referenced, not packed", id));
|
|
||||||
}
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
let mut stmt = self
|
|
||||||
.conn
|
|
||||||
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let rows = stmt
|
|
||||||
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let mut out = Vec::with_capacity(info.total_len as usize);
|
|
||||||
for row in rows {
|
|
||||||
out.extend_from_slice(&row.map_err(map_sql)?);
|
|
||||||
}
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Open a streaming reader over a packed media item. The reader owns its own
|
|
||||||
/// SQLite connection (read-only) so it can live on a separate thread (e.g.
|
|
||||||
/// the audio disk reader) independent of this archive handle.
|
|
||||||
pub fn open_blob_reader(&self, db_path: &Path, id: Uuid) -> Result<BlobReader, String> {
|
|
||||||
let info = self
|
|
||||||
.media_info(id)?
|
|
||||||
.ok_or_else(|| format!("Media {} not found", id))?;
|
|
||||||
if info.storage != MediaStorage::Packed {
|
|
||||||
return Err(format!("Media {} is referenced, not packed", id));
|
|
||||||
}
|
|
||||||
BlobReader::open(db_path, id, info.total_len, self.chunk_size)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Override the chunk size (testing / tuning). Affects subsequent writes.
|
|
||||||
#[doc(hidden)]
|
|
||||||
pub fn set_chunk_size(&mut self, chunk_size: u64) {
|
|
||||||
assert!(chunk_size > 0);
|
|
||||||
self.chunk_size = chunk_size;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Streaming reader (`Read + Seek`) over a packed media item's chunk rows.
|
|
||||||
///
|
|
||||||
/// Owns a dedicated read-only SQLite connection so it is independent of the
|
|
||||||
/// writing [`BeamArchive`] handle and can be moved to another thread. Each
|
|
||||||
/// `read` opens a blob handle on the current chunk's row via `blob_open` (no
|
|
||||||
/// per-read query — chunk rowids are resolved once up front) and reads up to the
|
|
||||||
/// chunk boundary; callers that issue many tiny reads should wrap this in a
|
|
||||||
/// `BufReader`.
|
|
||||||
pub struct BlobReader {
|
|
||||||
conn: Connection,
|
|
||||||
chunk_rowids: Vec<i64>,
|
|
||||||
chunk_size: u64,
|
|
||||||
total_len: u64,
|
|
||||||
pos: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl BlobReader {
|
|
||||||
fn open(db_path: &Path, id: Uuid, total_len: u64, chunk_size: u64) -> Result<Self, String> {
|
|
||||||
let conn = Connection::open_with_flags(
|
|
||||||
db_path,
|
|
||||||
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
let mut stmt = conn
|
|
||||||
.prepare("SELECT id FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let rows = stmt
|
|
||||||
.query_map([&id_bytes], |r| r.get::<_, i64>(0))
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let mut chunk_rowids = Vec::new();
|
|
||||||
for row in rows {
|
|
||||||
chunk_rowids.push(row.map_err(map_sql)?);
|
|
||||||
}
|
|
||||||
drop(stmt);
|
|
||||||
Ok(Self { conn, chunk_rowids, chunk_size, total_len, pos: 0 })
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Total length of the media payload in bytes.
|
|
||||||
pub fn len(&self) -> u64 {
|
|
||||||
self.total_len
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn is_empty(&self) -> bool {
|
|
||||||
self.total_len == 0
|
|
||||||
}
|
|
||||||
|
|
||||||
fn chunk_blob(&self, chunk_index: usize) -> io::Result<Blob<'_>> {
|
|
||||||
let rowid = *self
|
|
||||||
.chunk_rowids
|
|
||||||
.get(chunk_index)
|
|
||||||
.ok_or_else(|| io::Error::new(io::ErrorKind::UnexpectedEof, "chunk index out of range"))?;
|
|
||||||
self.conn
|
|
||||||
.blob_open(DatabaseName::Main, "media_chunk", "bytes", rowid, true)
|
|
||||||
.map_err(|e| io::Error::new(io::ErrorKind::Other, e))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Read for BlobReader {
|
|
||||||
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
|
|
||||||
if self.pos >= self.total_len || buf.is_empty() {
|
|
||||||
return Ok(0);
|
|
||||||
}
|
|
||||||
let chunk_index = (self.pos / self.chunk_size) as usize;
|
|
||||||
let off_in_chunk = self.pos % self.chunk_size;
|
|
||||||
|
|
||||||
// The chunk's length is derivable from total_len/chunk_size, so we don't
|
|
||||||
// depend on Blob::len(): every chunk but the last is exactly chunk_size.
|
|
||||||
let chunk_start = chunk_index as u64 * self.chunk_size;
|
|
||||||
let chunk_len = (self.total_len - chunk_start).min(self.chunk_size);
|
|
||||||
let avail_in_chunk = (chunk_len - off_in_chunk) as usize;
|
|
||||||
let avail_total = (self.total_len - self.pos) as usize;
|
|
||||||
let want = buf.len().min(avail_in_chunk).min(avail_total);
|
|
||||||
|
|
||||||
// Scope the blob borrow (it borrows `self.conn`) so it ends before we
|
|
||||||
// mutate `self.pos`.
|
|
||||||
let n = {
|
|
||||||
let mut blob = self.chunk_blob(chunk_index)?;
|
|
||||||
blob.seek(SeekFrom::Start(off_in_chunk))?;
|
|
||||||
blob.read(&mut buf[..want])?
|
|
||||||
};
|
|
||||||
self.pos += n as u64;
|
|
||||||
Ok(n)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Seek for BlobReader {
|
|
||||||
fn seek(&mut self, pos: SeekFrom) -> io::Result<u64> {
|
|
||||||
let new_pos = match pos {
|
|
||||||
SeekFrom::Start(n) => n as i64,
|
|
||||||
SeekFrom::End(n) => self.total_len as i64 + n,
|
|
||||||
SeekFrom::Current(n) => self.pos as i64 + n,
|
|
||||||
};
|
|
||||||
if new_pos < 0 {
|
|
||||||
return Err(io::Error::new(
|
|
||||||
io::ErrorKind::InvalidInput,
|
|
||||||
"seek before start of media",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
// Allow seeking to/past end (reads then return 0), matching File semantics.
|
|
||||||
self.pos = new_pos as u64;
|
|
||||||
Ok(self.pos)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A write transaction over a [`BeamArchive`]. All writes are buffered until
|
|
||||||
/// [`BeamTxn::commit`]; dropping without committing rolls back. Lets a save
|
|
||||||
/// touch only the rows that changed, in place, without rewriting unchanged media.
|
|
||||||
pub struct BeamTxn<'a> {
|
|
||||||
tx: rusqlite::Transaction<'a>,
|
|
||||||
chunk_size: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl BeamTxn<'_> {
|
|
||||||
pub fn put_media_packed(
|
|
||||||
&self,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
bytes: &[u8],
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
write_media_packed(&self.tx, self.chunk_size, id, kind, codec, bytes, meta)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn put_media_referenced(
|
|
||||||
&self,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
ext_path: &str,
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
write_media_referenced(&self.tx, id, kind, codec, ext_path, meta)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Like [`BeamTxn::put_media_packed`] but streams the bytes from a file on
|
|
||||||
/// disk chunk-by-chunk, so an arbitrarily large file is never fully loaded
|
|
||||||
/// into memory. `total_len` is taken from the bytes actually read.
|
|
||||||
pub fn put_media_packed_from_path(
|
|
||||||
&self,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
path: &Path,
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
write_media_packed_from_path(&self.tx, self.chunk_size, id, kind, codec, path, meta)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn set_project_json(&self, json: &str) -> Result<(), String> {
|
|
||||||
set_project_json_conn(&self.tx, json)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn set_meta(&self, key: &str, value: &str) -> Result<(), String> {
|
|
||||||
set_meta_conn(&self.tx, key, value)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Does a media row with this id already exist?
|
|
||||||
pub fn media_exists(&self, id: Uuid) -> Result<bool, String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
let n: i64 = self
|
|
||||||
.tx
|
|
||||||
.query_row("SELECT COUNT(*) FROM media WHERE id = ?1", [&id_bytes], |r| r.get(0))
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(n > 0)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Every media id currently in the archive.
|
|
||||||
pub fn all_media_ids(&self) -> Result<Vec<Uuid>, String> {
|
|
||||||
let mut stmt = self.tx.prepare("SELECT id FROM media").map_err(map_sql)?;
|
|
||||||
let rows = stmt.query_map([], |r| r.get::<_, Vec<u8>>(0)).map_err(map_sql)?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
for row in rows {
|
|
||||||
out.push(uuid_from_bytes(&row.map_err(map_sql)?)?);
|
|
||||||
}
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Delete a media row (and its chunks).
|
|
||||||
pub fn delete_media(&self, id: Uuid) -> Result<(), String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
self.tx
|
|
||||||
.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
self.tx
|
|
||||||
.execute("DELETE FROM media WHERE id = ?1", [&id_bytes])
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Delete every media row whose id is not in `keep` (orphan cleanup).
|
|
||||||
pub fn retain_media(&self, keep: &std::collections::HashSet<Uuid>) -> Result<usize, String> {
|
|
||||||
let mut removed = 0;
|
|
||||||
for id in self.all_media_ids()? {
|
|
||||||
if !keep.contains(&id) {
|
|
||||||
self.delete_media(id)?;
|
|
||||||
removed += 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(removed)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn commit(self) -> Result<(), String> {
|
|
||||||
self.tx.commit().map_err(map_sql)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// -- shared write helpers (used by both BeamArchive and BeamTxn) --------------
|
|
||||||
|
|
||||||
fn write_media_packed(
|
|
||||||
conn: &Connection,
|
|
||||||
chunk_size: u64,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
bytes: &[u8],
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
|
|
||||||
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO media
|
|
||||||
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
|
|
||||||
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
|
|
||||||
rusqlite::params![
|
|
||||||
id_bytes,
|
|
||||||
kind as i64,
|
|
||||||
codec,
|
|
||||||
MediaStorage::Packed as i64,
|
|
||||||
bytes.len() as i64,
|
|
||||||
meta.channels,
|
|
||||||
meta.sample_rate,
|
|
||||||
meta.width,
|
|
||||||
meta.height,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
for (chunk_index, chunk) in bytes.chunks(chunk_size as usize).enumerate() {
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
|
|
||||||
rusqlite::params![id_bytes, chunk_index as i64, chunk],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn write_media_packed_from_path(
|
|
||||||
conn: &Connection,
|
|
||||||
chunk_size: u64,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
path: &Path,
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
|
|
||||||
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
|
|
||||||
let file = std::fs::File::open(path).map_err(|e| format!("Failed to open {:?}: {}", path, e))?;
|
|
||||||
let mut reader = std::io::BufReader::new(file);
|
|
||||||
let mut buf = vec![0u8; chunk_size as usize];
|
|
||||||
let mut chunk_index: i64 = 0;
|
|
||||||
let mut total_len: u64 = 0;
|
|
||||||
|
|
||||||
loop {
|
|
||||||
// Fill `buf` up to chunk_size, tolerating short reads.
|
|
||||||
let mut filled = 0usize;
|
|
||||||
while filled < buf.len() {
|
|
||||||
let n = reader
|
|
||||||
.read(&mut buf[filled..])
|
|
||||||
.map_err(|e| format!("Failed to read {:?}: {}", path, e))?;
|
|
||||||
if n == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
filled += n;
|
|
||||||
}
|
|
||||||
if filled == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO media_chunk (media_id, chunk_index, bytes) VALUES (?1, ?2, ?3)",
|
|
||||||
rusqlite::params![id_bytes, chunk_index, &buf[..filled]],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
chunk_index += 1;
|
|
||||||
total_len += filled as u64;
|
|
||||||
if filled < buf.len() {
|
|
||||||
break; // reached EOF
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO media
|
|
||||||
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
|
|
||||||
VALUES (?1, ?2, ?3, ?4, NULL, ?5, ?6, ?7, ?8, ?9)",
|
|
||||||
rusqlite::params![
|
|
||||||
id_bytes,
|
|
||||||
kind as i64,
|
|
||||||
codec,
|
|
||||||
MediaStorage::Packed as i64,
|
|
||||||
total_len as i64,
|
|
||||||
meta.channels,
|
|
||||||
meta.sample_rate,
|
|
||||||
meta.width,
|
|
||||||
meta.height,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn write_media_referenced(
|
|
||||||
conn: &Connection,
|
|
||||||
id: Uuid,
|
|
||||||
kind: MediaKind,
|
|
||||||
codec: &str,
|
|
||||||
ext_path: &str,
|
|
||||||
meta: MediaMeta,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
conn.execute("DELETE FROM media WHERE id = ?1", [&id_bytes]).map_err(map_sql)?;
|
|
||||||
conn.execute("DELETE FROM media_chunk WHERE media_id = ?1", [&id_bytes])
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO media
|
|
||||||
(id, kind, codec, storage, ext_path, total_len, channels, sample_rate, width, height)
|
|
||||||
VALUES (?1, ?2, ?3, ?4, ?5, 0, ?6, ?7, ?8, ?9)",
|
|
||||||
rusqlite::params![
|
|
||||||
id_bytes,
|
|
||||||
kind as i64,
|
|
||||||
codec,
|
|
||||||
MediaStorage::Referenced as i64,
|
|
||||||
ext_path,
|
|
||||||
meta.channels,
|
|
||||||
meta.sample_rate,
|
|
||||||
meta.width,
|
|
||||||
meta.height,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn set_project_json_conn(conn: &Connection, json: &str) -> Result<(), String> {
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO project_json (id, data) VALUES (0, ?1)
|
|
||||||
ON CONFLICT(id) DO UPDATE SET data = excluded.data",
|
|
||||||
[json],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn set_meta_conn(conn: &Connection, key: &str, value: &str) -> Result<(), String> {
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO meta (key, value) VALUES (?1, ?2)
|
|
||||||
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
|
||||||
rusqlite::params![key, value],
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read a packed media item's full bytes via a fresh **read-only** connection,
|
|
||||||
/// without an open [`BeamArchive`] handle. Returns `Ok(None)` if the item has no
|
|
||||||
/// chunk rows. Used for on-demand raster fault-in: a read-only connection can't
|
|
||||||
/// conflict with an in-place save and needs no long-lived handle.
|
|
||||||
pub fn read_packed_media_readonly(db_path: &Path, id: Uuid) -> Result<Option<Vec<u8>>, String> {
|
|
||||||
let conn = Connection::open_with_flags(
|
|
||||||
db_path,
|
|
||||||
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
|
|
||||||
)
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let id_bytes = id.as_bytes().to_vec();
|
|
||||||
let mut stmt = conn
|
|
||||||
.prepare("SELECT bytes FROM media_chunk WHERE media_id = ?1 ORDER BY chunk_index")
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let rows = stmt
|
|
||||||
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
|
|
||||||
.map_err(map_sql)?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
let mut any = false;
|
|
||||||
for row in rows {
|
|
||||||
out.extend_from_slice(&row.map_err(map_sql)?);
|
|
||||||
any = true;
|
|
||||||
}
|
|
||||||
Ok(if any { Some(out) } else { None })
|
|
||||||
}
|
|
||||||
|
|
||||||
fn map_sql(e: rusqlite::Error) -> String {
|
|
||||||
format!("SQLite error: {}", e)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn uuid_from_bytes(bytes: &[u8]) -> Result<Uuid, String> {
|
|
||||||
let arr: [u8; 16] = bytes
|
|
||||||
.try_into()
|
|
||||||
.map_err(|_| format!("Invalid uuid blob length {}", bytes.len()))?;
|
|
||||||
Ok(Uuid::from_bytes(arr))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tests live in `tests/beam_archive.rs` (integration tests), so they compile the
|
|
||||||
// library in non-test mode and don't depend on the crate's other `#[cfg(test)]`
|
|
||||||
// modules.
|
|
||||||
|
|
@ -575,27 +575,6 @@ pub fn encode_png(img: &RgbaImage) -> Result<Vec<u8>, String> {
|
||||||
Ok(buf.into_inner())
|
Ok(buf.into_inner())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Long-edge cap (pixels) for raster keyframe proxies — low-res page-in placeholders.
|
|
||||||
pub const RASTER_PROXY_MAX_EDGE: u32 = 192;
|
|
||||||
|
|
||||||
/// Downscale a full RGBA buffer to a low-res proxy PNG (long edge ≤
|
|
||||||
/// `RASTER_PROXY_MAX_EDGE`). Returns `None` on invalid dims/length or encode failure.
|
|
||||||
pub fn encode_raster_proxy_png(raw: &[u8], width: u32, height: u32) -> Option<Vec<u8>> {
|
|
||||||
if width == 0 || height == 0 || raw.len() != (width as usize * height as usize * 4) {
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
let long = width.max(height);
|
|
||||||
let (pw, ph) = if long <= RASTER_PROXY_MAX_EDGE {
|
|
||||||
(width, height)
|
|
||||||
} else {
|
|
||||||
let s = RASTER_PROXY_MAX_EDGE as f32 / long as f32;
|
|
||||||
(((width as f32 * s).round() as u32).max(1), ((height as f32 * s).round() as u32).max(1))
|
|
||||||
};
|
|
||||||
let img = RgbaImage::from_raw(width, height, raw.to_vec())?;
|
|
||||||
let resized = image::imageops::resize(&img, pw, ph, image::imageops::FilterType::Triangle);
|
|
||||||
encode_png(&resized).ok()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Decode PNG bytes into an `RgbaImage`
|
/// Decode PNG bytes into an `RgbaImage`
|
||||||
pub fn decode_png(data: &[u8]) -> Result<RgbaImage, String> {
|
pub fn decode_png(data: &[u8]) -> Result<RgbaImage, String> {
|
||||||
image::load_from_memory(data)
|
image::load_from_memory(data)
|
||||||
|
|
|
||||||
|
|
@ -902,7 +902,7 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_audio_clip_midi() {
|
fn test_audio_clip_midi() {
|
||||||
let clip = AudioClip::new_midi("Piano Melody", 1, daw_backend::Beats(60.0));
|
let clip = AudioClip::new_midi("Piano Melody", 1, 60.0);
|
||||||
assert_eq!(clip.name, "Piano Melody");
|
assert_eq!(clip.name, "Piano Melody");
|
||||||
assert_eq!(clip.duration, 60.0);
|
assert_eq!(clip.duration, 60.0);
|
||||||
match &clip.clip_type {
|
match &clip.clip_type {
|
||||||
|
|
@ -952,10 +952,7 @@ mod tests {
|
||||||
|
|
||||||
assert_eq!(instance.trim_start, 2.0);
|
assert_eq!(instance.trim_start, 2.0);
|
||||||
assert_eq!(instance.trim_end, Some(8.0));
|
assert_eq!(instance.trim_end, Some(8.0));
|
||||||
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
|
assert_eq!(instance.effective_duration(10.0), 6.0);
|
||||||
// beats-domain effective duration equals the seconds content window.
|
|
||||||
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
|
|
||||||
assert_eq!(instance.effective_duration(10.0, &tempo_map), 6.0);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
@ -966,9 +963,7 @@ mod tests {
|
||||||
|
|
||||||
assert_eq!(instance.trim_start, 2.0);
|
assert_eq!(instance.trim_start, 2.0);
|
||||||
assert_eq!(instance.trim_end, None);
|
assert_eq!(instance.trim_end, None);
|
||||||
// At 60 BPM the tempo map is identity (1 beat == 1 second).
|
assert_eq!(instance.effective_duration(10.0), 8.0);
|
||||||
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
|
|
||||||
assert_eq!(instance.effective_duration(10.0, &tempo_map), 8.0);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
|
|
@ -650,29 +650,6 @@ impl Document {
|
||||||
layers
|
layers
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get mutable references to all layers across the entire document
|
|
||||||
/// (root + nested groups + inside all vector clips). Mirrors [`all_layers`].
|
|
||||||
pub fn all_layers_mut(&mut self) -> Vec<&mut AnyLayer> {
|
|
||||||
let mut layers: Vec<&mut AnyLayer> = Vec::new();
|
|
||||||
// Iterative walk with an explicit stack of child slices. Group layers are
|
|
||||||
// descended into but not themselves collected (they hold no keyframes).
|
|
||||||
let mut stack: Vec<&mut [AnyLayer]> = vec![&mut self.root.children];
|
|
||||||
while let Some(list) = stack.pop() {
|
|
||||||
for layer in list {
|
|
||||||
match layer {
|
|
||||||
AnyLayer::Group(g) => {
|
|
||||||
stack.push(&mut g.children);
|
|
||||||
}
|
|
||||||
other => layers.push(other),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for clip in self.vector_clips.values_mut() {
|
|
||||||
layers.extend(clip.layers.root_data_mut());
|
|
||||||
}
|
|
||||||
layers
|
|
||||||
}
|
|
||||||
|
|
||||||
// === CLIP LIBRARY METHODS ===
|
// === CLIP LIBRARY METHODS ===
|
||||||
|
|
||||||
/// Add a vector clip to the library
|
/// Add a vector clip to the library
|
||||||
|
|
|
||||||
|
|
@ -240,18 +240,14 @@ mod tests {
|
||||||
let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end
|
let effect2 = def.create_instance(3.0, 7.0); // 3.0 + 7.0 = 10.0 end
|
||||||
layer.add_clip_instance(effect2);
|
layer.add_clip_instance(effect2);
|
||||||
|
|
||||||
// At 60 BPM the tempo map is identity (1 beat == 1 second), so the
|
|
||||||
// query time in seconds equals the effect's beats-domain extents.
|
|
||||||
let tempo_map = crate::tempo_map::TempoMap::constant(60.0);
|
|
||||||
|
|
||||||
// At time 2: only effect1 active
|
// At time 2: only effect1 active
|
||||||
assert_eq!(layer.active_clip_instances_at(2.0, &tempo_map).len(), 1);
|
assert_eq!(layer.active_clip_instances_at(2.0, 60.0).len(), 1);
|
||||||
|
|
||||||
// At time 4: both effects active
|
// At time 4: both effects active
|
||||||
assert_eq!(layer.active_clip_instances_at(4.0, &tempo_map).len(), 2);
|
assert_eq!(layer.active_clip_instances_at(4.0, 60.0).len(), 2);
|
||||||
|
|
||||||
// At time 7: only effect2 active
|
// At time 7: only effect2 active
|
||||||
assert_eq!(layer.active_clip_instances_at(7.0, &tempo_map).len(), 1);
|
assert_eq!(layer.active_clip_instances_at(7.0, 60.0).len(), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
|
|
@ -1,25 +1,20 @@
|
||||||
//! File I/O for .beam project files
|
//! File I/O for .beam project files
|
||||||
//!
|
//!
|
||||||
//! The `.beam` format is a single **SQLite database** (see [`crate::beam_archive`]):
|
//! This module handles saving and loading Lightningbeam projects in the .beam format,
|
||||||
//! - `project_json` table — serialized project metadata and structure
|
//! which is a ZIP archive containing:
|
||||||
//! - `media` / `media_chunk` tables — audio and raster media (packed as chunked
|
//! - project.json (compressed) - Project metadata and structure
|
||||||
//! blobs, or referenced by external path)
|
//! - media/ directory (uncompressed) - Embedded media files (FLAC for audio)
|
||||||
//!
|
|
||||||
//! Older `.beam` files are ZIP archives; [`load_beam`] detects and reads those
|
|
||||||
//! too (via [`load_beam_zip_legacy`]). Saving always writes the SQLite form, so
|
|
||||||
//! opening a legacy file and saving migrates it.
|
|
||||||
|
|
||||||
use crate::beam_archive::{BeamArchive, MediaKind, MediaMeta, LARGE_MEDIA_THRESHOLD};
|
|
||||||
use crate::document::Document;
|
use crate::document::Document;
|
||||||
use daw_backend::audio::pool::AudioPoolEntry;
|
use daw_backend::audio::pool::AudioPoolEntry;
|
||||||
use daw_backend::audio::project::Project as AudioProject;
|
use daw_backend::audio::project::Project as AudioProject;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashSet;
|
|
||||||
use std::fs::File;
|
use std::fs::File;
|
||||||
use std::io::Read;
|
use std::io::{Read, Write};
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use uuid::Uuid;
|
use zip::write::FileOptions;
|
||||||
use zip::ZipArchive;
|
use zip::{CompressionMethod, ZipArchive, ZipWriter};
|
||||||
|
use flacenc::error::Verify;
|
||||||
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
|
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
|
||||||
|
|
||||||
/// File format version
|
/// File format version
|
||||||
|
|
@ -56,7 +51,9 @@ pub struct SerializedAudioBackend {
|
||||||
/// Audio project (tracks, MIDI clips, etc.)
|
/// Audio project (tracks, MIDI clips, etc.)
|
||||||
pub project: AudioProject,
|
pub project: AudioProject,
|
||||||
|
|
||||||
/// Audio pool entries (metadata and media references for audio files)
|
/// Audio pool entries (metadata and paths for audio files)
|
||||||
|
/// Note: embedded_data field from daw-backend is ignored; embedded files
|
||||||
|
/// are stored as FLAC in the ZIP's media/audio/ directory instead
|
||||||
pub audio_pool_entries: Vec<AudioPoolEntry>,
|
pub audio_pool_entries: Vec<AudioPoolEntry>,
|
||||||
|
|
||||||
/// Mapping from UI layer UUIDs to backend TrackIds
|
/// Mapping from UI layer UUIDs to backend TrackIds
|
||||||
|
|
@ -66,25 +63,6 @@ pub struct SerializedAudioBackend {
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// How to store a media file at or above [`LARGE_MEDIA_THRESHOLD`].
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
|
||||||
pub enum LargeMediaMode {
|
|
||||||
/// Not yet decided — prompt the user the first time a large file is imported.
|
|
||||||
/// Treated as [`LargeMediaMode::Reference`] at save time. Resetting the
|
|
||||||
/// preference to `Ask` re-triggers the prompt (useful for testing).
|
|
||||||
Ask,
|
|
||||||
/// Pack the bytes into the `.beam` container (chunked, streamed from disk).
|
|
||||||
Pack,
|
|
||||||
/// Keep the file external and store only a path reference.
|
|
||||||
Reference,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Default for LargeMediaMode {
|
|
||||||
fn default() -> Self {
|
|
||||||
LargeMediaMode::Ask
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Settings for saving a project
|
/// Settings for saving a project
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct SaveSettings {
|
pub struct SaveSettings {
|
||||||
|
|
@ -96,10 +74,6 @@ pub struct SaveSettings {
|
||||||
|
|
||||||
/// Force linking all media files (don't embed any)
|
/// Force linking all media files (don't embed any)
|
||||||
pub force_link_all: bool,
|
pub force_link_all: bool,
|
||||||
|
|
||||||
/// How to store files at/above [`LARGE_MEDIA_THRESHOLD`] (pack vs reference).
|
|
||||||
/// `Ask` behaves as `Reference` here (safe default: don't bloat the DB).
|
|
||||||
pub large_media_mode: LargeMediaMode,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for SaveSettings {
|
impl Default for SaveSettings {
|
||||||
|
|
@ -108,7 +82,6 @@ impl Default for SaveSettings {
|
||||||
auto_embed_threshold_bytes: 10_000_000, // 10 MB
|
auto_embed_threshold_bytes: 10_000_000, // 10 MB
|
||||||
force_embed_all: false,
|
force_embed_all: false,
|
||||||
force_link_all: false,
|
force_link_all: false,
|
||||||
large_media_mode: LargeMediaMode::Ask,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -127,11 +100,6 @@ pub struct LoadedProject {
|
||||||
/// Loaded audio pool entries
|
/// Loaded audio pool entries
|
||||||
pub audio_pool_entries: Vec<AudioPoolEntry>,
|
pub audio_pool_entries: Vec<AudioPoolEntry>,
|
||||||
|
|
||||||
/// Persisted video-thumbnail packs by clip id (opaque LBTN blobs; decoded and
|
|
||||||
/// inserted into the VideoManager by the editor). Clips present here don't need
|
|
||||||
/// their thumbnails regenerated on load.
|
|
||||||
pub thumbnail_blobs: std::collections::HashMap<uuid::Uuid, Vec<u8>>,
|
|
||||||
|
|
||||||
/// List of files that couldn't be found
|
/// List of files that couldn't be found
|
||||||
pub missing_files: Vec<MissingFileInfo>,
|
pub missing_files: Vec<MissingFileInfo>,
|
||||||
}
|
}
|
||||||
|
|
@ -157,319 +125,282 @@ pub enum MediaFileType {
|
||||||
Image,
|
Image,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Save a project to a `.beam` file (SQLite container).
|
/// Save a project to a .beam file
|
||||||
///
|
///
|
||||||
/// Re-saving an existing SQLite `.beam` updates it **in place** inside a single
|
/// This function:
|
||||||
/// transaction: unchanged (large) media is never rewritten, only changed rows
|
/// 1. Prepares audio project for save (saves AudioGraph presets)
|
||||||
/// are touched, and the commit is atomic/crash-safe. A brand-new file or a
|
/// 2. Serializes project data to JSON
|
||||||
/// legacy-ZIP migration is written to a temp file and atomically renamed (there
|
/// 3. Creates ZIP archive with compressed project.json
|
||||||
/// is no large existing container to copy in that case).
|
/// 4. Embeds media files as FLAC (for audio) in media/ directory
|
||||||
///
|
///
|
||||||
/// Audio and raster media become rows in the `media` table — packed as chunked
|
/// # Arguments
|
||||||
/// blobs, or referenced by external path for files at/above
|
/// * `path` - Path to save the .beam file
|
||||||
/// [`LARGE_MEDIA_THRESHOLD`]. `project.json` goes in the `project_json` table.
|
/// * `document` - UI document state
|
||||||
/// Whether a stored media codec is an audio format the disk reader (Symphonia)
|
/// * `audio_project` - Audio backend project
|
||||||
/// can stream directly from a packed blob. Video-container audio tracks and any
|
/// * `audio_pool_entries` - Serialized audio pool entries
|
||||||
/// unknown formats fall back to the legacy reconstitution-and-decode path.
|
/// * `settings` - Save settings (embedding preferences)
|
||||||
fn is_streamable_audio_codec(codec: &str) -> bool {
|
///
|
||||||
matches!(
|
/// # Returns
|
||||||
codec.to_lowercase().as_str(),
|
/// Ok(()) on success, or error message
|
||||||
"mp3" | "flac" | "ogg" | "oga" | "wav" | "wave" | "aiff" | "aif"
|
|
||||||
| "aac" | "m4a" | "opus" | "alac" | "caf"
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A `Sync` wrapper over core's `BlobReader` so it satisfies Symphonia's
|
|
||||||
/// `MediaSource: Send + Sync`. `BlobReader` holds a rusqlite `Connection`
|
|
||||||
/// (`Send` but `!Sync`); the disk reader uses it single-threaded, so the
|
|
||||||
/// hot Read/Seek path goes through `Mutex::get_mut` (no runtime locking).
|
|
||||||
struct SyncBlobReader {
|
|
||||||
inner: std::sync::Mutex<crate::beam_archive::BlobReader>,
|
|
||||||
len: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl std::io::Read for SyncBlobReader {
|
|
||||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
|
||||||
self.inner.get_mut().unwrap().read(buf)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl std::io::Seek for SyncBlobReader {
|
|
||||||
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
|
|
||||||
self.inner.get_mut().unwrap().seek(pos)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
impl daw_backend::audio::MediaByteSource for SyncBlobReader {
|
|
||||||
fn byte_len(&self) -> u64 {
|
|
||||||
self.len
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The host's packed-media byte-source factory: opens an independent streaming
|
|
||||||
/// reader over a `.beam` container's packed audio by media id. Installed into the
|
|
||||||
/// engine on load so container-packed audio streams without a full decode.
|
|
||||||
#[derive(Debug)]
|
|
||||||
struct BeamBlobFactory {
|
|
||||||
db_path: PathBuf,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl daw_backend::audio::AudioBlobSourceFactory for BeamBlobFactory {
|
|
||||||
fn open(
|
|
||||||
&self,
|
|
||||||
media_id: &str,
|
|
||||||
) -> Result<Box<dyn daw_backend::audio::MediaByteSource>, String> {
|
|
||||||
let id = Uuid::parse_str(media_id).map_err(|e| format!("bad media id {}: {}", media_id, e))?;
|
|
||||||
let archive = BeamArchive::open(&self.db_path)?;
|
|
||||||
let reader = archive.open_blob_reader(&self.db_path, id)?;
|
|
||||||
let len = reader.len();
|
|
||||||
Ok(Box::new(SyncBlobReader { inner: std::sync::Mutex::new(reader), len }))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Build a packed-media byte-source factory for a `.beam` file, to install into
|
|
||||||
/// the engine (`EngineController::set_blob_source_factory`) before loading so its
|
|
||||||
/// packed audio can be streamed.
|
|
||||||
pub fn blob_source_factory(
|
|
||||||
beam_path: &Path,
|
|
||||||
) -> std::sync::Arc<dyn daw_backend::audio::AudioBlobSourceFactory> {
|
|
||||||
std::sync::Arc::new(BeamBlobFactory { db_path: beam_path.to_path_buf() })
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Deterministic id for the waveform-pyramid media row of audio pool entry
|
|
||||||
/// `pool_index`, within a single project container. Stable across saves (so an
|
|
||||||
/// in-place re-save reuses the row instead of orphaning/rewriting it) and
|
|
||||||
/// independent of how the audio bytes are stored. The top 32 bits are a fixed
|
|
||||||
/// "LBWF" sentinel so it can't collide with the random v4 ids used elsewhere.
|
|
||||||
fn waveform_media_id(pool_index: usize) -> Uuid {
|
|
||||||
const SENTINEL: u128 = 0x4C42_5746u128 << 96; // "LBWF" in the high 32 bits
|
|
||||||
Uuid::from_u128(SENTINEL | (pool_index as u128))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Deterministic id for the thumbnail-pack media row of a video clip. Derived from
|
|
||||||
/// the clip id by XOR with a fixed constant — bijective and stable across saves, so
|
|
||||||
/// it reuses the row in place, and it can't (in practice) collide with the random
|
|
||||||
/// v4 ids used for other media of a different kind.
|
|
||||||
fn thumbnail_media_id(clip_id: Uuid) -> Uuid {
|
|
||||||
// "LBTN" repeated — moves clip ids into a distinct region of the id space.
|
|
||||||
const SENTINEL: u128 = 0x4C42_544E_4C42_544E_4C42_544E_4C42_544E;
|
|
||||||
Uuid::from_u128(clip_id.as_u128() ^ SENTINEL)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Derived id for a raster keyframe's low-res proxy row (distinct from the keyframe's
|
|
||||||
/// own full-res `Raster` row, which is keyed by the raw keyframe id).
|
|
||||||
fn raster_proxy_media_id(kf_id: Uuid) -> Uuid {
|
|
||||||
// "LBPX" repeated — distinct id region from the full raster + thumbnail rows.
|
|
||||||
const SENTINEL: u128 = 0x4C42_5058_4C42_5058_4C42_5058_4C42_5058;
|
|
||||||
Uuid::from_u128(kf_id.as_u128() ^ SENTINEL)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn save_beam(
|
pub fn save_beam(
|
||||||
path: &Path,
|
path: &Path,
|
||||||
document: &Document,
|
document: &Document,
|
||||||
audio_project: &mut AudioProject,
|
audio_project: &mut AudioProject,
|
||||||
audio_pool_entries: Vec<AudioPoolEntry>,
|
audio_pool_entries: Vec<AudioPoolEntry>,
|
||||||
layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>,
|
layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>,
|
||||||
thumbnail_blobs: &std::collections::HashMap<uuid::Uuid, Vec<u8>>,
|
|
||||||
_settings: &SaveSettings,
|
_settings: &SaveSettings,
|
||||||
) -> Result<(), String> {
|
) -> Result<(), String> {
|
||||||
let fn_start = std::time::Instant::now();
|
let fn_start = std::time::Instant::now();
|
||||||
eprintln!("📊 [SAVE_BEAM] Starting save_beam() (SQLite container)...");
|
eprintln!("📊 [SAVE_BEAM] Starting save_beam()...");
|
||||||
|
|
||||||
|
// 1. Create backup if file exists and open it for reading old audio files
|
||||||
|
let step1_start = std::time::Instant::now();
|
||||||
|
let mut old_zip = if path.exists() {
|
||||||
|
let backup_path = path.with_extension("beam.backup");
|
||||||
|
std::fs::copy(path, &backup_path)
|
||||||
|
.map_err(|e| format!("Failed to create backup: {}", e))?;
|
||||||
|
|
||||||
|
// Open the backup as a ZIP archive for reading
|
||||||
|
match File::open(&backup_path) {
|
||||||
|
Ok(file) => match ZipArchive::new(file) {
|
||||||
|
Ok(archive) => {
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup and open for reading took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
Some(archive)
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup as ZIP: {}, will not copy old audio files", e);
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
},
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("⚠️ [SAVE_BEAM] Failed to open backup: {}, will not copy old audio files", e);
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 1: Create backup took {:.2}ms", step1_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 1: No backup needed (new file)");
|
||||||
|
None
|
||||||
|
};
|
||||||
|
|
||||||
|
// 2. Graph presets are already populated by the engine thread (in GetProject handler)
|
||||||
|
// before cloning. Do NOT call prepare_for_save() here — the cloned project has
|
||||||
|
// default empty graphs (AudioTrack::clone() doesn't copy the graph), so calling
|
||||||
|
// prepare_for_save() would overwrite the good presets with empty ones.
|
||||||
|
let step2_start = std::time::Instant::now();
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 2: (graph presets already prepared) took {:.2}ms", step2_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
|
// 3. Create ZIP writer
|
||||||
|
let step3_start = std::time::Instant::now();
|
||||||
|
let file = File::create(path)
|
||||||
|
.map_err(|e| format!("Failed to create file: {}", e))?;
|
||||||
|
let mut zip = ZipWriter::new(file);
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 3: Create ZIP writer took {:.2}ms", step3_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
|
// 4. Process audio pool entries and write embedded audio files to ZIP
|
||||||
|
// Priority: old ZIP file > external file > encode PCM as FLAC
|
||||||
|
let step4_start = std::time::Instant::now();
|
||||||
|
let mut modified_entries = Vec::new();
|
||||||
|
let mut flac_encode_time = 0.0;
|
||||||
|
let mut zip_write_time = 0.0;
|
||||||
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
|
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
|
||||||
let in_place = path.exists() && BeamArchive::is_sqlite(path);
|
|
||||||
|
|
||||||
// In-place for an existing SQLite container (don't rewrite unchanged media);
|
|
||||||
// temp + atomic rename for new files / legacy-ZIP migration.
|
|
||||||
let tmp_path = path.with_extension("beam.tmp");
|
|
||||||
let mut archive = if in_place {
|
|
||||||
BeamArchive::open(path)?
|
|
||||||
} else {
|
|
||||||
BeamArchive::create(&tmp_path)?
|
|
||||||
};
|
|
||||||
|
|
||||||
let now = chrono::Utc::now().to_rfc3339();
|
|
||||||
let created = if in_place {
|
|
||||||
archive.get_meta("created").ok().flatten().unwrap_or_else(|| now.clone())
|
|
||||||
} else {
|
|
||||||
now.clone()
|
|
||||||
};
|
|
||||||
|
|
||||||
let txn = archive.transaction()?;
|
|
||||||
|
|
||||||
// --- audio pool entries -> media rows (packed) or external references ---
|
|
||||||
let mut modified_entries = Vec::with_capacity(audio_pool_entries.len());
|
|
||||||
let mut live_media: HashSet<Uuid> = HashSet::new();
|
|
||||||
|
|
||||||
for entry in &audio_pool_entries {
|
for entry in &audio_pool_entries {
|
||||||
let mut e = entry.clone();
|
let mut modified_entry = entry.clone();
|
||||||
let existing_id = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok());
|
|
||||||
|
|
||||||
// Already packed in this archive (in-place re-save): leave the bytes
|
// Try to get audio data from various sources (in priority order)
|
||||||
// untouched, just keep the reference.
|
let audio_source: Option<(Vec<u8>, String)> = if let Some(ref rel_path) = entry.relative_path {
|
||||||
if let Some(id) = existing_id {
|
// Priority 1: Check if file is in the old ZIP
|
||||||
if txn.media_exists(id)? {
|
if rel_path.starts_with("media/audio/") {
|
||||||
live_media.insert(id);
|
if let Some(ref mut old_zip_archive) = old_zip {
|
||||||
e.media_id = Some(id.to_string());
|
match old_zip_archive.by_name(rel_path) {
|
||||||
e.relative_path = None;
|
Ok(mut file) => {
|
||||||
e.embedded_data = None;
|
let mut bytes = Vec::new();
|
||||||
modified_entries.push(e);
|
if file.read_to_end(&mut bytes).is_ok() {
|
||||||
continue;
|
let extension = rel_path.split('.').last().unwrap_or("bin").to_string();
|
||||||
}
|
eprintln!("📊 [SAVE_BEAM] Copying from old ZIP: {}", rel_path);
|
||||||
}
|
Some((bytes, extension))
|
||||||
|
|
||||||
// Otherwise resolve the source: external file (Priority 2, streamed from
|
|
||||||
// disk so a huge file is never fully loaded), or embedded data (Priority 3).
|
|
||||||
let meta = MediaMeta {
|
|
||||||
channels: Some(entry.channels),
|
|
||||||
sample_rate: Some(entry.sample_rate),
|
|
||||||
..Default::default()
|
|
||||||
};
|
|
||||||
let mut wrote_packed: Option<Uuid> = None;
|
|
||||||
let mut referenced: Option<String> = None;
|
|
||||||
|
|
||||||
if let Some(rel) = entry.relative_path.as_ref() {
|
|
||||||
let full = if Path::new(rel).is_absolute() {
|
|
||||||
PathBuf::from(rel)
|
|
||||||
} else {
|
} else {
|
||||||
project_dir.join(rel)
|
eprintln!("⚠️ [SAVE_BEAM] Failed to read {} from old ZIP", rel_path);
|
||||||
};
|
None
|
||||||
// Require an actual file: an empty/blank `relative_path` resolves to the
|
}
|
||||||
// project directory itself (`join("")` == dir), which `exists()` accepts
|
}
|
||||||
// but can't be read as media. `is_file()` skips dirs + missing paths, so
|
Err(_) => {
|
||||||
// such an entry correctly falls through to embedded data below.
|
eprintln!("⚠️ [SAVE_BEAM] File {} not found in old ZIP", rel_path);
|
||||||
if full.is_file() {
|
None
|
||||||
let size = std::fs::metadata(&full).map(|m| m.len()).unwrap_or(0);
|
}
|
||||||
let codec = full
|
}
|
||||||
.extension()
|
} else {
|
||||||
.and_then(|x| x.to_str())
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Priority 2: Check external filesystem
|
||||||
|
else {
|
||||||
|
let full_path = project_dir.join(rel_path);
|
||||||
|
if full_path.exists() {
|
||||||
|
match std::fs::read(&full_path) {
|
||||||
|
Ok(bytes) => {
|
||||||
|
let extension = full_path.extension()
|
||||||
|
.and_then(|e| e.to_str())
|
||||||
.unwrap_or("bin")
|
.unwrap_or("bin")
|
||||||
.to_lowercase();
|
.to_string();
|
||||||
// Video-audio entries are always referenced (the video is already
|
eprintln!("📊 [SAVE_BEAM] Using external file: {:?}", full_path);
|
||||||
// referenced by its VideoClip; reloaded by re-probing via FFmpeg).
|
Some((bytes, extension))
|
||||||
// Otherwise large files honor the user's pack-vs-reference choice
|
}
|
||||||
// (`Ask` == reference); smaller files are always packed.
|
Err(e) => {
|
||||||
let reference_it = entry.is_video_audio
|
eprintln!("⚠️ [SAVE_BEAM] Failed to read {:?}: {}", full_path, e);
|
||||||
|| (size >= LARGE_MEDIA_THRESHOLD
|
None
|
||||||
&& _settings.large_media_mode != LargeMediaMode::Pack);
|
}
|
||||||
if reference_it {
|
}
|
||||||
referenced = Some(rel.clone());
|
|
||||||
} else {
|
} else {
|
||||||
let id = existing_id.unwrap_or_else(Uuid::new_v4);
|
eprintln!("⚠️ [SAVE_BEAM] External file not found: {:?}", full_path);
|
||||||
txn.put_media_packed_from_path(id, MediaKind::Audio, &codec, &full, meta)?;
|
None
|
||||||
wrote_packed = Some(id);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
};
|
||||||
|
|
||||||
|
if let Some((audio_bytes, extension)) = audio_source {
|
||||||
|
// We have the original file - copy it directly
|
||||||
|
let zip_filename = format!("media/audio/{}.{}", entry.pool_index, extension);
|
||||||
|
|
||||||
|
let file_options = FileOptions::default()
|
||||||
|
.compression_method(CompressionMethod::Stored);
|
||||||
|
|
||||||
|
zip.start_file(&zip_filename, file_options)
|
||||||
|
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?;
|
||||||
|
|
||||||
|
let write_start = std::time::Instant::now();
|
||||||
|
zip.write_all(&audio_bytes)
|
||||||
|
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?;
|
||||||
|
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0;
|
||||||
|
|
||||||
|
// Update entry to point to ZIP file
|
||||||
|
modified_entry.embedded_data = None;
|
||||||
|
modified_entry.relative_path = Some(zip_filename);
|
||||||
|
|
||||||
|
} else if let Some(ref embedded_data) = entry.embedded_data {
|
||||||
|
// Priority 3: No original file - encode PCM as FLAC
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Encoding PCM to FLAC for pool {} (no original file)", entry.pool_index);
|
||||||
|
// Embedded data is always PCM - encode as FLAC
|
||||||
|
let audio_bytes = BASE64_STANDARD.decode(&embedded_data.data_base64)
|
||||||
|
.map_err(|e| format!("Failed to decode base64 audio data for pool index {}: {}", entry.pool_index, e))?;
|
||||||
|
|
||||||
|
let zip_filename = format!("media/audio/{}.flac", entry.pool_index);
|
||||||
|
|
||||||
|
let file_options = FileOptions::default()
|
||||||
|
.compression_method(CompressionMethod::Stored);
|
||||||
|
|
||||||
|
zip.start_file(&zip_filename, file_options)
|
||||||
|
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_filename, e))?;
|
||||||
|
|
||||||
|
// Encode PCM samples to FLAC
|
||||||
|
let flac_start = std::time::Instant::now();
|
||||||
|
|
||||||
|
// The audio_bytes are raw PCM samples (interleaved f32 little-endian)
|
||||||
|
let samples: Vec<f32> = audio_bytes
|
||||||
|
.chunks_exact(4)
|
||||||
|
.map(|chunk| f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
// Convert f32 samples to i32 for FLAC encoding
|
||||||
|
let samples_i32: Vec<i32> = samples
|
||||||
|
.iter()
|
||||||
|
.map(|&s| {
|
||||||
|
let clamped = s.clamp(-1.0, 1.0);
|
||||||
|
(clamped * 8388607.0) as i32
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
// Configure FLAC encoder
|
||||||
|
let config = flacenc::config::Encoder::default()
|
||||||
|
.into_verified()
|
||||||
|
.map_err(|(_, e)| format!("FLAC encoder config error: {:?}", e))?;
|
||||||
|
|
||||||
|
let source = flacenc::source::MemSource::from_samples(
|
||||||
|
&samples_i32,
|
||||||
|
entry.channels as usize,
|
||||||
|
24,
|
||||||
|
entry.sample_rate as usize,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Encode to FLAC
|
||||||
|
let flac_stream = flacenc::encode_with_fixed_block_size(
|
||||||
|
&config,
|
||||||
|
source,
|
||||||
|
config.block_size,
|
||||||
|
).map_err(|e| format!("FLAC encoding failed: {:?}", e))?;
|
||||||
|
|
||||||
|
// Convert stream to bytes
|
||||||
|
use flacenc::component::BitRepr;
|
||||||
|
let mut sink = flacenc::bitsink::ByteSink::new();
|
||||||
|
flac_stream.write(&mut sink)
|
||||||
|
.map_err(|e| format!("Failed to write FLAC stream: {:?}", e))?;
|
||||||
|
let flac_bytes = sink.as_slice();
|
||||||
|
|
||||||
|
flac_encode_time += flac_start.elapsed().as_secs_f64() * 1000.0;
|
||||||
|
|
||||||
|
let write_start = std::time::Instant::now();
|
||||||
|
zip.write_all(flac_bytes)
|
||||||
|
.map_err(|e| format!("Failed to write {}: {}", zip_filename, e))?;
|
||||||
|
zip_write_time += write_start.elapsed().as_secs_f64() * 1000.0;
|
||||||
|
|
||||||
|
// Update entry to point to ZIP file instead of embedding data
|
||||||
|
modified_entry.embedded_data = None;
|
||||||
|
modified_entry.relative_path = Some(zip_filename);
|
||||||
}
|
}
|
||||||
|
|
||||||
if wrote_packed.is_none() && referenced.is_none() {
|
modified_entries.push(modified_entry);
|
||||||
if let Some(ed) = entry.embedded_data.as_ref() {
|
|
||||||
if let Ok(bytes) = BASE64_STANDARD.decode(&ed.data_base64) {
|
|
||||||
let id = existing_id.unwrap_or_else(Uuid::new_v4);
|
|
||||||
txn.put_media_packed(id, MediaKind::Audio, &ed.format.to_lowercase(), &bytes, meta)?;
|
|
||||||
wrote_packed = Some(id);
|
|
||||||
}
|
}
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 4: Process audio pool ({} entries) took {:.2}ms",
|
||||||
|
audio_pool_entries.len(), step4_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
if flac_encode_time > 0.0 {
|
||||||
|
eprintln!("📊 [SAVE_BEAM] - FLAC encoding: {:.2}ms", flac_encode_time);
|
||||||
}
|
}
|
||||||
|
if zip_write_time > 0.0 {
|
||||||
|
eprintln!("📊 [SAVE_BEAM] - ZIP writing: {:.2}ms", zip_write_time);
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Some(id) = wrote_packed {
|
// 4b. Write raster layer PNG buffers to ZIP (media/raster/<keyframe-uuid>.png)
|
||||||
live_media.insert(id);
|
let step4b_start = std::time::Instant::now();
|
||||||
e.media_id = Some(id.to_string());
|
let raster_file_options = FileOptions::default()
|
||||||
e.relative_path = None;
|
.compression_method(CompressionMethod::Stored); // PNG is already compressed
|
||||||
e.embedded_data = None;
|
|
||||||
} else if let Some(rel) = referenced {
|
|
||||||
e.media_id = None;
|
|
||||||
e.relative_path = Some(rel);
|
|
||||||
e.embedded_data = None;
|
|
||||||
} // else: nothing available — keep original references (reported missing on load)
|
|
||||||
|
|
||||||
// Persist this entry's waveform pyramid (keyed by pool index, independent
|
|
||||||
// of the audio storage above). Reuse the row in place on re-save.
|
|
||||||
let wf_id = waveform_media_id(entry.pool_index);
|
|
||||||
if let Some(blob) = entry.waveform_blob.as_ref() {
|
|
||||||
txn.put_media_packed(wf_id, MediaKind::Waveform, "lbwf", blob, MediaMeta::default())?;
|
|
||||||
live_media.insert(wf_id);
|
|
||||||
} else if txn.media_exists(wf_id)? {
|
|
||||||
// Unchanged this save — keep the stored waveform row.
|
|
||||||
live_media.insert(wf_id);
|
|
||||||
}
|
|
||||||
|
|
||||||
modified_entries.push(e);
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- raster keyframes -> media rows (PNG), keyed by keyframe id ---
|
|
||||||
// (Phase 0 writes all resident frames each save; a disk-dirty flag to skip
|
|
||||||
// unchanged frames in place is deferred to Phase 3.)
|
|
||||||
// Walk ALL layers (incl. nested in groups/clips) so nested raster keyframes
|
|
||||||
// are persisted too, and so `live_media` covers them — matching the load path,
|
|
||||||
// which arms `needs_fault_in` recursively. Top-level-only projects are unaffected.
|
|
||||||
let mut raster_count = 0usize;
|
let mut raster_count = 0usize;
|
||||||
for layer in document.all_layers() {
|
for layer in &document.root.children {
|
||||||
if let crate::layer::AnyLayer::Raster(rl) = layer {
|
if let crate::layer::AnyLayer::Raster(rl) = layer {
|
||||||
for kf in &rl.keyframes {
|
for kf in &rl.keyframes {
|
||||||
if !kf.raw_pixels.is_empty() {
|
if !kf.raw_pixels.is_empty() {
|
||||||
let img =
|
// Encode raw RGBA to PNG for storage
|
||||||
crate::brush_engine::image_from_raw(kf.raw_pixels.clone(), kf.width, kf.height);
|
let img = crate::brush_engine::image_from_raw(
|
||||||
|
kf.raw_pixels.clone(), kf.width, kf.height,
|
||||||
|
);
|
||||||
match crate::brush_engine::encode_png(&img) {
|
match crate::brush_engine::encode_png(&img) {
|
||||||
Ok(png_bytes) => {
|
Ok(png_bytes) => {
|
||||||
txn.put_media_packed(
|
let zip_path = kf.media_path.clone();
|
||||||
kf.id,
|
zip.start_file(&zip_path, raster_file_options)
|
||||||
MediaKind::Raster,
|
.map_err(|e| format!("Failed to create {} in ZIP: {}", zip_path, e))?;
|
||||||
"png",
|
zip.write_all(&png_bytes)
|
||||||
&png_bytes,
|
.map_err(|e| format!("Failed to write {}: {}", zip_path, e))?;
|
||||||
MediaMeta {
|
|
||||||
width: Some(kf.width),
|
|
||||||
height: Some(kf.height),
|
|
||||||
..Default::default()
|
|
||||||
},
|
|
||||||
)?;
|
|
||||||
live_media.insert(kf.id);
|
|
||||||
raster_count += 1;
|
raster_count += 1;
|
||||||
}
|
}
|
||||||
Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster {}: {}", kf.id, e),
|
Err(e) => eprintln!("⚠️ [SAVE_BEAM] Failed to encode raster PNG {}: {}", kf.media_path, e),
|
||||||
}
|
|
||||||
// Low-res proxy alongside the full PNG (shown while the full pages
|
|
||||||
// in on a later load). Regenerated from the resident pixels.
|
|
||||||
let proxy_id = raster_proxy_media_id(kf.id);
|
|
||||||
if let Some(proxy_png) =
|
|
||||||
crate::brush_engine::encode_raster_proxy_png(&kf.raw_pixels, kf.width, kf.height)
|
|
||||||
{
|
|
||||||
txn.put_media_packed(
|
|
||||||
proxy_id, MediaKind::RasterProxy, "png", &proxy_png, MediaMeta::default(),
|
|
||||||
)?;
|
|
||||||
live_media.insert(proxy_id);
|
|
||||||
}
|
|
||||||
} else if txn.media_exists(kf.id)? {
|
|
||||||
// Pixels not resident but already stored — keep both rows.
|
|
||||||
live_media.insert(kf.id);
|
|
||||||
let proxy_id = raster_proxy_media_id(kf.id);
|
|
||||||
if txn.media_exists(proxy_id)? {
|
|
||||||
live_media.insert(proxy_id);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 4b: Write {} raster PNG buffers took {:.2}ms",
|
||||||
|
raster_count, step4b_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
// --- video thumbnail packs -> media rows (opaque LBTN blob), keyed by a
|
// 5. Build BeamProject structure with modified entries
|
||||||
// sentinel-derived id from the video clip id ---
|
let step5_start = std::time::Instant::now();
|
||||||
for clip_id in document.video_clips.keys() {
|
let now = chrono::Utc::now().to_rfc3339();
|
||||||
let tn_id = thumbnail_media_id(*clip_id);
|
|
||||||
if let Some(blob) = thumbnail_blobs.get(clip_id) {
|
|
||||||
txn.put_media_packed(tn_id, MediaKind::Thumbnail, "lbtn", blob, MediaMeta::default())?;
|
|
||||||
live_media.insert(tn_id);
|
|
||||||
} else if txn.media_exists(tn_id)? {
|
|
||||||
// Not regenerated this session — keep the stored pack.
|
|
||||||
live_media.insert(tn_id);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- orphan cleanup: drop media for removed clips/keyframes ---
|
|
||||||
let removed = txn.retain_media(&live_media)?;
|
|
||||||
|
|
||||||
// --- project.json + meta ---
|
|
||||||
let beam_project = BeamProject {
|
let beam_project = BeamProject {
|
||||||
version: BEAM_VERSION.to_string(),
|
version: BEAM_VERSION.to_string(),
|
||||||
created: created.clone(),
|
created: now.clone(),
|
||||||
modified: now.clone(),
|
modified: now,
|
||||||
ui_state: document.clone(),
|
ui_state: document.clone(),
|
||||||
audio_backend: SerializedAudioBackend {
|
audio_backend: SerializedAudioBackend {
|
||||||
sample_rate: 48000, // TODO: Get from audio engine
|
sample_rate: 48000, // TODO: Get from audio engine
|
||||||
|
|
@ -478,217 +409,52 @@ pub fn save_beam(
|
||||||
layer_to_track_map: layer_to_track_map.clone(),
|
layer_to_track_map: layer_to_track_map.clone(),
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
let json = serde_json::to_string(&beam_project)
|
eprintln!("📊 [SAVE_BEAM] Step 5: Build BeamProject structure took {:.2}ms", step5_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
|
// 6. Write project.json (compressed with DEFLATE)
|
||||||
|
let step6_start = std::time::Instant::now();
|
||||||
|
let json_options = FileOptions::default()
|
||||||
|
.compression_method(CompressionMethod::Deflated)
|
||||||
|
.compression_level(Some(6));
|
||||||
|
|
||||||
|
zip.start_file("project.json", json_options)
|
||||||
|
.map_err(|e| format!("Failed to create project.json in ZIP: {}", e))?;
|
||||||
|
|
||||||
|
let json = serde_json::to_string_pretty(&beam_project)
|
||||||
.map_err(|e| format!("JSON serialization failed: {}", e))?;
|
.map_err(|e| format!("JSON serialization failed: {}", e))?;
|
||||||
txn.set_project_json(&json)?;
|
|
||||||
txn.set_meta("version", BEAM_VERSION)?;
|
|
||||||
txn.set_meta("created", &created)?;
|
|
||||||
txn.set_meta("modified", &now)?;
|
|
||||||
txn.commit()?;
|
|
||||||
|
|
||||||
// Close the connection before renaming (required on Windows; harmless elsewhere).
|
zip.write_all(json.as_bytes())
|
||||||
drop(archive);
|
.map_err(|e| format!("Failed to write project.json: {}", e))?;
|
||||||
if !in_place {
|
eprintln!("📊 [SAVE_BEAM] Step 6: Write project.json ({} bytes) took {:.2}ms", json.len(), step6_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
std::fs::rename(&tmp_path, path)
|
|
||||||
.map_err(|e| format!("Failed to finalize {:?}: {}", path, e))?;
|
// 7. Finalize ZIP
|
||||||
}
|
let step7_start = std::time::Instant::now();
|
||||||
|
zip.finish()
|
||||||
|
.map_err(|e| format!("Failed to finalize ZIP: {}", e))?;
|
||||||
|
eprintln!("📊 [SAVE_BEAM] Step 7: Finalize ZIP took {:.2}ms", step7_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
|
eprintln!("📊 [SAVE_BEAM] ✅ Total save_beam() time: {:.2}ms", fn_start.elapsed().as_secs_f64() * 1000.0);
|
||||||
|
|
||||||
eprintln!(
|
|
||||||
"📊 [SAVE_BEAM] ✅ Saved {} audio + {} raster media, {} orphans removed, in {:.2}ms",
|
|
||||||
audio_pool_entries.len(),
|
|
||||||
raster_count,
|
|
||||||
removed,
|
|
||||||
fn_start.elapsed().as_secs_f64() * 1000.0
|
|
||||||
);
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Load a project from a `.beam` file.
|
/// Load a project from a .beam file
|
||||||
///
|
///
|
||||||
/// Detects the container format: SQLite (current) or legacy ZIP, and dispatches
|
/// This function:
|
||||||
/// accordingly. Both produce an identical [`LoadedProject`].
|
/// 1. Opens ZIP archive and reads project.json
|
||||||
|
/// 2. Deserializes project data
|
||||||
|
/// 3. Loads embedded media files from archive
|
||||||
|
/// 4. Attempts to load external media files
|
||||||
|
/// 5. Rebuilds AudioGraphs from presets with correct sample_rate
|
||||||
|
///
|
||||||
|
/// # Arguments
|
||||||
|
/// * `path` - Path to the .beam file
|
||||||
|
///
|
||||||
|
/// # Returns
|
||||||
|
/// LoadedProject on success (with missing_files list), or error message
|
||||||
pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
|
pub fn load_beam(path: &Path) -> Result<LoadedProject, String> {
|
||||||
if BeamArchive::is_sqlite(path) {
|
|
||||||
load_beam_sqlite(path)
|
|
||||||
} else {
|
|
||||||
load_beam_zip_legacy(path)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Load a project from a SQLite `.beam` container.
|
|
||||||
///
|
|
||||||
/// Phase 0 reconstitutes packed audio into each entry's `embedded_data` so the
|
|
||||||
/// existing (full-decode) audio pool loader keeps working unchanged; Phase 1b
|
|
||||||
/// replaces this with streaming reads via `BlobReader`.
|
|
||||||
fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
|
|
||||||
let fn_start = std::time::Instant::now();
|
let fn_start = std::time::Instant::now();
|
||||||
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (SQLite container)...");
|
eprintln!("📊 [LOAD_BEAM] Starting load_beam()...");
|
||||||
|
|
||||||
let archive = BeamArchive::open(path)?;
|
|
||||||
let json = archive.get_project_json()?;
|
|
||||||
let beam_project: BeamProject = serde_json::from_str(&json)
|
|
||||||
.map_err(|e| format!("Failed to deserialize project.json: {}", e))?;
|
|
||||||
|
|
||||||
if beam_project.version != BEAM_VERSION {
|
|
||||||
return Err(format!(
|
|
||||||
"Unsupported file version: {} (expected {})",
|
|
||||||
beam_project.version, BEAM_VERSION
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut document = beam_project.ui_state;
|
|
||||||
document.tempo_map_mut().rebuild_seconds();
|
|
||||||
let mut audio_project = beam_project.audio_backend.project;
|
|
||||||
audio_project
|
|
||||||
.rebuild_audio_graphs(DEFAULT_BUFFER_SIZE)
|
|
||||||
.map_err(|e| format!("Failed to rebuild audio graphs: {}", e))?;
|
|
||||||
let layer_to_track_map = beam_project.audio_backend.layer_to_track_map;
|
|
||||||
|
|
||||||
// For each packed audio item: stream it (leave `embedded_data` empty so the
|
|
||||||
// pool builds a Compressed placeholder backed by the blob factory) when it's a
|
|
||||||
// recognized audio codec; otherwise fall back to the legacy reconstitution
|
|
||||||
// (whole bytes → base64 → decode), which still covers video-container audio
|
|
||||||
// tracks symphonia can't stream and any unknown formats.
|
|
||||||
let mut restored_entries = Vec::with_capacity(beam_project.audio_backend.audio_pool_entries.len());
|
|
||||||
for entry in &beam_project.audio_backend.audio_pool_entries {
|
|
||||||
let mut e = entry.clone();
|
|
||||||
if let Some(id) = entry.media_id.as_ref().and_then(|s| Uuid::parse_str(s).ok()) {
|
|
||||||
match archive.media_info(id) {
|
|
||||||
Ok(Some(info)) => {
|
|
||||||
if is_streamable_audio_codec(&info.codec) {
|
|
||||||
// Stream: keep media_id, no embedded bytes. The engine opens
|
|
||||||
// the packed blob via the factory at activation time.
|
|
||||||
e.embedded_data = None;
|
|
||||||
e.relative_path = None;
|
|
||||||
} else {
|
|
||||||
match archive.read_media_full(id) {
|
|
||||||
Ok(bytes) => {
|
|
||||||
e.embedded_data = Some(daw_backend::audio::pool::EmbeddedAudioData {
|
|
||||||
data_base64: BASE64_STANDARD.encode(&bytes),
|
|
||||||
format: info.codec,
|
|
||||||
});
|
|
||||||
e.relative_path = None;
|
|
||||||
}
|
|
||||||
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read audio media {}: {}", id, err),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(None) => eprintln!("⚠️ [LOAD_BEAM] Audio media {} missing from archive", id),
|
|
||||||
Err(err) => eprintln!("⚠️ [LOAD_BEAM] media_info({}) failed: {}", id, err),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Restore this entry's persisted waveform pyramid, if present — avoids
|
|
||||||
// re-decoding the source media just to redraw the overview.
|
|
||||||
let wf_id = waveform_media_id(entry.pool_index);
|
|
||||||
if let Ok(Some(_)) = archive.media_info(wf_id) {
|
|
||||||
match archive.read_media_full(wf_id) {
|
|
||||||
Ok(bytes) => e.waveform_blob = Some(bytes),
|
|
||||||
Err(err) => eprintln!("⚠️ [LOAD_BEAM] Failed to read waveform {}: {}", wf_id, err),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
restored_entries.push(e);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Raster keyframes are NOT eagerly decoded (Phase 3 paging): `raw_pixels` stays
|
|
||||||
// empty and is faulted in on demand from the container's `Raster` rows via the
|
|
||||||
// editor's `RasterStore` (keyed by `kf.id`). Loading a big paint project is now
|
|
||||||
// instant and only the resident window lives in RAM. Mark every keyframe
|
|
||||||
// `needs_fault_in` (recursively, incl. nested layers) so the renderer requests a
|
|
||||||
// page-in; a freshly-created keyframe stays `false` (blank-resident, nothing to load).
|
|
||||||
// Proxies (low-res, ~tens of KB each) ARE decoded eagerly so a cold scrub onto a
|
|
||||||
// not-yet-paged frame shows the proxy instantly instead of flashing blank.
|
|
||||||
let mut raster_load_count = 0usize;
|
|
||||||
let mut proxy_load_count = 0usize;
|
|
||||||
for layer in document.all_layers_mut() {
|
|
||||||
if let crate::layer::AnyLayer::Raster(rl) = layer {
|
|
||||||
for kf in &mut rl.keyframes {
|
|
||||||
kf.needs_fault_in = true;
|
|
||||||
raster_load_count += 1;
|
|
||||||
let proxy_id = raster_proxy_media_id(kf.id);
|
|
||||||
if let Ok(Some(_)) = archive.media_info(proxy_id) {
|
|
||||||
if let Ok(bytes) = archive.read_media_full(proxy_id) {
|
|
||||||
if let Ok(img) = crate::brush_engine::decode_png(&bytes) {
|
|
||||||
let (w, h) = (img.width(), img.height());
|
|
||||||
kf.proxy = Some(crate::raster_layer::RasterProxy {
|
|
||||||
width: w,
|
|
||||||
height: h,
|
|
||||||
pixels: img.into_raw(),
|
|
||||||
});
|
|
||||||
proxy_load_count += 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let _ = proxy_load_count;
|
|
||||||
|
|
||||||
// Missing external files (referenced entries whose file no longer exists).
|
|
||||||
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
|
|
||||||
let missing_files: Vec<MissingFileInfo> = restored_entries
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.filter_map(|(idx, entry)| {
|
|
||||||
if entry.embedded_data.is_none() && entry.media_id.is_none() {
|
|
||||||
if let Some(rel) = entry.relative_path.as_ref() {
|
|
||||||
let full = if Path::new(rel).is_absolute() {
|
|
||||||
PathBuf::from(rel)
|
|
||||||
} else {
|
|
||||||
project_dir.join(rel)
|
|
||||||
};
|
|
||||||
if !full.exists() {
|
|
||||||
return Some(MissingFileInfo {
|
|
||||||
pool_index: idx,
|
|
||||||
original_path: full,
|
|
||||||
file_type: MediaFileType::Audio,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None
|
|
||||||
})
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
// Persisted video thumbnail packs (opaque LBTN blobs), keyed by clip id. The
|
|
||||||
// editor decodes + inserts them and skips regeneration for these clips.
|
|
||||||
let mut thumbnail_blobs = std::collections::HashMap::new();
|
|
||||||
for clip_id in document.video_clips.keys() {
|
|
||||||
let tn_id = thumbnail_media_id(*clip_id);
|
|
||||||
if let Ok(Some(info)) = archive.media_info(tn_id) {
|
|
||||||
if info.kind == MediaKind::Thumbnail {
|
|
||||||
match archive.read_media_full(tn_id) {
|
|
||||||
Ok(bytes) => { thumbnail_blobs.insert(*clip_id, bytes); }
|
|
||||||
Err(e) => eprintln!("⚠️ [LOAD_BEAM] Failed to read thumbnails for {}: {}", clip_id, e),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
eprintln!(
|
|
||||||
"📊 [LOAD_BEAM] ✅ Loaded {} audio entries, {} raster frames, {} thumbnail packs in {:.2}ms",
|
|
||||||
restored_entries.len(),
|
|
||||||
raster_load_count,
|
|
||||||
thumbnail_blobs.len(),
|
|
||||||
fn_start.elapsed().as_secs_f64() * 1000.0
|
|
||||||
);
|
|
||||||
|
|
||||||
Ok(LoadedProject {
|
|
||||||
document,
|
|
||||||
audio_project,
|
|
||||||
layer_to_track_map,
|
|
||||||
audio_pool_entries: restored_entries,
|
|
||||||
thumbnail_blobs,
|
|
||||||
missing_files,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Load a project from a legacy ZIP `.beam` archive (pre-SQLite format).
|
|
||||||
/// Retained for backward compatibility; saving converts to SQLite.
|
|
||||||
fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
|
|
||||||
let fn_start = std::time::Instant::now();
|
|
||||||
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (legacy ZIP)...");
|
|
||||||
|
|
||||||
// 1. Open ZIP archive
|
// 1. Open ZIP archive
|
||||||
let step1_start = std::time::Instant::now();
|
let step1_start = std::time::Instant::now();
|
||||||
|
|
@ -907,7 +673,6 @@ fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
|
||||||
audio_project,
|
audio_project,
|
||||||
layer_to_track_map,
|
layer_to_track_map,
|
||||||
audio_pool_entries: restored_entries,
|
audio_pool_entries: restored_entries,
|
||||||
thumbnail_blobs: std::collections::HashMap::new(), // legacy ZIP has no thumbnail packs
|
|
||||||
missing_files,
|
missing_files,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,6 @@ pub mod segment_builder;
|
||||||
pub mod planar_graph;
|
pub mod planar_graph;
|
||||||
pub mod file_types;
|
pub mod file_types;
|
||||||
pub mod file_io;
|
pub mod file_io;
|
||||||
pub mod beam_archive;
|
|
||||||
pub mod export;
|
pub mod export;
|
||||||
pub mod clipboard;
|
pub mod clipboard;
|
||||||
pub(crate) mod clipboard_platform;
|
pub(crate) mod clipboard_platform;
|
||||||
|
|
@ -54,7 +53,6 @@ pub mod svg_export;
|
||||||
pub mod snap;
|
pub mod snap;
|
||||||
pub mod webcam;
|
pub mod webcam;
|
||||||
pub mod raster_layer;
|
pub mod raster_layer;
|
||||||
pub mod raster_store;
|
|
||||||
pub mod brush_settings;
|
pub mod brush_settings;
|
||||||
pub mod brush_engine;
|
pub mod brush_engine;
|
||||||
pub mod raster_draw;
|
pub mod raster_draw;
|
||||||
|
|
|
||||||
|
|
@ -92,16 +92,6 @@ impl Default for TweenType {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A low-res decoded RGBA proxy of a keyframe's pixels, shown while the full-res
|
|
||||||
/// buffer pages in from the container so cold scrubs don't flash blank.
|
|
||||||
#[derive(Clone, Debug)]
|
|
||||||
pub struct RasterProxy {
|
|
||||||
pub width: u32,
|
|
||||||
pub height: u32,
|
|
||||||
/// RGBA, `width * height * 4` bytes.
|
|
||||||
pub pixels: Vec<u8>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A single keyframe of a raster layer
|
/// A single keyframe of a raster layer
|
||||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||||
pub struct RasterKeyframe {
|
pub struct RasterKeyframe {
|
||||||
|
|
@ -127,24 +117,6 @@ pub struct RasterKeyframe {
|
||||||
/// Always `true` after load; cleared by the renderer after uploading.
|
/// Always `true` after load; cleared by the renderer after uploading.
|
||||||
#[serde(skip, default = "default_true")]
|
#[serde(skip, default = "default_true")]
|
||||||
pub texture_dirty: bool,
|
pub texture_dirty: bool,
|
||||||
/// Phase 3 paging: the keyframe's pixels live in the container and must be
|
|
||||||
/// faulted in (`raw_pixels` empty *and* this true ⇒ page in from the store).
|
|
||||||
/// A *new* keyframe is `false` (intentionally blank/resident, nothing to load);
|
|
||||||
/// set true on load and again when evicted. Never serialized.
|
|
||||||
#[serde(skip)]
|
|
||||||
pub needs_fault_in: bool,
|
|
||||||
/// Phase 3a eviction: set `true` whenever user editing mutates `raw_pixels`
|
|
||||||
/// (brush, fill, paint-bucket, floating-selection commit/lift, undo/redo of
|
|
||||||
/// those). A dirty keyframe's current pixels are NOT yet persisted in the
|
|
||||||
/// container, so it must NEVER be evicted (doing so would silently lose the
|
|
||||||
/// unsaved edit). Cleared on a successful save. Never serialized.
|
|
||||||
#[serde(skip)]
|
|
||||||
pub dirty: bool,
|
|
||||||
/// Phase 3a-3: low-res proxy decoded from the container on load, rendered while
|
|
||||||
/// the full pixels page in (removes the cold-scrub blank flash). `None` if the
|
|
||||||
/// keyframe has no persisted proxy yet (new/unsaved, or pre-proxy project).
|
|
||||||
#[serde(skip)]
|
|
||||||
pub proxy: Option<RasterProxy>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn default_true() -> bool { true }
|
fn default_true() -> bool { true }
|
||||||
|
|
@ -168,9 +140,6 @@ impl RasterKeyframe {
|
||||||
tween_after: TweenType::Hold,
|
tween_after: TweenType::Hold,
|
||||||
raw_pixels: Vec::new(),
|
raw_pixels: Vec::new(),
|
||||||
texture_dirty: true,
|
texture_dirty: true,
|
||||||
needs_fault_in: false,
|
|
||||||
dirty: false,
|
|
||||||
proxy: None,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -235,35 +204,6 @@ impl RasterLayer {
|
||||||
&mut self.keyframes[insert_idx]
|
&mut self.keyframes[insert_idx]
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Insert a blank keyframe at `time` if none exists there (within tolerance).
|
|
||||||
/// Returns the new keyframe's id if one was created, `None` if a keyframe already
|
|
||||||
/// existed. Used by the explicit "New Keyframe" command (blank cel).
|
|
||||||
pub fn insert_blank_keyframe_at(&mut self, time: f64, width: u32, height: u32) -> Option<Uuid> {
|
|
||||||
if self.keyframe_index_at_exact(time, 0.001).is_some() {
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
let (w, h) = if width == 0 || height == 0 {
|
|
||||||
self.keyframe_at(time)
|
|
||||||
.map(|kf| (kf.width, kf.height))
|
|
||||||
.unwrap_or((1920, 1080))
|
|
||||||
} else {
|
|
||||||
(width, height)
|
|
||||||
};
|
|
||||||
let insert_idx = self.keyframes.partition_point(|kf| kf.time < time);
|
|
||||||
let kf = RasterKeyframe::new(time, w, h);
|
|
||||||
let id = kf.id;
|
|
||||||
self.keyframes.insert(insert_idx, kf);
|
|
||||||
Some(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Remove the keyframe with the given id, returning it if found.
|
|
||||||
pub fn remove_keyframe(&mut self, id: Uuid) -> Option<RasterKeyframe> {
|
|
||||||
self.keyframes
|
|
||||||
.iter()
|
|
||||||
.position(|kf| kf.id == id)
|
|
||||||
.map(|pos| self.keyframes.remove(pos))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`.
|
/// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`.
|
||||||
pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> {
|
pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> {
|
||||||
self.keyframe_at(time).map(|kf| kf.media_path.as_str())
|
self.keyframe_at(time).map(|kf| kf.media_path.as_str())
|
||||||
|
|
|
||||||
|
|
@ -1,56 +0,0 @@
|
||||||
//! On-demand loader for raster keyframe pixels backed by the project `.beam`
|
|
||||||
//! container (Phase 3 paging).
|
|
||||||
//!
|
|
||||||
//! Raster keyframes are no longer eagerly decoded at load; `raw_pixels` stays
|
|
||||||
//! empty until something needs the frame, then it is faulted in from the
|
|
||||||
//! container's `Raster` media row (keyed by the keyframe id). The store holds only
|
|
||||||
//! the container path and reads through a fresh **read-only** connection per call,
|
|
||||||
//! so it never conflicts with an in-place save and keeps no long-lived handle.
|
|
||||||
//! `None` path = an unsaved document (nothing to fault in).
|
|
||||||
|
|
||||||
use std::path::PathBuf;
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
/// Faults in raster keyframe pixels from the project container on demand.
|
|
||||||
#[derive(Default, Clone)]
|
|
||||||
pub struct RasterStore {
|
|
||||||
path: Option<PathBuf>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl RasterStore {
|
|
||||||
pub fn new(path: Option<PathBuf>) -> Self {
|
|
||||||
Self { path }
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Point the store at a (possibly new) container path, or `None` for an
|
|
||||||
/// unsaved document. Call on load and on save-as.
|
|
||||||
pub fn set_path(&mut self, path: Option<PathBuf>) {
|
|
||||||
self.path = path;
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn has_path(&self) -> bool {
|
|
||||||
self.path.is_some()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Decode the keyframe's full RGBA pixels from the container, or `None` if the
|
|
||||||
/// container has no row for it (or decoding fails). The returned buffer is the
|
|
||||||
/// working `raw_pixels` representation (`width*height*4` sRGB-premultiplied RGBA).
|
|
||||||
pub fn load_pixels(&self, kf_id: Uuid) -> Option<Vec<u8>> {
|
|
||||||
let path = self.path.as_ref()?;
|
|
||||||
let png = match crate::beam_archive::read_packed_media_readonly(path, kf_id) {
|
|
||||||
Ok(Some(bytes)) => bytes,
|
|
||||||
Ok(None) => return None,
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[RasterStore] read {} failed: {}", kf_id, e);
|
|
||||||
return None;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
match crate::brush_engine::decode_png(&png) {
|
|
||||||
Ok(img) => Some(img.into_raw()),
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("[RasterStore] decode {} failed: {}", kf_id, e);
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -423,25 +423,11 @@ pub fn render_layer_isolated(
|
||||||
* Affine::scale_non_uniform(scale_x, scale_y)
|
* Affine::scale_non_uniform(scale_x, scale_y)
|
||||||
* skew_transform;
|
* skew_transform;
|
||||||
|
|
||||||
// The decoded frame is scaled down to fit the document (decoder caps
|
|
||||||
// at the canvas size), so its pixel size is smaller than the clip's
|
|
||||||
// native dimensions. The instance is blitted treating the texture as
|
|
||||||
// `frame.width × frame.height`, while `clip_transform` is expressed in
|
|
||||||
// the clip's native space — so scale frame-px → clip-native-px first,
|
|
||||||
// else the frame renders small in a corner with its edges streaked.
|
|
||||||
let frame_to_clip = if frame.width > 0 && frame.height > 0 {
|
|
||||||
Affine::scale_non_uniform(
|
|
||||||
video_clip.width / frame.width as f64,
|
|
||||||
video_clip.height / frame.height as f64,
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
Affine::IDENTITY
|
|
||||||
};
|
|
||||||
instances.push(VideoRenderInstance {
|
instances.push(VideoRenderInstance {
|
||||||
rgba_data: frame.rgba_data.clone(),
|
rgba_data: frame.rgba_data.clone(),
|
||||||
width: frame.width,
|
width: frame.width,
|
||||||
height: frame.height,
|
height: frame.height,
|
||||||
transform: base_transform * clip_transform * frame_to_clip,
|
transform: base_transform * clip_transform,
|
||||||
opacity: (layer_opacity * inst_opacity) as f32,
|
opacity: (layer_opacity * inst_opacity) as f32,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
@ -1042,26 +1028,12 @@ fn render_video_layer(
|
||||||
// Create rectangle path for the video frame
|
// Create rectangle path for the video frame
|
||||||
let video_rect = Rect::new(0.0, 0.0, video_clip.width, video_clip.height);
|
let video_rect = Rect::new(0.0, 0.0, video_clip.width, video_clip.height);
|
||||||
|
|
||||||
// The decoded frame is scaled down to fit the document (the decoder caps at
|
|
||||||
// the canvas size to bound memory), so its pixel dimensions are smaller than
|
|
||||||
// the clip's native display size. Scale the image brush from frame-pixel
|
|
||||||
// space to the clip rect; without this the image is drawn 1:1 in a corner
|
|
||||||
// and its edge pixels pad the rest (small frame with "stretched corners").
|
|
||||||
let brush_transform = if frame.width > 0 && frame.height > 0 {
|
|
||||||
Affine::scale_non_uniform(
|
|
||||||
video_clip.width / frame.width as f64,
|
|
||||||
video_clip.height / frame.height as f64,
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
Affine::IDENTITY
|
|
||||||
};
|
|
||||||
|
|
||||||
// Render video frame as image fill
|
// Render video frame as image fill
|
||||||
scene.fill(
|
scene.fill(
|
||||||
Fill::NonZero,
|
Fill::NonZero,
|
||||||
instance_transform,
|
instance_transform,
|
||||||
&image_with_alpha,
|
&image_with_alpha,
|
||||||
Some(brush_transform),
|
None,
|
||||||
&video_rect,
|
&video_rect,
|
||||||
);
|
);
|
||||||
clip_rendered = true;
|
clip_rendered = true;
|
||||||
|
|
|
||||||
|
|
@ -395,7 +395,7 @@ mod tests {
|
||||||
let shape = Shape::new(path);
|
let shape = Shape::new(path);
|
||||||
|
|
||||||
assert_eq!(shape.versions.len(), 1);
|
assert_eq!(shape.versions.len(), 1);
|
||||||
assert!(shape.fill_color.is_none());
|
assert!(shape.fill_color.is_some());
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|
|
||||||
|
|
@ -97,7 +97,7 @@ impl VideoDecoder {
|
||||||
// Optionally build keyframe index for fast seeking
|
// Optionally build keyframe index for fast seeking
|
||||||
let keyframe_positions = if build_keyframes {
|
let keyframe_positions = if build_keyframes {
|
||||||
eprintln!("[Video Decoder] Building keyframe index for {}", path);
|
eprintln!("[Video Decoder] Building keyframe index for {}", path);
|
||||||
let positions = Self::scan_keyframes(&path, stream_index)?;
|
let positions = Self::build_keyframe_index(&path, stream_index)?;
|
||||||
eprintln!("[Video Decoder] Found {} keyframes", positions.len());
|
eprintln!("[Video Decoder] Found {} keyframes", positions.len());
|
||||||
positions
|
positions
|
||||||
} else {
|
} else {
|
||||||
|
|
@ -125,19 +125,14 @@ impl VideoDecoder {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Source file path this decoder reads from.
|
/// Build keyframe index for this decoder
|
||||||
pub fn path(&self) -> &str {
|
/// This can be called asynchronously after decoder creation
|
||||||
&self.path
|
fn build_and_set_keyframe_index(&mut self) -> Result<(), String> {
|
||||||
}
|
eprintln!("[Video Decoder] Building keyframe index for {}", self.path);
|
||||||
|
let positions = Self::build_keyframe_index(&self.path, self.stream_index)?;
|
||||||
/// Parameters needed to scan keyframes off-thread (path + video stream index).
|
eprintln!("[Video Decoder] Found {} keyframes", positions.len());
|
||||||
pub fn keyframe_scan_params(&self) -> (String, usize) {
|
|
||||||
(self.path.clone(), self.stream_index)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Replace the keyframe index (built off-thread via [`VideoDecoder::scan_keyframes`]).
|
|
||||||
pub fn set_keyframe_index(&mut self, positions: Vec<i64>) {
|
|
||||||
self.keyframe_positions = positions;
|
self.keyframe_positions = positions;
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get the output width (scaled dimensions)
|
/// Get the output width (scaled dimensions)
|
||||||
|
|
@ -155,10 +150,9 @@ impl VideoDecoder {
|
||||||
self.get_frame(timestamp)
|
self.get_frame(timestamp)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Build an index of all keyframe positions in the video by scanning packets
|
/// Build an index of all keyframe positions in the video
|
||||||
/// from a fresh input. Does not touch `self` — call it off-thread (it is slow
|
/// This enables fast seeking by knowing exactly where keyframes are
|
||||||
/// on long videos) and hand the result to [`VideoDecoder::set_keyframe_index`].
|
fn build_keyframe_index(path: &str, stream_index: usize) -> Result<Vec<i64>, String> {
|
||||||
pub fn scan_keyframes(path: &str, stream_index: usize) -> Result<Vec<i64>, String> {
|
|
||||||
let mut input = ffmpeg::format::input(path)
|
let mut input = ffmpeg::format::input(path)
|
||||||
.map_err(|e| format!("Failed to open video for indexing: {}", e))?;
|
.map_err(|e| format!("Failed to open video for indexing: {}", e))?;
|
||||||
|
|
||||||
|
|
@ -346,58 +340,6 @@ impl VideoDecoder {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Generate timeline thumbnails for a video using a **dedicated** decoder that
|
|
||||||
/// is independent of any shared playback decoder — so thumbnail work never holds
|
|
||||||
/// a lock the UI/playback needs.
|
|
||||||
///
|
|
||||||
/// Thumbnails are sampled at keyframes ~`interval_secs` apart. Decoding at a
|
|
||||||
/// keyframe is cheap (≈one frame) versus decoding forward to an arbitrary
|
|
||||||
/// timestamp (the whole GOP). Frames are decoded directly at `thumb_width` (so
|
|
||||||
/// `get_thumbnail_at`'s 128-wide assumption holds) and tightly packed RGBA is
|
|
||||||
/// handed to `on_thumb` as `(timestamp_secs, data)`.
|
|
||||||
pub fn generate_keyframe_thumbnails(
|
|
||||||
path: &str,
|
|
||||||
interval_secs: f64,
|
|
||||||
thumb_width: u32,
|
|
||||||
mut should_skip: impl FnMut(f64) -> bool,
|
|
||||||
mut on_thumb: impl FnMut(f64, Arc<Vec<u8>>),
|
|
||||||
) -> Result<(), String> {
|
|
||||||
// Own decoder at thumbnail resolution; builds its own keyframe index. The
|
|
||||||
// large max-height lets width be the constraining dimension, so output width
|
|
||||||
// is exactly `thumb_width`.
|
|
||||||
let mut decoder = VideoDecoder::new(
|
|
||||||
path.to_string(),
|
|
||||||
4,
|
|
||||||
Some(thumb_width),
|
|
||||||
Some(100_000),
|
|
||||||
true, // build keyframe index (needed to sample at keyframes)
|
|
||||||
)?;
|
|
||||||
|
|
||||||
let keyframe_secs: Vec<f64> = decoder
|
|
||||||
.keyframe_positions
|
|
||||||
.iter()
|
|
||||||
.map(|&ts| ts as f64 * decoder.time_base)
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
let mut last_emitted = f64::NEG_INFINITY;
|
|
||||||
for ks in keyframe_secs {
|
|
||||||
if ks - last_emitted < interval_secs {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// This keyframe is a target slot; advance regardless of skip so the chosen
|
|
||||||
// slots are deterministic (lets a resumed pass target the same timestamps).
|
|
||||||
last_emitted = ks;
|
|
||||||
// Skip slots already covered (resume after a partial save / dedup).
|
|
||||||
if should_skip(ks) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if let Ok(rgba) = decoder.get_frame(ks) {
|
|
||||||
on_thumb(ks, Arc::new(rgba));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Probe video file for metadata without creating a full decoder
|
/// Probe video file for metadata without creating a full decoder
|
||||||
pub fn probe_video(path: &str) -> Result<VideoMetadata, String> {
|
pub fn probe_video(path: &str) -> Result<VideoMetadata, String> {
|
||||||
ffmpeg::init().map_err(|e| e.to_string())?;
|
ffmpeg::init().map_err(|e| e.to_string())?;
|
||||||
|
|
@ -465,33 +407,18 @@ pub struct VideoManager {
|
||||||
/// Pool of video decoders, one per clip
|
/// Pool of video decoders, one per clip
|
||||||
decoders: HashMap<Uuid, Arc<Mutex<VideoDecoder>>>,
|
decoders: HashMap<Uuid, Arc<Mutex<VideoDecoder>>>,
|
||||||
|
|
||||||
/// Frame cache: (clip_id, timestamp_ms) -> frame. Stores decoded RGBA for
|
/// Frame cache: (clip_id, timestamp_ms) -> frame
|
||||||
/// zero-copy rendering. Bounded by a **byte budget** (not a frame count, which
|
/// Stores raw RGBA data for zero-copy rendering
|
||||||
/// would be unsafe across resolutions — a 4K frame is ~33MB vs ~2MB at 800x600)
|
frame_cache: HashMap<(Uuid, i64), Arc<VideoFrame>>,
|
||||||
/// so playback of arbitrarily long video never grows unbounded.
|
|
||||||
frame_cache: LruCache<(Uuid, i64), Arc<VideoFrame>>,
|
|
||||||
/// Running total of bytes held in `frame_cache` (sum of each frame's RGBA len),
|
|
||||||
/// kept in sync on insert/evict/remove so eviction is O(1) per frame.
|
|
||||||
frame_cache_bytes: usize,
|
|
||||||
|
|
||||||
/// Thumbnail cache: clip_id -> Vec of (timestamp, rgba_data)
|
/// Thumbnail cache: clip_id -> Vec of (timestamp, rgba_data)
|
||||||
/// Low-resolution (64px width) thumbnails for scrubbing
|
/// Low-resolution (64px width) thumbnails for scrubbing
|
||||||
thumbnail_cache: HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>>,
|
thumbnail_cache: HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>>,
|
||||||
|
|
||||||
/// Clips whose thumbnail generation finished. Only complete sets are worth
|
|
||||||
/// persisting — a partial set (saved mid-generation) is dropped so the load
|
|
||||||
/// regenerates it fully rather than leaving it permanently incomplete.
|
|
||||||
thumbnails_complete: std::collections::HashSet<Uuid>,
|
|
||||||
|
|
||||||
/// Maximum number of frames to cache per decoder
|
/// Maximum number of frames to cache per decoder
|
||||||
cache_size: usize,
|
cache_size: usize,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Byte budget for [`VideoManager::frame_cache`] (decoded full-resolution frames).
|
|
||||||
/// At ~2MB/frame (800x600) this holds ~128 frames; at ~33MB/frame (4K) ~8 — in
|
|
||||||
/// both cases enough for the current frame plus a scrub window, while bounding RAM.
|
|
||||||
const FRAME_CACHE_BYTE_BUDGET: usize = 256 * 1024 * 1024;
|
|
||||||
|
|
||||||
impl VideoManager {
|
impl VideoManager {
|
||||||
/// Create a new video manager with default cache size
|
/// Create a new video manager with default cache size
|
||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
|
|
@ -502,10 +429,8 @@ impl VideoManager {
|
||||||
pub fn with_cache_size(cache_size: usize) -> Self {
|
pub fn with_cache_size(cache_size: usize) -> Self {
|
||||||
Self {
|
Self {
|
||||||
decoders: HashMap::new(),
|
decoders: HashMap::new(),
|
||||||
frame_cache: LruCache::unbounded(),
|
frame_cache: HashMap::new(),
|
||||||
frame_cache_bytes: 0,
|
|
||||||
thumbnail_cache: HashMap::new(),
|
thumbnail_cache: HashMap::new(),
|
||||||
thumbnails_complete: std::collections::HashSet::new(),
|
|
||||||
cache_size,
|
cache_size,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -515,9 +440,8 @@ impl VideoManager {
|
||||||
/// `target_width` and `target_height` specify the maximum dimensions
|
/// `target_width` and `target_height` specify the maximum dimensions
|
||||||
/// for decoded frames. Video will be scaled down if larger.
|
/// for decoded frames. Video will be scaled down if larger.
|
||||||
///
|
///
|
||||||
/// The keyframe index is NOT built during this call — scan it off-thread via
|
/// The keyframe index is NOT built during this call - use `build_keyframe_index_async`
|
||||||
/// [`VideoDecoder::scan_keyframes`] and store it with
|
/// in a background thread to build it asynchronously.
|
||||||
/// [`VideoDecoder::set_keyframe_index`] so the slow scan never blocks playback.
|
|
||||||
pub fn load_video(
|
pub fn load_video(
|
||||||
&mut self,
|
&mut self,
|
||||||
clip_id: Uuid,
|
clip_id: Uuid,
|
||||||
|
|
@ -543,6 +467,20 @@ impl VideoManager {
|
||||||
Ok(metadata)
|
Ok(metadata)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Build keyframe index for a loaded video asynchronously
|
||||||
|
///
|
||||||
|
/// This should be called from a background thread after load_video()
|
||||||
|
/// to avoid blocking the UI during import.
|
||||||
|
pub fn build_keyframe_index(&self, clip_id: &Uuid) -> Result<(), String> {
|
||||||
|
let decoder_arc = self.decoders.get(clip_id)
|
||||||
|
.ok_or_else(|| format!("Video clip {} not found", clip_id))?;
|
||||||
|
|
||||||
|
let mut decoder = decoder_arc.lock()
|
||||||
|
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
|
||||||
|
|
||||||
|
decoder.build_and_set_keyframe_index()
|
||||||
|
}
|
||||||
|
|
||||||
/// Get a decoded frame for a specific clip at a specific timestamp
|
/// Get a decoded frame for a specific clip at a specific timestamp
|
||||||
///
|
///
|
||||||
/// Returns None if the clip is not loaded or decoding fails.
|
/// Returns None if the clip is not loaded or decoding fails.
|
||||||
|
|
@ -557,16 +495,14 @@ impl VideoManager {
|
||||||
return Some(Arc::clone(cached_frame));
|
return Some(Arc::clone(cached_frame));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get decoder for this clip. Clone the Arc so we don't hold a borrow of
|
// Get decoder for this clip
|
||||||
// `self.decoders` across the `&mut self` cache insert below.
|
let decoder_arc = self.decoders.get(clip_id)?;
|
||||||
let decoder_arc = Arc::clone(self.decoders.get(clip_id)?);
|
|
||||||
let mut decoder = decoder_arc.lock().ok()?;
|
let mut decoder = decoder_arc.lock().ok()?;
|
||||||
|
|
||||||
// Decode the frame
|
// Decode the frame
|
||||||
let rgba_data = decoder.get_frame(timestamp).ok()?;
|
let rgba_data = decoder.get_frame(timestamp).ok()?;
|
||||||
let width = decoder.output_width;
|
let width = decoder.output_width;
|
||||||
let height = decoder.output_height;
|
let height = decoder.output_height;
|
||||||
drop(decoder); // release the lock before touching `self`
|
|
||||||
|
|
||||||
// Create VideoFrame and cache it
|
// Create VideoFrame and cache it
|
||||||
let frame = Arc::new(VideoFrame {
|
let frame = Arc::new(VideoFrame {
|
||||||
|
|
@ -576,27 +512,65 @@ impl VideoManager {
|
||||||
timestamp,
|
timestamp,
|
||||||
});
|
});
|
||||||
|
|
||||||
self.cache_frame(cache_key, Arc::clone(&frame));
|
self.frame_cache.insert(cache_key, Arc::clone(&frame));
|
||||||
|
|
||||||
Some(frame)
|
Some(frame)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Insert a frame into the byte-budgeted cache, evicting least-recently-used
|
/// Generate thumbnails for a video clip (single batch version - use generate_thumbnails_progressive instead)
|
||||||
/// frames until the total is within [`FRAME_CACHE_BYTE_BUDGET`].
|
///
|
||||||
fn cache_frame(&mut self, key: (Uuid, i64), frame: Arc<VideoFrame>) {
|
/// Thumbnails are generated every 5 seconds at 128px width.
|
||||||
let bytes = frame.rgba_data.len();
|
/// This should be called in a background thread to avoid blocking.
|
||||||
if let Some(old) = self.frame_cache.put(key, frame) {
|
/// Thumbnails are inserted into the cache progressively as they're generated,
|
||||||
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(old.rgba_data.len());
|
/// allowing the UI to display them immediately.
|
||||||
}
|
///
|
||||||
self.frame_cache_bytes += bytes;
|
/// DEPRECATED: Use generate_thumbnails_progressive which releases the lock between thumbnails.
|
||||||
// Keep at least one frame resident even if it alone exceeds the budget.
|
pub fn generate_thumbnails(&mut self, clip_id: &Uuid, duration: f64) -> Result<(), String> {
|
||||||
while self.frame_cache_bytes > FRAME_CACHE_BYTE_BUDGET && self.frame_cache.len() > 1 {
|
let decoder_arc = self.decoders.get(clip_id)
|
||||||
if let Some((_, evicted)) = self.frame_cache.pop_lru() {
|
.ok_or("Clip not loaded")?
|
||||||
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(evicted.rgba_data.len());
|
.clone();
|
||||||
} else {
|
|
||||||
break;
|
let mut decoder = decoder_arc.lock()
|
||||||
|
.map_err(|e| format!("Failed to lock decoder: {}", e))?;
|
||||||
|
|
||||||
|
// Initialize thumbnail cache entry with empty vec
|
||||||
|
self.thumbnail_cache.insert(*clip_id, Vec::new());
|
||||||
|
|
||||||
|
let interval = 5.0; // Generate thumbnail every 5 seconds
|
||||||
|
let mut t = 0.0;
|
||||||
|
|
||||||
|
while t < duration {
|
||||||
|
// Decode frame at this timestamp
|
||||||
|
if let Ok(rgba_data) = decoder.get_frame(t) {
|
||||||
|
// Decode already scaled to output dimensions, but we want 128px width for thumbnails
|
||||||
|
// We need to scale down further
|
||||||
|
let current_width = decoder.output_width;
|
||||||
|
let current_height = decoder.output_height;
|
||||||
|
|
||||||
|
// Calculate thumbnail dimensions (128px width, maintain aspect ratio)
|
||||||
|
let thumb_width = 128u32;
|
||||||
|
let aspect_ratio = current_height as f32 / current_width as f32;
|
||||||
|
let thumb_height = (thumb_width as f32 * aspect_ratio) as u32;
|
||||||
|
|
||||||
|
// Simple nearest-neighbor downsampling for thumbnails
|
||||||
|
let thumb_data = downsample_rgba(
|
||||||
|
&rgba_data,
|
||||||
|
current_width,
|
||||||
|
current_height,
|
||||||
|
thumb_width,
|
||||||
|
thumb_height,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Insert thumbnail into cache immediately so UI can display it
|
||||||
|
if let Some(thumbnails) = self.thumbnail_cache.get_mut(clip_id) {
|
||||||
|
thumbnails.push((t, Arc::new(thumb_data)));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
t += interval;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get the decoder Arc for a clip (for external thumbnail generation)
|
/// Get the decoder Arc for a clip (for external thumbnail generation)
|
||||||
|
|
@ -605,57 +579,18 @@ impl VideoManager {
|
||||||
self.decoders.get(clip_id).cloned()
|
self.decoders.get(clip_id).cloned()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Snapshot all cached thumbnails for persistence (clip id -> sorted
|
/// Insert a thumbnail into the cache (for external thumbnail generation)
|
||||||
/// (timestamp, rgba) pairs). Cheap: clones the `Arc`s, not the pixel data.
|
|
||||||
/// Partial sets are persisted too — pair with [`complete_thumbnail_clips`] so
|
|
||||||
/// the load knows which clips still need generation resumed.
|
|
||||||
pub fn snapshot_all_thumbnails(&self) -> HashMap<Uuid, Vec<(f64, Arc<Vec<u8>>)>> {
|
|
||||||
self.thumbnail_cache.clone()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// The set of clips whose thumbnail generation has finished (a full keyframe
|
|
||||||
/// pass). A persisted set flagged incomplete is resumed on load.
|
|
||||||
pub fn complete_thumbnail_clips(&self) -> std::collections::HashSet<Uuid> {
|
|
||||||
self.thumbnails_complete.clone()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Mark a clip's thumbnail generation as complete (called when the background
|
|
||||||
/// generator finishes the full keyframe pass).
|
|
||||||
pub fn mark_thumbnails_complete(&mut self, clip_id: &Uuid) {
|
|
||||||
self.thumbnails_complete.insert(*clip_id);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Whether the clip already has a thumbnail within `tol` seconds of `ts`.
|
|
||||||
/// Lets the generator skip keyframes already covered (resume / dedup).
|
|
||||||
pub fn has_thumbnail_near(&self, clip_id: &Uuid, ts: f64, tol: f64) -> bool {
|
|
||||||
self.thumbnail_cache
|
|
||||||
.get(clip_id)
|
|
||||||
.map_or(false, |v| v.iter().any(|(t, _)| (t - ts).abs() < tol))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Insert a thumbnail into the cache, keeping it **sorted by timestamp** and
|
|
||||||
/// **deduped** (an existing entry at the same timestamp is replaced). Sorted
|
|
||||||
/// order is required by `get_thumbnail_at`'s binary search, and dedup makes
|
|
||||||
/// concurrent restore + resumed generation idempotent (no double inserts).
|
|
||||||
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) {
|
pub fn insert_thumbnail(&mut self, clip_id: &Uuid, timestamp: f64, data: Arc<Vec<u8>>) {
|
||||||
let vec = self.thumbnail_cache.entry(*clip_id).or_default();
|
self.thumbnail_cache
|
||||||
match vec.binary_search_by(|(t, _)| {
|
.entry(*clip_id)
|
||||||
t.partial_cmp(×tamp).unwrap_or(std::cmp::Ordering::Equal)
|
.or_insert_with(Vec::new)
|
||||||
}) {
|
.push((timestamp, data));
|
||||||
Ok(i) => vec[i] = (timestamp, data),
|
|
||||||
Err(i) => vec.insert(i, (timestamp, data)),
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get the thumbnail closest to the specified timestamp.
|
/// Get the thumbnail closest to the specified timestamp
|
||||||
///
|
///
|
||||||
/// Returns `(actual_timestamp, width, height, data)` — `actual_timestamp` is
|
|
||||||
/// the time of the thumbnail actually chosen (which may differ from the
|
|
||||||
/// requested `timestamp`, and changes as closer thumbnails finish generating).
|
|
||||||
/// Callers key their GPU texture cache on it so the on-clip strip refreshes as
|
|
||||||
/// better thumbnails load instead of freezing on the first one.
|
|
||||||
/// Returns None if no thumbnails have been generated for this clip.
|
/// Returns None if no thumbnails have been generated for this clip.
|
||||||
pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(f64, u32, u32, Arc<Vec<u8>>)> {
|
pub fn get_thumbnail_at(&self, clip_id: &Uuid, timestamp: f64) -> Option<(u32, u32, Arc<Vec<u8>>)> {
|
||||||
let thumbnails = self.thumbnail_cache.get(clip_id)?;
|
let thumbnails = self.thumbnail_cache.get(clip_id)?;
|
||||||
|
|
||||||
if thumbnails.is_empty() {
|
if thumbnails.is_empty() {
|
||||||
|
|
@ -683,43 +618,30 @@ impl VideoManager {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
let (actual_ts, rgba_data) = &thumbnails[idx];
|
let (_, rgba_data) = &thumbnails[idx];
|
||||||
|
|
||||||
// Return (actual_timestamp, width, height, data)
|
// Return (width, height, data)
|
||||||
// Thumbnails are always 128px width
|
// Thumbnails are always 128px width
|
||||||
let thumb_width = 128;
|
let thumb_width = 128;
|
||||||
let thumb_height = (rgba_data.len() / (thumb_width * 4)) as u32;
|
let thumb_height = (rgba_data.len() / (thumb_width * 4)) as u32;
|
||||||
|
|
||||||
Some((*actual_ts, thumb_width as u32, thumb_height, Arc::clone(rgba_data)))
|
Some((thumb_width as u32, thumb_height, Arc::clone(rgba_data)))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Remove a video clip and its cached data
|
/// Remove a video clip and its cached data
|
||||||
pub fn unload_video(&mut self, clip_id: &Uuid) {
|
pub fn unload_video(&mut self, clip_id: &Uuid) {
|
||||||
self.decoders.remove(clip_id);
|
self.decoders.remove(clip_id);
|
||||||
|
|
||||||
// Remove all cached frames for this clip (LruCache has no retain; collect
|
// Remove all cached frames for this clip
|
||||||
// matching keys, then pop each, keeping the byte total in sync).
|
self.frame_cache.retain(|(id, _), _| id != clip_id);
|
||||||
let keys: Vec<(Uuid, i64)> = self
|
|
||||||
.frame_cache
|
|
||||||
.iter()
|
|
||||||
.filter(|((id, _), _)| id == clip_id)
|
|
||||||
.map(|(k, _)| *k)
|
|
||||||
.collect();
|
|
||||||
for key in keys {
|
|
||||||
if let Some(frame) = self.frame_cache.pop(&key) {
|
|
||||||
self.frame_cache_bytes = self.frame_cache_bytes.saturating_sub(frame.rgba_data.len());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Remove thumbnails
|
// Remove thumbnails
|
||||||
self.thumbnail_cache.remove(clip_id);
|
self.thumbnail_cache.remove(clip_id);
|
||||||
self.thumbnails_complete.remove(clip_id);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Clear all frame caches (useful for memory management)
|
/// Clear all frame caches (useful for memory management)
|
||||||
pub fn clear_frame_cache(&mut self) {
|
pub fn clear_frame_cache(&mut self) {
|
||||||
self.frame_cache.clear();
|
self.frame_cache.clear();
|
||||||
self.frame_cache_bytes = 0;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -728,3 +650,211 @@ impl Default for VideoManager {
|
||||||
Self::new()
|
Self::new()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Simple nearest-neighbor downsampling for RGBA images
|
||||||
|
pub fn downsample_rgba_public(
|
||||||
|
src: &[u8],
|
||||||
|
src_width: u32,
|
||||||
|
src_height: u32,
|
||||||
|
dst_width: u32,
|
||||||
|
dst_height: u32,
|
||||||
|
) -> Vec<u8> {
|
||||||
|
downsample_rgba(src, src_width, src_height, dst_width, dst_height)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Simple nearest-neighbor downsampling for RGBA images (internal)
|
||||||
|
fn downsample_rgba(
|
||||||
|
src: &[u8],
|
||||||
|
src_width: u32,
|
||||||
|
src_height: u32,
|
||||||
|
dst_width: u32,
|
||||||
|
dst_height: u32,
|
||||||
|
) -> Vec<u8> {
|
||||||
|
let mut dst = Vec::with_capacity((dst_width * dst_height * 4) as usize);
|
||||||
|
|
||||||
|
let x_ratio = src_width as f32 / dst_width as f32;
|
||||||
|
let y_ratio = src_height as f32 / dst_height as f32;
|
||||||
|
|
||||||
|
for y in 0..dst_height {
|
||||||
|
for x in 0..dst_width {
|
||||||
|
let src_x = (x as f32 * x_ratio) as u32;
|
||||||
|
let src_y = (y as f32 * y_ratio) as u32;
|
||||||
|
|
||||||
|
let src_idx = ((src_y * src_width + src_x) * 4) as usize;
|
||||||
|
|
||||||
|
// Copy RGBA bytes
|
||||||
|
dst.push(src[src_idx]); // R
|
||||||
|
dst.push(src[src_idx + 1]); // G
|
||||||
|
dst.push(src[src_idx + 2]); // B
|
||||||
|
dst.push(src[src_idx + 3]); // A
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
dst
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extracted audio data from a video file
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct ExtractedAudio {
|
||||||
|
pub samples: Vec<f32>,
|
||||||
|
pub channels: u32,
|
||||||
|
pub sample_rate: u32,
|
||||||
|
pub duration: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract audio from a video file
|
||||||
|
///
|
||||||
|
/// This function performs the slow FFmpeg decoding without holding any locks.
|
||||||
|
/// The caller can then quickly add the audio to the DAW backend in a background thread.
|
||||||
|
///
|
||||||
|
/// Returns None if the video has no audio stream.
|
||||||
|
pub fn extract_audio_from_video(path: &str) -> Result<Option<ExtractedAudio>, String> {
|
||||||
|
ffmpeg::init().map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
// Open video file
|
||||||
|
let mut input = ffmpeg::format::input(path)
|
||||||
|
.map_err(|e| format!("Failed to open video: {}", e))?;
|
||||||
|
|
||||||
|
// Find audio stream
|
||||||
|
let audio_stream_opt = input.streams()
|
||||||
|
.best(ffmpeg::media::Type::Audio);
|
||||||
|
|
||||||
|
// Return None if no audio stream
|
||||||
|
if audio_stream_opt.is_none() {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
|
||||||
|
let audio_stream = audio_stream_opt.unwrap();
|
||||||
|
let audio_index = audio_stream.index();
|
||||||
|
|
||||||
|
// Get audio properties
|
||||||
|
let context_decoder = ffmpeg::codec::context::Context::from_parameters(
|
||||||
|
audio_stream.parameters()
|
||||||
|
).map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let mut audio_decoder = context_decoder.decoder().audio()
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let sample_rate = audio_decoder.rate();
|
||||||
|
let channels = audio_decoder.channels() as u32;
|
||||||
|
|
||||||
|
// Decode all audio frames
|
||||||
|
let mut audio_samples: Vec<f32> = Vec::new();
|
||||||
|
|
||||||
|
for (stream, packet) in input.packets() {
|
||||||
|
if stream.index() == audio_index {
|
||||||
|
audio_decoder.send_packet(&packet)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
|
||||||
|
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
|
||||||
|
// Convert audio to f32 packed format
|
||||||
|
let format = audio_frame.format();
|
||||||
|
let frame_channels = audio_frame.channels() as usize;
|
||||||
|
|
||||||
|
// Create resampler to convert to f32 packed
|
||||||
|
let mut resampler = ffmpeg::software::resampling::context::Context::get(
|
||||||
|
format,
|
||||||
|
audio_frame.channel_layout(),
|
||||||
|
sample_rate,
|
||||||
|
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
|
||||||
|
audio_frame.channel_layout(),
|
||||||
|
sample_rate,
|
||||||
|
).map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
|
||||||
|
resampler.run(&audio_frame, &mut resampled_frame)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
// Extract f32 samples (interleaved format)
|
||||||
|
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
|
||||||
|
let total_samples = resampled_frame.samples() * frame_channels;
|
||||||
|
|
||||||
|
// Safety checks before creating slice from FFmpeg data
|
||||||
|
// 1. Verify f32 alignment (required: 4 bytes)
|
||||||
|
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
|
||||||
|
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Verify the frame actually has enough data
|
||||||
|
let byte_size = resampled_frame.data(0).len();
|
||||||
|
let expected_bytes = total_samples * std::mem::size_of::<f32>();
|
||||||
|
if byte_size < expected_bytes {
|
||||||
|
return Err(format!(
|
||||||
|
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
|
||||||
|
byte_size, expected_bytes
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
// SAFETY: We verified alignment and bounds above.
|
||||||
|
// The slice lifetime is tied to resampled_frame which lives until
|
||||||
|
// after extend_from_slice completes.
|
||||||
|
let samples_slice = unsafe {
|
||||||
|
std::slice::from_raw_parts(data_ptr, total_samples)
|
||||||
|
};
|
||||||
|
|
||||||
|
audio_samples.extend_from_slice(samples_slice);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flush audio decoder
|
||||||
|
audio_decoder.send_eof().map_err(|e| e.to_string())?;
|
||||||
|
let mut audio_frame = ffmpeg::util::frame::Audio::empty();
|
||||||
|
while audio_decoder.receive_frame(&mut audio_frame).is_ok() {
|
||||||
|
let format = audio_frame.format();
|
||||||
|
let frame_channels = audio_frame.channels() as usize;
|
||||||
|
|
||||||
|
let mut resampler = ffmpeg::software::resampling::context::Context::get(
|
||||||
|
format,
|
||||||
|
audio_frame.channel_layout(),
|
||||||
|
sample_rate,
|
||||||
|
ffmpeg::format::Sample::F32(ffmpeg::format::sample::Type::Packed),
|
||||||
|
audio_frame.channel_layout(),
|
||||||
|
sample_rate,
|
||||||
|
).map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let mut resampled_frame = ffmpeg::util::frame::Audio::empty();
|
||||||
|
resampler.run(&audio_frame, &mut resampled_frame)
|
||||||
|
.map_err(|e| e.to_string())?;
|
||||||
|
|
||||||
|
let data_ptr = resampled_frame.data(0).as_ptr() as *const f32;
|
||||||
|
let total_samples = resampled_frame.samples() * frame_channels;
|
||||||
|
|
||||||
|
// Safety checks before creating slice from FFmpeg data
|
||||||
|
// 1. Verify f32 alignment (required: 4 bytes)
|
||||||
|
if data_ptr.align_offset(std::mem::align_of::<f32>()) != 0 {
|
||||||
|
return Err("FFmpeg audio data is not properly aligned for f32".to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Verify the frame actually has enough data
|
||||||
|
let byte_size = resampled_frame.data(0).len();
|
||||||
|
let expected_bytes = total_samples * std::mem::size_of::<f32>();
|
||||||
|
if byte_size < expected_bytes {
|
||||||
|
return Err(format!(
|
||||||
|
"FFmpeg frame buffer too small: {} bytes, need {} bytes",
|
||||||
|
byte_size, expected_bytes
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
// SAFETY: We verified alignment and bounds above.
|
||||||
|
// The slice lifetime is tied to resampled_frame which lives until
|
||||||
|
// after extend_from_slice completes.
|
||||||
|
let samples_slice = unsafe {
|
||||||
|
std::slice::from_raw_parts(data_ptr, total_samples)
|
||||||
|
};
|
||||||
|
|
||||||
|
audio_samples.extend_from_slice(samples_slice);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate duration
|
||||||
|
let total_samples_per_channel = audio_samples.len() / channels as usize;
|
||||||
|
let duration = total_samples_per_channel as f64 / sample_rate as f64;
|
||||||
|
|
||||||
|
Ok(Some(ExtractedAudio {
|
||||||
|
samples: audio_samples,
|
||||||
|
channels,
|
||||||
|
sample_rate,
|
||||||
|
duration,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,207 +0,0 @@
|
||||||
//! Integration tests for the SQLite-backed `.beam` container.
|
|
||||||
//!
|
|
||||||
//! These are integration tests (not `#[cfg(test)]` unit tests) so they build the
|
|
||||||
//! library in normal mode and exercise only the public API — independent of any
|
|
||||||
//! pre-existing breakage in the crate's internal test modules.
|
|
||||||
|
|
||||||
use lightningbeam_core::beam_archive::{BeamArchive, MediaKind, MediaMeta, MediaStorage};
|
|
||||||
use std::io::{Read, Seek, SeekFrom};
|
|
||||||
use std::sync::atomic::{AtomicU64, Ordering};
|
|
||||||
use uuid::Uuid;
|
|
||||||
|
|
||||||
fn temp_db_path(tag: &str) -> std::path::PathBuf {
|
|
||||||
static N: AtomicU64 = AtomicU64::new(0);
|
|
||||||
let n = N.fetch_add(1, Ordering::Relaxed);
|
|
||||||
let mut p = std::env::temp_dir();
|
|
||||||
p.push(format!("beam_archive_it_{}_{}_{}.beam", std::process::id(), tag, n));
|
|
||||||
let _ = std::fs::remove_file(&p);
|
|
||||||
p
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn project_json_roundtrip() {
|
|
||||||
let path = temp_db_path("json");
|
|
||||||
let archive = BeamArchive::create(&path).unwrap();
|
|
||||||
archive.set_project_json("{\"hello\":\"world\"}").unwrap();
|
|
||||||
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
|
|
||||||
drop(archive);
|
|
||||||
let archive = BeamArchive::open(&path).unwrap();
|
|
||||||
assert_eq!(archive.get_project_json().unwrap(), "{\"hello\":\"world\"}");
|
|
||||||
assert!(BeamArchive::is_sqlite(&path));
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn packed_media_roundtrip_full() {
|
|
||||||
let path = temp_db_path("full");
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
let id = Uuid::from_u128(0x1234);
|
|
||||||
archive.set_chunk_size(1000);
|
|
||||||
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
|
|
||||||
archive
|
|
||||||
.put_media_packed(
|
|
||||||
id,
|
|
||||||
MediaKind::Audio,
|
|
||||||
"flac",
|
|
||||||
&data,
|
|
||||||
MediaMeta { channels: Some(2), sample_rate: Some(44100), ..Default::default() },
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let info = archive.media_info(id).unwrap().unwrap();
|
|
||||||
assert_eq!(info.kind, MediaKind::Audio);
|
|
||||||
assert_eq!(info.codec, "flac");
|
|
||||||
assert_eq!(info.storage, MediaStorage::Packed);
|
|
||||||
assert_eq!(info.total_len, 3500);
|
|
||||||
assert_eq!(info.channels, Some(2));
|
|
||||||
assert_eq!(info.sample_rate, Some(44100));
|
|
||||||
|
|
||||||
assert_eq!(archive.read_media_full(id).unwrap(), data);
|
|
||||||
assert_eq!(archive.media_ids_of_kind(MediaKind::Audio).unwrap(), vec![id]);
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn blob_reader_streams_and_seeks() {
|
|
||||||
let path = temp_db_path("stream");
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
archive.set_chunk_size(1000);
|
|
||||||
let id = Uuid::from_u128(0xBEEF);
|
|
||||||
let data: Vec<u8> = (0..3500u32).map(|i| (i % 251) as u8).collect();
|
|
||||||
archive
|
|
||||||
.put_media_packed(id, MediaKind::Audio, "mp3", &data, MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let mut reader = archive.open_blob_reader(&path, id).unwrap();
|
|
||||||
assert_eq!(reader.len(), 3500);
|
|
||||||
|
|
||||||
// Sequential read in odd-sized buffers crosses chunk boundaries.
|
|
||||||
let mut got = Vec::new();
|
|
||||||
let mut buf = [0u8; 333];
|
|
||||||
loop {
|
|
||||||
let n = reader.read(&mut buf).unwrap();
|
|
||||||
if n == 0 {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
got.extend_from_slice(&buf[..n]);
|
|
||||||
}
|
|
||||||
assert_eq!(got, data);
|
|
||||||
|
|
||||||
// Seek to a mid-chunk position and read across a boundary.
|
|
||||||
reader.seek(SeekFrom::Start(990)).unwrap();
|
|
||||||
let mut window = [0u8; 20];
|
|
||||||
let mut filled = 0;
|
|
||||||
while filled < window.len() {
|
|
||||||
let n = reader.read(&mut window[filled..]).unwrap();
|
|
||||||
assert!(n > 0);
|
|
||||||
filled += n;
|
|
||||||
}
|
|
||||||
assert_eq!(&window[..], &data[990..1010]);
|
|
||||||
|
|
||||||
// Seek from end and read the tail.
|
|
||||||
reader.seek(SeekFrom::End(-10)).unwrap();
|
|
||||||
let mut tail = Vec::new();
|
|
||||||
reader.read_to_end(&mut tail).unwrap();
|
|
||||||
assert_eq!(tail, &data[3490..]);
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn referenced_media_records_path() {
|
|
||||||
let path = temp_db_path("ref");
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
let id = Uuid::from_u128(0xCAFE);
|
|
||||||
archive
|
|
||||||
.put_media_referenced(
|
|
||||||
id,
|
|
||||||
MediaKind::Video,
|
|
||||||
"mp4",
|
|
||||||
"/mnt/share/big.mp4",
|
|
||||||
MediaMeta { width: Some(3840), height: Some(2160), ..Default::default() },
|
|
||||||
)
|
|
||||||
.unwrap();
|
|
||||||
let info = archive.media_info(id).unwrap().unwrap();
|
|
||||||
assert_eq!(info.storage, MediaStorage::Referenced);
|
|
||||||
assert_eq!(info.ext_path.as_deref(), Some("/mnt/share/big.mp4"));
|
|
||||||
assert_eq!(info.width, Some(3840));
|
|
||||||
// Streaming a referenced item is an error (caller opens the path directly).
|
|
||||||
assert!(archive.open_blob_reader(&path, id).is_err());
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn transaction_groups_writes_and_orphan_cleanup() {
|
|
||||||
let path = temp_db_path("txn");
|
|
||||||
let keep = Uuid::from_u128(1);
|
|
||||||
let orphan = Uuid::from_u128(2);
|
|
||||||
|
|
||||||
// First save: two media items committed in one transaction.
|
|
||||||
{
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
let txn = archive.transaction().unwrap();
|
|
||||||
txn.put_media_packed(keep, MediaKind::Audio, "flac", &vec![9u8; 10], MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
txn.put_media_packed(orphan, MediaKind::Audio, "mp3", &vec![8u8; 10], MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
txn.set_project_json("{}").unwrap();
|
|
||||||
txn.commit().unwrap();
|
|
||||||
}
|
|
||||||
{
|
|
||||||
let archive = BeamArchive::open(&path).unwrap();
|
|
||||||
assert!(archive.media_info(keep).unwrap().is_some());
|
|
||||||
assert!(archive.media_info(orphan).unwrap().is_some());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Second save (in place): keep only `keep`; `orphan` should be retained-out.
|
|
||||||
{
|
|
||||||
let mut archive = BeamArchive::open(&path).unwrap();
|
|
||||||
let txn = archive.transaction().unwrap();
|
|
||||||
// `keep` already present → in-place save leaves it untouched.
|
|
||||||
assert!(txn.media_exists(keep).unwrap());
|
|
||||||
let mut live = std::collections::HashSet::new();
|
|
||||||
live.insert(keep);
|
|
||||||
let removed = txn.retain_media(&live).unwrap();
|
|
||||||
assert_eq!(removed, 1);
|
|
||||||
txn.commit().unwrap();
|
|
||||||
}
|
|
||||||
{
|
|
||||||
let archive = BeamArchive::open(&path).unwrap();
|
|
||||||
assert!(archive.media_info(keep).unwrap().is_some());
|
|
||||||
assert!(archive.media_info(orphan).unwrap().is_none());
|
|
||||||
// `keep`'s bytes survived untouched.
|
|
||||||
assert_eq!(archive.read_media_full(keep).unwrap(), vec![9u8; 10]);
|
|
||||||
}
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn rolled_back_transaction_writes_nothing() {
|
|
||||||
let path = temp_db_path("rollback");
|
|
||||||
let id = Uuid::from_u128(42);
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
{
|
|
||||||
let txn = archive.transaction().unwrap();
|
|
||||||
txn.put_media_packed(id, MediaKind::Audio, "flac", &vec![1u8; 5], MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
// Drop without commit → rollback.
|
|
||||||
}
|
|
||||||
assert!(archive.media_info(id).unwrap().is_none());
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn overwrite_media_replaces_chunks() {
|
|
||||||
let path = temp_db_path("overwrite");
|
|
||||||
let mut archive = BeamArchive::create(&path).unwrap();
|
|
||||||
archive.set_chunk_size(100);
|
|
||||||
let id = Uuid::from_u128(7);
|
|
||||||
archive
|
|
||||||
.put_media_packed(id, MediaKind::Raster, "png", &vec![1u8; 250], MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
// Overwrite with shorter data — stale chunks must be gone.
|
|
||||||
archive
|
|
||||||
.put_media_packed(id, MediaKind::Raster, "png", &vec![2u8; 50], MediaMeta::default())
|
|
||||||
.unwrap();
|
|
||||||
assert_eq!(archive.read_media_full(id).unwrap(), vec![2u8; 50]);
|
|
||||||
let _ = std::fs::remove_file(&path);
|
|
||||||
}
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use crate::keymap::KeybindingConfig;
|
use crate::keymap::KeybindingConfig;
|
||||||
use lightningbeam_core::file_io::LargeMediaMode;
|
|
||||||
|
|
||||||
/// Application configuration (persistent)
|
/// Application configuration (persistent)
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
|
@ -58,18 +57,6 @@ pub struct AppConfig {
|
||||||
/// Custom keyboard shortcut overrides (sparse — only non-default bindings stored)
|
/// Custom keyboard shortcut overrides (sparse — only non-default bindings stored)
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub keybindings: KeybindingConfig,
|
pub keybindings: KeybindingConfig,
|
||||||
|
|
||||||
/// How to store media files at/above the large-media threshold (~2GB).
|
|
||||||
/// `Ask` (default) prompts the first time such a file is imported, then the
|
|
||||||
/// chosen mode is persisted here. Reset to `Ask` to be prompted again.
|
|
||||||
#[serde(default)]
|
|
||||||
pub large_media_default: LargeMediaMode,
|
|
||||||
|
|
||||||
/// Finest-level resolution of the waveform LOD pyramid: source frames per
|
|
||||||
/// floor texel (`B`). Smaller = larger on-disk pyramid but zoom-in re-decodes
|
|
||||||
/// sooner; larger = smaller pyramid, wider re-decode span. Default 256.
|
|
||||||
#[serde(default = "defaults::waveform_floor_samples_per_texel")]
|
|
||||||
pub waveform_floor_samples_per_texel: u32,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for AppConfig {
|
impl Default for AppConfig {
|
||||||
|
|
@ -88,8 +75,6 @@ impl Default for AppConfig {
|
||||||
waveform_stereo: defaults::waveform_stereo(),
|
waveform_stereo: defaults::waveform_stereo(),
|
||||||
theme_mode: defaults::theme_mode(),
|
theme_mode: defaults::theme_mode(),
|
||||||
keybindings: KeybindingConfig::default(),
|
keybindings: KeybindingConfig::default(),
|
||||||
large_media_default: LargeMediaMode::default(),
|
|
||||||
waveform_floor_samples_per_texel: defaults::waveform_floor_samples_per_texel(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -291,5 +276,4 @@ mod defaults {
|
||||||
pub fn debug() -> bool { false }
|
pub fn debug() -> bool { false }
|
||||||
pub fn waveform_stereo() -> bool { false }
|
pub fn waveform_stereo() -> bool { false }
|
||||||
pub fn theme_mode() -> String { "system".to_string() }
|
pub fn theme_mode() -> String { "system".to_string() }
|
||||||
pub fn waveform_floor_samples_per_texel() -> u32 { 256 }
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -39,33 +39,6 @@ pub fn update_prepare_timing(
|
||||||
t.composite_ms = composite_ms;
|
t.composite_ms = composite_ms;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// GPU memory the editor tracks itself (wgpu has no allocator query). Currently the
|
|
||||||
/// raster-layer texture cache — the only unbounded-by-default VRAM consumer.
|
|
||||||
#[derive(Debug, Clone, Default)]
|
|
||||||
pub struct GpuMemoryStats {
|
|
||||||
pub raster_cache_entries: usize,
|
|
||||||
pub raster_cache_bytes: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
static GPU_MEMORY: OnceLock<Mutex<GpuMemoryStats>> = OnceLock::new();
|
|
||||||
|
|
||||||
/// Called by the GPU brush whenever the raster-layer cache changes.
|
|
||||||
pub fn update_gpu_memory(raster_cache_entries: usize, raster_cache_bytes: usize) {
|
|
||||||
let cell = GPU_MEMORY.get_or_init(|| Mutex::new(GpuMemoryStats::default()));
|
|
||||||
if let Ok(mut s) = cell.lock() {
|
|
||||||
s.raster_cache_entries = raster_cache_entries;
|
|
||||||
s.raster_cache_bytes = raster_cache_bytes;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn get_gpu_memory() -> GpuMemoryStats {
|
|
||||||
GPU_MEMORY
|
|
||||||
.get_or_init(|| Mutex::new(GpuMemoryStats::default()))
|
|
||||||
.lock()
|
|
||||||
.map(|s| s.clone())
|
|
||||||
.unwrap_or_default()
|
|
||||||
}
|
|
||||||
|
|
||||||
const DEVICE_REFRESH_INTERVAL: Duration = Duration::from_secs(2); // Refresh devices every 2 seconds
|
const DEVICE_REFRESH_INTERVAL: Duration = Duration::from_secs(2); // Refresh devices every 2 seconds
|
||||||
const MEMORY_REFRESH_INTERVAL: Duration = Duration::from_millis(500); // Refresh memory every 500ms
|
const MEMORY_REFRESH_INTERVAL: Duration = Duration::from_millis(500); // Refresh memory every 500ms
|
||||||
|
|
||||||
|
|
@ -79,7 +52,6 @@ pub struct DebugStats {
|
||||||
pub frame_time_ms: f32, // Current frame time in milliseconds
|
pub frame_time_ms: f32, // Current frame time in milliseconds
|
||||||
pub memory_physical_mb: usize,
|
pub memory_physical_mb: usize,
|
||||||
pub memory_virtual_mb: usize,
|
pub memory_virtual_mb: usize,
|
||||||
pub gpu_memory: GpuMemoryStats,
|
|
||||||
pub gpu_name: String,
|
pub gpu_name: String,
|
||||||
pub gpu_backend: String,
|
pub gpu_backend: String,
|
||||||
pub gpu_driver: String,
|
pub gpu_driver: String,
|
||||||
|
|
@ -246,7 +218,6 @@ impl DebugStatsCollector {
|
||||||
frame_time_ms,
|
frame_time_ms,
|
||||||
memory_physical_mb,
|
memory_physical_mb,
|
||||||
memory_virtual_mb,
|
memory_virtual_mb,
|
||||||
gpu_memory: get_gpu_memory(),
|
|
||||||
gpu_name,
|
gpu_name,
|
||||||
gpu_backend,
|
gpu_backend,
|
||||||
gpu_driver,
|
gpu_driver,
|
||||||
|
|
@ -315,11 +286,6 @@ pub fn render_debug_overlay(ctx: &egui::Context, stats: &DebugStats) {
|
||||||
ui.colored_label(egui::Color32::YELLOW, format!("Memory: ({}µs)", stats.timing_memory_us));
|
ui.colored_label(egui::Color32::YELLOW, format!("Memory: ({}µs)", stats.timing_memory_us));
|
||||||
ui.label(format!("Physical: {} MB", stats.memory_physical_mb));
|
ui.label(format!("Physical: {} MB", stats.memory_physical_mb));
|
||||||
ui.label(format!("Virtual: {} MB", stats.memory_virtual_mb));
|
ui.label(format!("Virtual: {} MB", stats.memory_virtual_mb));
|
||||||
ui.label(format!(
|
|
||||||
"VRAM (raster cache): {:.1} MB ({} frames)",
|
|
||||||
stats.gpu_memory.raster_cache_bytes as f64 / (1024.0 * 1024.0),
|
|
||||||
stats.gpu_memory.raster_cache_entries,
|
|
||||||
));
|
|
||||||
|
|
||||||
ui.add_space(8.0);
|
ui.add_space(8.0);
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -384,90 +384,90 @@ impl ExportOrchestrator {
|
||||||
println!("🎵 [MUX] Audio stream - Input TB: {}/{}, Output TB: {}/{}",
|
println!("🎵 [MUX] Audio stream - Input TB: {}/{}, Output TB: {}/{}",
|
||||||
audio_input_tb.0, audio_input_tb.1, audio_output_tb.0, audio_output_tb.1);
|
audio_input_tb.0, audio_input_tb.1, audio_output_tb.0, audio_output_tb.1);
|
||||||
|
|
||||||
// Stream-merge the two inputs by PTS, writing each packet as it's read —
|
// Collect all packets with their stream info and timestamps
|
||||||
// O(1) memory (one pending packet per stream) instead of collecting every
|
let mut video_packets = Vec::new();
|
||||||
// packet first, so muxing a long export never grows unbounded.
|
for (stream, packet) in video_input.packets() {
|
||||||
let video_idx = video_stream_index;
|
if stream.index() == video_stream_index {
|
||||||
let audio_idx = audio_stream_index;
|
video_packets.push(packet);
|
||||||
let mut v_iter = video_input.packets();
|
}
|
||||||
let mut a_iter = audio_input.packets();
|
}
|
||||||
|
|
||||||
// Pull the next packet belonging to the desired stream from each input.
|
let mut audio_packets = Vec::new();
|
||||||
let mut next_video = move || -> Option<ffmpeg::Packet> {
|
for (stream, packet) in audio_input.packets() {
|
||||||
loop {
|
if stream.index() == audio_stream_index {
|
||||||
match v_iter.next() {
|
audio_packets.push(packet);
|
||||||
Some((stream, packet)) => {
|
|
||||||
if stream.index() == video_idx {
|
|
||||||
return Some(packet);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
None => return None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
let mut next_audio = move || -> Option<ffmpeg::Packet> {
|
|
||||||
loop {
|
|
||||||
match a_iter.next() {
|
|
||||||
Some((stream, packet)) => {
|
|
||||||
if stream.index() == audio_idx {
|
|
||||||
return Some(packet);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
None => return None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut pending_v = next_video();
|
println!("🎬 [MUX] Collected {} video packets, {} audio packets",
|
||||||
let mut pending_a = next_audio();
|
video_packets.len(), audio_packets.len());
|
||||||
let mut v_count = 0usize;
|
|
||||||
let mut a_count = 0usize;
|
|
||||||
let mut log_count = 0;
|
|
||||||
|
|
||||||
loop {
|
// Report first and last timestamps
|
||||||
// Write whichever pending packet has the earlier PTS (in a common
|
if !video_packets.is_empty() {
|
||||||
// microsecond base); when one stream is exhausted, drain the other.
|
println!("🎬 [MUX] Video PTS range: {} to {}",
|
||||||
let write_video = match (&pending_v, &pending_a) {
|
video_packets[0].pts().unwrap_or(0),
|
||||||
(None, None) => break,
|
video_packets[video_packets.len()-1].pts().unwrap_or(0));
|
||||||
(Some(_), None) => true,
|
|
||||||
(None, Some(_)) => false,
|
|
||||||
(Some(v), Some(a)) => {
|
|
||||||
let v_us = v.pts().unwrap_or(0) * 1_000_000 * video_input_tb.0 as i64
|
|
||||||
/ video_input_tb.1 as i64;
|
|
||||||
let a_us = a.pts().unwrap_or(0) * 1_000_000 * audio_input_tb.0 as i64
|
|
||||||
/ audio_input_tb.1 as i64;
|
|
||||||
v_us <= a_us
|
|
||||||
}
|
}
|
||||||
|
if !audio_packets.is_empty() {
|
||||||
|
println!("🎵 [MUX] Audio PTS range: {} to {}",
|
||||||
|
audio_packets[0].pts().unwrap_or(0),
|
||||||
|
audio_packets[audio_packets.len()-1].pts().unwrap_or(0));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Interleave packets by comparing timestamps in a common time base (use microseconds)
|
||||||
|
let mut v_idx = 0;
|
||||||
|
let mut a_idx = 0;
|
||||||
|
let mut interleave_log_count = 0;
|
||||||
|
|
||||||
|
while v_idx < video_packets.len() || a_idx < audio_packets.len() {
|
||||||
|
let write_video = if v_idx >= video_packets.len() {
|
||||||
|
false // No more video
|
||||||
|
} else if a_idx >= audio_packets.len() {
|
||||||
|
true // No more audio, write video
|
||||||
|
} else {
|
||||||
|
// Compare timestamps - convert both to microseconds
|
||||||
|
let v_pts = video_packets[v_idx].pts().unwrap_or(0);
|
||||||
|
let a_pts = audio_packets[a_idx].pts().unwrap_or(0);
|
||||||
|
|
||||||
|
// Convert to microseconds: pts * 1000000 * tb.num / tb.den
|
||||||
|
let v_us = v_pts * 1_000_000 * video_input_tb.0 as i64 / video_input_tb.1 as i64;
|
||||||
|
let a_us = a_pts * 1_000_000 * audio_input_tb.0 as i64 / audio_input_tb.1 as i64;
|
||||||
|
|
||||||
|
v_us <= a_us // Write video if it comes before or at same time as audio
|
||||||
};
|
};
|
||||||
|
|
||||||
if write_video {
|
if write_video {
|
||||||
let mut packet = pending_v.take().unwrap();
|
let mut packet = video_packets[v_idx].clone();
|
||||||
packet.set_stream(0);
|
packet.set_stream(0);
|
||||||
packet.rescale_ts(video_input_tb, video_output_tb);
|
packet.rescale_ts(video_input_tb, video_output_tb);
|
||||||
if log_count < 10 {
|
|
||||||
println!("🎬 [MUX] Writing V packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
|
if interleave_log_count < 10 {
|
||||||
log_count += 1;
|
println!("🎬 [MUX] Writing V packet {} - PTS={:?}, DTS={:?}, Duration={:?}",
|
||||||
|
v_idx, packet.pts(), packet.dts(), packet.duration());
|
||||||
|
interleave_log_count += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
packet.write_interleaved(&mut output)
|
packet.write_interleaved(&mut output)
|
||||||
.map_err(|e| format!("Failed to write video packet: {}", e))?;
|
.map_err(|e| format!("Failed to write video packet: {}", e))?;
|
||||||
v_count += 1;
|
v_idx += 1;
|
||||||
pending_v = next_video();
|
|
||||||
} else {
|
} else {
|
||||||
let mut packet = pending_a.take().unwrap();
|
let mut packet = audio_packets[a_idx].clone();
|
||||||
packet.set_stream(1);
|
packet.set_stream(1);
|
||||||
packet.rescale_ts(audio_input_tb, audio_output_tb);
|
packet.rescale_ts(audio_input_tb, audio_output_tb);
|
||||||
if log_count < 10 {
|
|
||||||
println!("🎵 [MUX] Writing A packet - PTS={:?}, DTS={:?}", packet.pts(), packet.dts());
|
if interleave_log_count < 10 {
|
||||||
log_count += 1;
|
println!("🎵 [MUX] Writing A packet {} - PTS={:?}, DTS={:?}, Duration={:?}",
|
||||||
|
a_idx, packet.pts(), packet.dts(), packet.duration());
|
||||||
|
interleave_log_count += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
packet.write_interleaved(&mut output)
|
packet.write_interleaved(&mut output)
|
||||||
.map_err(|e| format!("Failed to write audio packet: {}", e))?;
|
.map_err(|e| format!("Failed to write audio packet: {}", e))?;
|
||||||
a_count += 1;
|
a_idx += 1;
|
||||||
pending_a = next_audio();
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_count, a_count);
|
println!("🎬 [MUX] Wrote {} video packets, {} audio packets", v_idx, a_idx);
|
||||||
|
|
||||||
// Write trailer
|
// Write trailer
|
||||||
output.write_trailer().map_err(|e| format!("Failed to write trailer: {}", e))?;
|
output.write_trailer().map_err(|e| format!("Failed to write trailer: {}", e))?;
|
||||||
|
|
@ -528,7 +528,6 @@ impl ExportOrchestrator {
|
||||||
image_cache: &mut ImageCache,
|
image_cache: &mut ImageCache,
|
||||||
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
|
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
|
||||||
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
|
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
|
||||||
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
|
|
||||||
) -> Result<bool, String> {
|
) -> Result<bool, String> {
|
||||||
if self.cancel_flag.load(Ordering::Relaxed) {
|
if self.cancel_flag.load(Ordering::Relaxed) {
|
||||||
self.image_state = None;
|
self.image_state = None;
|
||||||
|
|
@ -576,7 +575,6 @@ impl ExportOrchestrator {
|
||||||
output_view,
|
output_view,
|
||||||
floating_selection,
|
floating_selection,
|
||||||
state.settings.allow_transparency,
|
state.settings.allow_transparency,
|
||||||
raster_store,
|
|
||||||
)?;
|
)?;
|
||||||
queue.submit(Some(encoder.finish()));
|
queue.submit(Some(encoder.finish()));
|
||||||
|
|
||||||
|
|
@ -1031,7 +1029,6 @@ impl ExportOrchestrator {
|
||||||
renderer: &mut vello::Renderer,
|
renderer: &mut vello::Renderer,
|
||||||
image_cache: &mut ImageCache,
|
image_cache: &mut ImageCache,
|
||||||
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
|
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
|
||||||
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
|
|
||||||
) -> Result<bool, String> {
|
) -> Result<bool, String> {
|
||||||
use std::time::Instant;
|
use std::time::Instant;
|
||||||
|
|
||||||
|
|
@ -1129,7 +1126,6 @@ impl ExportOrchestrator {
|
||||||
gpu_resources, &acquired.rgba_texture_view,
|
gpu_resources, &acquired.rgba_texture_view,
|
||||||
None, // No floating selection during video export
|
None, // No floating selection during video export
|
||||||
false, // Video export is never transparent
|
false, // Video export is never transparent
|
||||||
raster_store,
|
|
||||||
)?;
|
)?;
|
||||||
let render_end = Instant::now();
|
let render_end = Instant::now();
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1179,39 +1179,6 @@ pub fn render_frame_to_rgba_hdr(
|
||||||
///
|
///
|
||||||
/// # Returns
|
/// # Returns
|
||||||
/// Command encoder ready for submission (caller submits via ReadbackPipeline)
|
/// Command encoder ready for submission (caller submits via ReadbackPipeline)
|
||||||
/// Fault in raster keyframe pixels needed to composite the document at its current
|
|
||||||
/// time, decoding them from the project `.beam` container via `raster_store`.
|
|
||||||
///
|
|
||||||
/// Mutates the document in place: for every raster layer's active keyframe whose
|
|
||||||
/// `raw_pixels` are empty, loads + sets them (and marks `texture_dirty`). A no-op
|
|
||||||
/// when `raster_store` is `None`/unsaved or everything is already resident.
|
|
||||||
fn fault_in_raster_for_frame(
|
|
||||||
document: &mut Document,
|
|
||||||
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
|
|
||||||
) {
|
|
||||||
let store = match raster_store {
|
|
||||||
Some(s) if s.has_path() => s,
|
|
||||||
_ => return,
|
|
||||||
};
|
|
||||||
let now = document.current_time;
|
|
||||||
for layer in document.all_layers_mut() {
|
|
||||||
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
|
|
||||||
// Resolve the active keyframe id at the current time, then fault it in.
|
|
||||||
let kf_id = match rl.keyframe_at(now) {
|
|
||||||
Some(kf) if kf.raw_pixels.is_empty() && kf.needs_fault_in => kf.id,
|
|
||||||
_ => continue,
|
|
||||||
};
|
|
||||||
if let Some(kf) = rl.keyframes.iter_mut().find(|kf| kf.id == kf_id) {
|
|
||||||
if let Some(pixels) = store.load_pixels(kf_id) {
|
|
||||||
kf.raw_pixels = pixels;
|
|
||||||
kf.texture_dirty = true;
|
|
||||||
}
|
|
||||||
kf.needs_fault_in = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn render_frame_to_gpu_rgba(
|
pub fn render_frame_to_gpu_rgba(
|
||||||
document: &mut Document,
|
document: &mut Document,
|
||||||
timestamp: f64,
|
timestamp: f64,
|
||||||
|
|
@ -1226,19 +1193,12 @@ pub fn render_frame_to_gpu_rgba(
|
||||||
rgba_texture_view: &wgpu::TextureView,
|
rgba_texture_view: &wgpu::TextureView,
|
||||||
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
|
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
|
||||||
allow_transparency: bool,
|
allow_transparency: bool,
|
||||||
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
|
|
||||||
) -> Result<wgpu::CommandEncoder, String> {
|
) -> Result<wgpu::CommandEncoder, String> {
|
||||||
use vello::kurbo::Affine;
|
use vello::kurbo::Affine;
|
||||||
|
|
||||||
// Set document time to the frame timestamp
|
// Set document time to the frame timestamp
|
||||||
document.current_time = timestamp;
|
document.current_time = timestamp;
|
||||||
|
|
||||||
// Fault in raster keyframe pixels for this frame (Phase 3 paging). Offline
|
|
||||||
// export renders synchronously with no "next frame", so unlike the live canvas
|
|
||||||
// we must page the pixels in here, before compositing. Cheap no-op when every
|
|
||||||
// keyframe is already resident or when the document is unsaved (no store path).
|
|
||||||
fault_in_raster_for_frame(document, raster_store);
|
|
||||||
|
|
||||||
// Scale the document to the export resolution. The core renderer bakes this
|
// Scale the document to the export resolution. The core renderer bakes this
|
||||||
// base transform into every layer (vector scenes, raster and video layer
|
// base transform into every layer (vector scenes, raster and video layer
|
||||||
// transforms), so the whole stage scales up/down to fill the output. When the
|
// transforms), so the whole stage scales up/down to fill the output. When the
|
||||||
|
|
|
||||||
|
|
@ -1049,17 +1049,6 @@ pub struct GpuBrushEngine {
|
||||||
/// once when `RasterKeyframe::texture_dirty` is set, then reused every frame.
|
/// once when `RasterKeyframe::texture_dirty` is set, then reused every frame.
|
||||||
/// Separate from `canvases` so tool teardown never accidentally removes them.
|
/// Separate from `canvases` so tool teardown never accidentally removes them.
|
||||||
pub raster_layer_cache: HashMap<Uuid, CanvasPair>,
|
pub raster_layer_cache: HashMap<Uuid, CanvasPair>,
|
||||||
/// Recency order for `raster_layer_cache` (least-recent first), bumped on every
|
|
||||||
/// `ensure_layer_texture` so the visible frames stay at the back. Scrubbing a long
|
|
||||||
/// timeline would otherwise grow this map without bound (~w·h·16 bytes of VRAM per
|
|
||||||
/// entry); we evict the oldest once it exceeds `RASTER_LAYER_CACHE_MAX`.
|
|
||||||
raster_layer_lru: Vec<Uuid>,
|
|
||||||
/// Low-res proxy textures (one small `CanvasPair` per keyframe), shown while the
|
|
||||||
/// full-res pixels page in. Keyed by keyframe id; separate from
|
|
||||||
/// `raster_layer_cache` so a proxy and its full frame never collide. Bounded by
|
|
||||||
/// its own (generous, since each is tiny) recency LRU.
|
|
||||||
proxy_layer_cache: HashMap<Uuid, CanvasPair>,
|
|
||||||
proxy_layer_lru: Vec<Uuid>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// CPU-side parameters uniform for the compute shader.
|
/// CPU-side parameters uniform for the compute shader.
|
||||||
|
|
@ -1077,16 +1066,6 @@ struct DabParams {
|
||||||
}
|
}
|
||||||
|
|
||||||
impl GpuBrushEngine {
|
impl GpuBrushEngine {
|
||||||
/// Max number of idle raster-layer textures kept resident in VRAM. Scrubbing a
|
|
||||||
/// long timeline would otherwise cache one ~`w·h·16`-byte pair per visited frame
|
|
||||||
/// without bound; the least-recently-used are evicted past this (re-uploaded on
|
|
||||||
/// revisit from the faulted-in pixels).
|
|
||||||
const RASTER_LAYER_CACHE_MAX: usize = 12;
|
|
||||||
|
|
||||||
/// Proxies are ~1/100th the VRAM of a full frame, so we can keep many resident
|
|
||||||
/// for instant cold-scrub display before evicting the least-recently-used.
|
|
||||||
const RASTER_PROXY_CACHE_MAX: usize = 64;
|
|
||||||
|
|
||||||
/// Create the pipeline. Returns `Err` if the device lacks the required
|
/// Create the pipeline. Returns `Err` if the device lacks the required
|
||||||
/// storage-texture capability for `Rgba16Float`.
|
/// storage-texture capability for `Rgba16Float`.
|
||||||
pub fn new(device: &wgpu::Device) -> Self {
|
pub fn new(device: &wgpu::Device) -> Self {
|
||||||
|
|
@ -1181,9 +1160,6 @@ impl GpuBrushEngine {
|
||||||
canvases: HashMap::new(),
|
canvases: HashMap::new(),
|
||||||
displacement_bufs: HashMap::new(),
|
displacement_bufs: HashMap::new(),
|
||||||
raster_layer_cache: HashMap::new(),
|
raster_layer_cache: HashMap::new(),
|
||||||
raster_layer_lru: Vec::new(),
|
|
||||||
proxy_layer_cache: HashMap::new(),
|
|
||||||
proxy_layer_lru: Vec::new(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1557,33 +1533,6 @@ impl GpuBrushEngine {
|
||||||
}
|
}
|
||||||
self.raster_layer_cache.insert(kf_id, canvas);
|
self.raster_layer_cache.insert(kf_id, canvas);
|
||||||
}
|
}
|
||||||
// Bump recency (the frame was used this render) and evict the least-recently
|
|
||||||
// used textures over budget. The current frame is pushed to the back, so it
|
|
||||||
// (and every other frame rendered this pass) is never the eviction victim.
|
|
||||||
if let Some(pos) = self.raster_layer_lru.iter().position(|id| *id == kf_id) {
|
|
||||||
self.raster_layer_lru.remove(pos);
|
|
||||||
}
|
|
||||||
self.raster_layer_lru.push(kf_id);
|
|
||||||
let mut evicted = false;
|
|
||||||
while self.raster_layer_lru.len() > Self::RASTER_LAYER_CACHE_MAX {
|
|
||||||
let old = self.raster_layer_lru.remove(0);
|
|
||||||
self.raster_layer_cache.remove(&old);
|
|
||||||
evicted = true;
|
|
||||||
}
|
|
||||||
if needs_new || evicted {
|
|
||||||
self.report_raster_cache_vram();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Estimated VRAM footprint of `raster_layer_cache` (two `Rgba16Float` textures =
|
|
||||||
/// `w·h·16` bytes per entry), published to the F3 debug overlay.
|
|
||||||
fn report_raster_cache_vram(&self) {
|
|
||||||
let bytes = |cache: &HashMap<Uuid, CanvasPair>| -> usize {
|
|
||||||
cache.values().map(|c| (c.width as usize) * (c.height as usize) * 16).sum()
|
|
||||||
};
|
|
||||||
let total = bytes(&self.raster_layer_cache) + bytes(&self.proxy_layer_cache);
|
|
||||||
let count = self.raster_layer_cache.len() + self.proxy_layer_cache.len();
|
|
||||||
crate::debug_overlay::update_gpu_memory(count, total);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get the cached display texture for a raster layer keyframe.
|
/// Get the cached display texture for a raster layer keyframe.
|
||||||
|
|
@ -1591,55 +1540,9 @@ impl GpuBrushEngine {
|
||||||
self.raster_layer_cache.get(kf_id)
|
self.raster_layer_cache.get(kf_id)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Ensure a low-res proxy texture exists for `kf_id` (uploaded once; proxies are
|
|
||||||
/// immutable). Bumps recency and evicts the least-recently-used past the budget.
|
|
||||||
/// `pixels` is sRGB-premultiplied RGBA of length `w * h * 4`.
|
|
||||||
pub fn ensure_proxy_texture(
|
|
||||||
&mut self,
|
|
||||||
device: &wgpu::Device,
|
|
||||||
queue: &wgpu::Queue,
|
|
||||||
kf_id: Uuid,
|
|
||||||
pixels: &[u8],
|
|
||||||
w: u32,
|
|
||||||
h: u32,
|
|
||||||
) {
|
|
||||||
if pixels.len() != (w * h * 4) as usize {
|
|
||||||
return; // malformed proxy — skip rather than panic in the render loop
|
|
||||||
}
|
|
||||||
let mut changed = false;
|
|
||||||
if !self.proxy_layer_cache.contains_key(&kf_id) {
|
|
||||||
let canvas = CanvasPair::new(device, w, h);
|
|
||||||
canvas.upload(queue, pixels);
|
|
||||||
self.proxy_layer_cache.insert(kf_id, canvas);
|
|
||||||
changed = true;
|
|
||||||
}
|
|
||||||
if let Some(pos) = self.proxy_layer_lru.iter().position(|id| *id == kf_id) {
|
|
||||||
self.proxy_layer_lru.remove(pos);
|
|
||||||
}
|
|
||||||
self.proxy_layer_lru.push(kf_id);
|
|
||||||
while self.proxy_layer_lru.len() > Self::RASTER_PROXY_CACHE_MAX {
|
|
||||||
let old = self.proxy_layer_lru.remove(0);
|
|
||||||
self.proxy_layer_cache.remove(&old);
|
|
||||||
changed = true;
|
|
||||||
}
|
|
||||||
if changed {
|
|
||||||
self.report_raster_cache_vram();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Get the cached low-res proxy texture for a raster keyframe.
|
|
||||||
pub fn get_proxy_texture(&self, kf_id: &Uuid) -> Option<&CanvasPair> {
|
|
||||||
self.proxy_layer_cache.get(kf_id)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted).
|
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted).
|
||||||
pub fn remove_layer_texture(&mut self, kf_id: &Uuid) {
|
pub fn remove_layer_texture(&mut self, kf_id: &Uuid) {
|
||||||
if self.raster_layer_cache.remove(kf_id).is_some() {
|
self.raster_layer_cache.remove(kf_id);
|
||||||
if let Some(pos) = self.raster_layer_lru.iter().position(|id| id == kf_id) {
|
|
||||||
self.raster_layer_lru.remove(pos);
|
|
||||||
}
|
|
||||||
self.report_raster_cache_vram();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Composite the accumulated-dab scratch buffer C over the source A, writing the
|
/// Composite the accumulated-dab scratch buffer C over the source A, writing the
|
||||||
|
|
@ -1970,9 +1873,6 @@ pub struct CanvasBlitPipeline {
|
||||||
pub pipeline: wgpu::RenderPipeline,
|
pub pipeline: wgpu::RenderPipeline,
|
||||||
pub bg_layout: wgpu::BindGroupLayout,
|
pub bg_layout: wgpu::BindGroupLayout,
|
||||||
pub sampler: wgpu::Sampler,
|
pub sampler: wgpu::Sampler,
|
||||||
/// Bilinear sampler for smooth upscaling (used by `blit_smooth`, e.g. low-res
|
|
||||||
/// proxies). The default `sampler` stays nearest to keep the real canvas crisp.
|
|
||||||
pub linear_sampler: wgpu::Sampler,
|
|
||||||
/// Nearest-neighbour sampler used for the selection mask texture.
|
/// Nearest-neighbour sampler used for the selection mask texture.
|
||||||
pub mask_sampler: wgpu::Sampler,
|
pub mask_sampler: wgpu::Sampler,
|
||||||
}
|
}
|
||||||
|
|
@ -2148,17 +2048,6 @@ impl CanvasBlitPipeline {
|
||||||
..Default::default()
|
..Default::default()
|
||||||
});
|
});
|
||||||
|
|
||||||
let linear_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
|
|
||||||
label: Some("canvas_blit_linear_sampler"),
|
|
||||||
address_mode_u: wgpu::AddressMode::ClampToEdge,
|
|
||||||
address_mode_v: wgpu::AddressMode::ClampToEdge,
|
|
||||||
address_mode_w: wgpu::AddressMode::ClampToEdge,
|
|
||||||
mag_filter: wgpu::FilterMode::Linear,
|
|
||||||
min_filter: wgpu::FilterMode::Linear,
|
|
||||||
mipmap_filter: wgpu::FilterMode::Linear,
|
|
||||||
..Default::default()
|
|
||||||
});
|
|
||||||
|
|
||||||
let mask_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
|
let mask_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
|
||||||
label: Some("canvas_mask_sampler"),
|
label: Some("canvas_mask_sampler"),
|
||||||
address_mode_u: wgpu::AddressMode::ClampToEdge,
|
address_mode_u: wgpu::AddressMode::ClampToEdge,
|
||||||
|
|
@ -2170,7 +2059,7 @@ impl CanvasBlitPipeline {
|
||||||
..Default::default()
|
..Default::default()
|
||||||
});
|
});
|
||||||
|
|
||||||
Self { pipeline, bg_layout, sampler, linear_sampler, mask_sampler }
|
Self { pipeline, bg_layout, sampler, mask_sampler }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Render the canvas texture into `target_view` (Rgba16Float) with the given camera.
|
/// Render the canvas texture into `target_view` (Rgba16Float) with the given camera.
|
||||||
|
|
@ -2178,7 +2067,6 @@ impl CanvasBlitPipeline {
|
||||||
/// `target_view` is cleared to transparent before writing.
|
/// `target_view` is cleared to transparent before writing.
|
||||||
/// `mask_view` is an R8Unorm texture in canvas-pixel space: 255 = keep, 0 = discard.
|
/// `mask_view` is an R8Unorm texture in canvas-pixel space: 255 = keep, 0 = discard.
|
||||||
/// Pass `None` to use the built-in 1×1 all-white default (no masking).
|
/// Pass `None` to use the built-in 1×1 all-white default (no masking).
|
||||||
/// Blit with the default nearest-neighbour sampler (crisp; for real canvas pixels).
|
|
||||||
pub fn blit(
|
pub fn blit(
|
||||||
&self,
|
&self,
|
||||||
device: &wgpu::Device,
|
device: &wgpu::Device,
|
||||||
|
|
@ -2187,32 +2075,6 @@ impl CanvasBlitPipeline {
|
||||||
target_view: &wgpu::TextureView,
|
target_view: &wgpu::TextureView,
|
||||||
transform: &BlitTransform,
|
transform: &BlitTransform,
|
||||||
mask_view: Option<&wgpu::TextureView>,
|
mask_view: Option<&wgpu::TextureView>,
|
||||||
) {
|
|
||||||
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.sampler);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Blit with a bilinear sampler — smooth upscaling for low-res sources (proxies).
|
|
||||||
pub fn blit_smooth(
|
|
||||||
&self,
|
|
||||||
device: &wgpu::Device,
|
|
||||||
queue: &wgpu::Queue,
|
|
||||||
canvas_view: &wgpu::TextureView,
|
|
||||||
target_view: &wgpu::TextureView,
|
|
||||||
transform: &BlitTransform,
|
|
||||||
mask_view: Option<&wgpu::TextureView>,
|
|
||||||
) {
|
|
||||||
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler);
|
|
||||||
}
|
|
||||||
|
|
||||||
fn blit_with(
|
|
||||||
&self,
|
|
||||||
device: &wgpu::Device,
|
|
||||||
queue: &wgpu::Queue,
|
|
||||||
canvas_view: &wgpu::TextureView,
|
|
||||||
target_view: &wgpu::TextureView,
|
|
||||||
transform: &BlitTransform,
|
|
||||||
mask_view: Option<&wgpu::TextureView>,
|
|
||||||
canvas_sampler: &wgpu::Sampler,
|
|
||||||
) {
|
) {
|
||||||
// When no mask is provided, create a temporary 1×1 all-white texture.
|
// When no mask is provided, create a temporary 1×1 all-white texture.
|
||||||
// (queue is already available here, unlike in new())
|
// (queue is already available here, unlike in new())
|
||||||
|
|
@ -2265,7 +2127,7 @@ impl CanvasBlitPipeline {
|
||||||
},
|
},
|
||||||
wgpu::BindGroupEntry {
|
wgpu::BindGroupEntry {
|
||||||
binding: 1,
|
binding: 1,
|
||||||
resource: wgpu::BindingResource::Sampler(canvas_sampler),
|
resource: wgpu::BindingResource::Sampler(&self.sampler),
|
||||||
},
|
},
|
||||||
wgpu::BindGroupEntry {
|
wgpu::BindGroupEntry {
|
||||||
binding: 2,
|
binding: 2,
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -242,16 +242,6 @@ pub struct SharedPaneState<'a> {
|
||||||
pub raw_audio_cache: &'a std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
|
pub raw_audio_cache: &'a std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
|
||||||
/// Pool indices needing GPU waveform texture upload
|
/// Pool indices needing GPU waveform texture upload
|
||||||
pub waveform_gpu_dirty: &'a mut std::collections::HashSet<usize>,
|
pub waveform_gpu_dirty: &'a mut std::collections::HashSet<usize>,
|
||||||
/// Pools whose `raw_audio_cache` entry is a packed min/max floor rather than
|
|
||||||
/// raw samples (pool_index -> `B`, floor frames-per-texel). Drives the GPU
|
|
||||||
/// min/max upload path and the floor's effective rate `sr/B` in the renderer.
|
|
||||||
pub waveform_minmax_pools: &'a std::collections::HashMap<usize, u32>,
|
|
||||||
/// Miss-sink for on-demand raster keyframe pixel faulting (Phase 3 paging).
|
|
||||||
/// The canvas inserts the id of any raster keyframe it wants to upload whose
|
|
||||||
/// `raw_pixels` aren't resident; the App drains this at the top of the next
|
|
||||||
/// `update()` and faults the pixels in from the project container.
|
|
||||||
pub raster_fault_requests:
|
|
||||||
&'a std::sync::Arc<std::sync::Mutex<std::collections::HashSet<uuid::Uuid>>>,
|
|
||||||
/// Effect ID to load into shader editor (set by asset library, consumed by shader editor)
|
/// Effect ID to load into shader editor (set by asset library, consumed by shader editor)
|
||||||
pub effect_to_load: &'a mut Option<Uuid>,
|
pub effect_to_load: &'a mut Option<Uuid>,
|
||||||
/// Queue for effect thumbnail requests (effect IDs to generate thumbnails for)
|
/// Queue for effect thumbnail requests (effect IDs to generate thumbnails for)
|
||||||
|
|
|
||||||
|
|
@ -77,29 +77,27 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
|
||||||
let frames_per_pixel = params.sample_rate / params.pixels_per_second;
|
let frames_per_pixel = params.sample_rate / params.pixels_per_second;
|
||||||
// Each mip level reduces by 4x in sample count (2x in each texture dimension)
|
// Each mip level reduces by 4x in sample count (2x in each texture dimension)
|
||||||
let mip_f = max(0.0, log2(frames_per_pixel) / 2.0);
|
let mip_f = max(0.0, log2(frames_per_pixel) / 2.0);
|
||||||
|
let max_mip = f32(textureNumLevels(peak_tex) - 1u);
|
||||||
|
let mip = min(mip_f, max_mip);
|
||||||
|
|
||||||
// Pick the NEAREST INTEGER LOD and read its exact texel. Sampling at a
|
// Frame index at the chosen mip level
|
||||||
// fractional mip (trilinear) blends level N and N+1, but each level has its
|
let mip_floor = u32(mip);
|
||||||
// own 1D→2D row-major linearization (width halves per level), so the two
|
let reduction = pow(4.0, f32(mip_floor));
|
||||||
// levels disagree on which audio frame a given screen column maps to. The
|
|
||||||
// blend then reads horizontally-offset neighbours, and because a 2x zoom step
|
|
||||||
// shifts mip_f by exactly 0.5, alternate zoom levels land on a clean integer
|
|
||||||
// (correct) vs a 50/50 blend (offset) — the "every other zoom level" artifact.
|
|
||||||
// textureLoad at one integer level keeps the frame→texel mapping exact.
|
|
||||||
let max_mip = i32(textureNumLevels(peak_tex)) - 1;
|
|
||||||
let mip_i = clamp(i32(mip_f + 0.5), 0, max_mip);
|
|
||||||
let reduction = pow(4.0, f32(mip_i));
|
|
||||||
let mip_frame = frame_f / reduction;
|
let mip_frame = frame_f / reduction;
|
||||||
|
|
||||||
// Convert 1D mip-space index to 2D texel coords using this level's actual
|
// Convert 1D mip-space index to 2D UV coordinates
|
||||||
// dimensions (texture may be pre-allocated larger, e.g. for live recording).
|
// Use actual texture dimensions (not computed from total_frames) because the
|
||||||
let mip_dims = textureDimensions(peak_tex, mip_i);
|
// texture may be pre-allocated larger for live recording.
|
||||||
|
let mip_dims = textureDimensions(peak_tex, mip_floor);
|
||||||
let mip_tex_width = f32(mip_dims.x);
|
let mip_tex_width = f32(mip_dims.x);
|
||||||
let texel_x = i32(mip_frame % mip_tex_width);
|
let mip_tex_height = f32(mip_dims.y);
|
||||||
let texel_y = i32(floor(mip_frame / mip_tex_width));
|
let texel_x = mip_frame % mip_tex_width;
|
||||||
|
let texel_y = floor(mip_frame / mip_tex_width);
|
||||||
|
let uv = vec2((texel_x + 0.5) / mip_tex_width, (texel_y + 0.5) / mip_tex_height);
|
||||||
|
|
||||||
|
// Sample the peak texture at computed mip level
|
||||||
// R = left_min, G = left_max, B = right_min, A = right_max
|
// R = left_min, G = left_max, B = right_min, A = right_max
|
||||||
let peak = textureLoad(peak_tex, vec2<i32>(texel_x, texel_y), mip_i);
|
let peak = textureSampleLevel(peak_tex, peak_sampler, uv, mip);
|
||||||
|
|
||||||
let clip_height = params.clip_rect.w - params.clip_rect.y;
|
let clip_height = params.clip_rect.w - params.clip_rect.y;
|
||||||
let clip_top = params.clip_rect.y;
|
let clip_top = params.clip_rect.y;
|
||||||
|
|
|
||||||
|
|
@ -523,12 +523,6 @@ struct VelloRenderContext {
|
||||||
/// When `Some`, readback this B-canvas into `RASTER_READBACK_RESULTS` after
|
/// When `Some`, readback this B-canvas into `RASTER_READBACK_RESULTS` after
|
||||||
/// dispatching GPU tool work. Set on mouseup by the unified raster tool commit path.
|
/// dispatching GPU tool work. Set on mouseup by the unified raster tool commit path.
|
||||||
pending_tool_readback_b: Option<uuid::Uuid>,
|
pending_tool_readback_b: Option<uuid::Uuid>,
|
||||||
/// Miss-sink for on-demand raster keyframe pixel faulting (Phase 3 paging).
|
|
||||||
/// When the compositor wants to upload an idle raster keyframe whose `raw_pixels`
|
|
||||||
/// aren't resident, it inserts the keyframe id here; the App drains this at the
|
|
||||||
/// top of the next `update()` and faults the pixels in from the project container.
|
|
||||||
raster_fault_requests:
|
|
||||||
std::sync::Arc<std::sync::Mutex<std::collections::HashSet<uuid::Uuid>>>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Callback for Vello rendering within egui
|
/// Callback for Vello rendering within egui
|
||||||
|
|
@ -1175,9 +1169,6 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
|
||||||
|
|
||||||
// 4. Raster layer texture cache: for idle raster layers (no active tool canvas).
|
// 4. Raster layer texture cache: for idle raster layers (no active tool canvas).
|
||||||
// Upload raw_pixels to the cache if texture_dirty; then use the cache entry.
|
// Upload raw_pixels to the cache if texture_dirty; then use the cache entry.
|
||||||
// If the full pixels aren't resident, fall back to the low-res proxy:
|
|
||||||
// (kf_id, logical_w, logical_h) so the blit upscales it to full size.
|
|
||||||
let mut raster_proxy_blit: Option<(uuid::Uuid, u32, u32)> = None;
|
|
||||||
let raster_cache_kf: Option<uuid::Uuid> = if gpu_canvas_kf.is_none() {
|
let raster_cache_kf: Option<uuid::Uuid> = if gpu_canvas_kf.is_none() {
|
||||||
// Find the active keyframe for this raster layer.
|
// Find the active keyframe for this raster layer.
|
||||||
let doc = &self.ctx.document;
|
let doc = &self.ctx.document;
|
||||||
|
|
@ -1215,24 +1206,6 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
|
||||||
);
|
);
|
||||||
Some(kf_id)
|
Some(kf_id)
|
||||||
} else {
|
} else {
|
||||||
// Empty pixels: if the frame is paged out (lives in the
|
|
||||||
// container), record a fault-in request so the App pages it
|
|
||||||
// in at the top of the next frame. A new blank keyframe
|
|
||||||
// (needs_fault_in == false) has nothing to load — skip it.
|
|
||||||
if kf.needs_fault_in {
|
|
||||||
if let Ok(mut reqs) = self.ctx.raster_fault_requests.lock() {
|
|
||||||
reqs.insert(kf_id);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Show the low-res proxy (if decoded) while the full
|
|
||||||
// pages in, so the cold scrub doesn't flash blank.
|
|
||||||
if let Some(proxy) = &kf.proxy {
|
|
||||||
gpu_brush.ensure_proxy_texture(
|
|
||||||
device, queue, kf_id,
|
|
||||||
&proxy.pixels, proxy.width, proxy.height,
|
|
||||||
);
|
|
||||||
raster_proxy_blit = Some((kf_id, kf.width, kf.height));
|
|
||||||
}
|
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
|
@ -1248,9 +1221,7 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
|
||||||
None
|
None
|
||||||
};
|
};
|
||||||
|
|
||||||
if !rendered_layer.has_content && gpu_canvas_kf.is_none() && raster_cache_kf.is_none()
|
if !rendered_layer.has_content && gpu_canvas_kf.is_none() && raster_cache_kf.is_none() {
|
||||||
&& raster_proxy_blit.is_none()
|
|
||||||
{
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1297,47 +1268,27 @@ impl egui_wgpu::CallbackTrait for VelloCallback {
|
||||||
}
|
}
|
||||||
RenderedLayerType::Raster { transform: layer_transform, .. } => {
|
RenderedLayerType::Raster { transform: layer_transform, .. } => {
|
||||||
// Raster layer — GPU canvas blit directly to HDR (bypasses Vello).
|
// Raster layer — GPU canvas blit directly to HDR (bypasses Vello).
|
||||||
// Tool override canvas (gpu_canvas_kf) takes priority over cached
|
// Tool override canvas (gpu_canvas_kf) takes priority over cached texture.
|
||||||
// texture; if neither full-res source is present, the low-res proxy.
|
if let Some(use_kf_id) = gpu_canvas_kf.or(raster_cache_kf) {
|
||||||
let full_kf = gpu_canvas_kf.or(raster_cache_kf);
|
|
||||||
if full_kf.is_some() || raster_proxy_blit.is_some() {
|
|
||||||
let hdr_layer_handle = buffer_pool.acquire(device, hdr_spec);
|
let hdr_layer_handle = buffer_pool.acquire(device, hdr_spec);
|
||||||
if let (Some(hdr_layer_view), Some(hdr_view)) = (
|
if let (Some(hdr_layer_view), Some(hdr_view)) = (
|
||||||
buffer_pool.get_view(hdr_layer_handle),
|
buffer_pool.get_view(hdr_layer_handle),
|
||||||
&instance_resources.hdr_texture_view,
|
&instance_resources.hdr_texture_view,
|
||||||
) {
|
) {
|
||||||
if let Ok(gpu_brush) = shared.gpu_brush.lock() {
|
if let Ok(gpu_brush) = shared.gpu_brush.lock() {
|
||||||
// Pick the source texture and the LOGICAL dims the blit
|
let canvas = gpu_brush.canvases.get(&use_kf_id)
|
||||||
// should map it to. A full texture uses its own dims; a
|
.or_else(|| gpu_brush.raster_layer_cache.get(&use_kf_id));
|
||||||
// proxy uses the keyframe's full dims so it upscales.
|
if let Some(canvas) = canvas {
|
||||||
let blit = if let Some(use_kf_id) = full_kf {
|
|
||||||
gpu_brush.canvases.get(&use_kf_id)
|
|
||||||
.or_else(|| gpu_brush.raster_layer_cache.get(&use_kf_id))
|
|
||||||
.map(|c| (c, c.width, c.height, false))
|
|
||||||
} else if let Some((pkf, lw, lh)) = raster_proxy_blit {
|
|
||||||
gpu_brush.get_proxy_texture(&pkf).map(|c| (c, lw, lh, true))
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
if let Some((canvas, logical_w, logical_h, is_proxy)) = blit {
|
|
||||||
let bt = crate::gpu_brush::BlitTransform::new(
|
let bt = crate::gpu_brush::BlitTransform::new(
|
||||||
*layer_transform,
|
*layer_transform,
|
||||||
logical_w, logical_h,
|
canvas.width, canvas.height,
|
||||||
width, height,
|
width, height,
|
||||||
);
|
);
|
||||||
// Proxies are upscaled, so sample them bilinearly;
|
|
||||||
// the real canvas stays nearest (crisp pixels).
|
|
||||||
if is_proxy {
|
|
||||||
shared.canvas_blit.blit_smooth(
|
|
||||||
device, queue, canvas.src_view(), hdr_layer_view, &bt, None,
|
|
||||||
);
|
|
||||||
} else {
|
|
||||||
shared.canvas_blit.blit(
|
shared.canvas_blit.blit(
|
||||||
device, queue, canvas.src_view(), hdr_layer_view, &bt, None,
|
device, queue, canvas.src_view(), hdr_layer_view, &bt, None,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
let compositor_layer = lightningbeam_core::gpu::CompositorLayer::new(
|
let compositor_layer = lightningbeam_core::gpu::CompositorLayer::new(
|
||||||
hdr_layer_handle,
|
hdr_layer_handle,
|
||||||
rendered_layer.opacity,
|
rendered_layer.opacity,
|
||||||
|
|
@ -5624,8 +5575,15 @@ impl StagePane {
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
|
|
||||||
// Don't create a keyframe — explicit "New Keyframe" only. The read below
|
// Ensure the keyframe exists before reading its ID.
|
||||||
// returns None (no workspace) if there's no active keyframe to lift from.
|
{
|
||||||
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
|
||||||
|
rl.ensure_keyframe_at(time, doc_w, doc_h);
|
||||||
|
} else {
|
||||||
|
return None; // not a raster layer
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Read keyframe id and pixels.
|
// Read keyframe id and pixels.
|
||||||
let (kf_id, w, h, pixels) = {
|
let (kf_id, w, h, pixels) = {
|
||||||
|
|
@ -5848,9 +5806,6 @@ impl StagePane {
|
||||||
kf.raw_pixels[si..si + 4].fill(0);
|
kf.raw_pixels[si..si + 4].fill(0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Punching the hole edits raw_pixels but is NOT committed through an
|
|
||||||
// action yet — mark dirty so eviction can't drop the un-persisted hole.
|
|
||||||
kf.dirty = true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-set selection (commit_raster_floating_now cleared it) and create float.
|
// Re-set selection (commit_raster_floating_now cleared it) and create float.
|
||||||
|
|
@ -6059,12 +6014,21 @@ impl StagePane {
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
|
|
||||||
// Paint into the ACTIVE keyframe (the one at-or-before the playhead) —
|
// Ensure the keyframe exists BEFORE reading its ID, so we always get
|
||||||
// do NOT create one. Keyframes are made explicitly via "New Keyframe"
|
// the real UUID. Previously we read the ID first and fell back to a
|
||||||
// (a new layer already seeds one). If none exists at-or-before the
|
// randomly-generated UUID when no keyframe existed; that fake UUID was
|
||||||
// playhead, there's nothing to paint into; bail.
|
// stored in painting_canvas but subsequent drag frames used the real UUID
|
||||||
let _ = (doc_width, doc_height);
|
// from keyframe_at(), causing the GPU canvas to be a different object from
|
||||||
let (keyframe_id, kf_time, canvas_width, canvas_height, buffer_before, initial_pixels) = {
|
// the one being composited.
|
||||||
|
{
|
||||||
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&active_layer_id) {
|
||||||
|
rl.ensure_keyframe_at(*shared.playback_time, doc_width, doc_height);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Now read the guaranteed-to-exist keyframe to get the real UUID.
|
||||||
|
let (keyframe_id, canvas_width, canvas_height, buffer_before, initial_pixels) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
if let Some(AnyLayer::Raster(rl)) = doc.get_layer(&active_layer_id) {
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer(&active_layer_id) {
|
||||||
if let Some(kf) = rl.keyframe_at(*shared.playback_time) {
|
if let Some(kf) = rl.keyframe_at(*shared.playback_time) {
|
||||||
|
|
@ -6074,9 +6038,9 @@ impl StagePane {
|
||||||
} else {
|
} else {
|
||||||
raw.clone()
|
raw.clone()
|
||||||
};
|
};
|
||||||
(kf.id, kf.time, kf.width, kf.height, raw, init)
|
(kf.id, kf.width, kf.height, raw, init)
|
||||||
} else {
|
} else {
|
||||||
return; // no keyframe at/before the playhead — nothing to paint
|
return; // shouldn't happen after ensure_keyframe_at
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
return;
|
return;
|
||||||
|
|
@ -6108,7 +6072,7 @@ impl StagePane {
|
||||||
self.painting_canvas = Some((active_layer_id, keyframe_id));
|
self.painting_canvas = Some((active_layer_id, keyframe_id));
|
||||||
self.pending_undo_before = Some((
|
self.pending_undo_before = Some((
|
||||||
active_layer_id,
|
active_layer_id,
|
||||||
kf_time,
|
*shared.playback_time,
|
||||||
canvas_width,
|
canvas_width,
|
||||||
canvas_height,
|
canvas_height,
|
||||||
buffer_before,
|
buffer_before,
|
||||||
|
|
@ -6116,7 +6080,7 @@ impl StagePane {
|
||||||
self.pending_raster_dabs = Some(PendingRasterDabs {
|
self.pending_raster_dabs = Some(PendingRasterDabs {
|
||||||
keyframe_id,
|
keyframe_id,
|
||||||
layer_id: active_layer_id,
|
layer_id: active_layer_id,
|
||||||
time: kf_time,
|
time: *shared.playback_time,
|
||||||
canvas_width,
|
canvas_width,
|
||||||
canvas_height,
|
canvas_height,
|
||||||
initial_pixels: Some(initial_pixels),
|
initial_pixels: Some(initial_pixels),
|
||||||
|
|
@ -6126,7 +6090,7 @@ impl StagePane {
|
||||||
});
|
});
|
||||||
self.raster_stroke_state = Some((
|
self.raster_stroke_state = Some((
|
||||||
active_layer_id,
|
active_layer_id,
|
||||||
kf_time,
|
*shared.playback_time,
|
||||||
stroke_state,
|
stroke_state,
|
||||||
Vec::new(), // buffer_before now lives in pending_undo_before
|
Vec::new(), // buffer_before now lives in pending_undo_before
|
||||||
));
|
));
|
||||||
|
|
@ -6352,13 +6316,17 @@ impl StagePane {
|
||||||
|
|
||||||
let time = *shared.playback_time;
|
let time = *shared.playback_time;
|
||||||
// Canvas dimensions (to create keyframe if needed).
|
// Canvas dimensions (to create keyframe if needed).
|
||||||
let (_doc_w, _doc_h) = {
|
let (doc_w, doc_h) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
// Ensure a keyframe exists at the current time.
|
// Ensure a keyframe exists at the current time.
|
||||||
// Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
|
{
|
||||||
// The snapshot below edits the active keyframe and bails if none exists.
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
|
||||||
|
rl.ensure_keyframe_at(time, doc_w, doc_h);
|
||||||
|
}
|
||||||
|
}
|
||||||
// Snapshot the pixel buffer before drawing.
|
// Snapshot the pixel buffer before drawing.
|
||||||
let (buffer_before, w, h) = {
|
let (buffer_before, w, h) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
|
|
@ -6701,12 +6669,16 @@ impl StagePane {
|
||||||
let time = *shared.playback_time;
|
let time = *shared.playback_time;
|
||||||
|
|
||||||
// Ensure a keyframe exists at the current time.
|
// Ensure a keyframe exists at the current time.
|
||||||
let (_doc_w, _doc_h) = {
|
let (doc_w, doc_h) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
// Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
|
{
|
||||||
// The snapshot below edits the active keyframe and bails if none exists.
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
|
||||||
|
rl.ensure_keyframe_at(time, doc_w, doc_h);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Snapshot current pixels.
|
// Snapshot current pixels.
|
||||||
let (buffer_before, width, height) = {
|
let (buffer_before, width, height) = {
|
||||||
|
|
@ -6778,12 +6750,16 @@ impl StagePane {
|
||||||
let time = *shared.playback_time;
|
let time = *shared.playback_time;
|
||||||
|
|
||||||
// Ensure keyframe exists.
|
// Ensure keyframe exists.
|
||||||
let (_doc_w, _doc_h) = {
|
let (doc_w, doc_h) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
// Don't create a keyframe — keyframes are made explicitly via "New Keyframe".
|
{
|
||||||
// The snapshot below edits the active keyframe and bails if none exists.
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
|
||||||
|
rl.ensure_keyframe_at(time, doc_w, doc_h);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let (pixels, width, height) = {
|
let (pixels, width, height) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
|
|
@ -6860,11 +6836,16 @@ impl StagePane {
|
||||||
Self::commit_raster_floating_now(shared);
|
Self::commit_raster_floating_now(shared);
|
||||||
|
|
||||||
// Ensure the keyframe exists.
|
// Ensure the keyframe exists.
|
||||||
let (_doc_w, _doc_h) = {
|
let (doc_w, doc_h) = {
|
||||||
let doc = shared.action_executor.document();
|
let doc = shared.action_executor.document();
|
||||||
(doc.width as u32, doc.height as u32)
|
(doc.width as u32, doc.height as u32)
|
||||||
};
|
};
|
||||||
// Don't create a keyframe — explicit "New Keyframe" only; bail below if none.
|
{
|
||||||
|
let doc = shared.action_executor.document_mut();
|
||||||
|
if let Some(AnyLayer::Raster(rl)) = doc.get_layer_mut(&layer_id) {
|
||||||
|
rl.ensure_keyframe_at(time, doc_w, doc_h);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Snapshot canvas pixels.
|
// Snapshot canvas pixels.
|
||||||
let (pixels, width, height) = {
|
let (pixels, width, height) = {
|
||||||
|
|
@ -11921,7 +11902,6 @@ impl PaneRenderer for StagePane {
|
||||||
.and_then(|(tool, _)| tool.take_pending_gpu_work()),
|
.and_then(|(tool, _)| tool.take_pending_gpu_work()),
|
||||||
pending_layer_cache_removals: std::mem::take(&mut self.pending_layer_cache_removals),
|
pending_layer_cache_removals: std::mem::take(&mut self.pending_layer_cache_removals),
|
||||||
pending_tool_readback_b: self.pending_tool_readback_b.take(),
|
pending_tool_readback_b: self.pending_tool_readback_b.take(),
|
||||||
raster_fault_requests: std::sync::Arc::clone(shared.raster_fault_requests),
|
|
||||||
}};
|
}};
|
||||||
|
|
||||||
let cb = egui_wgpu::Callback::new_paint_callback(
|
let cb = egui_wgpu::Callback::new_paint_callback(
|
||||||
|
|
|
||||||
|
|
@ -99,94 +99,6 @@ fn clip_instance_y_bounds(row: usize, total_rows: usize) -> (f32, f32) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Draw the strip of video thumbnails across a clip.
|
|
||||||
///
|
|
||||||
/// Thumbnails are tiled from the clip's **true** (unclamped) screen origin
|
|
||||||
/// `clip_origin_x` over its full width `clip_true_width_px`, and only the tiles
|
|
||||||
/// intersecting `visible_rect` are drawn. This makes the strip scroll naturally
|
|
||||||
/// with the timeline and show the correct content when the clip is partly off the
|
|
||||||
/// left edge (a tile's content time is derived from its offset from the true
|
|
||||||
/// origin, not from the clamped visible edge).
|
|
||||||
///
|
|
||||||
/// Each tile's GPU texture is cached by the **actual** thumbnail timestamp (from
|
|
||||||
/// [`VideoManager::get_thumbnail_at`]), so as closer thumbnails finish generating
|
|
||||||
/// the strip refreshes instead of freezing on whatever loaded first.
|
|
||||||
#[allow(clippy::too_many_arguments)]
|
|
||||||
fn draw_video_thumbnail_strip(
|
|
||||||
painter: &egui::Painter,
|
|
||||||
ui: &egui::Ui,
|
|
||||||
video_mgr: &lightningbeam_core::video::VideoManager,
|
|
||||||
textures: &mut std::collections::HashMap<(uuid::Uuid, i64), egui::TextureHandle>,
|
|
||||||
clip_id: uuid::Uuid,
|
|
||||||
trim_start: f64,
|
|
||||||
clip_origin_x: f32,
|
|
||||||
clip_true_width_px: f32,
|
|
||||||
visible_rect: egui::Rect,
|
|
||||||
thumb_top_y: f32,
|
|
||||||
thumb_height: f32,
|
|
||||||
pixels_per_second: f64,
|
|
||||||
) {
|
|
||||||
if clip_true_width_px <= 0.0 || thumb_height <= 0.0 {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
// Aspect ratio from any available thumbnail; sets the per-tile width.
|
|
||||||
let aspect = match video_mgr.get_thumbnail_at(&clip_id, 0.0) {
|
|
||||||
Some((_, tw, th, _)) if th > 0 => tw as f32 / th as f32,
|
|
||||||
_ => return,
|
|
||||||
};
|
|
||||||
let step = (thumb_height * aspect).max(1.0);
|
|
||||||
let num_thumbs = ((clip_true_width_px / step).ceil() as usize).max(1);
|
|
||||||
|
|
||||||
// Iterate only the tiles that intersect the visible rect.
|
|
||||||
let first = (((visible_rect.min.x - clip_origin_x) / step).floor() as i64).max(0) as usize;
|
|
||||||
let last = (((visible_rect.max.x - clip_origin_x) / step).ceil() as i64).max(0) as usize;
|
|
||||||
let last = last.min(num_thumbs);
|
|
||||||
|
|
||||||
for i in first..last {
|
|
||||||
let thumb_x = clip_origin_x + i as f32 * step;
|
|
||||||
// Content time at the tile centre, measured from the TRUE clip origin so
|
|
||||||
// it's stable as the clip scrolls.
|
|
||||||
let content_time =
|
|
||||||
trim_start + (i as f64 * step as f64 + step as f64 * 0.5) / pixels_per_second;
|
|
||||||
|
|
||||||
let Some((actual_ts, tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&clip_id, content_time)
|
|
||||||
else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
let ts_key = (actual_ts * 1000.0) as i64;
|
|
||||||
let texture = textures.entry((clip_id, ts_key)).or_insert_with(|| {
|
|
||||||
let image = egui::ColorImage::from_rgba_unmultiplied([tw as usize, th as usize], &rgba_data);
|
|
||||||
ui.ctx().load_texture(
|
|
||||||
format!("vthumb_{}_{}", clip_id, ts_key),
|
|
||||||
image,
|
|
||||||
egui::TextureOptions::LINEAR,
|
|
||||||
)
|
|
||||||
});
|
|
||||||
|
|
||||||
let full_rect = egui::Rect::from_min_size(
|
|
||||||
egui::pos2(thumb_x, thumb_top_y),
|
|
||||||
egui::vec2(step, thumb_height),
|
|
||||||
);
|
|
||||||
let thumb_rect = full_rect.intersect(visible_rect);
|
|
||||||
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
|
|
||||||
let uv_min = egui::pos2(
|
|
||||||
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
|
|
||||||
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
|
|
||||||
);
|
|
||||||
let uv_max = egui::pos2(
|
|
||||||
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
|
|
||||||
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
|
|
||||||
);
|
|
||||||
painter.image(
|
|
||||||
texture.id(),
|
|
||||||
thumb_rect,
|
|
||||||
egui::Rect::from_min_max(uv_min, uv_max),
|
|
||||||
egui::Color32::WHITE,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Get the effective clip duration for a clip instance on a given layer.
|
/// Get the effective clip duration for a clip instance on a given layer.
|
||||||
/// For groups on vector layers, the duration spans all consecutive keyframes
|
/// For groups on vector layers, the duration spans all consecutive keyframes
|
||||||
/// where the group is present. For regular clips, returns the clip's internal duration.
|
/// where the group is present. For regular clips, returns the clip's internal duration.
|
||||||
|
|
@ -251,11 +163,6 @@ pub struct TimelinePane {
|
||||||
/// Vertical scroll offset (in pixels)
|
/// Vertical scroll offset (in pixels)
|
||||||
viewport_scroll_y: f32,
|
viewport_scroll_y: f32,
|
||||||
|
|
||||||
/// Clickable keyframe-diamond hit targets `(screen_rect, keyframe_time)` collected
|
|
||||||
/// during the last `render_layers`. Used by `handle_input` (next frame) to snap the
|
|
||||||
/// playhead exactly to a keyframe when its diamond is clicked.
|
|
||||||
keyframe_diamond_hits: Vec<(egui::Rect, f64)>,
|
|
||||||
|
|
||||||
/// Total duration of the animation
|
/// Total duration of the animation
|
||||||
duration: f64,
|
duration: f64,
|
||||||
|
|
||||||
|
|
@ -778,7 +685,6 @@ impl TimelinePane {
|
||||||
pixels_per_second: 100.0,
|
pixels_per_second: 100.0,
|
||||||
viewport_start_time: 0.0,
|
viewport_start_time: 0.0,
|
||||||
viewport_scroll_y: 0.0,
|
viewport_scroll_y: 0.0,
|
||||||
keyframe_diamond_hits: Vec::new(),
|
|
||||||
duration: 10.0, // Default 10 seconds
|
duration: 10.0, // Default 10 seconds
|
||||||
is_scrubbing: false,
|
is_scrubbing: false,
|
||||||
is_panning: false,
|
is_panning: false,
|
||||||
|
|
@ -2726,21 +2632,18 @@ impl TimelinePane {
|
||||||
midi_event_cache: &std::collections::HashMap<u32, Vec<daw_backend::audio::midi::MidiEvent>>,
|
midi_event_cache: &std::collections::HashMap<u32, Vec<daw_backend::audio::midi::MidiEvent>>,
|
||||||
raw_audio_cache: &std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
|
raw_audio_cache: &std::collections::HashMap<usize, (std::sync::Arc<Vec<f32>>, u32, u32)>,
|
||||||
waveform_gpu_dirty: &mut std::collections::HashSet<usize>,
|
waveform_gpu_dirty: &mut std::collections::HashSet<usize>,
|
||||||
waveform_minmax_pools: &std::collections::HashMap<usize, u32>,
|
|
||||||
target_format: wgpu::TextureFormat,
|
target_format: wgpu::TextureFormat,
|
||||||
waveform_stereo: bool,
|
waveform_stereo: bool,
|
||||||
context_layers: &[&lightningbeam_core::layer::AnyLayer],
|
context_layers: &[&lightningbeam_core::layer::AnyLayer],
|
||||||
video_manager: &std::sync::Arc<std::sync::Mutex<lightningbeam_core::video::VideoManager>>,
|
video_manager: &std::sync::Arc<std::sync::Mutex<lightningbeam_core::video::VideoManager>>,
|
||||||
audio_cache: &HashMap<uuid::Uuid, Vec<ClipInstance>>,
|
audio_cache: &HashMap<uuid::Uuid, Vec<ClipInstance>>,
|
||||||
playback_time: f64,
|
playback_time: f64,
|
||||||
) -> (Vec<(egui::Rect, uuid::Uuid, f64, f32)>, Vec<AutomationLaneRender>) {
|
) -> (Vec<(egui::Rect, uuid::Uuid, f64, f64)>, Vec<AutomationLaneRender>) {
|
||||||
let painter = ui.painter().clone();
|
let painter = ui.painter().clone();
|
||||||
let mut pending_lane_renders: Vec<AutomationLaneRender> = Vec::new();
|
let mut pending_lane_renders: Vec<AutomationLaneRender> = Vec::new();
|
||||||
// Rebuilt each frame; consumed by handle_input (next frame) for click-to-seek.
|
|
||||||
self.keyframe_diamond_hits.clear();
|
|
||||||
|
|
||||||
// Collect video clip rects for hover detection (to avoid borrow conflicts)
|
// Collect video clip rects for hover detection (to avoid borrow conflicts)
|
||||||
let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f32)> = Vec::new();
|
let mut video_clip_hovers: Vec<(egui::Rect, uuid::Uuid, f64, f64)> = Vec::new();
|
||||||
|
|
||||||
// Track visible video clip IDs for texture cache cleanup
|
// Track visible video clip IDs for texture cache cleanup
|
||||||
let mut visible_video_clip_ids: std::collections::HashSet<uuid::Uuid> = std::collections::HashSet::new();
|
let mut visible_video_clip_ids: std::collections::HashSet<uuid::Uuid> = std::collections::HashSet::new();
|
||||||
|
|
@ -2996,9 +2899,8 @@ impl TimelinePane {
|
||||||
theme.text_color(&["#timeline", ".group-bar"], ui.ctx(), egui::Color32::from_rgb(100, 220, 220))
|
theme.text_color(&["#timeline", ".group-bar"], ui.ctx(), egui::Color32::from_rgb(100, 220, 220))
|
||||||
};
|
};
|
||||||
for (s, e) in &merged {
|
for (s, e) in &merged {
|
||||||
// `merged` ranges are in beats; convert to seconds for time_to_x.
|
let sx = self.time_to_x(*s);
|
||||||
let sx = self.time_to_x(document.tempo_map().transform(*s));
|
let ex = self.time_to_x(*e).max(sx + MIN_CLIP_WIDTH_PX);
|
||||||
let ex = self.time_to_x(document.tempo_map().transform(*e)).max(sx + MIN_CLIP_WIDTH_PX);
|
|
||||||
if ex >= 0.0 && sx <= rect.width() {
|
if ex >= 0.0 && sx <= rect.width() {
|
||||||
let vsx = sx.max(0.0);
|
let vsx = sx.max(0.0);
|
||||||
let vex = ex.min(rect.width());
|
let vex = ex.min(rect.width());
|
||||||
|
|
@ -3038,9 +2940,8 @@ impl TimelinePane {
|
||||||
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
|
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
|
||||||
let ci_end = ci_start + ci_duration;
|
let ci_end = ci_start + ci_duration;
|
||||||
|
|
||||||
// ci_start/ci_end are in beats; convert to seconds for time_to_x.
|
let sx = self.time_to_x(ci_start);
|
||||||
let sx = self.time_to_x(document.tempo_map().transform(ci_start));
|
let ex = self.time_to_x(ci_end);
|
||||||
let ex = self.time_to_x(document.tempo_map().transform(ci_end));
|
|
||||||
if ex < 0.0 || sx > rect.width() { continue; }
|
if ex < 0.0 || sx > rect.width() { continue; }
|
||||||
|
|
||||||
let ci_rect = egui::Rect::from_min_max(
|
let ci_rect = egui::Rect::from_min_max(
|
||||||
|
|
@ -3055,29 +2956,69 @@ impl TimelinePane {
|
||||||
egui::pos2(ci_rect.min.x, span_y_min),
|
egui::pos2(ci_rect.min.x, span_y_min),
|
||||||
egui::pos2(ci_rect.max.x, span_y_max),
|
egui::pos2(ci_rect.max.x, span_y_max),
|
||||||
);
|
);
|
||||||
// 4th elem = clip's TRUE (unclamped) origin x, for correct
|
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, ci_start));
|
||||||
// hover content time when scrolled partly off the left.
|
|
||||||
video_clip_hovers.push((hover_rect, ci.clip_id, ci.trim_start, rect.min.x + sx));
|
|
||||||
|
|
||||||
let thumb_display_height = (thumb_y_max - span_y_min) - 4.0;
|
let thumb_display_height = (thumb_y_max - span_y_min) - 4.0;
|
||||||
if thumb_display_height > 8.0 {
|
if thumb_display_height > 8.0 {
|
||||||
let video_mgr = video_manager.lock().unwrap();
|
let video_mgr = video_manager.lock().unwrap();
|
||||||
// Tile from the clip's true origin (sx unclamped; ci_rect is
|
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&ci.clip_id, 0.0) {
|
||||||
// the clamped visible rect) — scrolls correctly off-screen.
|
let aspect = tw as f32 / th as f32;
|
||||||
draw_video_thumbnail_strip(
|
let thumb_display_width = thumb_display_height * aspect;
|
||||||
&painter,
|
let ci_width = ci_rect.width();
|
||||||
ui,
|
let num_thumbs = ((ci_width / thumb_display_width).ceil() as usize).max(1);
|
||||||
&video_mgr,
|
|
||||||
&mut self.video_thumbnail_textures,
|
for ti in 0..num_thumbs {
|
||||||
ci.clip_id,
|
let x_offset = ti as f32 * thumb_display_width;
|
||||||
ci.trim_start,
|
if x_offset >= ci_width { break; }
|
||||||
rect.min.x + sx,
|
|
||||||
ex - sx,
|
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5)
|
||||||
ci_rect,
|
/ self.pixels_per_second as f64;
|
||||||
ci_rect.min.y + 2.0,
|
let content_time = ci.trim_start + time_offset;
|
||||||
thumb_display_height,
|
|
||||||
self.pixels_per_second as f64,
|
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(&ci.clip_id, content_time) {
|
||||||
|
let ts_key = (content_time * 1000.0) as i64;
|
||||||
|
let cache_key = (ci.clip_id, ts_key);
|
||||||
|
|
||||||
|
let texture = self.video_thumbnail_textures
|
||||||
|
.entry(cache_key)
|
||||||
|
.or_insert_with(|| {
|
||||||
|
let image = egui::ColorImage::from_rgba_unmultiplied(
|
||||||
|
[tw as usize, th as usize],
|
||||||
|
&rgba_data,
|
||||||
);
|
);
|
||||||
|
ui.ctx().load_texture(
|
||||||
|
format!("vthumb_{}_{}", ci.clip_id, ts_key),
|
||||||
|
image,
|
||||||
|
egui::TextureOptions::LINEAR,
|
||||||
|
)
|
||||||
|
});
|
||||||
|
|
||||||
|
let full_rect = egui::Rect::from_min_size(
|
||||||
|
egui::pos2(ci_rect.min.x + x_offset, ci_rect.min.y + 2.0),
|
||||||
|
egui::vec2(thumb_display_width, thumb_display_height),
|
||||||
|
);
|
||||||
|
let thumb_rect = full_rect.intersect(ci_rect);
|
||||||
|
|
||||||
|
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
|
||||||
|
let uv_min = egui::pos2(
|
||||||
|
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
|
||||||
|
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
|
||||||
|
);
|
||||||
|
let uv_max = egui::pos2(
|
||||||
|
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
|
||||||
|
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
|
||||||
|
);
|
||||||
|
|
||||||
|
painter.image(
|
||||||
|
texture.id(),
|
||||||
|
thumb_rect,
|
||||||
|
egui::Rect::from_min_max(uv_min, uv_max),
|
||||||
|
egui::Color32::WHITE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -3107,17 +3048,8 @@ impl TimelinePane {
|
||||||
None => continue,
|
None => continue,
|
||||||
};
|
};
|
||||||
|
|
||||||
// Min/max overview pools store 4 f32 per texel at the
|
let total_frames = samples.len() / (*ch).max(1) as usize;
|
||||||
// floor rate sr/B; raw pools store interleaved samples.
|
let audio_file_duration = total_frames as f64 / *sr as f64;
|
||||||
let minmax_b = waveform_minmax_pools.get(&audio_pool_index).copied();
|
|
||||||
let is_minmax = minmax_b.is_some();
|
|
||||||
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
|
|
||||||
let total_frames = samples.len() / frame_stride;
|
|
||||||
let eff_sr: f32 = match minmax_b {
|
|
||||||
Some(b) => *sr as f32 / b.max(1) as f32,
|
|
||||||
None => *sr as f32,
|
|
||||||
};
|
|
||||||
let audio_file_duration = total_frames as f64 / eff_sr as f64;
|
|
||||||
|
|
||||||
let clip_dur = audio_clip.duration;
|
let clip_dur = audio_clip.duration;
|
||||||
let mut ci_start = ci.effective_start();
|
let mut ci_start = ci.effective_start();
|
||||||
|
|
@ -3126,9 +3058,8 @@ impl TimelinePane {
|
||||||
}
|
}
|
||||||
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
|
let ci_duration = ci.total_duration(clip_dur, document.tempo_map());
|
||||||
|
|
||||||
// ci_start/ci_duration are in beats; convert to seconds for time_to_x.
|
let ci_screen_start = rect.min.x + self.time_to_x(ci_start);
|
||||||
let ci_screen_start = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start));
|
let ci_screen_end = ci_screen_start + (ci_duration * self.pixels_per_second as f64) as f32;
|
||||||
let ci_screen_end = rect.min.x + self.time_to_x(document.tempo_map().transform(ci_start + ci_duration));
|
|
||||||
|
|
||||||
let waveform_rect = egui::Rect::from_min_max(
|
let waveform_rect = egui::Rect::from_min_max(
|
||||||
egui::pos2(ci_screen_start.max(rect.min.x), wave_y_min),
|
egui::pos2(ci_screen_start.max(rect.min.x), wave_y_min),
|
||||||
|
|
@ -3150,10 +3081,9 @@ impl TimelinePane {
|
||||||
}
|
}
|
||||||
Some(crate::waveform_gpu::PendingUpload {
|
Some(crate::waveform_gpu::PendingUpload {
|
||||||
samples: samples.clone(),
|
samples: samples.clone(),
|
||||||
sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
|
sample_rate: *sr,
|
||||||
channels: *ch,
|
channels: *ch,
|
||||||
frame_limit,
|
frame_limit,
|
||||||
minmax: is_minmax,
|
|
||||||
})
|
})
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
|
|
@ -3168,7 +3098,7 @@ impl TimelinePane {
|
||||||
viewport_start_time: self.viewport_start_time as f32,
|
viewport_start_time: self.viewport_start_time as f32,
|
||||||
pixels_per_second: self.pixels_per_second as f32,
|
pixels_per_second: self.pixels_per_second as f32,
|
||||||
audio_duration: audio_file_duration as f32,
|
audio_duration: audio_file_duration as f32,
|
||||||
sample_rate: eff_sr,
|
sample_rate: *sr as f32,
|
||||||
clip_start_time: ci_screen_start,
|
clip_start_time: ci_screen_start,
|
||||||
trim_start: ci.trim_start as f32,
|
trim_start: ci.trim_start as f32,
|
||||||
tex_width: crate::waveform_gpu::tex_width() as f32,
|
tex_width: crate::waveform_gpu::tex_width() as f32,
|
||||||
|
|
@ -3667,16 +3597,8 @@ impl TimelinePane {
|
||||||
// Sampled Audio: Draw waveform via GPU
|
// Sampled Audio: Draw waveform via GPU
|
||||||
lightningbeam_core::clip::AudioClipType::Sampled { audio_pool_index } => {
|
lightningbeam_core::clip::AudioClipType::Sampled { audio_pool_index } => {
|
||||||
if let Some((samples, sr, ch)) = raw_audio_cache.get(audio_pool_index) {
|
if let Some((samples, sr, ch)) = raw_audio_cache.get(audio_pool_index) {
|
||||||
// Min/max overview pools: 4 f32/texel at rate sr/B.
|
let total_frames = samples.len() / (*ch).max(1) as usize;
|
||||||
let minmax_b = waveform_minmax_pools.get(audio_pool_index).copied();
|
let audio_file_duration = total_frames as f64 / *sr as f64;
|
||||||
let is_minmax = minmax_b.is_some();
|
|
||||||
let frame_stride = if is_minmax { 4 } else { (*ch).max(1) as usize };
|
|
||||||
let total_frames = samples.len() / frame_stride;
|
|
||||||
let eff_sr: f32 = match minmax_b {
|
|
||||||
Some(b) => *sr as f32 / b.max(1) as f32,
|
|
||||||
None => *sr as f32,
|
|
||||||
};
|
|
||||||
let audio_file_duration = total_frames as f64 / eff_sr as f64;
|
|
||||||
let screen_size = ui.ctx().content_rect().size();
|
let screen_size = ui.ctx().content_rect().size();
|
||||||
|
|
||||||
let pending_upload = if waveform_gpu_dirty.contains(audio_pool_index) {
|
let pending_upload = if waveform_gpu_dirty.contains(audio_pool_index) {
|
||||||
|
|
@ -3698,10 +3620,9 @@ impl TimelinePane {
|
||||||
|
|
||||||
Some(crate::waveform_gpu::PendingUpload {
|
Some(crate::waveform_gpu::PendingUpload {
|
||||||
samples: samples.clone(),
|
samples: samples.clone(),
|
||||||
sample_rate: if is_minmax { eff_sr.round().max(1.0) as u32 } else { *sr },
|
sample_rate: *sr,
|
||||||
channels: *ch,
|
channels: *ch,
|
||||||
frame_limit,
|
frame_limit,
|
||||||
minmax: is_minmax,
|
|
||||||
})
|
})
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
|
|
@ -3763,7 +3684,7 @@ impl TimelinePane {
|
||||||
viewport_start_time: self.viewport_start_time as f32,
|
viewport_start_time: self.viewport_start_time as f32,
|
||||||
pixels_per_second: self.pixels_per_second as f32,
|
pixels_per_second: self.pixels_per_second as f32,
|
||||||
audio_duration: audio_file_duration as f32,
|
audio_duration: audio_file_duration as f32,
|
||||||
sample_rate: eff_sr,
|
sample_rate: *sr as f32,
|
||||||
clip_start_time: iter_screen_start,
|
clip_start_time: iter_screen_start,
|
||||||
trim_start: preview_trim_start as f32,
|
trim_start: preview_trim_start as f32,
|
||||||
tex_width: crate::waveform_gpu::tex_width() as f32,
|
tex_width: crate::waveform_gpu::tex_width() as f32,
|
||||||
|
|
@ -3809,7 +3730,6 @@ impl TimelinePane {
|
||||||
sample_rate: *sr,
|
sample_rate: *sr,
|
||||||
channels: *ch,
|
channels: *ch,
|
||||||
frame_limit: None, // recording uses incremental path
|
frame_limit: None, // recording uses incremental path
|
||||||
minmax: false,
|
|
||||||
})
|
})
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
|
|
@ -3874,31 +3794,75 @@ impl TimelinePane {
|
||||||
let thumb_display_height = clip_rect.height() - 4.0;
|
let thumb_display_height = clip_rect.height() - 4.0;
|
||||||
if thumb_display_height > 8.0 {
|
if thumb_display_height > 8.0 {
|
||||||
let video_mgr = video_manager.lock().unwrap();
|
let video_mgr = video_manager.lock().unwrap();
|
||||||
// Tile from the clip's true origin (start_x is unclamped;
|
if let Some((tw, th, _)) = video_mgr.get_thumbnail_at(&clip_instance.clip_id, 0.0) {
|
||||||
// clip_rect is the clamped visible rect) so the strip scrolls
|
let aspect = tw as f32 / th as f32;
|
||||||
// correctly and shows the right content when partly off-screen.
|
let thumb_display_width = thumb_display_height * aspect;
|
||||||
draw_video_thumbnail_strip(
|
let thumb_step_px = thumb_display_width;
|
||||||
&painter,
|
|
||||||
ui,
|
let clip_width = clip_rect.width();
|
||||||
&video_mgr,
|
let num_thumbs = ((clip_width / thumb_step_px).ceil() as usize).max(1);
|
||||||
&mut self.video_thumbnail_textures,
|
|
||||||
clip_instance.clip_id,
|
for i in 0..num_thumbs {
|
||||||
clip_instance.trim_start,
|
let x_offset = i as f32 * thumb_step_px;
|
||||||
rect.min.x + start_x,
|
if x_offset >= clip_width { break; }
|
||||||
end_x - start_x,
|
|
||||||
clip_rect,
|
// Map pixel position to content time
|
||||||
clip_rect.min.y + 2.0,
|
let time_offset = (x_offset as f64 + thumb_display_width as f64 * 0.5)
|
||||||
thumb_display_height,
|
/ self.pixels_per_second as f64;
|
||||||
self.pixels_per_second as f64,
|
let content_time = clip_instance.trim_start + time_offset;
|
||||||
|
|
||||||
|
if let Some((tw, th, rgba_data)) = video_mgr.get_thumbnail_at(
|
||||||
|
&clip_instance.clip_id, content_time
|
||||||
|
) {
|
||||||
|
let ts_key = (content_time * 1000.0) as i64;
|
||||||
|
let cache_key = (clip_instance.clip_id, ts_key);
|
||||||
|
|
||||||
|
let texture = self.video_thumbnail_textures
|
||||||
|
.entry(cache_key)
|
||||||
|
.or_insert_with(|| {
|
||||||
|
let image = egui::ColorImage::from_rgba_unmultiplied(
|
||||||
|
[tw as usize, th as usize],
|
||||||
|
&rgba_data,
|
||||||
|
);
|
||||||
|
ui.ctx().load_texture(
|
||||||
|
format!("vthumb_{}_{}", clip_instance.clip_id, ts_key),
|
||||||
|
image,
|
||||||
|
egui::TextureOptions::LINEAR,
|
||||||
|
)
|
||||||
|
});
|
||||||
|
|
||||||
|
let full_rect = egui::Rect::from_min_size(
|
||||||
|
egui::pos2(clip_rect.min.x + x_offset, clip_rect.min.y + 2.0),
|
||||||
|
egui::vec2(thumb_display_width, thumb_display_height),
|
||||||
|
);
|
||||||
|
let thumb_rect = full_rect.intersect(clip_rect);
|
||||||
|
|
||||||
|
if thumb_rect.width() > 2.0 && thumb_rect.height() > 2.0 {
|
||||||
|
let uv_min = egui::pos2(
|
||||||
|
(thumb_rect.min.x - full_rect.min.x) / full_rect.width(),
|
||||||
|
(thumb_rect.min.y - full_rect.min.y) / full_rect.height(),
|
||||||
|
);
|
||||||
|
let uv_max = egui::pos2(
|
||||||
|
(thumb_rect.max.x - full_rect.min.x) / full_rect.width(),
|
||||||
|
(thumb_rect.max.y - full_rect.min.y) / full_rect.height(),
|
||||||
|
);
|
||||||
|
|
||||||
|
painter.image(
|
||||||
|
texture.id(),
|
||||||
|
thumb_rect,
|
||||||
|
egui::Rect::from_min_max(uv_min, uv_max),
|
||||||
|
egui::Color32::WHITE,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// VIDEO PREVIEW: Collect clip rect for hover detection. Store the
|
// VIDEO PREVIEW: Collect clip rect for hover detection
|
||||||
// clip's TRUE (unclamped) origin x so the hover content time is
|
|
||||||
// correct even when the clip is scrolled partly off the left.
|
|
||||||
if let lightningbeam_core::layer::AnyLayer::Video(_) = layer {
|
if let lightningbeam_core::layer::AnyLayer::Video(_) = layer {
|
||||||
video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, rect.min.x + start_x));
|
video_clip_hovers.push((clip_rect, clip_instance.clip_id, clip_instance.trim_start, instance_start));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Draw border per segment (per loop iteration for looping clips)
|
// Draw border per segment (per loop iteration for looping clips)
|
||||||
|
|
@ -3980,40 +3944,6 @@ impl TimelinePane {
|
||||||
color,
|
color,
|
||||||
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
|
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
|
||||||
));
|
));
|
||||||
// Record a (slightly enlarged) clickable area for click-to-seek.
|
|
||||||
self.keyframe_diamond_hits.push((
|
|
||||||
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
|
|
||||||
kf.time,
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Draw keyframe markers for raster layers (same diamond as vector).
|
|
||||||
if let lightningbeam_core::layer::AnyLayer::Raster(rl) = layer {
|
|
||||||
for kf in &rl.keyframes {
|
|
||||||
let x = self.time_to_x(kf.time);
|
|
||||||
if x >= 0.0 && x <= rect.width() {
|
|
||||||
let cx = rect.min.x + x;
|
|
||||||
let cy = y + LAYER_HEIGHT - 8.0;
|
|
||||||
let size = 5.0;
|
|
||||||
let diamond = [
|
|
||||||
egui::pos2(cx, cy - size),
|
|
||||||
egui::pos2(cx + size, cy),
|
|
||||||
egui::pos2(cx, cy + size),
|
|
||||||
egui::pos2(cx - size, cy),
|
|
||||||
];
|
|
||||||
let color = theme.bg_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(255, 220, 100));
|
|
||||||
painter.add(egui::Shape::convex_polygon(
|
|
||||||
diamond.to_vec(),
|
|
||||||
color,
|
|
||||||
egui::Stroke::new(1.0, theme.border_color(&["#timeline", ".keyframe-diamond"], ui.ctx(), egui::Color32::from_rgb(180, 150, 50))),
|
|
||||||
));
|
|
||||||
// Record a (slightly enlarged) clickable area for click-to-seek.
|
|
||||||
self.keyframe_diamond_hits.push((
|
|
||||||
egui::Rect::from_center_size(egui::pos2(cx, cy), egui::vec2(13.0, 13.0)),
|
|
||||||
kf.time,
|
|
||||||
));
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -4867,30 +4797,6 @@ impl TimelinePane {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Pointing-hand cursor when hovering a clickable keyframe diamond.
|
|
||||||
if let Some(hover) = response.hover_pos() {
|
|
||||||
if self.keyframe_diamond_hits.iter().any(|(r, _)| r.contains(hover)) {
|
|
||||||
ui.ctx().set_cursor_icon(egui::CursorIcon::PointingHand);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Click a keyframe diamond → snap the playhead exactly to that keyframe.
|
|
||||||
// (Hit targets come from the previous frame's render_layers; diamonds don't
|
|
||||||
// move between frames, so a click lands on the right one.)
|
|
||||||
if response.clicked() && !alt_held {
|
|
||||||
if let Some(pos) = response.interact_pointer_pos() {
|
|
||||||
if let Some(&(_, kf_time)) =
|
|
||||||
self.keyframe_diamond_hits.iter().find(|(r, _)| r.contains(pos))
|
|
||||||
{
|
|
||||||
*playback_time = kf_time;
|
|
||||||
if let Some(controller_arc) = audio_controller {
|
|
||||||
let mut controller = controller_arc.lock().unwrap();
|
|
||||||
controller.seek(kf_time);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get mouse position relative to content area
|
// Get mouse position relative to content area
|
||||||
let mouse_pos = response.hover_pos().unwrap_or(content_rect.center());
|
let mouse_pos = response.hover_pos().unwrap_or(content_rect.center());
|
||||||
let mouse_x = (mouse_pos.x - content_rect.min.x).max(0.0);
|
let mouse_x = (mouse_pos.x - content_rect.min.x).max(0.0);
|
||||||
|
|
@ -5512,7 +5418,7 @@ impl PaneRenderer for TimelinePane {
|
||||||
|
|
||||||
// Render layer rows with clipping
|
// Render layer rows with clipping
|
||||||
ui.set_clip_rect(content_rect.intersect(original_clip_rect));
|
ui.set_clip_rect(content_rect.intersect(original_clip_rect));
|
||||||
let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.waveform_minmax_pools, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time);
|
let (video_clip_hovers, pending_lane_renders) = self.render_layers(ui, content_rect, shared.theme, document, shared.active_layer_id, shared.focus, shared.selection, shared.midi_event_cache, shared.raw_audio_cache, shared.waveform_gpu_dirty, shared.target_format, shared.waveform_stereo, &context_layers, shared.video_manager, &audio_cache, *shared.playback_time);
|
||||||
|
|
||||||
// Render playhead on top (clip to timeline area)
|
// Render playhead on top (clip to timeline area)
|
||||||
ui.set_clip_rect(timeline_rect.intersect(original_clip_rect));
|
ui.set_clip_rect(timeline_rect.intersect(original_clip_rect));
|
||||||
|
|
@ -5988,24 +5894,26 @@ impl PaneRenderer for TimelinePane {
|
||||||
|
|
||||||
// VIDEO HOVER DETECTION: Handle video clip hover tooltips AFTER input handling
|
// VIDEO HOVER DETECTION: Handle video clip hover tooltips AFTER input handling
|
||||||
// This ensures hover events aren't consumed by the main input handler
|
// This ensures hover events aren't consumed by the main input handler
|
||||||
for (clip_rect, clip_id, trim_start, clip_origin_x) in video_clip_hovers {
|
for (clip_rect, clip_id, trim_start, instance_start) in video_clip_hovers {
|
||||||
let hover_response = ui.allocate_rect(clip_rect, egui::Sense::hover());
|
let hover_response = ui.allocate_rect(clip_rect, egui::Sense::hover());
|
||||||
|
|
||||||
if hover_response.hovered() {
|
if hover_response.hovered() {
|
||||||
if let Some(hover_pos) = hover_response.hover_pos() {
|
if let Some(hover_pos) = hover_response.hover_pos() {
|
||||||
// Content time from the clip's TRUE origin. `clip_rect` is clamped
|
// Calculate timestamp at hover position
|
||||||
// to the viewport, so using its left edge mislabels the frame when
|
let hover_offset_pixels = hover_pos.x - clip_rect.min.x;
|
||||||
// the clip is scrolled partly off the left (same bug the strip had).
|
let hover_offset_time = (hover_offset_pixels as f64) / (self.pixels_per_second as f64);
|
||||||
let hover_offset_pixels = hover_pos.x - clip_origin_x;
|
let hover_timestamp = instance_start + hover_offset_time;
|
||||||
let clip_content_time = trim_start + (hover_offset_pixels as f64) / (self.pixels_per_second as f64);
|
|
||||||
|
// Remap to clip content time accounting for trim
|
||||||
|
let clip_content_time = trim_start + (hover_timestamp - instance_start);
|
||||||
|
|
||||||
// Try to get thumbnail from video manager
|
// Try to get thumbnail from video manager
|
||||||
let thumbnail_data: Option<(f64, u32, u32, std::sync::Arc<Vec<u8>>)> = {
|
let thumbnail_data: Option<(u32, u32, std::sync::Arc<Vec<u8>>)> = {
|
||||||
let video_mgr = shared.video_manager.lock().unwrap();
|
let video_mgr = shared.video_manager.lock().unwrap();
|
||||||
video_mgr.get_thumbnail_at(&clip_id, clip_content_time)
|
video_mgr.get_thumbnail_at(&clip_id, clip_content_time)
|
||||||
};
|
};
|
||||||
|
|
||||||
if let Some((_, thumb_width, thumb_height, ref thumb_data)) = thumbnail_data {
|
if let Some((thumb_width, thumb_height, ref thumb_data)) = thumbnail_data {
|
||||||
// Create texture from thumbnail
|
// Create texture from thumbnail
|
||||||
let color_image = egui::ColorImage::from_rgba_unmultiplied(
|
let color_image = egui::ColorImage::from_rgba_unmultiplied(
|
||||||
[thumb_width as usize, thumb_height as usize],
|
[thumb_width as usize, thumb_height as usize],
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,6 @@ use crate::config::AppConfig;
|
||||||
use crate::keymap::{self, AppAction, KeymapManager};
|
use crate::keymap::{self, AppAction, KeymapManager};
|
||||||
use crate::menu::{MenuSystem, Shortcut, ShortcutKey};
|
use crate::menu::{MenuSystem, Shortcut, ShortcutKey};
|
||||||
use crate::theme::{Theme, ThemeMode};
|
use crate::theme::{Theme, ThemeMode};
|
||||||
use lightningbeam_core::file_io::LargeMediaMode;
|
|
||||||
|
|
||||||
/// Which tab is selected in the preferences dialog
|
/// Which tab is selected in the preferences dialog
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
|
@ -60,7 +59,6 @@ struct PreferencesState {
|
||||||
debug: bool,
|
debug: bool,
|
||||||
waveform_stereo: bool,
|
waveform_stereo: bool,
|
||||||
theme_mode: ThemeMode,
|
theme_mode: ThemeMode,
|
||||||
large_media_default: LargeMediaMode,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl From<(&AppConfig, &Theme)> for PreferencesState {
|
impl From<(&AppConfig, &Theme)> for PreferencesState {
|
||||||
|
|
@ -77,7 +75,6 @@ impl From<(&AppConfig, &Theme)> for PreferencesState {
|
||||||
debug: config.debug,
|
debug: config.debug,
|
||||||
waveform_stereo: config.waveform_stereo,
|
waveform_stereo: config.waveform_stereo,
|
||||||
theme_mode: theme.mode(),
|
theme_mode: theme.mode(),
|
||||||
large_media_default: config.large_media_default,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -96,7 +93,6 @@ impl Default for PreferencesState {
|
||||||
debug: false,
|
debug: false,
|
||||||
waveform_stereo: false,
|
waveform_stereo: false,
|
||||||
theme_mode: ThemeMode::System,
|
theme_mode: ThemeMode::System,
|
||||||
large_media_default: LargeMediaMode::default(),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -571,24 +567,6 @@ impl PreferencesDialog {
|
||||||
&mut self.working_prefs.waveform_stereo,
|
&mut self.working_prefs.waveform_stereo,
|
||||||
"Show waveforms as stacked stereo",
|
"Show waveforms as stacked stereo",
|
||||||
);
|
);
|
||||||
ui.horizontal(|ui| {
|
|
||||||
let threshold_gb = lightningbeam_core::beam_archive::LARGE_MEDIA_THRESHOLD
|
|
||||||
as f64
|
|
||||||
/ (1024.0 * 1024.0 * 1024.0);
|
|
||||||
ui.label(format!("Large media (>{:.0} GB):", threshold_gb));
|
|
||||||
let label = |m: LargeMediaMode| match m {
|
|
||||||
LargeMediaMode::Ask => "Ask each time",
|
|
||||||
LargeMediaMode::Pack => "Pack into project",
|
|
||||||
LargeMediaMode::Reference => "Reference external file",
|
|
||||||
};
|
|
||||||
egui::ComboBox::from_id_salt("large_media_default")
|
|
||||||
.selected_text(label(self.working_prefs.large_media_default))
|
|
||||||
.show_ui(ui, |ui| {
|
|
||||||
for mode in [LargeMediaMode::Ask, LargeMediaMode::Pack, LargeMediaMode::Reference] {
|
|
||||||
ui.selectable_value(&mut self.working_prefs.large_media_default, mode, label(mode));
|
|
||||||
}
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -651,7 +629,6 @@ impl PreferencesDialog {
|
||||||
config.debug = self.working_prefs.debug;
|
config.debug = self.working_prefs.debug;
|
||||||
config.waveform_stereo = self.working_prefs.waveform_stereo;
|
config.waveform_stereo = self.working_prefs.waveform_stereo;
|
||||||
config.theme_mode = self.working_prefs.theme_mode.to_string_lower();
|
config.theme_mode = self.working_prefs.theme_mode.to_string_lower();
|
||||||
config.large_media_default = self.working_prefs.large_media_default;
|
|
||||||
config.keybindings = keybinding_config;
|
config.keybindings = keybinding_config;
|
||||||
|
|
||||||
// Apply theme immediately
|
// Apply theme immediately
|
||||||
|
|
|
||||||
|
|
@ -112,37 +112,6 @@ pub struct PendingUpload {
|
||||||
/// The texture is allocated at full size, but total_frames is set to
|
/// The texture is allocated at full size, but total_frames is set to
|
||||||
/// the limited count so subsequent calls use the incremental path.
|
/// the limited count so subsequent calls use the incremental path.
|
||||||
pub frame_limit: Option<usize>,
|
pub frame_limit: Option<usize>,
|
||||||
/// When true, `samples` is interpreted as **pre-packed min/max texels**:
|
|
||||||
/// 4 floats per "frame" = `[l_min, l_max, r_min, r_max]` (a waveform-pyramid
|
|
||||||
/// floor level). The caller passes the *effective* `sample_rate` (`sr / B`)
|
|
||||||
/// so the shader's time→texel mapping covers `B` source samples per texel.
|
|
||||||
/// When false, `samples` is raw interleaved audio (min = max per texel).
|
|
||||||
pub minmax: bool,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Pack one source "frame" into an `Rgba16Float` texel `(Lmin,Lmax,Rmin,Rmax)`.
|
|
||||||
/// Raw audio sets min = max per channel; min/max input copies the 4 values.
|
|
||||||
#[inline]
|
|
||||||
fn pack_texel(samples: &[f32], global_frame: usize, channels: usize, minmax: bool) -> [half::f16; 4] {
|
|
||||||
if minmax {
|
|
||||||
let o = global_frame * 4;
|
|
||||||
[
|
|
||||||
half::f16::from_f32(samples.get(o).copied().unwrap_or(0.0)),
|
|
||||||
half::f16::from_f32(samples.get(o + 1).copied().unwrap_or(0.0)),
|
|
||||||
half::f16::from_f32(samples.get(o + 2).copied().unwrap_or(0.0)),
|
|
||||||
half::f16::from_f32(samples.get(o + 3).copied().unwrap_or(0.0)),
|
|
||||||
]
|
|
||||||
} else {
|
|
||||||
let so = global_frame * channels;
|
|
||||||
let left = samples.get(so).copied().unwrap_or(0.0);
|
|
||||||
let right = if channels >= 2 {
|
|
||||||
samples.get(so + 1).copied().unwrap_or(left)
|
|
||||||
} else {
|
|
||||||
left
|
|
||||||
};
|
|
||||||
let (l, r) = (half::f16::from_f32(left), half::f16::from_f32(right));
|
|
||||||
[l, l, r, r]
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Maximum frames to convert and upload per frame (~250K frames ≈ 5.6s at 44.1kHz).
|
/// Maximum frames to convert and upload per frame (~250K frames ≈ 5.6s at 44.1kHz).
|
||||||
|
|
@ -322,11 +291,8 @@ impl WaveformGpuResources {
|
||||||
sample_rate: u32,
|
sample_rate: u32,
|
||||||
channels: u32,
|
channels: u32,
|
||||||
frame_limit: Option<usize>,
|
frame_limit: Option<usize>,
|
||||||
minmax: bool,
|
|
||||||
) -> Vec<wgpu::CommandBuffer> {
|
) -> Vec<wgpu::CommandBuffer> {
|
||||||
// For min/max input each "frame" is 4 floats; for raw it's `channels`.
|
let new_total_frames = samples.len() / channels.max(1) as usize;
|
||||||
let frame_stride = if minmax { 4 } else { channels.max(1) as usize };
|
|
||||||
let new_total_frames = samples.len() / frame_stride;
|
|
||||||
if new_total_frames == 0 {
|
if new_total_frames == 0 {
|
||||||
return Vec::new();
|
return Vec::new();
|
||||||
}
|
}
|
||||||
|
|
@ -363,9 +329,22 @@ impl WaveformGpuResources {
|
||||||
if global_frame >= effective_frames {
|
if global_frame >= effective_frames {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
let sample_offset = global_frame * channels as usize;
|
||||||
|
let left = if sample_offset < samples.len() {
|
||||||
|
samples[sample_offset]
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
|
||||||
|
samples[sample_offset + 1]
|
||||||
|
} else {
|
||||||
|
left
|
||||||
|
};
|
||||||
let texel_offset = frame * 4;
|
let texel_offset = frame * 4;
|
||||||
let t = pack_texel(samples, global_frame, channels as usize, minmax);
|
row_data[texel_offset] = half::f16::from_f32(left);
|
||||||
row_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
|
row_data[texel_offset + 1] = half::f16::from_f32(left);
|
||||||
|
row_data[texel_offset + 2] = half::f16::from_f32(right);
|
||||||
|
row_data[texel_offset + 3] = half::f16::from_f32(right);
|
||||||
}
|
}
|
||||||
|
|
||||||
let entry = self.entries.get(&pool_index).unwrap();
|
let entry = self.entries.get(&pool_index).unwrap();
|
||||||
|
|
@ -487,9 +466,24 @@ impl WaveformGpuResources {
|
||||||
|
|
||||||
for frame in 0..seg_upload_count as usize {
|
for frame in 0..seg_upload_count as usize {
|
||||||
let global_frame = seg_start_frame as usize + frame;
|
let global_frame = seg_start_frame as usize + frame;
|
||||||
|
let sample_offset = global_frame * channels as usize;
|
||||||
|
|
||||||
|
let left = if sample_offset < samples.len() {
|
||||||
|
samples[sample_offset]
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
let right = if channels >= 2 && sample_offset + 1 < samples.len() {
|
||||||
|
samples[sample_offset + 1]
|
||||||
|
} else {
|
||||||
|
left
|
||||||
|
};
|
||||||
|
|
||||||
let texel_offset = frame * 4;
|
let texel_offset = frame * 4;
|
||||||
let t = pack_texel(samples, global_frame, channels as usize, minmax);
|
mip0_data[texel_offset] = half::f16::from_f32(left);
|
||||||
mip0_data[texel_offset..texel_offset + 4].copy_from_slice(&t);
|
mip0_data[texel_offset + 1] = half::f16::from_f32(left);
|
||||||
|
mip0_data[texel_offset + 2] = half::f16::from_f32(right);
|
||||||
|
mip0_data[texel_offset + 3] = half::f16::from_f32(right);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Upload mip 0 (only rows with actual data)
|
// Upload mip 0 (only rows with actual data)
|
||||||
|
|
@ -707,7 +701,6 @@ impl egui_wgpu::CallbackTrait for WaveformCallback {
|
||||||
upload.sample_rate,
|
upload.sample_rate,
|
||||||
upload.channels,
|
upload.channels,
|
||||||
upload.frame_limit,
|
upload.frame_limit,
|
||||||
upload.minmax,
|
|
||||||
);
|
);
|
||||||
cmds.extend(new_cmds);
|
cmds.extend(new_cmds);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue