docs: mark Phase 3a-1/3a-2/3b done; note deferred raster-keyframe-UI bugs
This commit is contained in:
Skyler Lehmkuhl
parent
d001ca1083
commit
62d6cd0c84
|
|
@ -0,0 +1,740 @@
|
|||
# Streaming Media To/From Disk — Plan
|
||||
|
||||
**Goal:** Lightningbeam must handle audio and video files (and raster animation, and
|
||||
image assets) of *arbitrary length/size*. Anywhere we touch media we should stream from
|
||||
and to disk when the data is too large to fit comfortably in memory, rather than loading
|
||||
the entire file regardless of size.
|
||||
|
||||
**Scope of this document:** audio, video, raster frames, image-asset paging, **and the
|
||||
`.beam` container format** — these turned out to be one problem, not two. Streaming on load
|
||||
is impossible while the container forces a full decode, so the container decision (below)
|
||||
is now part of this plan.
|
||||
|
||||
## Deferred bugs (do at the end)
|
||||
- [x] **Timeline thumbnail scroll (FIXED):** the strip tiled from the *clamped* visible-left of the
|
||||
clip, so when a clip was scrolled partly off the left it showed the clip's start content at the
|
||||
viewport edge. Now tiled from the clip's **true (unclamped) origin** over its full width, drawing
|
||||
only the tiles intersecting the visible rect (`draw_video_thumbnail_strip` in timeline.rs). Both
|
||||
render sites (collapsed-group + expanded-track) share the helper. *(Compiles; needs in-app check.)*
|
||||
- [x] **Clip thumbnails stop updating (FIXED):** the GPU texture cache was keyed by the *requested*
|
||||
content time, so once a tile cached the first (often far-off) thumbnail it never refreshed as
|
||||
closer ones loaded. `VideoManager::get_thumbnail_at` now also returns the **actual** thumbnail
|
||||
timestamp, and the cache keys on that — so a tile picks up a new texture when a closer thumbnail
|
||||
finishes generating. Existing `retain`-by-visible-clip cleanup keeps it bounded. *(Needs in-app check.)*
|
||||
|
||||
## Deferred raster-keyframe-UI bugs (pre-existing; found during Phase 3 testing)
|
||||
Both stem from the **timeline having no model for raster keyframes** — it renders *clip
|
||||
instances* (`layer_clips`/`collect_clip_instances` return `&[ClipInstance]`), but raster layers use
|
||||
`keyframes: Vec<RasterKeyframe>`; every raster arm in timeline.rs is a stub (`Raster => &[]`/`{}`).
|
||||
- **(a) `Timeline > New Keyframe` doesn't refresh the canvas** until you draw. The menu action
|
||||
creates a blank raster keyframe but doesn't trigger a canvas repaint / GPU-cache invalidation for
|
||||
the new (different) `kf.id`, so the stale previous-frame texture stays until a stroke dirties it.
|
||||
- **(b) Raster keyframes never render on the timeline** — no code walks `RasterLayer::keyframes` to
|
||||
draw markers. Needs a raster-keyframe strip (display + click-to-navigate + insert).
|
||||
Confirmed not paging regressions (same before 3a-1). A focused "raster keyframe timeline UI" task.
|
||||
|
||||
## Noted enhancements (later, after the phases)
|
||||
- [x] **Surround → stereo downmix (DONE).** Done uniformly in `render_from_file` (`pool.rs`) so it
|
||||
covers every storage type (PCM/InMemory, compressed via symphonia, video-audio via ffmpeg — all
|
||||
flow through this mixer with the source kept multichannel in the read-ahead buffer). New
|
||||
`stereo_downmix_matrix(src_channels)` gives `[L][src]`/`[R][src]` coefficients for the conventional
|
||||
interleave order (FL FR FC LFE BL BR SL SR…) for 3/4/5/5.1/6.1/7.1: full level for the matching
|
||||
front, `1/√2` for centre + each surround, LFE dropped; each row normalized so |coef| sum ≤ 1 to
|
||||
prevent clipping (matches ffmpeg's default). Applied in both the direct-copy and sinc-resample
|
||||
paths (only when `dst==2 && src>2`; unknown layouts fall back to front L/R). Compiles clean.
|
||||
*(Needs in-app check: a 5.1 file now has centre/dialog present and isn't thin; not distorted/clipping.)*
|
||||
Native multichannel support remains a separate, larger project.
|
||||
- **Export speed:** a 1:14 1080p MP4 took ~9:06 to export (~7.4x slower than realtime). The video
|
||||
export pipeline re-seeks + decodes per output frame (see `[Video Seek]`/`[Video Timing]` logs) and
|
||||
does CPU YUV conversion; likely wins from sequential decode (avoid per-frame seeks), reusing the
|
||||
decode cache, and/or GPU-side color conversion. Profile before optimizing.
|
||||
- **AAC export NaN guard (done):** `convert_chunk_to_planar_f32` now sanitizes non-finite samples
|
||||
(NaN/Inf → 0, finite clamped to [-1,1]) like the integer paths, with a one-time warning — a stray
|
||||
non-finite render sample no longer fails the whole export. Upstream NaN source (effect/automation/
|
||||
decode) still worth chasing if it recurs.
|
||||
- [x] **Persist video thumbnails (DONE).** Mirrors waveform persistence: each clip's thumbnails are
|
||||
PNG-encoded + packed into one opaque `LBTN` blob (editor owns the format; `encode/decode_thumbnail_blob`
|
||||
in main.rs), stored as a `MediaKind::Thumbnail` row keyed by `thumbnail_media_id(clip_id)` (clip id XOR
|
||||
a fixed sentinel). Save: a cheap Arc-clone snapshot (`VideoManager::snapshot_all_thumbnails`) rides the
|
||||
`FileCommand::Save`, PNG-encoded off the UI thread in the worker, written by `save_beam` (kept in place
|
||||
on re-save). Load: `load_beam_sqlite` reads the packs into `LoadedProject.thumbnail_blobs`; the editor
|
||||
decodes + `insert_thumbnail`s them on a background thread and **gates regeneration** (`register_loaded_videos`
|
||||
skips clips with persisted thumbnails). Bonus: thumbnails show even if the source video file is missing.
|
||||
**Partial sets are persisted and resumed** (not thrown away): the `LBTN` blob (v2) carries a `complete`
|
||||
flag (`VideoManager.thumbnails_complete`, marked when the keyframe pass finishes). On load, complete
|
||||
packs are restored + skip regeneration; *partial* packs are restored AND generation is resumed —
|
||||
`generate_keyframe_thumbnails` takes a `should_skip` predicate (`has_thumbnail_near`) so it only decodes
|
||||
the keyframes not already covered. `insert_thumbnail` is now sorted + idempotent (fixes a latent
|
||||
unsorted-`binary_search` bug and makes concurrent restore + resume race-safe). So a save 50 min into a
|
||||
2 h video keeps that work and continues from there on reload.
|
||||
Container tests still green; all crates compile. *(Needs in-app check: reload = instant thumbnails for
|
||||
complete clips; a mid-generation save resumes from where it left off on reload.)*
|
||||
**Size assessment (done):** thumbnails are 128px wide, height by aspect (72px at 16:9 →
|
||||
128×72×4 ≈ **36 KB raw** each; 4:3 ≈ 49 KB), generated **one per ~5 s** (capped `interval_secs`,
|
||||
at keyframes — so ~12/min). Raw: ~0.5 MB per 1:14 clip, ~26 MB/hour, ~52 MB/2 h. Compressed for
|
||||
on-disk: JPEG ~3–6 KB/thumb → **~6 MB/2 h**; PNG ~8–15 KB → ~14 MB/2 h. So persistence is cheap
|
||||
(≤ the waveform's ~36 MB/2 h), especially as JPEG. Plan: encode each clip's thumbnails (JPEG) +
|
||||
their timestamps into one blob, a new `MediaKind::Thumbnail` row keyed by the clip/media id (mirror
|
||||
the waveform persistence: write on save, restore via `insert_thumbnail` on load, regenerate if
|
||||
absent). The 5 s interval already bounds count; no extra budget needed.
|
||||
- **Progressive waveform on first import:** generation streams the whole file before the
|
||||
waveform appears (several seconds for large files). Since `build_waveform_pyramid` already
|
||||
streams, emit partial floors as it advances (e.g. flush every N seconds of decoded audio via
|
||||
the existing `waveform_result` channel + chunked GPU upload) so the overview fills in across
|
||||
the clip left-to-right instead of appearing all at once. Persistence saves only the final
|
||||
complete pyramid.
|
||||
|
||||
## Guiding principle
|
||||
Three subsystems already have the right streaming primitive; most of the work is wiring,
|
||||
bounding caches, and adding a residency window. The recurring pattern:
|
||||
|
||||
> Keep tiny metadata always-resident, fault the heavy payload in on demand keyed by a
|
||||
> stable ID, and evict everything outside a window around the playhead.
|
||||
|
||||
---
|
||||
|
||||
## Audit summary (where we stand today)
|
||||
|
||||
### Correctly streaming / bounded
|
||||
- Video frame decode/seek/playback (`lightningbeam-core/src/video.rs:191` `get_frame` —
|
||||
keyframe-index seek + decode-until-target, one frame resident).
|
||||
- WAV/AIFF import via mmap (`daw-backend/src/audio/engine.rs:2328`).
|
||||
- Webcam capture encodes directly to disk (`lightningbeam-core/src/webcam.rs`).
|
||||
- `WaveformCache` (100MB cap), decoder `LruCache` (20 frames), export render loop (≤3
|
||||
frames in flight).
|
||||
- The compressed-audio disk reader `daw-backend/src/audio/disk_reader.rs`
|
||||
(`CompressedReader` + 3s `ReadAheadBuffer`) — **correct but never activated** (Phase 1a).
|
||||
|
||||
### Fully-loaded, unbounded by file length (the problems)
|
||||
| Site | Issue |
|
||||
|---|---|
|
||||
| `daw-backend/src/io/audio_file.rs:344` `decode_progressive` | Decodes whole compressed file into a `Vec<f32>`; de-facto playback source. |
|
||||
| `daw-backend/src/audio/pool.rs:1071` `load_file_into_pool` | Every audio file in a saved project fully decoded to `InMemory` on open. |
|
||||
| `lightningbeam-core/src/video.rs:711` `extract_audio_from_video` | Whole video audio track into one `Vec<f32>`. |
|
||||
| `lightningbeam-core/src/video.rs:412` `VideoManager.frame_cache` | Unbounded `HashMap` of full-res RGBA frames; grows while scrubbing. |
|
||||
| `export/mod.rs:388-400` | Mux step buffers all compressed packets into `Vec`s; O(duration). |
|
||||
| `lightningbeam-core/src/raster_layer.rs:115` `RasterKeyframe.raw_pixels` | ~8MB/frame at 1080p; all keyframes decoded from PNG at load (`file_io.rs:611-640`), never evicted. |
|
||||
| `lightningbeam-editor/src/gpu_brush.rs:1051` `raster_layer_cache` | Unbounded GPU texture `HashMap`. |
|
||||
| `lightningbeam-core/src/renderer.rs:25` `ImageCache` | Unbounded decoded image cache (asset textures). |
|
||||
| `Document.image_assets` (`document.rs:206`) | Every image asset's compressed bytes resident for document life. |
|
||||
|
||||
---
|
||||
|
||||
## Container format decision: `.beam` → SQLite *(DECIDED)*
|
||||
|
||||
The `.beam` container moves from a **ZIP archive** to a **SQLite database file** (same
|
||||
`.beam` extension). This is the foundation the rest of the plan builds on.
|
||||
|
||||
### Why
|
||||
ZIP can stream `Stored` entries in place (via `data_start()`), but it has **no in-place
|
||||
mutation** — every save and every raster frame write-back rewrites the whole archive — and
|
||||
embedded PCM is rarely mmap-aligned. The current load path is even worse: it reads each
|
||||
ZIP audio entry fully, decodes FLAC → re-encodes WAV → base64 → base64-decodes → temp file
|
||||
→ full Symphonia decode → resident `Vec<f32>` (`file_io.rs:513-604`, `pool.rs:1071`).
|
||||
|
||||
SQLite dissolves the single-file-vs-performance tension:
|
||||
- **Single file** — beginner-friendly, behaves like a file on every OS (no package-folder
|
||||
confusion; we have no bundle magic on Linux/Windows).
|
||||
- **Streaming reads** — `sqlite3_blob_open` / `blob_read(offset, len)` gives seekable,
|
||||
chunked reads through the pager (mmap mode for the DB). For chunked streaming the
|
||||
pager-copy is negligible vs. decode cost, so the lack of zero-copy mmap doesn't matter.
|
||||
- **Cheap, crash-safe mutation** — raster frame write-back is a transactional `UPDATE`;
|
||||
save is a metadata write + dirty-blob updates. **ACID** means a force-quit / power loss /
|
||||
crash mid-save can't corrupt the project (ZIP and package-dirs both have to hand-roll
|
||||
atomicity).
|
||||
- **Inspectable / scriptable** — `sqlite3` CLI; `beam_inspector.py` can read it directly.
|
||||
|
||||
**Net effect: there is no scratch directory anywhere in this plan.** Media stream via blob
|
||||
reads (or external paths); raster frames live in blob rows and write back transactionally.
|
||||
|
||||
### Large-media policy: packed OR referenced
|
||||
Two storage modes per media item, both supported:
|
||||
- **Packed** — bytes live in the DB. To stay under SQLite's ~2GB per-blob ceiling (and to
|
||||
make reads naturally chunked), large media is split into **multiple blob-chunk rows**
|
||||
(e.g. 64 MB/chunk); streaming reads address `(chunk_index, offset)`.
|
||||
- **Referenced** — the DB stores only a path; bytes stay on disk (useful for shared media
|
||||
on a network drive, or media too large/volatile to pack).
|
||||
|
||||
**Default-mode preference for files over the per-blob limit (~2GB):**
|
||||
- A user preference `large_media_default: Pack | Reference` controls what happens to
|
||||
imports above the threshold.
|
||||
- The **first time** the user imports a media file over the limit, **prompt** them
|
||||
(Pack vs Reference), apply it, and **persist the choice** as the preference for future
|
||||
large imports (changeable later in settings).
|
||||
- Files under the limit are packed by default (chunked only if needed).
|
||||
|
||||
### Schema sketch
|
||||
```
|
||||
media(
|
||||
id BLOB PRIMARY KEY, -- stable Uuid
|
||||
kind INTEGER, -- audio | video | raster | image-asset
|
||||
codec TEXT, -- "flac","mp3","png",... (original, lossless-preserving)
|
||||
storage INTEGER, -- 0 = packed, 1 = referenced
|
||||
ext_path TEXT, -- set when storage = referenced
|
||||
total_len INTEGER, -- bytes (packed) for chunk math
|
||||
channels INTEGER, sample_rate INTEGER, width INTEGER, height INTEGER -- kind-specific meta
|
||||
)
|
||||
media_chunk(
|
||||
media_id BLOB, chunk_index INTEGER, bytes BLOB,
|
||||
PRIMARY KEY (media_id, chunk_index)
|
||||
)
|
||||
project_json(id INTEGER PRIMARY KEY CHECK (id = 0), data TEXT) -- existing project.json, verbatim
|
||||
meta(key TEXT PRIMARY KEY, value TEXT) -- version, created, modified
|
||||
```
|
||||
`project.json` stays the same serialized `BeamProject` for now — only its container and the
|
||||
media storage change. A migration reads a legacy ZIP `.beam` and writes the SQLite form on
|
||||
first open/save.
|
||||
|
||||
### Streaming reads from packed media
|
||||
A `BlobReader` implementing `Read + Seek` over `media_chunk` rows feeds the existing
|
||||
streaming consumers unchanged: `CompressedReader` (audio) decodes from it instead of a
|
||||
`File`; the video decoder seeks within it; raster `UPDATE`s a chunk. Referenced media uses a
|
||||
plain `File` exactly as `do_import_audio` already does for originals today.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Audio: activate what already exists *(highest impact, lowest effort)*
|
||||
|
||||
### 1a. Turn on the compressed-audio disk reader
|
||||
The `CompressedReader` + 3-second `ReadAheadBuffer` in `disk_reader.rs` is complete but
|
||||
never invoked (`DiskReaderCommand::ActivateFile` / `DiskReader::create_buffer` are never
|
||||
called; `AudioClip::read_ahead` at `clip.rs:63` is hard-wired to `None`).
|
||||
- On compressed import (`engine.rs:2381`) and during playback setup, activate the file and
|
||||
assign `AudioClip::read_ahead`.
|
||||
- Change `decode_progressive` (`io/audio_file.rs:344`) to produce only the downsampled
|
||||
waveform overview (min/max peaks) the UI needs, then drop decoded PCM. Playback comes
|
||||
from the ring buffer, not RAM.
|
||||
- Verify `render_from_file` (`pool.rs:449`) reads from `read_ahead` when `data()` is empty.
|
||||
|
||||
**Risk:** the real-time thread must never block on disk. The ring buffer prefetches ~2s
|
||||
ahead; underruns degrade to silence (live) or block-wait (export), which `disk_reader.rs`
|
||||
already distinguishes.
|
||||
|
||||
### 1b. Stream on project load *(depends on the SQLite container)*
|
||||
Three coupled changes (none works alone):
|
||||
1. Replace `load_file_into_pool`'s full decode (`pool.rs:1071`) with the same branching as
|
||||
`do_import_audio`: PCM → mmap (referenced) or in-memory for tiny packed PCM; compressed
|
||||
(incl. FLAC) → `from_compressed` placeholder backed by a `BlobReader` (packed) or `File`
|
||||
(referenced). The claxon FLAC→WAV→base64 round-trip in `file_io.rs:533-591` is deleted.
|
||||
2. **Bulk read-ahead activation:** loaded clips are deserialized directly
|
||||
(`audio_backend.project`), bypassing `AddAudioClip`, so the Phase 1a wiring never fires
|
||||
for them. After the engine installs the project, walk all audio clips and
|
||||
`create_buffer` + `ActivateFile` + set `read_ahead` for every clip referencing a
|
||||
`Compressed` pool entry. (`CompressedReader::open` needs a variant that takes a
|
||||
`BlobReader` instead of a path for packed media.)
|
||||
3. Pool entries carry storage mode (packed-chunks vs referenced path) from the `media`
|
||||
table instead of base64 `embedded_data`.
|
||||
|
||||
### 1c. Video's embedded audio track — stream from the video via ffmpeg
|
||||
|
||||
**Interim stopgap (shipped):** `extract_audio_from_video_to_wav` streams the decoded audio to
|
||||
a temp WAV, imported via `import_audio_sync` (mmap). Fixes the RAM OOM but writes the whole
|
||||
uncompressed track to `/tmp` (fills small temp partitions) and the temp path doesn't survive
|
||||
save/reload. **Superseded by the design below.**
|
||||
|
||||
**Proper design — stream the video's audio track on demand, never materialized.**
|
||||
|
||||
*Enabler:* `daw-backend` already depends on `ffmpeg-next` (used for MP3/AAC encoding), so the
|
||||
ffmpeg audio decoder lives beside `CompressedReader` in `daw-backend/src/audio/`. No
|
||||
cross-crate work (`core → daw-backend` is one-way). `CompressedReader` already has the needed
|
||||
interface.
|
||||
|
||||
1. **`VideoAudioReader` (ffmpeg)** — mirrors `CompressedReader`:
|
||||
`open(path)`, `decode_next(&mut Vec<f32>) -> frames` (resample → interleaved f32 at native
|
||||
rate; reuse the old extraction resampler), `seek(target_frame) -> actual`,
|
||||
`sample_rate`/`channels`/`total_frames`.
|
||||
2. **Source dispatch:** `enum StreamSource { Compressed(CompressedReader), Video(VideoAudioReader) }`
|
||||
(or a small `trait AudioFrameSource`) held by the reader thread; ring buffer / prefetch /
|
||||
export-blocking unchanged. `DiskReaderCommand::ActivateFile` gains a `kind: SourceKind`.
|
||||
3. **Pool model:** `AudioStorage::VideoAudio { video_path, decoded_for_waveform, decoded_frames,
|
||||
total_frames }` (near-copy of `Compressed`); `data()` empty, playback via `read_ahead`. Pool
|
||||
entry `path` = the video file.
|
||||
4. **Engine API:** `EngineController::add_video_audio_sync(video_path) -> usize` — ffmpeg-probe
|
||||
the audio track (rate/channels/frames/duration, no decode), build the pool entry, return index.
|
||||
5. **Clip activation:** extend the Phase 1a `AddAudioClip` wiring — if entry is `VideoAudio`,
|
||||
make the buffer + `ActivateFile{kind:VideoAudio, path:video_path}` + set `clip.read_ahead`.
|
||||
One ffmpeg context + 3 s buffer per active clip instance.
|
||||
6. **Import flow:** `import_video` calls `add_video_audio_sync(video_path)` →
|
||||
`AudioClip::new_sampled`. **Remove** `extract_audio_from_video_to_wav`, the temp-WAV
|
||||
handling, and the now-dead `add_audio_file_sync`. No WAV / `/tmp` / RAM.
|
||||
7. **Save/load:** the `VideoAudio` entry serializes as a path reference to the video (no media
|
||||
bytes — the video is already referenced by its `VideoClip`); reconstruct on load by
|
||||
re-probing. Fixes the stopgap's reload fragility (nothing to persist).
|
||||
8. **Waveform overview:** background ffmpeg pass emitting **downsampled peaks only** (bounded
|
||||
memory) into the existing waveform path — shared with the Phase 1a `decode_progressive`
|
||||
cleanup.
|
||||
|
||||
**Sample accuracy (required — video audio must stay frame-synced with other clips):**
|
||||
Coarse ffmpeg seeks are NOT sufficient. `VideoAudioReader::seek(target_frame)` must:
|
||||
- coarse-seek to a point ≤ target, then **decode-and-discard** to land exactly on
|
||||
`target_frame`, tracking the absolute sample position from decoded-frame PTS (discard whole
|
||||
frames before target; for the frame straddling target, drop its leading samples). After
|
||||
`seek`, `decode_next` yields samples starting at exactly `target_frame`.
|
||||
- This makes frame N of the video-audio pool entry correspond to the exact timeline position,
|
||||
so it mixes sample-aligned with mmap/InMemory clips. Continuous decode advances frame-exact.
|
||||
- *Consistency note:* `CompressedReader` should get the same decode-discard alignment (its
|
||||
current coarse-seek-then-write-at-target can misalign by up to a GOP after a seek). Fold in
|
||||
while here, or at least flag.
|
||||
|
||||
*Model decision (confirmed):* the video's audio stays a **separate, editable `AudioClip`** on
|
||||
an audio track, backed by the `VideoAudio` pool entry — users can move/trim/mute/detach it.
|
||||
|
||||
*Build order:* `VideoAudioReader` + `StreamSource` → pool `VideoAudio` variant →
|
||||
`add_video_audio_sync` + activation → swap `import_video` (remove WAV path) → sample-accurate
|
||||
seek (both readers) → waveform-peaks pass.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Video: bound the caches *(small, isolated)*
|
||||
|
||||
### 2a. Bound `VideoManager.frame_cache`
|
||||
`video.rs:412` — convert the unbounded `HashMap<(Uuid,i64), Arc<VideoFrame>>` to an LRU
|
||||
mirroring the decoder-level cache (`video.rs:34`). Frame-count or byte budget.
|
||||
|
||||
### 2b. Stream the export mux
|
||||
`export/mod.rs:388-400` — interleave-write packets to the output as produced (compare PTS,
|
||||
write the earlier stream) instead of collecting all then writing. O(duration) → O(1).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Raster: disk-backed keyframe paging *(the heavy one)* **[locked design]**
|
||||
|
||||
Today `load_beam_sqlite` (`file_io.rs:564`) eagerly `decode_png`s **every** raster keyframe's
|
||||
`Raster` media row into `RasterKeyframe.raw_pixels` (`raster_layer.rs:115`, `w·h·4` ≈ 8 MB @
|
||||
1080p, `#[serde(skip)]`), never evicts, has an unbounded GPU texture cache, and holds full-frame
|
||||
undo snapshots. `raw_pixels` is the working rep (edits write it, save reads it, render reads it),
|
||||
`has_pixels()` = `!raw_pixels.is_empty()`, `keyframe_at` is a `partition_point` binary search, and
|
||||
the container is opened only at load/save (no live handle).
|
||||
|
||||
**Design (confirmed with user):** keep `raw_pixels` as the working rep; make residency explicit
|
||||
via a `RasterStore` + an editor-run fault-in/evict pass *before* the immutable render. Async
|
||||
fault-in (no scrub hitch), with a **low-res image proxy** shown until the full frame lands.
|
||||
Decisions: small window (±~2 keyframes); **dirty (edited-unsaved) frames stay fully resident**
|
||||
(spill-to-scratch deferred); fault-in is **async**; proxy is a **per-keyframe low-res RGBA image**
|
||||
(PNG/WebP, correct alpha), NOT a video (VP9-alpha was rejected as finicky for negligible disk win).
|
||||
|
||||
### Drive-by (Arc pixels): DROPPED
|
||||
Investigated and rejected: `raw_pixels` has ~64 access sites, and most `.clone()`s genuinely need
|
||||
an owned `Vec<u8>` (undo buffers, export, GPU readback) so `Arc<Vec<u8>>` would force `(*p).clone()`
|
||||
and still copy. The only beneficiary, the per-frame `renderer.rs:550` Vello clone, is on the
|
||||
**legacy/dead** path — the live HDR canvas renders raster as `RenderedLayerType::Raster` → GPU
|
||||
upload in `stage.rs` which passes a `&[u8]` slice and uploads only on cache-miss (no per-frame
|
||||
clone). Not worth 64 edits. Start at 3a.
|
||||
|
||||
### 3a. Lazy async fault-in + image proxy
|
||||
- **[DONE 3a-1]** Lazy load: full-decode removed; `raw_pixels` empty on load, `needs_fault_in`
|
||||
armed recursively; canvas records misses → App pages in via `RasterStore.load_pixels`.
|
||||
- **[DONE 3a-2]** Async: page-in runs on a background thread (deduped via `raster_loads_inflight`);
|
||||
results applied at top of `update()`. No UI block on cold scrub.
|
||||
- **[TODO 3a-3]** Image proxy (below) to remove the brief blank gap before a full PNG lands.
|
||||
|
||||
- **`RasterStore`** (core): current `.beam` path + a read-only connection; `load_pixels(kf_id,w,h)`
|
||||
reads the `Raster` row and `decode_png`s it. Set/cleared by the editor on load + save-as.
|
||||
- **Save:** alongside the full PNG, write a low-res RGBA proxy per resident keyframe
|
||||
(`MediaKind::RasterProxy`, ≤~480px long edge, keyed by `kf.id`).
|
||||
- **Load:** stop eager full-decode; decode **proxies** eagerly (cheap → instant scrub everywhere);
|
||||
leave full `raw_pixels` empty.
|
||||
- **Fault-in pass** (editor, `&mut document` + store, each frame before render): for each raster
|
||||
layer ensure the active keyframe ±N is requested; load full PNGs on a **background thread pool**;
|
||||
on arrival, set `raw_pixels` + `texture_dirty`. Render uses full `raw_pixels` if resident, else the
|
||||
upscaled proxy. Reused by the exporter (already frame-by-frame).
|
||||
|
||||
### 3b. Residency window + eviction **[DONE]**
|
||||
- Added `#[serde(skip)] dirty: bool` (edited-since-persist; distinct from `texture_dirty`). Set on
|
||||
stroke/fill/paint-bucket/floating-lift commits + undo/redo; cleared on save (which re-arms the LRU).
|
||||
- Implemented as a fault-in-recency **LRU** (`RASTER_RESIDENT_MAX = 12`), not a strict ±N window:
|
||||
evict the oldest **clean** frame (drop `raw_pixels`, re-arm `needs_fault_in`); the shown frame is
|
||||
always most-recent so it's protected; **dirty frames never evicted**. Save preserves evicted frames'
|
||||
rows via `media_exists` (no data loss) and walks all layers to match load.
|
||||
*(Refinement deferred: count budget → byte budget for 4K resolution-robustness.)*
|
||||
|
||||
### 3c. Bound the GPU cache
|
||||
`raster_layer_cache` (`gpu_brush.rs:1051`, `HashMap<Uuid,CanvasPair>`, Rgba16Float ping-pong
|
||||
≈ `w·h·16`/entry, **unbounded**) → evict textures for keyframes outside every layer's window
|
||||
(drive from the same pass) / LRU byte budget. Export `raster_cache` (`video_exporter.rs:85`) lives
|
||||
one export — bound trivially. (Vello `ImageCache` is image *assets* → Phase 4.)
|
||||
|
||||
### 3d. Undo memory
|
||||
`RasterStrokeAction`/`RasterFillAction` hold `buffer_before`+`buffer_after` full frames
|
||||
(`raster_stroke.rs:20`). Switch to **dirty-rect diffs** (store only the changed bbox before/after;
|
||||
full-canvas fills compressed). Independent of 3a–3c.
|
||||
|
||||
### Build order & tests
|
||||
1. Arc drive-by — COW make_mut test. 2. 3a fault-in + store + proxy — load→empty-until-faulted,
|
||||
PNG round-trip, proxy-then-swap. 3. 3b window/evict/dirty — residency ≤ window while scrubbing,
|
||||
dirty never evicted. 4. 3c GPU bound. 5. 3d undo diffs reproduce pre-stroke buffer exactly.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Asset paging by usage + LRU *(vector's real cost is assets, not geometry)*
|
||||
|
||||
Vector geometry is compact flat POD (tens of KB/frame, no cached tessellation/DCEL) — leave
|
||||
it resident. The heavy, evictable thing is the **image assets** referenced by fills.
|
||||
|
||||
**Data model.**
|
||||
- `ImageAsset` (`clip.rs:250`): `path: PathBuf` + `data: Option<Vec<u8>>` (whole compressed
|
||||
file bytes) + dims. Imported fully into `data` at `main.rs:3936`.
|
||||
- All assets resident in `Document.image_assets: HashMap<Uuid, ImageAsset>` (`document.rs:206`).
|
||||
- Decoded form in `ImageCache` (`renderer.rs:25`): `HashMap<Uuid, Arc<ImageBrush>>` + CPU
|
||||
`Pixmap` map, keyed by asset id, **unbounded**.
|
||||
- A `Fill` references an asset by `image_fill: Option<Uuid>` (`vector_graph/mod.rs:110`).
|
||||
Same UUID may appear in many fills/keyframes/layers and recursively through clip instances.
|
||||
**No asset→frame or frame→asset index exists today.**
|
||||
|
||||
**Two evictable tiers:** Tier 1 = compressed bytes (`ImageAsset.data`, droppable, reload
|
||||
from blob row or external `path`); Tier 2 = decoded pixels (`ImageCache` + GPU textures —
|
||||
the heavy one).
|
||||
|
||||
### 4a. Frame→asset enumeration (incl. nested clips — see note below)
|
||||
A function `assets_needed_at(time) -> HashSet<Uuid>`: walk each visible vector layer's active
|
||||
`ShapeKeyframe`, collect `fill.image_fill` across its `VectorGraph.fills`, **recursing into
|
||||
clip instances** with the outer→inner local-time mapping. This is "needed now". Scanning
|
||||
upcoming keyframes (and upcoming nested-clip keyframes) gives "needed soon" for prefetch.
|
||||
|
||||
### 4b. Usage bookkeeping (the multi-frame problem)
|
||||
Maintain a reverse index `asset_id → usage count` (fills referencing it across the whole
|
||||
document), updated incrementally as edits add/remove `image_fill`s (hook the fill-mutation
|
||||
paths in `vector_graph` and the relevant actions).
|
||||
- count 0 → dead, fully evictable / GC candidate.
|
||||
- count > 0 → keep metadata; residency of `data`/decoded pixels driven by **proximity to
|
||||
playhead**, not by count (a high-count asset far from the playhead is still evicted).
|
||||
|
||||
Residency decision: `resident = needed-now ∪ needed-soon`; beyond that, an **LRU with a byte
|
||||
budget** for referenced-but-distant assets (covers scrubbing back without a reload).
|
||||
Eviction never touches an asset in needed-now.
|
||||
|
||||
### 4c. Bound the decoded tier
|
||||
Convert `ImageCache`'s two maps to LRU/byte-budgeted (`renderer.rs:25`) and bound the GPU
|
||||
image-texture cache the same way, keyed to the residency window.
|
||||
|
||||
### Nested-clip prefetch (important)
|
||||
A clip instance placed on an outer frame has its **own internal timeline of keyframes**,
|
||||
each of which can reference its own image assets. Prefetch must therefore:
|
||||
- Recurse through clip instances when computing both needed-now and needed-soon.
|
||||
- Map outer playhead time → each nested clip's local time, and look ahead along the
|
||||
**nested** timeline (not just the outer one) so assets used by an upcoming *inner*
|
||||
keyframe are loaded before the nested clip reaches it.
|
||||
- Deduplicate across the whole recursion (an asset shared by outer and inner frames counts
|
||||
once); the usage index handles refcounting.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting: a shared residency abstraction
|
||||
|
||||
A generic **`PagedStore<Id, Payload>`** with three consumers — always-resident metadata,
|
||||
disk backing, residency = window/needed-set around playhead + LRU byte budget:
|
||||
|
||||
| Consumer | Metadata kept | Paged payload | Backing | "Needed now" key |
|
||||
|---|---|---|---|---|
|
||||
| Raster keyframes (Ph 3) | id, dims, time | `raw_pixels` + GPU texture | SQLite blob row (`UPDATE` on write-back) | active keyframe per layer |
|
||||
| Image assets (Ph 4) | id, dims, storage | `data` bytes + decoded pixels/texture | SQLite blob row or external path | fills' `image_fill` set at time (recursive) |
|
||||
| Video frames (Ph 2a) | — | RGBA frame | source via ffmpeg seek | requested timestamps |
|
||||
|
||||
Audio stays separate (real-time ring buffer, different constraints). The frame→asset
|
||||
enumeration + usage index is unique to Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing
|
||||
1. **Phase 1a** — done; independent of the container, works with the current ZIP loader.
|
||||
2. **Phase 2** — small, isolated, independently shippable; container-independent.
|
||||
3. **Phase 0 (container)** — `.beam` ZIP → SQLite + `BlobReader` + large-media policy +
|
||||
legacy-ZIP migration. Prerequisite for 1b/1c/3/4.
|
||||
4. **Phase 1b** — streaming pool loader + bulk read-ahead activation (on the SQLite store).
|
||||
5. **Phase 1c** — depends on 1b's pool path.
|
||||
6. **Phase 3** — the substantial build; implement `PagedStore` over blob rows.
|
||||
7. **Phase 4** — thin layer on the same abstraction + the frame→asset/usage index.
|
||||
|
||||
Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the container).
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
- [~] Phase 1a — activate compressed-audio disk reader ← **in progress**
|
||||
- [x] Wire `ActivateFile` + assign `clip.read_ahead` on `AddAudioClip` for compressed
|
||||
pool files (`engine.rs:909`). Per-clip reader keyed by `clip_id`; matches the
|
||||
existing `DeactivateFile` convention in `RemoveAudioClip`. Compiles clean.
|
||||
- [ ] Stop `decode_progressive` (`io/audio_file.rs:344`) from accumulating/streaming the
|
||||
full PCM; emit only the downsampled waveform overview. (Crosses into the UI
|
||||
waveform pipeline — `AudioDecodeProgress` consumer — so handled as its own step.)
|
||||
- [ ] Runtime verification: confirm a compressed clip actually plays from the ring
|
||||
buffer (was effectively silent before, since `read_ahead` was always `None`).
|
||||
- [~] **Phase 0 — container migration `.beam` ZIP → SQLite** ← **in progress**
|
||||
- [x] SQLite schema (`media`, `media_chunk`, `project_json`, `meta`) + `rusqlite` dep
|
||||
(bundled) — `lightningbeam-core/src/beam_archive.rs`
|
||||
- [x] `BlobReader` (`Read + Seek` over `media_chunk`, owns its own read-only connection,
|
||||
opens a blob handle per read with rowids resolved once) — for `CompressedReader` /
|
||||
video decoder in 1b. 5 integration tests pass (`tests/beam_archive.rs`): json
|
||||
round-trip, packed full read, streaming reads + seeks across chunk boundaries,
|
||||
referenced-path, overwrite-replaces-chunks.
|
||||
- [x] Packed (chunked) + referenced media write/read API; `is_sqlite()` format detection;
|
||||
`MediaKind`/`MediaStorage`/`MediaMeta`/`MediaInfo`.
|
||||
- [x] `BeamArchive::transaction()` / `BeamTxn` — in-place transactional save (only
|
||||
changed rows written; unchanged large media never rewritten); orphan cleanup via
|
||||
`retain_media`. 7 archive tests pass (added txn-grouping + rollback). Per user: save
|
||||
must NOT copy+rename for existing SQLite files.
|
||||
- [x] Wire `save_beam` to `BeamArchive` — in-place txn for existing SQLite, temp+rename
|
||||
only for new/migrated files. Audio → packed (or referenced ≥2GB) `media` rows;
|
||||
raster → PNG `media` rows keyed by keyframe id. FLAC→WAV→base64 save round-trip
|
||||
deleted (now packs original bytes with their codec).
|
||||
- [x] Wire `load_beam` — format dispatch: SQLite (`load_beam_sqlite`) vs legacy ZIP
|
||||
(`load_beam_zip_legacy`, kept verbatim). SQLite load reconstitutes packed audio into
|
||||
`embedded_data` so the existing pool loader is unchanged (streaming = Phase 1b).
|
||||
- [x] Legacy ZIP `.beam` → SQLite migration: `is_sqlite()` routes load; saving a
|
||||
ZIP-loaded project writes SQLite (migrates on save). Editor compiles end-to-end.
|
||||
- [x] Large-media policy: packed (chunked) vs referenced — `LargeMediaMode {Ask,Pack,
|
||||
Reference}`; save honors it for files ≥`LARGE_MEDIA_THRESHOLD`. Packing streams from
|
||||
disk via `put_media_packed_from_path` (chunk-by-chunk, never loads the whole file).
|
||||
`Ask` behaves as `Reference` at save time.
|
||||
- [x] `large_media_default` user preference: persisted in `AppConfig`, editable in
|
||||
Preferences → Advanced (incl. resetting to `Ask` to re-trigger the prompt).
|
||||
- [x] First-import-over-threshold prompt: `note_possible_large_media` (hooked into
|
||||
import_audio/video/image) queues a one-time modal; choice persists to config.
|
||||
Threshold shown in the modal is derived from the constant.
|
||||
- [ ] Runtime verification: save a real project, reopen it, confirm audio + raster survive
|
||||
round-trip; confirm an old ZIP `.beam` still opens and migrates on save.
|
||||
- [ ] (Optimization, later) FLAC-compress packed PCM/WAV audio; raster disk-dirty flag to
|
||||
skip unchanged frames on in-place save (Phase 3).
|
||||
|
||||
> Note: the crate's internal `#[cfg(test)]` modules (`clip.rs`, `effect_layer.rs`) have
|
||||
> pre-existing compile breakage (old `Beats`/`TempoMap` API) unrelated to this work; it
|
||||
> blocks `cargo test --lib`, so `beam_archive` tests live in `tests/` (integration) which
|
||||
> build the lib in normal mode. Worth fixing separately.
|
||||
- [x] Phase 1b — stream on project load (PACKED audio path complete & user-verified: streams on load,
|
||||
waveform generates + persists, sample-accurate seeking). Referenced-path streaming + MP3 seek index
|
||||
+ proper video-audio reload remain as noted follow-ups.
|
||||
- **Decision (user):** cross-crate packed streaming via an **inversion-of-control factory** —
|
||||
daw-backend defines the interface, core implements it over `BlobReader`. Keeps the audio
|
||||
engine container-agnostic. (Alternatives rejected: daw-backend owning rusqlite = layering
|
||||
violation; referenced-only-first = leaves packed <2GB in RAM.)
|
||||
- **Current load reality (why this is needed):** *nothing* streams on load today — every entry
|
||||
is fully decoded to a PCM `Vec<f32>`. Packed audio is base64-reconstituted into `embedded_data`
|
||||
(`load_beam_sqlite`) → written to a temp file → `load_file_into_pool` full-decodes; referenced
|
||||
audio also full-decodes via `load_file_into_pool`; and the Phase 1a/1c disk-reader activation
|
||||
never fires for loaded clips (they bypass `AddAudioClip`).
|
||||
- [x] **B1/B2 foundation (DONE, headless-tested):** in `disk_reader.rs` — `trait MediaByteSource:
|
||||
Read+Seek+Send+Sync { byte_len }` + `trait AudioBlobSourceFactory: Send+Sync { open(media_id)
|
||||
-> Box<dyn MediaByteSource> }`; `SymphoniaByteSource` adapter (impl `MediaSource`,
|
||||
is_seekable/byte_len); `CompressedReader::open_source(src, ext)` sharing probe via a
|
||||
refactored `from_mss`; `enum StreamOpen { Path, Source{src,ext} }`; `StreamSource::open` and
|
||||
`DiskReaderCommand::ActivateFile` now take `StreamOpen` (engine site wraps `Path`); re-exported
|
||||
`AudioBlobSourceFactory`/`MediaByteSource` at `daw_backend::audio`. Test
|
||||
`tests/compressed_source_stream.rs` decodes an in-memory WAV through a `Cursor`-backed
|
||||
`MediaByteSource` (proves probe+decode+seek over a byte stream). daw-backend compiles clean.
|
||||
- [x] **B3 (engine, DONE):** `Engine.blob_source_factory: Option<Arc<dyn AudioBlobSourceFactory>>` +
|
||||
`EngineController::set_blob_source_factory` (via `Query::SetBlobSourceFactory`, ordered before
|
||||
`SetProject` on the same queue). `AudioFile.packed_media_id: Option<String>` (Some ⇒ open via
|
||||
factory using `original_format` as the ext hint; None ⇒ `StreamOpen::Path`). Activation factored
|
||||
into `Engine::activate_streaming_for(reader_id, pool_index)`, used by `AddAudioClip` and bulk.
|
||||
- [x] **C (core factory, DONE):** `file_io::blob_source_factory(beam_path)` → `BeamBlobFactory`
|
||||
implementing `AudioBlobSourceFactory` over `BeamArchive::open_blob_reader`. `BlobReader` holds a
|
||||
`!Sync` rusqlite `Connection`, so it's wrapped in `SyncBlobReader` (a `Mutex` used via `get_mut`
|
||||
on the hot path — no runtime locking) to satisfy Symphonia's `MediaSource: Send + Sync`. Installed
|
||||
by the editor between `load_audio_pool` and `set_project`.
|
||||
- [x] **D (load-path, DONE — packed audio):** `load_beam_sqlite` now streams packed audio whose codec
|
||||
is recognized (`is_streamable_audio_codec`) — leaves `embedded_data` empty so the pool builds a
|
||||
Compressed placeholder with `packed_media_id`; no base64, no temp file, no decode. `serialize`
|
||||
round-trips packed entries by media id (so in-place re-save keeps the row). Non-audio codecs
|
||||
(video-container audio tracks) keep the legacy reconstitution path → **no regression**.
|
||||
- [x] **E (bulk activation, DONE):** `SetProject` calls `Engine::activate_all_streaming_clips` —
|
||||
walks every loaded audio clip and `activate_streaming_for` (create_buffer + `ActivateFile` + set
|
||||
`read_ahead`), the loaded-clip equivalent of the Phase 1a wiring.
|
||||
- [x] **Waveform-on-load for streamed audio (DONE):** streaming broke the old waveform path (it came
|
||||
from the full in-RAM decode, which no longer happens). Added
|
||||
`disk_reader::build_waveform_pyramid_from_source(Box<dyn MediaByteSource>, ext, B)` (load-time
|
||||
counterpart of the path-based builder). On load, the editor background-generates a pyramid for any
|
||||
streamed entry lacking a persisted one (opens the packed blob via a local factory), sending the
|
||||
floor through the same `waveform_result` channel `update()` drains; the next save persists it.
|
||||
Verified in-app: packed MP3 **streams + plays** (`Activated reader=0, kind=CompressedAudio`); the
|
||||
overview now fills in shortly after load.
|
||||
- **Headless tests pass** (compressed_source_stream, video_audio_stream, waveform_pyramid); all three
|
||||
crates compile clean. **Needs in-app verification:** the waveform appears after load (background gen),
|
||||
then instantly on subsequent loads once saved; RAM stays flat on a big project.
|
||||
- [x] **Seek alignment fix (DONE):** streamed compressed audio was ~1.2s off *after seeking*
|
||||
(fine from the start). `CompressedReader::seek` used `SeekMode::Coarse`, which for MP3
|
||||
byte-estimates the position and seeds the timestamp from that estimate — wrong for VBR / files
|
||||
whose header padding the estimate ignores, so `actual_ts` (and thus the buffer's frame labels)
|
||||
landed ~1.2s early. Switched to `SeekMode::Accurate`: Symphonia counts frame *headers* (no
|
||||
decode) from a true anchor (current pos, or rewind-to-0 for backward seeks) → exact `actual_ts`;
|
||||
the existing sub-frame `pending_discard` finishes the job. FLAC/OGG seek cheaply (seek tables);
|
||||
a long MP3 backward seek walks headers from 0 (I/O, not decode). Tests still green.
|
||||
- [ ] **Deferred (follow-up):** per-file **seek index** for elementary streams (MP3) — a one-time
|
||||
header scan (ts↔byte map) to make far seeks O(1) instead of an Accurate header-walk from the
|
||||
anchor. Matters for multi-hour MP3s; song-length files are fine as-is.
|
||||
- [x] **Proper video-audio reload (DONE):** a video's audio is now stored as a **path reference** to
|
||||
the video (never packed/embedded as audio media) and **re-probed via FFmpeg** on load into a
|
||||
streaming `VideoAudio` entry — `AudioPoolEntry.is_video_audio` flag drives both `serialize`
|
||||
(reference, not pack), `save_beam` (`reference_it |= is_video_audio`), and `load_from_serialized`
|
||||
(`VideoAudioReader::open` → `from_video_audio`). Fixes 5.1 audio losing its channels on reload
|
||||
(the old Symphonia reconstitution collapsed it); also no more decode-whole-video-to-RAM / temp
|
||||
files on load. Old saves (video mis-packed as audio) self-heal on the next save.
|
||||
- [ ] **Deferred (follow-up):** stream *referenced* (external-path) **audio** on load too — replace
|
||||
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
|
||||
→ `from_compressed` placeholder). Higher risk (touches the working referenced path); packed
|
||||
covers the common <2GB case first.
|
||||
- [ ] **Deferred (follow-up): packed video streaming.** Let small videos be packed into the `.beam`
|
||||
(a `MediaKind::Video` blob, `VideoClip` referencing it by id) and stream **both frames and audio**
|
||||
from the DB blob via FFmpeg. ffmpeg-next has no custom-I/O wrapper, so this needs an
|
||||
`AVIOContext`-over-`BlobReader` shim via raw FFI. **Decision (user):** that FFI wrapper lives in
|
||||
its **own crate, version-pinned to the ffmpeg version**, isolating the unsafe + the ABI coupling.
|
||||
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
|
||||
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
|
||||
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills
|
||||
small temp partitions) and the temp path doesn't survive reload.
|
||||
- [~] **Proper design** (see "Phase 1c" body): stream the video's audio on demand via a new
|
||||
ffmpeg `VideoAudioReader` in the disk reader — no extraction, no `/tmp`, no RAM; path
|
||||
reference survives save/load.
|
||||
- [x] **Step 1 (DONE):** `VideoAudioReader` (ffmpeg) + `StreamSource` enum + `SourceKind`
|
||||
in `disk_reader.rs`. Sample-accurate seek (coarse seek + decode-discard to exact
|
||||
frame via PTS). 2 integration tests pass (`daw-backend/tests/video_audio_stream.rs`):
|
||||
in-order decode + sample-exact seek at several targets. (Found: mono frames have an
|
||||
empty channel layout → must `set_channel_layout` before resampling, else swr returns
|
||||
AVERROR_INPUT_CHANGED.) Lib compiles clean; `StreamSource` `#[allow(dead_code)]`
|
||||
until wired. `VideoAudioReader` made `pub` for the integration test.
|
||||
- [x] **Step 2 (DONE):** `AudioStorage::VideoAudio { decoded_for_waveform, decoded_frames,
|
||||
total_frames }` + `AudioFile::from_video_audio` (path = the video file). `data()`
|
||||
empty / `read_samples()` 0 (streamed). `Query::AddVideoAudioSync` +
|
||||
`do_add_video_audio` (probes via `VideoAudioReader::open`, no decode) +
|
||||
`EngineController::add_video_audio_sync`. `GetPoolAudioSamples` surfaces VideoAudio's
|
||||
waveform overview too. daw-backend compiles clean; probe `total_frames` test passes.
|
||||
- [x] **Step 3 (DONE):** reader thread now holds `StreamSource` (opens via
|
||||
`StreamSource::open(path, kind)`, dispatches `sample_rate()/channels()/seek/decode_next`);
|
||||
`ActivateFile` carries `kind: SourceKind`; `#[allow(dead_code)]` removed. `AddAudioClip`
|
||||
activation maps `Compressed`→`CompressedAudio`, `VideoAudio`→`VideoAudio`, creates the
|
||||
read-ahead buffer + `ActivateFile{kind}` + sets `clip.read_ahead`. Compressed path is
|
||||
behaviorally identical (StreamSource::Compressed wraps the same CompressedReader).
|
||||
daw-backend + editor compile clean; VideoAudioReader tests still pass.
|
||||
⚠️ Not runtime-verified — needs in-app check that compressed audio still plays (no
|
||||
regression) and that an activated VideoAudio clip produces sound.
|
||||
- [x] **Step 4 (DONE):** `import_video` now calls `add_video_audio_sync(video_path)` →
|
||||
pool index, fetches channels/sample_rate via `get_pool_file_info`, makes the
|
||||
`AudioClip` with the video's duration. **No WAV / /tmp / RAM.** Removed the stopgap
|
||||
(`extract_audio_from_video_to_wav` + WAV helpers + `ExtractedAudioInfo`), dead
|
||||
`add_audio_file_sync` (+ `Query::AddAudioFileSync` / `QueryResponse::AudioFileAddedSync`
|
||||
/ handler), and the now-unreachable `AudioExtractionResult::NoAudio`. Kept
|
||||
`import_audio_sync` (still used by normal audio import). daw-backend + editor clean.
|
||||
**→ Feature is live end-to-end; ready for in-app testing.**
|
||||
- [x] **Step 5 (DONE):** `CompressedReader` now seeks sample-accurately too — coarse
|
||||
symphonia seek + decode-discard (`pending_discard` set from `seeked.actual_ts` in
|
||||
`seek`, applied in `decode_next`, which continues rather than reporting EOF when a
|
||||
whole packet is discarded). So compressed clips no longer drift vs video audio after
|
||||
a seek. Test `compressed_reader_seek_is_sample_accurate` passes (the WAV coarse seek
|
||||
lands pre-target, exercising the discard). `CompressedReader` made `pub` for the test.
|
||||
- [~] Step 6: **bounded waveform overview** — replaces today's full-resolution
|
||||
`raw_audio_cache`/GPU waveform (which doesn't scale: it stores every sample at mip 0,
|
||||
so a long file is multi-GB on GPU + RAM — the same memory issue, and the Phase 1a
|
||||
`decode_progressive` leftover). Design below. Slices: (1a) streaming pyramid builder
|
||||
+ (1b) persistence + (1c) min/max GPU upload, then (2) LRU tile cache + re-decode floor.
|
||||
- [x] **Slice 1a (DONE):** `daw-backend/src/audio/waveform_pyramid.rs` —
|
||||
`WaveformPyramidBuilder` streams interleaved samples, accumulates the floor, and
|
||||
reduces `BRANCH(4):1` at `finish` into a root-first pyramid (convention B:
|
||||
`levels[0]`=root envelope, `levels.last()`=floor, `.root()`/`.floor()` accessors).
|
||||
Ragged last buckets reduce over available children (no value padding). Bounded
|
||||
(~22 MB/2 h @ B=256). 7 integration tests pass (`tests/waveform_pyramid.rs`):
|
||||
bucket min/max, partial flush, multi-level envelope == global min/max, root-first
|
||||
ordering, stereo channels, size bound, chunk-agnostic.
|
||||
- [~] **Slice 1b (data layer DONE; orchestration folded into 1c):**
|
||||
- [x] Generation bridge `disk_reader::build_waveform_pyramid(path, kind, B)` — streams
|
||||
a decode (`StreamSource` over symphonia/ffmpeg) into the builder; bounded
|
||||
memory (one chunk + the pyramid). Test: envelope matches the signal through
|
||||
both backends.
|
||||
- [x] Serialization `WaveformPyramid::to_bytes`/`from_bytes` (LBWF blob; f32 texels —
|
||||
f16 a later size optimization). Round-trip test + rejects truncated/garbage.
|
||||
- [x] `MediaKind::Waveform` in the SQLite container (keyed by the audio item's id).
|
||||
- [ ] Orchestration (with 1c).
|
||||
- [~] **Slice 1c (in-memory floor overview DONE; persistence next):**
|
||||
- [x] `waveform_gpu`: `PendingUpload.minmax` flag + `pack_texel` helper; `upload_audio`
|
||||
threads `minmax` (frame_stride 4, packs `(Lmin,Lmax,Rmin,Rmax)` directly).
|
||||
The texture is already Rgba16Float and the GPU mipgen builds zoom-out levels, so
|
||||
only the texel-packing differs. Render the floor at **effective rate `sr/B`** (so
|
||||
time→texel maps B samples/texel) and `total_frames = floor_texel_count`.
|
||||
- [x] `AppConfig.waveform_floor_samples_per_texel` (default 256, user-configurable).
|
||||
- [x] App: `waveform_minmax_pools: HashMap<usize, u32>` (pool → `B`, carries the floor rate
|
||||
with full float precision) + a `(pool, packed_floor, sr, channels, B)` results channel;
|
||||
drained in `update()` → `raw_audio_cache.insert(floor)` + flag pool + `waveform_gpu_dirty`.
|
||||
- [x] Generation: on video-audio import Success, the same bg thread streams
|
||||
`disk_reader::build_waveform_pyramid(path, VideoAudio, B)` once and sends the packed
|
||||
`floor()`. (Video-audio has no in-RAM samples, so this is what makes its waveform appear.)
|
||||
- [x] Threaded `waveform_minmax_pools` through the pane-context (`panes/mod.rs` +
|
||||
main.rs construction) → `render_layers` → **both** render sites (collapsed-group
|
||||
~timeline.rs:3048 AND expanded-track ~3613): compute `total_frames = len/4`,
|
||||
`eff_sr = sr/B`, set `minmax`. Compiles clean (editor `cargo check` = 0 errors).
|
||||
- [x] Shader fix: `waveform.wgsl` now reads the **nearest integer LOD via `textureLoad`**
|
||||
instead of sampling a fractional mip. Trilinear blends two levels whose row-major
|
||||
linearizations differ → horizontal shift that flips each 0.5 of `mip_f` (= each 2x
|
||||
zoom step), the "every other zoom level is offset" artifact. **User-confirmed fixed:**
|
||||
features hold position at every zoom and line up with playback.
|
||||
See memory `waveform-shader-fractional-mip-offset`.
|
||||
- [x] **Persistence (done):** the full pyramid is serialized (`to_bytes`) on generation and
|
||||
kept in `App.waveform_pyramid_blobs`. `save_beam` writes it as a `MediaKind::Waveform`
|
||||
row keyed by a **deterministic id derived from the pool index** (`file_io::waveform_media_id`,
|
||||
"LBWF" sentinel in the high 32 bits) — independent of how the audio bytes are stored, so
|
||||
it works for packed/referenced/video-audio alike, and an in-place re-save reuses the row.
|
||||
Carried in/out via a transient `#[serde(skip)] AudioPoolEntry.waveform_blob` and a
|
||||
`waveform_blobs` field on `FileCommand::Save`. `load_beam_sqlite` reads the row back;
|
||||
the editor restores `raw_audio_cache`/`waveform_minmax_pools`/`waveform_pyramid_blobs`
|
||||
+ flags `waveform_gpu_dirty` after the backend loads the pool (using each entry's
|
||||
`sample_rate` for `eff_sr`, the stored `B` for the rate). No re-decode on load.
|
||||
`register_loaded_videos` only loads frames (not audio), so there is no redundant
|
||||
regeneration to suppress. Compiles clean across all three crates.
|
||||
|
||||
### Waveform LOD pyramid design (step 6)
|
||||
A min/max LOD pyramid (tree of zoom-level textures): fully zoomed out → envelope; fully zoomed
|
||||
in → per-sample; seamless between.
|
||||
|
||||
- **One streaming decode pass** builds the whole pyramid down to a configurable **floor**
|
||||
`B` samples/texel (default 256), via a hierarchical reduction (each sample updates a running
|
||||
per-level min/max accumulator; a filled bucket emits a texel and folds into its parent —
|
||||
`branch` 4:1). Bounded memory: holds only the pyramid (~`N/B·4/3` texels ≈ **~14 MB / 2 h
|
||||
stereo @ B=256**), never the full samples. Full-res (B=1 ≈ 2.7 GB) is the only level NOT
|
||||
stored.
|
||||
- **Persist the pyramid** in the `.beam` SQLite container (a `waveform` media kind; session
|
||||
temp before first save). `B` is stored with it (preference is just the default for new gen).
|
||||
Persistence is load-bearing: it makes mid-zoom a cheap **disk read**, not a re-decode.
|
||||
- **Runtime = LRU tile cache** (GPU textures) loaded from the persisted pyramid on demand.
|
||||
Eviction is **ancestor-closed**: only evict an LRU node with no resident children ("a node is
|
||||
cleared only after its children") — so rendering can always walk up to a resident ancestor;
|
||||
detail sharpens in, never blanks. Root is tiny/hot → effectively pinned for free.
|
||||
- **Re-decode only below the floor** (texel < `B` samples): by then the visible window spans a
|
||||
tiny time range, so decoding it (via the sample-accurate seekable readers from steps 1–5 —
|
||||
the payoff) for true per-sample detail is cheap. This removes the large-span re-decode gap:
|
||||
above the floor it's a disk read; below it the span is already small.
|
||||
- **Why a deep floor (not a coarse cutoff):** a coarse-only pinned set would force the first
|
||||
on-demand level to re-reduce a huge time span per tile. Persisting deep makes every level a
|
||||
disk read; `B` is a size-vs-crossover knob (smaller B = bigger pyramid, cheaper re-decode).
|
||||
- `waveform_gpu` needs a **min/max texel upload** (`Lmin,Lmax,Rmin,Rmax` per texel) instead of
|
||||
min=max-per-sample; the existing compute mipgen still builds the mip chain *within* a tile.
|
||||
|
||||
**Decisions (locked):** branch 4:1; floor `B≈256` samples/texel, **user-configurable**
|
||||
(`AppConfig.waveform_floor_samples_per_texel`, stored per-pyramid); 8192-wide tiles; LRU ~4
|
||||
viewports of fine tiles; persist pyramid in `.beam`.
|
||||
- [x] Video decoder concurrency (movie-length lag/freeze): keyframe-index scan now runs
|
||||
holding no VideoManager/decoder lock (brief locks only bracket it) → no more multi-second
|
||||
UI freeze on import/load; thumbnail generation uses a **dedicated** decoder and samples
|
||||
at keyframes (≈1 frame each vs whole-GOP) → no playback contention. Removed dead
|
||||
`VideoManager::build_keyframe_index`, `build_and_set_keyframe_index`, `downsample_rgba*`.
|
||||
- [x] Phase 2a — bound video frame cache. `VideoManager.frame_cache` (was an unbounded
|
||||
`HashMap<(Uuid,i64), Arc<VideoFrame>>` that grew per distinct frame during playback) is now an
|
||||
`LruCache` evicted by a **byte budget** (`FRAME_CACHE_BYTE_BUDGET` = 256 MB) rather than a frame
|
||||
count — robust across resolutions (a 4K frame is ~33 MB vs ~2 MB at 800×600). Byte total tracked
|
||||
on insert/evict/remove; `unload_video` pops per-clip keys (LruCache has no `retain`). Decoder-level
|
||||
cache was already LRU. Editor compiles clean. *(Not yet runtime-verified.)*
|
||||
- [x] Phase 2b — stream export mux. `export/mod.rs::mux_video_and_audio` no longer collects every
|
||||
packet into two `Vec`s before interleaving; it stream-merges the two inputs by PTS with one pending
|
||||
packet per stream (O(1) memory vs O(duration)). Same tie-break (`v_us <= a_us`) and drain-on-EOF
|
||||
behavior; output is byte-identical. Editor compiles clean. *(Not yet runtime-verified — needs an
|
||||
in-app export to confirm A/V sync.)*
|
||||
- [ ] Phase 3a — lazy raster fault-in from blob store
|
||||
- [ ] Phase 3b — raster residency window + eviction
|
||||
- [ ] Phase 3c — bound raster GPU/CPU caches
|
||||
- [ ] Phase 3d — spill undo snapshots
|
||||
- [ ] Phase 4a — frame→asset enumeration (recursive)
|
||||
- [ ] Phase 4b — usage bookkeeping + LRU residency
|
||||
- [ ] Phase 4c — bound decoded image tier
|
||||
- [x] Phase 5 — fixed the broken `#[cfg(test)]` unit tests; **`cargo test --lib` green again**
|
||||
(daw-backend 17 passed, lightningbeam-core 264 passed). Wrapped stale raw-`f64` time literals
|
||||
in `Beats(...)` / passed `&TempoMap` to changed signatures (automation.rs, clip.rs,
|
||||
effect_layer.rs); fixed stale test setup (register a vector clip so `get_clip_duration` resolves)
|
||||
and a stale default expectation (shape `fill_color` defaults `None`). Surfaced + fixed one **real
|
||||
undo bug**: `DeleteFolderAction(MoveToParent)` reparented child subfolders but never restored them
|
||||
on rollback (orphaned them) — now tracked and restored. Production code otherwise untouched.
|
||||
Loading…
Reference in New Issue