Compare commits

..

77 Commits

Author SHA1 Message Date
Skyler Lehmkuhl bb6b6fa9e3 Mobile: address code-review findings
Fixes from a review of the mobile-UI branch.

- Guard the inspector size clamps so min<=max (f32::clamp panics otherwise) and set a
  window min_inner_size; prevents a crash / degenerate layout at small sizes. Same
  small-size guards for the piano-roll landscape toggle bar and the intent grid.
- Landscape 2->1 edge reveal: snap to fullscreen at frac>=0.5 (the preview's phase
  boundary) instead of COLLAPSE_HI, so releasing mid-drag no longer jumps backward.
  Both edges.
- Gate the double-tap-drag marquee to mobile so it can't hijack a desktop Brush/Draw
  drag.
- Timeline mobile long-press: request a repaint while counting down so it fires even
  when the finger is held perfectly still.
- Landscape folded top bar: lay the filename+search+overflow cluster out within a
  reserved center span (gutters for the pane's own label/buttons) and elide the
  filename, so it can't overlap or overflow the header.
- Node editor connect/disconnect: assert + document the top-level-graph assumption.
- Palette search field focuses only when unfocused (no per-frame focus stomp); reset
  the inspector tap-anchor on orientation change; remove stale main.rs.backup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 01:55:40 -04:00
Skyler Lehmkuhl 9646eef487 Mobile: hide the desktop menu bar
Gate the top menu bar on !mobile_active(). Its commands are all reachable from the
mobile shell (⌕ command palette, ⋯ overflow, and the stage/timeline context menus),
so the desktop menu bar is redundant on mobile and just wastes a strip of vertical
space. Desktop unchanged.
2026-07-03 01:08:56 -04:00
Skyler Lehmkuhl 6f67ddb709 Mobile P7: landscape / orientation support
Make the phone shell orientation-aware (portrait vs landscape). Orientation is
aspect-based (rotation = a resize the shell re-lays-out each frame); LB_MOBILE_UI=2
opens a landscape dev window. `SharedPaneState.is_portrait` is published to panes.

- Stack caps at 2 panes in landscape (max_panes through the R1–R6 drag ops), with a
  per-frame clamp so rotating while 3 are open drops to 2.
- Edge drags are a continuous full-height reveal instead of stalling at the capped
  trigger: 1-pane → split → next-pane-fullscreen, and (landscape 2-pane) a two-phase
  slide-then-collapse to 1 pane. Both top and bottom edges, mirrored. Release snaps to
  stay / split / fullscreen. Dividers resize and collapse-to-1 at the extremes.
- Inspector becomes a right-side vertical column in landscape (inspector_width_frac,
  left-edge horizontal resize); portrait keeps the bottom sheet.
- Piano roll: vertical "Synthesia" view in portrait; landscape shows a Keys/Notes
  segmented toggle over the conventional horizontal roll / full keyboard.
- Start screen reflows to 2 rows × 3 with the recent list beside it in landscape.
- Landscape headers are shorter (52→34px) and the top bar folds into the middle of the
  top pane header (no separate band), reclaiming vertical space.

All gated to mobile; desktop unchanged (is_portrait defaults true).
2026-07-03 01:07:58 -04:00
Skyler Lehmkuhl 2cbc35f181 Mobile context-menu coverage + z-order
Hook up menu-bar actions that lacked a mobile affordance, and implement z-order.

- Stage/object long-press menu (stage.rs): add Paste (on clip, geometry, and
  empty stage) and Send to Back / Bring to Front on the selection.
- Timeline long-press menu (timeline.rs): the timeline had no mobile context
  menu — add one via manual long-press detection. Clip actions (Split,
  Duplicate, Cut, Copy, Paste, Delete) on a clip; animation actions (New/Blank
  keyframe, Add keyframe at playhead, Duplicate keyframe, Delete frame, Add
  motion/shape tween) on an empty lane. Gate the desktop right-click menu to
  !is_mobile so mobile shows only the long-press menu.
- Implement Send to Back / Bring to Front (were // TODO no-ops): new undoable
  ReorderClipInstancesAction in lightningbeam-core reorders the selected
  instances within their layer's clip_instances Vec (stacking order; last =
  on top; geometry stays flattened underneath). Wired via handle_menu_action.
2026-07-02 10:07:53 -04:00
Skyler Lehmkuhl 4f66ddb515 P6b: mobile node editor — Focus & Patch views
Replace the desktop node-graph canvas with touch-native views on mobile
(wireframe Plate 07), gated on is_mobile in NodeGraphPane::render_content; the
desktop draw_graph_editor path is untouched. New submodule
panes/node_graph/mobile.rs.

Focus view:
- The focused module's parameters as touch controls (slider / dropdown / field
  by the desktop widget rule, with a visible slider rail); editing mutates the
  graph ValueType so the existing check_parameter_changes dispatches
  SetParameterAction.
- A full-width minimap strip (tap the nearest node to focus it), travel chips
  naming connected endpoints (tap to jump), and an Add-node picker that creates
  the frontend node + AddNodeAction and focuses it.
- Bespoke nodes (Sampler/Script/Sequencer/AmpSim/Oscilloscope) render their
  existing desktop bottom_ui; sampler/script-sample loads are wired (other
  custom interactions are a case-by-case follow-up).

Patch view:
- Per-section egui::Grid so input/output port arrows align in a column. Each
  port's Lucide direction arrow (from-line/to-line, tinted by DataType) is
  clickable to arm a cable; a compatibility-filtered picker completes it
  (ConnectAction). Cable chips (remote-node · remote-port) tap to disconnect
  (DisconnectAction). Parallel cables allowed.

Also: add ARROW_RIGHT_FROM_LINE / ARROW_RIGHT_TO_LINE to mobile/icons.rs
(codepoints extracted from lucide.ttf via ttx).
2026-07-02 09:26:11 -04:00
Skyler Lehmkuhl b14972f657 P6a: mobile music surface (keyboard-primary instrument pane)
The mobile Piano Roll becomes a unified, keyboard-primary instrument surface
(wireframe Plate 04/08): a playable keyboard as the base, revealing a
Synthesia-style falling-notes roll above it when the pane is tall enough.

- Shared keyboard geometry (panes/keyboard_layout.rs): width-driven, pan-aware
  pitch->x, so the roll's note columns stay aligned with the Virtual Piano keys.
  Shared keyboard_octave + keyboard_pan_x in SharedPaneState.
- Piano Roll vertical mode (is_mobile): notes as columns falling toward an amber
  now-line by the keys, tempo-map converted (beats->seconds) so onsets cross the
  line exactly when they sound. Vertical drag scrubs the timeline; horizontal
  drag smoothly scrolls the keys (snaps to nearest key on release). Long-press
  creates a note (drag to size) or resizes an existing one; pan suppresses it.
  The keyboard is embedded (reuses Virtual Piano render + MIDI); the standalone
  VirtualPiano stack slot is removed. show_roll is driven by the snapped pane
  size-class so the keyboard<->roll reveal lands on a stack snap point.
- Virtual Piano: renders via the shared layout on mobile, colors playback_notes
  like pressed keys, gates note-on/glissando to presses that start on the keys
  (never gating release), and hides QWERTY hints on mobile.
- Transport formats by document.timeline_mode (Measures->bar.beat.tick,
  Frames->MM:SS:FF, Seconds->MM:SS.mmm) — per project type, like desktop.
- In-pane instrument header (name + Presets + REC). Recording is driven from the
  app each frame (not the Timeline pane's render) so REC works regardless of
  visible panes, and stopping playback stops recording.
- Compose/Record intent opens Timeline + instrument pane; mobile central panel is
  full-bleed (no inner margin) so panes sit flush.
- phone-ui-sketches.html: inst-bar moved to the top of the music surface.
2026-07-02 07:50:37 -04:00
Skyler Lehmkuhl 77eb2c2d33 Fix oscillator/synth phase wraparound drift
Replace conditional-subtraction phase wrapping with a single stable wrap:
- OscillatorNode: `(phase + freq_mod/sr).rem_euclid(1.0)` — also wraps correctly
  when FM drives freq_mod negative (the old `if >= 1.0` wouldn't).
- SynthVoice: `(phase + frequency/sr).fract()`.
Repeated `if phase >= 1.0 { phase -= 1.0 }` accumulates f32 rounding error over
long-held notes, drifting the timbre.
2026-07-02 01:15:45 -04:00
Skyler Lehmkuhl 4f58edf436 P5 follow-ups: double-tap priority, inspector dismiss, breadcrumb path
- Mobile double-tap is now a single tool-independent priority chain: object →
  enter (movie clip or group); empty inside a clip → exit one level; empty at
  root → zoom-to-fit. Desktop keeps its own select-tool double-click (gated).
- Inspector "tap outside to dismiss" now only HIDES the sheet (with a dismissed
  flag reset on selection change) instead of clearing the selection — the earlier
  clear clobbered the selection before menu/omnibutton actions ran, breaking
  Group/Convert/Cut/Copy. Delete also clears focus so it dismisses the inspector.
- Breadcrumb shows the full editing path (Scene 1 › … › current) via a new
  SharedPaneState.editing_clip_path / EditingContext::clip_path(); each ancestor
  crumb jumps to that level (pending_exit_to_depth → EditingContext::exit_to_depth).
- Remove the "Vello Stage" debug overlay; move the breadcrumb up to the top-left.
2026-07-02 01:14:38 -04:00
Skyler Lehmkuhl 01b20165cc P5 mobile gestures + long-press context menus
Gestures (Stage + Timeline):
- Pinch-zoom via zoom_delta() (touch pinch + Ctrl+wheel), unified with the raw-wheel
  path so Ctrl+wheel zooms exactly once; plain wheel zoom + trackpad pan preserved.
- Double-tap empty → zoom-to-fit (Stage: artboard via zoom_to_fit; Timeline: new
  fit_to_project over the full duration).
- Double-tap-drag on empty → transient marquee regardless of the active tool.

Long-press context menu: a shared SharedPaneState.mobile_context_menu that panes
populate with (label, MenuAction) items on secondary_clicked(); the mobile shell
renders one persistent popup (styled like the timeline menu) and dispatches via
pending_menu_actions. Stage offers Cut/Copy/Duplicate/Delete for clip instances and
Cut/Copy/Convert-to-movie-clip/Delete for geometry.

Fixes surfaced along the way:
- Geometry delete now frees selected fills (free_fill) and GCs isolated vertices
  (new VectorGraph::gc_isolated_vertices) — no more orphaned fills / phantom snap
  points; applies to the Delete key too.
- clipboard_delete_selection clears focus so deleting dismisses the mobile inspector.
- Mobile inspector: appears on pointer release (not press) so drags aren't
  interrupted; reflows only when the tapped point is actually behind the sheet; taps
  outside the sheet dismiss it; rounded-corner border no longer detaches.
2026-07-02 00:11:49 -04:00
Skyler Lehmkuhl 4b2802076b Touch-friendly egui widgets on mobile; egui-based inspector header
Add mobile::apply_touch_style(ctx) — applied every frame when mobile is active —
enlarging egui's global spacing/sizing (interact_size, button padding, slider
width/rail, combo width/height, icon sizes, scrollbar, body/button text) so the
standard widgets in panes and dialogs are finger-sized. Desktop is untouched.

Rework the mobile inspector header into a single row using real egui widgets:
title (left, truncating) + Timeline/Nodes jump buttons + ✕ close, laid out with
egui layouts so they inherit the touch sizing and theme visuals instead of being
hand-painted.
2026-07-01 07:28:13 -04:00
Skyler Lehmkuhl 943ff7c15f Unify colors via theme CSS variables; theme egui visuals
Add Theme::var(name, ctx) (CSS custom-property getter) and Theme::apply_to_egui(ctx)
which maps the palette onto egui's global Visuals so standard widgets share the
theme colors. Add a mobile::palette::Palette built from the theme each frame and
replace the duplicated per-file C_* color constants across all mobile modules
(omni, topbar, stack, inspector, transport, intent, shell) with it. Add --scrim
and --accent-* category vars to styles.css. Apply the theme visuals every frame in
update(). Mobile and the main UI now share one CSS-variable-driven palette that
responds to light/dark/user themes.
2026-07-01 07:08:00 -04:00
Skyler Lehmkuhl f13a127c9d Fit modal dialogs to the phone screen on mobile
Add mobile::dialog_width(ctx, desired) — a min()-clamp that's a no-op on wide
desktop screens. Clamp the Export and Export Progress modals and the Sample
Import window to it. Render Preferences as an egui::Modal (dim backdrop, centered,
no title bar) with a screen-fit scroll height when mobile_active(); desktop keeps
its window. Extracted the preferences body into render_body() shared by both paths.
2026-07-01 06:28:04 -04:00
Skyler Lehmkuhl a0ac4f2d35 Add mobile top bar: filename + command palette (⌕) + overflow (⋯)
Reserve a slim top bar in the mobile shell showing the project filename (or
"Lightningbeam" when unsaved) on the left, and ⌕/⋯ on the right. ⋯ opens a
curated overflow sheet (Save/Save As/Open/New/Import/Export/Undo/Redo/
Preferences); ⌕ opens a command palette that searches the whole menu tree
(flattened from MenuItemDef::menu_structure()). Both dispatch MenuActions.
Completes the P4 omnibutton + global menu.
2026-07-01 06:09:26 -04:00
Skyler Lehmkuhl b8419778a5 Add omnibutton FAB (P4); fix Stage playback freeze when timeline hidden
Omnibutton (mobile/omni.rs): a context FAB whose radial holds the active layer's
drawing tools laid out in a staggered inner/outer arc, plus a "+New" petal
(create Vector/Audio/MIDI/Raster/Video layer, Group, Import, and Split/Duplicate
when a clip is selected — dispatched as MenuActions) and a "more" petal that
opens a full-tools grid when the layer has extra tools. Contextual: the tool ring
only appears when the Stage is in the visible window; off-Stage the FAB is a
"+New" create button. Tools/creates use Lucide icons. Pane-internal adds
(nodes/notes/instruments) deferred to P6.

Playback fix: document.current_time (read by the renderer for animation eval and
video-frame decode) was only synced from the audio playback position inside
TimelinePane::render_content. In the mobile shell, panes render on demand, so with
the timeline off-screen current_time went stale and the Stage froze during
playback. Move the sync into the per-frame block in update() so it runs regardless
of which panes are visible.
2026-07-01 05:42:53 -04:00
Skyler Lehmkuhl 08b6aa0139 Compress mobile layer headers to a swatch; conditional inspector reflow
- Add `is_mobile` to SharedPaneState (set in build_frame). On mobile the timeline
  layer-header column collapses to a minimal type-colored swatch (1:2 w:h, ~30px),
  brighter/outlined when active/selected; tapping it selects the layer (handled by
  the existing header-column input path, which now uses the narrower width).
  Desktop headers unchanged.
- Inspector reflow: the sheet overlaps the stack by default, but when the selected
  pane would be covered by the sheet, the stack reflows (renders above the sheet)
  so it stays visible. Restore on dismiss is automatic since reflow only changes
  the render rect; manual divider moves while the sheet is up persist. Adds
  stack::pane_bottom_in and inspector::target_slot.
2026-07-01 01:14:37 -04:00
Skyler Lehmkuhl 87263489c0 Add mobile selection inspector sheet (P3); editable layer properties
- New mobile/inspector.rs: when something is selected/focused, a bottom sheet
  overlays the lower stack (no reflow) above the transport, showing the object's
  properties by reusing InfopanelPane full-bleed. Header shows a title + ✕ to
  deselect; jump-to chips (Timeline, Nodes) reframe the stack to that surface
  with the selection kept. Grab handle resizes the sheet.
- Make the Infopanel Layer section editable (benefits desktop too): Name (text,
  commit on Enter/blur), Opacity and Volume sliders (live preview, single-undo
  on release), and Mute/Solo/Lock toggles. Backs the plan to reduce mobile layer
  headers to a minimal swatch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 22:16:14 -04:00
Skyler Lehmkuhl f2cb516e8a Add mobile new-file intent picker (P2)
When LB_MOBILE_UI is set, the desktop start screen is replaced by a phone
intent picker: a 2x3 grid of icon cards (Draw / Animate / Compose / Record /
Edit video / Blank) taking the top ~2/3, and a scrollable Recent-projects list
in the bottom ~1/3.

Each intent reuses EditorApp::create_new_project_with_focus and then sets the
initial stack window (e.g. Draw -> fullscreen Stage, Compose -> Timeline +
Piano Roll + Keys). Recent rows load the project and enter the editor. Cards are
icon-only (Lucide glyph + label) for legibility on a phone. Adds the intent
icons to the Lucide set.
2026-06-30 22:16:09 -04:00
Skyler Lehmkuhl b25e639a1c Linked-divider preset snapping + animated transitions for the stack
Rework 3-pane divider behavior and animate every layout change:

- All transitions now ease over ~100ms via a LayoutAnim between two full stack
  configs (resize, slide, collapse, fullscreen) instead of jumping; edge
  transitions continue from the drag's progress.
- 3-pane dividers are linked along a single path parameter through the three
  presets ([.5,.25,.25] / even / [.25,.25,.5]) so both dividers move together
  and reach their snaps simultaneously. Over-grow past the extreme preset is
  anchored to the path endpoint so the non-dragged divider stays continuous.
- Growing a pane past ~66% collapses the window to 2 panes (dropping the
  opposite-end pane); the surviving divider snaps to the nearest 2-pane preset
  (1/4, 1/2, 3/4) based on the drop position. Dragging to the slide extreme
  still slides the window.
- 2-pane snaps to 1/4, 1/2, 3/4.
2026-06-30 02:07:31 -04:00
Skyler Lehmkuhl 01fe14f197 Ease divider snap into place over ~100ms
After releasing a divider, the boundary now eases (cubic ease-out, ~100ms) from
where the finger lifted to the snapped fraction instead of jumping. Starting a
new drag cancels an in-flight ease; membership changes still apply instantly.
2026-06-30 01:39:53 -04:00
Skyler Lehmkuhl 3b39e80e40 Add intermediate divider snapping (uneven pane sizes)
Dragging a divider now resizes its two adjacent panes live and, on release,
snaps the boundary to the nearest of 1/4, 1/3, 1/2, 2/3, 3/4 — so one pane can
be made dominant within a 2- or 3-pane window. Dragging far enough to squeeze a
pane past the collapse threshold still triggers the membership transition
(slide/shrink per R1-R6), after which sizing resets to even.

MobileState gains a `weights: [f32;3]` (normalized on use); config_rects is
weight-aware; edge drags keep animating toward the even-split target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 01:18:13 -04:00
Skyler Lehmkuhl 0e3c437d50 Round the top corners of stack pane headers
Each header now has rounded top corners (square at the bottom where it meets
the pane content) so it reads as a tab atop the pane.
2026-06-30 01:02:36 -04:00
Skyler Lehmkuhl 76e7963f03 Add Lucide icon font for the mobile UI
Bundle Lucide (ISC license) at assets/fonts/lucide.ttf and register it as the
egui font family "lucide" via mobile/icons.rs::install (called in EditorApp::new).
Icons are drawn as text with icons::font(size) + PUA codepoint constants.

Switch the stack headers (grip, maximize/minimize, node⇄instrument), the footer
chevron, and the transport play/pause to Lucide glyphs instead of relying on
egui's limited built-in emoji-icon-font.
2026-06-30 01:02:03 -04:00
Skyler Lehmkuhl 597697d0a6 Taller stack headers, fullscreen toggle, icon-font glyphs
- Double header height (52px) so the drag targets are easy to grab; footer
  stays compact (28px).
- Add a fullscreen button on the right of each header that collapses the
  window to that single pane; dragging the sole pane's header down or the
  footer up splits it with the adjacent pane (count can now be 1). The button
  toggles back to a 2-pane split.
- Use egui's bundled emoji-icon-font glyphs (⛶ fullscreen / ▣ restore) instead
  of hand-drawn primitives.
- Cap the drag-to-commit distance (TRIGGER_MAX) so transitions don't require
  dragging half the screen.
2026-06-30 00:33:47 -04:00
Skyler Lehmkuhl 0c59ed8b4b Fix stack overflow at list ends; full-height drag headers
- Fix panic "attempt to subtract with overflow" when sliding to the end:
  the R-ops used then_some(), which eagerly evaluates top-1 even when the
  guard is false. Switch to lazy .then(|| ...).
- Replace the thin divider/edge grab bars with full-width header bars: each
  band has a labeled header (top of band) that is its drag handle, plus a
  reserved bottom footer for the BottomEdge "pull up for more" handle. Band 0's
  header = TopEdge, band i's header = the divider above it, footer = BottomEdge.
  The Node/Instrument ⇄ toggle lives in that band's header.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 00:13:08 -04:00
Skyler Lehmkuhl 79f063d00c Replace mobile tabbed shell with vertical sliding-window stack
Per design feedback, drop the top-tabs + single-ribbon model. The mobile UI
is now one vertical stack of all panes in a fixed order (Outliner, Asset
Library, Stage, Timeline, Piano Roll, Virtual Piano, Node/Instrument, Script
editor) with a window of 2-3 consecutive panes visible at a time.

Dragging the dividers between panes and the top/bottom screen edges grows,
shrinks, or slides the window via a small state machine (R1-R6 in stack.rs),
with continuous interpolation during the drag and snap-on-release. The
Node/Instrument band toggles between the node editor and the preset browser.
The transport stays pinned as the bottom floor.

Each visible band reuses PaneInstance::render_content full-bleed (surface.rs).
Removes topbar.rs and ribbon.rs; rewrites mod.rs state (window_top,
window_count, show_instruments, drag); adds stack.rs.
2026-06-30 00:12:57 -04:00
Skyler Lehmkuhl 5ee2df5db7 Add mobile UI shell scaffold behind LB_MOBILE_UI
New src/mobile/ module renders a phone-style shell: top surface tabs
(Stage/Time/Nodes/Mixer/Tree), a single full-bleed hero pane reusing the
existing PaneInstance renderers, a resizable timeline ribbon (peek/half/full
snap tiers riding above the floor), and a fixed transport bar wired to the
audio controller (play/pause, timecode, project scrub).

Gated entirely on the LB_MOBILE_UI env var: when set, main() opens a
phone-aspect window and update() routes the central panel through
render_mobile_shell instead of render_layout_node, sharing the same
build_frame context and post-render drain. Desktop is unchanged when unset.

Tabs mirror the pane list; per the wireframe the Toolbar and Infopanel do
not become tabs (they become the omnibutton and inspector in later phases),
and the Nodes surface is a placeholder pending the focus/patch rework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 22:40:54 -04:00
Skyler Lehmkuhl da0b39fef0 Refactor RenderContext construction into EditorApp::build_frame
Extract the ~80-field SharedPaneState/RenderContext literal from update()
into EditorApp::build_frame, returning a FrameBundle that also carries the
layout-editing borrows (current_layout, drag_state, etc.). Per-frame locals
move into a FrameScratch struct so the bundle can borrow both self and the
scratch under one lifetime.

This is a behavior-preserving refactor that gives the upcoming mobile UI
path a single shared construction site for SharedPaneState instead of
duplicating the literal.
2026-06-29 21:42:40 -04:00
Skyler Lehmkuhl fd582828c2 Add text layers
Introduce editable text layers: a resizable text box with editable text,
font size, color, font family, and alignment.

Core:
- New TextLayer/TextContent (text_layer.rs), wired into AnyLayer/LayerType
  and all the exhaustive match sites; structured so content can be keyframed
  later via content_at().
- fonts.rs: thread-local parley FontContext with three bundled fonts
  (Liberation Sans/Serif/Mono, SIL OFL), system-font enumeration consolidated
  to base families, document-embedded fonts, glyph/caret/selection geometry,
  and a background preloader for the picker fonts.
- Rendering via parley layout + Scene::draw_glyphs (renderer.rs); text
  composites through the vector path.
- Actions: CreateTextClipAction (vector-layer branch, undoable),
  SetTextContentAction, ResizeTextBoxAction.
- .beam font embedding: MediaKind::Font rows (content-hash dedupe) written on
  save and registered on load, with bundled-default fallback.
- VectorClip content bounds include text boxes so text-only clips are
  selectable/draggable.

Editor:
- Text tool: click empty/raster/video to create a top-level text layer, or a
  vector layer to create+enter a clip containing the text; click an existing
  box to edit it.
- Hybrid in-place editing: a hidden egui TextEdit drives input/IME/caret while
  the text and caret/selection render in Vello; empty just-created layers are
  removed on commit.
- Selection outline + 8 resize handles (re-wrap text) with hover cursors;
  factored the corner/edge resize-cursor mapping shared with the Transform tool.
- Info panel: edit text, size, color, alignment, box size, and a font-family
  picker that previews each entry in its own font (fonts preloaded in the
  background to avoid hitches).

Deps: add parley (git, pinned to match vello's peniko); bundle Liberation
fonts under lightningbeam-core/assets/fonts. Gitignore the local
.cargo/config.toml used to select a machine's ffmpeg.
2026-06-27 18:14:47 -04:00
Skyler Lehmkuhl 58a3c829d7 Bump version to 1.0.7-alpha 2026-06-26 21:54:50 -04:00
Skyler Lehmkuhl 9717289271 Implement layer visibility toggle; log export color range
- Wire MenuAction::ToggleLayerVisibility (was a stub) to flip the active
  layer's `visible` flag via SetLayerPropertiesAction (undoable). This is
  what the SVG-export visibility fix depends on, and the "Hide/Show Layer"
  menu item now works.
- Log the resolved color range in the zero-copy H.264 path to diagnose
  whether `full_range` reaches the encoder (the VAAPI driver may still omit
  the SPS VUI full_range flag).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 21:38:07 -04:00
Skyler Lehmkuhl b301c63387 Address code-review findings across export, decode, and data model
Export correctness:
- Honor the user's color-range (Limited/Full) on the software encode path:
  thread full_range through gpu_yuv (new shader range uniform), the CPU
  swscale fallback (sws_setColorspaceDetails → BT.709 + range, fixing the
  BT.601 hue/level shift on odd-width exports), and the encoder color tags.
- Reject HDR + WebM up front with a clear message and log the forced HEVC
  override instead of producing an unplayable file.
- Delete dead render_frame_to_rgba_hdr (hardcoded Stretch; live HDR path
  already honors the fit mode).

Decode/playback (video.rs):
- Drain the decoder at EOF (send_eof + flush) so the final B-frame-delayed
  frames render instead of erroring; per-frame logic extracted to a helper.
- Missing-PTS frames continue monotonically rather than snapping to ts=0.
- Force exact thumbnail width so sub-128px sources aren't shown stretched.

Resource leaks (gpu-video-encoder):
- dmabuf import_raw: RAII guard frees the duped fd + partial VkImages/memory
  on every error path.
- vaapi alloc: free device/frames-ctx/AVFrames on the unexpected-DRM path.

Data model / robustness:
- collapse_boundary_spikes requires a full curve reversal (all control
  points) so it no longer deletes a real lens/sliver and drops the fill.
- Export audio spin-wait ignores a stale `finished` flag when a forward
  seek is pending (was rendering silence over real audio).
- RasterDiff apply_before/after take current dims and skip on a post-resize
  mismatch.
- beam_archive read_media_full caps the preallocation from untrusted total_len.

UI/visual:
- SVG export skips hidden layers/empty groups; import folds fill-opacity into
  gradients and surfaces failures as a notification.
- Active raster-layer border uses playback_time + overlay_transform.
- gpu_brush remove_layer_texture also evicts the stale low-res proxy.
- ensure_raster_resident_for_undo registers faulted frames in the LRU so
  resident RAM stays bounded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:47:32 -04:00
Skyler Lehmkuhl 1fa4d744be Add SVG vector import and export
Export (lightningbeam-core/svg_export.rs): document_to_svg walks vector
layers/groups at a given frame and emits <path> per fill (solid or
gradient via <defs>) and per stroked edge. Wired into the export dialog
as an "SVG" tab; written synchronously. Raster/video/effect layers are
skipped (vector-only, lossless), structured for a later rasterize pass.

Import (lightningbeam-editor/svg_import.rs): import_svg parses via usvg,
bakes each path's absolute transform into geometry, converts segments to
cubic edges, and maps solid/linear/radial paint to ShapeColor/
ShapeGradient. .svg is detected in the Ctrl+I Import handler and added as
a new vector layer (keyframe at the playhead). file_types gains
FileType::Vector + VECTOR_EXTENSIONS.

Tests: svg_export::export_tests (core) and svg_import::tests (editor).
2026-06-26 16:40:39 -04:00
Skyler Lehmkuhl 5869e3ced1 raster: info-panel size + "Layer to document size" (scale/expand-crop)
Resizing the document leaves raster layers as-is (canvas keeps its old pixel size,
anchored top-left). To reconcile, the info panel now shows the active raster
layer's canvas size — driven by the *active* layer, not selection focus, since
painting doesn't focus the layer — and, when it differs from the document, a
Scale / Expand-Crop toggle + a "Layer to document size" button.

- RasterKeyframe::resize_to(w, h, mode): always applies the new declared size;
  resamples (Lanczos3) when Scale, top-left pad/trim when Canvas, and only touches
  the buffer when pixels are resident (a blank canvas just takes the new size).
  Sets texture_dirty so the stage's dirty-scan refreshes the GPU texture.
- ResizeRasterLayerAction resizes every keyframe with undo, holding a (read-only)
  RasterStore so paged-out keyframes are loaded one at a time rather than bulk.

Resized keyframes stay resident + dirty and persist on the next full save (no
incremental store write to page them back out). Scale is lossy/compounds;
Expand-Crop is lossless.
2026-06-26 15:40:23 -04:00
Skyler Lehmkuhl 69939c066d stage: dashed black/yellow border around the active raster layer
Outline the active raster layer's canvas bounds (document space) so its size is
visible, especially when it differs from the document after a resize. Two strokes
sharing one dash pattern, the yellow offset by a dash to fill the black's gaps -
interlocking marching-ants. Sizes divided by zoom so the dashes stay ~constant on
screen.
2026-06-26 15:00:31 -04:00
Skyler Lehmkuhl 5bc117c518 export: add Fit mode to image export too
ImageExportSettings gains `fit: ExportFitMode` (default Letterbox) + a "Fit"
dropdown in advanced image settings; the image render path uses it instead of the
hardcoded Letterbox. Image export supports a resolution override, so the mode
matters when the override aspect differs from the document.
2026-06-26 14:42:05 -04:00
Skyler Lehmkuhl fffcf0679c aspect ratio: unify video placement + add export fit modes
A) Imported video had a different aspect ratio depending on how it was placed:
direct import used a uniform scale + center (preserve aspect), while the
asset-library timeline drag used an independent scale_x/scale_y (stretch to fill).
Add Transform::fit_centered (uniform scale, centered, aspect-preserving) and route
both paths through it, so a clip looks identical however it's added.

B) Exported video was stretched when the export resolution's aspect differed from
the document's (base_transform was always scale_non_uniform). Add
ExportFitMode {Stretch, Letterbox (default), Crop} on VideoExportSettings + a "Fit"
dropdown in advanced export settings, and a shared export_base_transform() helper:
Letterbox = uniform fit centered (black bars), Crop = uniform fill centered (trim),
Stretch = the old distort-to-fill. Threaded through the software, HDR, and
zero-copy export render paths (image export is doc-sized → identity).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 06:54:49 -04:00
Skyler Lehmkuhl 02566d571b video: fix sped-up + jerky 4K playback (frame-index cache + request-based seek)
Two bugs made 4K (and any) video playback wrong:

1. Sped-up playback. The VideoManager frame cache was keyed by milliseconds, but
   GPU (hardware-decoded) frames bypass the decoder's internal cache. A UI that
   refreshes finer than the video frame rate (a 60Hz canvas on a 30fps clip)
   therefore missed the cache on every sub-frame request and decoded the NEXT
   frame each time — advancing the video ~2x faster than the clock, and racing the
   decoder toward EOF. Key the cache on the video frame index (round(ts·fps))
   instead, so all requests within one frame share an entry and each frame decodes
   exactly once → correct speed.

2. Periodic seek/jerk. The seek decision compared the rounded request frame_ts to
   the decoded frame's exact PTS, which sit on slightly different grids — so
   frame_ts landed ~1 frame "behind" the just-decoded PTS every ~10 frames and
   falsely read as a backward seek (40ms seek + GOP re-decode). Track the previous
   request (last_requested_ts), which is strictly monotonic during forward
   playback, and detect backward from that instead. Real scrubs still seek.

Per-frame HW decode is ~3-5ms; both bugs were spurious decode/seek work on top.
Keeps a gated LB_VIDEO_DEBUG [Video Seek?] diagnostic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 06:27:01 -04:00
Skyler Lehmkuhl 701b57bfe8 video: thumbnails force CPU frames (fix black thumbs + decoder thrash)
The asset-library thumbnail called VideoManager::get_frame, which honors the
render-pass hardware flag — left ON by the preview, so the thumbnail got a GPU
frame with empty rgba_data → an all-black thumbnail. The thumbnail is also the
only consumer that requests a fixed low timestamp (1.0s) on the shared per-clip
decoder, so when it re-decodes during playback it yanks the decoder back to ~1s;
the next playback frame (6.x) is then ">2s forward" and re-seeks to the keyframe
+ re-decodes the whole GOP (the jerk: per-frame decode is ~5ms, but these seeks
cost 40ms + N-frame catch-up).

Add VideoManager::get_frame_cpu (forces want_gpu=false regardless of the render
flag); the thumbnail uses it. Now it produces a real RGBA thumbnail that the
editor texture-caches once, instead of re-decoding black frames.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 05:39:35 -04:00
Skyler Lehmkuhl ca9a70e10a export: run the zero-copy H.264 encoder on the shared device (Stage 3c-export pt 2)
Makes the common Linux H.264 export fully GPU-resident: decode (HW NV12) →
composite → VAAPI encode all on one device, no CPU round-trips.

- gpu-video-encoder: ZeroCopyEncoder now holds wgpu (device,queue,adapter) handles
  instead of owning a DrmDevice. New `new_on_device(device,queue,adapter,...)` runs
  the RGBA→NV12 render + DMA-BUF import on a passed device; `new` keeps building its
  own. The encoder only *imports* VAAPI surfaces (not export), so the shared
  import-capable device works directly.
- editor: stash the shared device handles on EditorApp (set in the creation closure
  from the eframe render_state when the shared device is active) and thread them
  through start_video_export / start_video_with_audio_export → try_build_zero_copy,
  which uses new_on_device when available. ZeroCopyVideo carries on_shared_device →
  the export composite uses hardware_ok=true (consuming HW-decoded GPU frames from
  pt 1) only then.

Falls back to the own-device encoder (decode downloads to CPU) when no shared
device. Tradeoff: the export now shares the GPU with the UI thread (was a separate
device); acceptable for the GPU-resident-decode win. Compiles; crate tests build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 05:18:23 -04:00
Skyler Lehmkuhl bb3369b709 export: GPU-resident decode for software + HDR export (Stage 3c-export pt 1)
The software and HDR export paths already composite on the shared device (the
eframe device, via render_next_video_frame), so they can consume the same
hardware-decoded NV12 GPU frames the preview uses — no CPU download/upload:

- ExportGpuResources gains the Nv12BlitPipeline; the export compositor's Video arm
  branches on inst.gpu (NV12 blit, mirroring stage.rs) vs the RGBA upload path.
- render_frame_to_gpu_rgba takes a hardware_ok flag: true for the software video +
  image export (shared device), false for the zero-copy path (own device, must
  download). render_frame_to_yuv10_hdr sets it true.

Drops the 4K software/HDR-export decode wall (HW decode + GPU composite, no
per-frame RGBA upload). The common Linux H.264 zero-copy path is unchanged
(separate device) — that's pt 2. Falls back to software when no shared device /
importer (flag is harmless then: get_frame returns CPU anyway). Compiles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 05:07:02 -04:00
Skyler Lehmkuhl 33dec6f327 export: 10-bit HDR video frame path + UI (Stage B pt 2)
Wire HDR frame production end-to-end (encoder side was pt 1):
- linear_to_pq.wgsl: linear scene HDR (BT.709, white=1.0) → BT.2020 primaries →
  PQ (203-nit) or HLG OETF → gamma-encoded R'G'B'. Inverse of the nv12 decode, so
  a decode→encode round-trip is the identity.
- hdr_frame.rs (isolated module): runs that pass into an Rgba16Float target, does a
  synchronous readback (f16-decoded on CPU), then BT.2020 R'G'B'→Y'CbCr limited
  4:2:0 10-bit pack → YUV420P10LE planes. Rgba16Float avoids the 16BIT_NORM device
  feature dependency.
- video_exporter::render_frame_to_yuv10_hdr: composite + HDR encode, returning
  10-bit planes; lazily builds the pipeline.
- orchestrator: HDR uses a synchronous 1-frame-per-call path (the async RGBA
  pipeline is 8-bit only); zero-copy is skipped for HDR.
- dialog: "Dynamic range" dropdown (SDR / HDR10 PQ / HLG).

Software HEVC Main10 path; favors correctness over throughput. Compiles; runtime
verification needs a GPU + HDR-capable player.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 04:47:35 -04:00
Skyler Lehmkuhl 41e4f3b12b export: HDR encoder scaffolding — 10-bit HEVC + BT.2020/PQ/HLG tags (Stage B pt 1)
Settings + encoder side of HDR video export (frame data still SDR until pt 2):
- core: HdrExportMode {Sdr (default), Pq, Hlg} on VideoExportSettings (serde
  default), with transfer-name helpers.
- setup_video_encoder takes the mode: HDR → YUV420P10LE, BT.2020 NCL matrix,
  limited range, color_primaries=bt2020, color_trc=smpte2084/arib-std-b67,
  profile=main10. SDR path unchanged (8-bit full-range BT.709).
- run_video_encoder forces HEVC when HDR is selected (the only 10-bit codec wired
  up). encode_frame is parameterized by pixel format and now copies planes
  row-by-row honoring stride (10-bit / non-aligned widths can have row padding).

Dormant: no UI exposes the mode yet and the frame data is still 8-bit SDR, so
selecting HDR is not yet wired. The render→PQ/HLG 10-bit frame path is pt 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 04:32:32 -04:00
Skyler Lehmkuhl 28b14b2ad0 hw: surface import_raw failures (diagnose washed-out 10-bit HDR)
A failed DMA-BUF import returned None silently → core falls back to software,
whose RGBA path does no BT.2020→709 gamut conversion, so HDR clips render washed
out. The detection log fires before import_raw, so it didn't reveal this. Log the
actual import error (with ten_bit) instead of swallowing it, to confirm whether
10-bit P010 import is failing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:59:39 -04:00
Skyler Lehmkuhl 7c2ac0e4d8 hw: propagate stream colour tags to VAAPI frames (fixes washed-out HDR)
Symptom: properly-converted BT.2020 PQ/HLG clips rendered progressively
desaturated vs the SDR reference — the signature of the transfer/primaries tags
not reaching the NV12→RGB shader, so HDR code values were decoded as plain
sRGB/BT.709 (no gamut expansion, wrong EOTF).

VAAPI hardware-decoded frames frequently leave color_trc/color_primaries/
colorspace/color_range unspecified — the authoritative values live on the codec
context (parsed from the bitstream), which the importer never sees. Before
importing, copy any unspecified colour field from the codec context onto the
frame, so the importer detects PQ/HLG + BT.2020 correctly.

Also add a one-time importer log (LB_VIDEO_DEBUG) of the detected
ten_bit/full_range/colorspace/primaries/trc to confirm what's picked up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:35:16 -04:00
Skyler Lehmkuhl 83410bd4b4 fix: P010 plane textures are sample-only (R16 isn't renderable)
R16Unorm/Rg16Unorm aren't renderable, but the import gave every plane texture
RENDER_ATTACHMENT/COLOR_ATTACHMENT usage (needed only for the encoder's RGBA→NV12
write path) — so create_view panicked on 10-bit decode ("R16Unorm is not
renderable"). Gate the render-target usage (vk COLOR_ATTACHMENT, hal COLOR_TARGET,
wgpu RENDER_ATTACHMENT) on 8-bit; 16-bit P010 planes are sample-only
(TEXTURE_BINDING/RESOURCE + COPY_SRC), which is all the decode path needs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:20:27 -04:00
Skyler Lehmkuhl 6c8cb5b9e0 hw: P010 (10-bit) DMA-BUF import for real HDR video
10/12-bit HDR decodes to P010-style VAAPI surfaces (16-bit planes), which the
8-bit NV12 import couldn't handle — real HDR clips fell back to software (losing
the HDR). Import them as R16/Rg16 plane textures:

- dmabuf::Nv12DmaBuf gains `ten_bit`; import_raw picks R16_UNORM/R16G16_UNORM (vk)
  + R16Unorm/Rg16Unorm (wgpu) when set, else the existing R8/Rg8.
- hw_video.rs detects 10-bit from the hw frames context sw_format (P010/P012/P016)
  and sets it. The NV12→RGB shader is unchanged: it samples normalized floats, and
  the limited/full de-quant lands at the same values regardless of bit depth.
- vk_device requests TEXTURE_FORMAT_16BIT_NORM when the adapter supports it (R16/
  Rg16 need it); absent → 10-bit falls back to software, 8-bit unaffected.

Pairs with the PQ/HLG/BT.2020 colour math so HDR10/HLG clips now reach the GPU
path end-to-end. NV12 callers (encoder, decode primitive) pass ten_bit=false.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:16:28 -04:00
Skyler Lehmkuhl b76eefb404 hdr: per-document Clip vs Highlight-rolloff output mode (Stage A pt 2)
Add a per-document HDR→SDR mapping applied at the final linear→sRGB encode, so
super-white (HDR) video highlights can be recovered instead of hard-clipped:

- core: HdrOutputMode {Clip (default), HighlightRolloff} on Document (serde
  default), with a SetDocumentPropertiesAction variant for undoable edits.
- shaders: a fs_main_rolloff entry point (preview linear_to_srgb.wgsl + the export
  inline shader) applying a C1 highlight knee — identity below 0.8, smooth rolloff
  [0.8,∞)→[0.8,1). SDR below the knee is untouched; Clip stays the historical path.
- preview (stage.rs) and both export encodes (video_exporter.rs) pick the pipeline
  variant from document.hdr_output_mode — one value per frame, so no per-pixel
  uniform; mirrors the existing fs_main_straight pattern.
- UI: an "HDR output" dropdown in the Document section of the info panel.

Default (Clip) is bit-identical to previous behaviour. Completes Stage A:
HDR-correct input (pt 1) + SDR-safe output mapping. HDR export (10-bit P010/PQ)
and HDR display remain Stages B/C.
2026-06-26 03:01:15 -04:00
Skyler Lehmkuhl ff490ab9ae nv12: HDR-correct input — PQ/HLG EOTF + BT.2020→709 gamut (Stage A pt 1)
Decode HDR video into the linear compositor correctly instead of approximating
everything as sRGB/BT.709:

- Read the frame's color_trc and color_primaries in the importer → VideoTransfer
  {Gamma,Pq,Hlg} + VideoPrimaries {Bt709,Bt2020} on GpuVideoFrame.
- nv12_blit.wgsl: branch the EOTF — sRGB gamma (SDR), SMPTE2084 PQ (normalized so
  203-nit graphics white = 1.0; highlights exceed 1.0), or HLG inverse-OETF
  (reference white ≈ 1.0). Then BT.2020→BT.709 primaries in linear light when
  wide-gamut, clamping out-of-709 colours.

Establishes the white=1.0 scene-linear convention: SDR content is unchanged
(stays in [0,1]); HDR video carries super-white highlights through compositing.
SDR-output mapping (clip default vs highlight rolloff) is Part 2. HLG's display
OOTF is omitted (scene-referred) — approximate but reasonable for SDR-out.
2026-06-26 02:47:09 -04:00
Skyler Lehmkuhl 6348e57de0 nv12: convert with the source colorspace matrix (not hardcoded BT.709)
The NV12→RGB pass hardcoded BT.709, so SD (BT.601) clips had slightly wrong hues.
Read each frame's AVColorSpace in the importer and derive the Y'CbCr→R'G'B'
matrix (BT.709/601/240M/2020; Unspecified guessed by height like swscale/players),
carry the four coefficients on GpuVideoFrame, and apply them in the shader.

- core: GpuVideoFrame.coeffs + ycbcr_coeffs(kr, kb) helper.
- hw_video.rs: map AVColorSpace → (kr, kb) → coeffs.
- nv12_blit{.rs,.wgsl}: uniform grows to 80 bytes (adds a coeffs vec4); the matrix
  multiply uses params.coeffs instead of literals.

BT.2020's transfer is still approximated as sRGB. The DRM-modifier-without-SAMPLED case stays a graceful software fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 02:32:57 -04:00
Skyler Lehmkuhl 1c537d99da fix: nv12_blit uniform size mismatch (64 vs 80 bytes)
The WGSL struct trailed `full_range: u32` with `_pad: vec3<u32>`, but vec3 has
16-byte alignment so the struct rounded up to 80 bytes while the Rust-side
Nv12Params (BlitTransform + u32 + [u32;3]) is 64 — wgpu rejected the draw
("Buffer is bound with size 64 where the shader expects 80"). Pack the flag as a
single `flags: vec4<u32>` (.x = full_range) so both sides are 64 bytes.
2026-06-26 02:26:51 -04:00
Skyler Lehmkuhl 1848c920d9 editor: wire hardware video decode + NV12 preview compositing (Stage 3c-preview)
Make the dormant core HW-decode engine live for the preview path:

- hw_video.rs: editor's HwVideoImporter — maps a decoded VAAPI surface to a
  DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane textures on the *shared*
  device (the only one with the import extensions). install() creates the VAAPI
  device and injects it + the importer into the VideoManager.
- main.rs: track whether the shared device is actually in use; only then (Linux,
  not LB_NO_SHARED_DEVICE) install hardware decode, using the CreationContext's
  shared device + adapter.
- nv12_blit.rs + nv12_blit.wgsl: NV12 plane textures → BT.709 → sRGB-encoded →
  linear, written straight into the Rgba16Float HDR layer (no CPU upload). Colour
  math mirrors the software path so HW/SW video match; honours full_range.
- stage.rs: the preview Video arm branches on inst.gpu (NV12 blit) vs rgba_data
  (existing upload+blit_straight); sets render_hardware_ok = !cpu_renderer so the
  CPU fallback still gets software frames.
- video_exporter.rs: sets render_hardware_ok(false) before both compositing
  passes — export composites on the encoder's separate device, so a hardware
  decoder downloads to CPU instead (export stays software, correct).
- dmabuf.rs: imported plane textures now also carry SAMPLED/TEXTURE_BINDING so
  they can be sampled by the NV12 blit (they were render-target-only); into_planes
  hands the textures to the longer-lived GpuVideoFrame.
- video.rs: cache-key the GPU/CPU representation on want_gpu (HW-configured AND
  render_hardware_ok) so software-only decode keeps a single cache entry.

Preview only this pass; export GPU-residency is the 3c-export follow-up. Untested
at runtime here (no GPU/display in container) — both crates compile.
2026-06-26 02:21:22 -04:00
Skyler Lehmkuhl 863edc80fc core: hardware video decode engine in VideoDecoder (Stage 3b)
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):

- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
  implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
  core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
  decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
  decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
  an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
  whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
  downloads the surface via av_hwframe_transfer_data then swscales — so export
  stays software/correct and only the preview goes GPU-resident. HW init or
  import failure falls back to software per clip.

Dormant until the editor injects an importer (next): no importer => software,
unchanged.
2026-06-26 02:06:31 -04:00
Skyler Lehmkuhl 7909b51df1 gpu-video-encoder: decouple dmabuf import from DrmDevice (Stage 3b foundation)
import_raw now takes (&wgpu::Device, &wgpu::Adapter) and extracts the raw Vulkan
handles via as_hal, instead of taking the crate's own DrmDevice. This lets a
VAAPI surface be imported onto ANY DMA-BUF-import-capable wgpu device — the
encoder/decoder's own device today, and the editor's shared device (Stage 3a)
next, so hardware-decoded frames are usable by the preview compositor.

Encoder/decoder pass their DrmDevice's device+adapter; round-trip decode test
still passes.
2026-06-26 01:21:08 -04:00
Skyler Lehmkuhl ca612d7807 editor: run on a shared VAAPI-capable wgpu device (Stage 3a)
Foundation for hardware video decode in BOTH preview and export: wgpu textures
can't cross devices, and a hardware-decoded frame is a DMA-BUF-imported texture
that needs the import extensions (only addable via wgpu-hal device_from_raw). So
eframe + the compositor + decode + encode must share ONE custom device.

- vk_device::create_windowed(): the existing import-capable DrmDevice plus
  VK_KHR_swapchain (the WSI surface instance extensions are already enabled by
  Instance::init), for use as the editor's main device.
- main.rs: on Linux, build the shared device and inject it into eframe via
  WgpuSetup::Existing (the egui fork supports it); fall back to wgpu's normal
  device + software decode on any failure, on other platforms, or via
  LB_NO_SHARED_DEVICE. The DrmDevice's wgpu handles are cloned into eframe
  (Arc-backed), so the VkDevice persists with them.

Runtime-verified: the editor launches and renders identically (canvas/vello,
video, panels) on the shared device, and the env-var fallback works.
2026-06-26 01:11:41 -04:00
Skyler Lehmkuhl 255e16434e gpu-video-encoder: VAAPI hardware decode → wgpu texture (Stage 2)
Headless decode primitive, the mirror of the encoder: VaapiDecoder opens a file,
hardware-decodes H.264 into VAAPI NV12 surfaces (hw_device_ctx + a get_format
callback selecting AV_PIX_FMT_VAAPI), maps each surface to a DRM-PRIME DMA-BUF,
and imports it as two wgpu plane textures via the existing dmabuf::import_raw —
the same path the encoder uses, in the read direction. Frames stay GPU-resident
(no CPU copy).

Validated by a round-trip test: encode solid gray with ZeroCopyEncoder, decode
it back, read the Y plane (mean ≈ 128). All 9 crate tests pass on the container's
Intel GPU.

Next (Stage 3): wire this into VideoManager/get_frame so the compositor consumes
a GPU texture directly (no write_texture upload), with software decode fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 00:46:53 -04:00
Skyler Lehmkuhl 9411145ce9 export: H.264 color range toggle (fix dark/oversaturated output)
The zero-copy VAAPI encoder emitted full-range BT.709 NV12 but wrote no color
tags, so players assumed limited range and stretched it — the H.264 output
looked dark and oversaturated (preview and VP9/software were fine).

- Rgba2Nv12 takes `full_range` and applies the matching Y/chroma scale+offset
  (limited 16-235 / full 0-255) via a uniform; the encoder sets color_range +
  bt709 colorspace/primaries/transfer tags to match. ffprobe-confirmed.
- New ColorRange { Limited, Full } on VideoExportSettings (default Limited, the
  universally-correct choice; serde(default) so saved settings still load),
  surfaced as a "Color range" dropdown in advanced export settings for H.264.

The swscale software fallback still emits Limited regardless of the toggle
(Full only affects the VAAPI zero-copy path).
2026-06-26 00:38:19 -04:00
Skyler Lehmkuhl f1fba186c1 video: decode at the consumer's target resolution (Stage 1)
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.

Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
2026-06-26 00:36:35 -04:00
Skyler Lehmkuhl aa7d3a3bf4 export: cache the static background + bilinear video, add render profiling
The export render bucket was dominated by re-rendering the static document
background through Vello every frame and by nearest-sampling the video on
upscale.

- Background cache: render the (static) background through Vello once, snapshot
  the composited HDR accumulator, and restore it with a single texture copy on
  every later frame instead of a Vello render + sRGB-convert + composite (+2
  submits). Invalidated on resize. background-render 3.6ms -> 0.56ms (1080p),
  7.5ms -> ~0.5ms (4K).
- blit_straight now uses the bilinear sampler — video frames are scaled to the
  output size, and nearest made that blocky. Fixes export and live preview.
- LB_RENDER_PROFILE: gated per-frame timing split (build/decode vs composite/
  upload vs srgb, and background vs layers) used to find all of the above. Kept
  as a debug aid for the remaining decode stages.

Net: export render ~12.8ms -> ~7.4ms/frame on a 1080p video clip.
2026-06-25 23:30:51 -04:00
Skyler Lehmkuhl 7525a604f2 video: reuse the swscale context + gate per-frame decode tracing
get_frame rebuilt the RGBA swscale context on every decoded frame and printed
[Video Timing] lines unconditionally. A stream's frames share one input
format/size, so build the scaler once (keyed on format+dims, rebuilt only if
they change) and reuse it; gate the per-frame traces behind LB_VIDEO_DEBUG.

Cuts export wall time ~10% on a 1080p video clip (the scaler rebuild was the
bulk of the per-frame "scale" cost; it's now ~0ms). SwsContext is !Send, so the
cached scaler is wrapped in a SendScaler — sound because a VideoDecoder is only
ever touched under the VideoManager mutex (same invariant as its decoder/input).
2026-06-25 22:44:25 -04:00
Skyler Lehmkuhl e7c239f203 ci: enable VAAPI in the from-source ffmpeg build
The libva guard fired: the release ffmpeg had no h264_vaapi. ffmpeg-sys-next
configures with --disable-autodetect and only passes --enable-vaapi when its
build-vaapi feature is set; ffmpeg-next 8.0 doesn't forward it, so installing
libva-dev alone did nothing.

Inject a direct ffmpeg-sys-next dependency with the build-vaapi feature
(Linux only) in both the GitHub Actions and container build paths. With it,
FFmpeg gets --enable-vaapi (and fails loudly if libva-dev is missing rather
than silently shipping software-only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 21:27:06 -04:00
Skyler Lehmkuhl 7ebd64391c editor: gate zero-copy VAAPI export to Linux (fix macOS/Windows build)
gpu_video_encoder's encoder/vaapi/vk_device/dmabuf modules are #[cfg(linux)]
(VAAPI is Linux-only), but the editor referenced them unconditionally, so
macOS/Windows failed to compile (cannot find `encoder` in `gpu_video_encoder`).

Gate the zero-copy path (ZeroCopyVideo, try_build_zero_copy,
run_zerocopy_video_export, and the branches in start_video_export /
start_video_with_audio_export) behind cfg(target_os = "linux"). Factor the
software encoder-thread spawn into spawn_software_video so both the None arm
and the non-Linux path share it without duplication. Non-Linux always uses
the software path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 21:27:06 -04:00
Skyler Lehmkuhl 05b5bf5f85 editor: zero-copy H.264 for video-only export too
Video-only (no-audio) H.264 export now uses the same background-thread zero-copy
VAAPI path as video+audio, instead of the software encoder. It writes the output
.mp4 directly (no mux step) and reports through the single-export progress
channel; completion clears that channel so the dialog closes cleanly.

Factored the encoder/renderer/resources setup shared by both entry points into
ExportOrchestrator::try_build_zero_copy. start_video_export gains the document/
video_manager/raster_store/container_path inputs to seed the off-thread render
from a Document snapshot. Non-H264 / non-VAAPI still falls back to software.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 20:55:43 -04:00
Skyler Lehmkuhl 5bed2e8adb Bump version to 1.0.6-alpha 2026-06-25 18:37:55 -04:00
Skyler Lehmkuhl 911d896610 packaging: build ffmpeg with VAAPI and declare libva runtime deps
The Linux release ffmpeg is built statically from source, and ffmpeg only
autodetects --enable-vaapi when libva headers are present at configure time.
Neither build path installed them, so every release shipped a static ffmpeg
with no h264_vaapi encoder -- the zero-copy export silently fell back to
software 100% of the time.

- Containerfile + build.yml: install libva-dev + libdrm-dev so the from-source
  ffmpeg gets VAAPI.
- build.yml: assert the release binary links libva, so a missing dep can't
  silently regress to a software-only build again.
- deb/rpm: libva (libva2/libva-drm2/libdrm2, libva/libdrm) is a hard runtime
  dep -- the vaapi-enabled ffmpeg DT_NEEDEDs it, so the app won't launch
  without it. The VA driver is a soft recommends (absent it, export falls back
  to software): va-driver-all (deb), intel-media-driver/mesa-va-drivers (rpm).

build.rs needs no change: releases link ffmpeg statically (its bundling path
is skipped) and the AppImage is thin, taking libva from the host like libvulkan.
2026-06-25 18:28:43 -04:00
Skyler Lehmkuhl ecfa192245 editor: run zero-copy H.264 export on a background thread
The zero-copy VAAPI export previously rendered one frame per egui repaint on
the UI thread, which pinned throughput to the 60Hz vsync of the present loop
(measured exactly 16ms/frame) -- so the near-free hardware encode bought
nothing. Because the export runs on its own VAAPI device (independent of
eframe), it can run entirely off the UI thread.

- run_zerocopy_video_export: a background thread owning the encoder + its own
  vello renderer/device, a Document snapshot (Document is Clone+Send; the UI
  keeps the live one), its own ImageCache, and a RasterStore clone. Renders +
  hardware-encodes every frame and reports through the same video_progress
  channel the software encoder thread uses.
- start_video_with_audio_export takes the document/video_manager/raster_store/
  container_path to seed the thread; video_state is None for this path.
- Throttle export-time UI repaints (~6Hz) and the thread's progress sends so
  the render thread keeps the cores; the breakdown print stays.
- cancel() tears down parallel_export (detaches threads, removes temp files)
  so the progress dialog dismisses; the call site closes the dialog.
- Gate the progress poll loop on has_pending_progress() so it stops once the
  export ends instead of polling/logging every repaint forever; the single
  export path clears its channel on the terminal event.

Vsync overhead is gone (0.1ms/frame); export is now render-bound (~11ms/frame
Vello scene-build). ~1:50 -> ~56s (~2x) on the validation clip.
2026-06-25 18:15:39 -04:00
Skyler Lehmkuhl 2bce5e93a6 gpu-video-encoder: VAAPI driver retry, Vello-capable device, Send
Three fixes found while running the zero-copy export on real Intel hardware:

- vaapi::create_device() retries LIBVA_DRIVER_NAME in order iHD -> auto ->
  i965 -> radeonsi. libva was auto-selecting the legacy i965 driver, which
  fails on newer Intel GPUs; the modern iHD (intel-media-driver) is needed.
  encoder.rs now builds its hwdevice through this helper.
- vk_device: request the adapter's full limits instead of downlevel_defaults.
  Vello's compute pipelines need max_storage_buffers_per_shader_stage >= 5
  (downlevel caps at 4), which panicked Vello's shader init on the export
  device. This device only ever runs on a real VAAPI GPU.
- ZeroCopyEncoder: unsafe impl Send. It owns its FFmpeg/Vulkan handles
  exclusively and is only moved (onto the export thread), never shared.
2026-06-25 18:15:05 -04:00
Skyler Lehmkuhl 3e4f29c297 Wire zero-copy VAAPI H.264 into video+audio export
When exporting H.264 with audio, try the gpu-video-encoder ZeroCopyEncoder:
render each frame to RGBA and hardware-encode it into a VAAPI surface
inline, on the encoder's own VAAPI-capable wgpu device — no GPU->CPU
readback, no swscale, no software-encoder thread. Falls back to the
existing software path verbatim when VAAPI/the device is unavailable
(non-Linux, non-H264, or init failure), so it's additive.

- VideoExportState gains zero_copy: Option<ZeroCopyVideo> (encoder + its own
  vello renderer + ExportGpuResources + a reused RGBA target, all on the
  encoder's device).
- start_video_with_audio_export builds it for H.264 and skips spawning the
  software encoder thread when present.
- render_next_video_frame routes to a zero-copy arm that reuses
  render_frame_to_gpu_rgba on the encoder's device, then encode_rgba; on the
  last frame finish() writes the temp .mp4 and sets video_progress=Complete
  so the existing mux runs. video_thread=None makes the mux join a no-op.

Separate export device (vs modifying the eframe device) keeps this contained
to export. Video-only export stays on the software path for now. Runtime
verification (an actual H.264 export) is pending — cannot run the editor in
the dev container.
2026-06-25 15:44:50 -04:00
Skyler Lehmkuhl a00e73c4b3 gpu-video-encoder: mux the zero-copy encode into a container file
ZeroCopyEncoder::new now takes an output path and writes a real container
(format inferred from the extension, e.g. .mp4): create an output format
context, add the h264 stream from the encoder, write header; encode_rgba
rescales each packet's ts and av_interleaved_write_frame's it; finish
flushes + writes the trailer + closes. Sets AV_CODEC_FLAG_GLOBAL_HEADER for
mp4/mov so SPS/PPS land in extradata. This lets the editor's existing
mux_video_and_audio consume the temp video file unchanged.

The zerocopy_encode test now writes a .mp4 and ffprobe-verifies the codec,
dimensions, and frame count. Also let wgpu own the imported plane-image
destruction via texture_from_raw drop callbacks (clears two warnings).
2026-06-25 15:43:39 -04:00
Skyler Lehmkuhl ba897eaea2 gpu-video-encoder: complete zero-copy H.264 encode pipeline
Build the full end-to-end zero-copy encoder, validated on Intel/VAAPI:

- render_nv12: fragment-shader RGBA->NV12 that renders luma/chroma into
  the imported R8/RG8 plane render targets (compute storage can't write
  the DMA-BUF-backed planes; render attachments can).
- dmabuf: import_raw imports an NV12 DMA-BUF by explicit layout; the two
  plane images + shared memory are now destroyed by wgpu via texture_from_raw
  drop callbacks (Arc MemoryGuard frees the memory once both images are
  gone, in wgpu's wait-idle'd deferred pass) -- fixes the teardown segfault.
- encoder::ZeroCopyEncoder: renders an RGBA texture straight into a pooled
  VAAPI surface (imports cached by VASurface id) and encodes with h264_vaapi.
  encode_rgba + finish; the caller renders on device().

Tests: real-frame render into the surface matches the CPU NV12 reference,
and a 30-frame encode produces valid H.264 (ffprobe-verified) with clean
teardown. Not yet wired into the editor.
2026-06-23 19:28:57 -04:00
Skyler Lehmkuhl 5917ce7921 Add gpu-video-encoder crate: zero-copy VAAPI encode (validated)
New workspace crate isolating the unsafe GPU<->encoder interop for
zero-copy hardware video encoding. Every link is validated by a test on
real Intel/Mesa/iHD hardware:

- nv12: GPU RGBA->NV12 compute (BT.709 full-range), byte-exact vs a CPU
  reference.
- vaapi: VAAPI hwcontext + h264_vaapi encode (CPU-fed NV12 -> valid H.264),
  and DRM-PRIME surface layout probing.
- vk_device: a custom wgpu Vulkan device that adds
  VK_EXT_image_drm_format_modifier (+ external-memory fd/dma-buf) via the
  wgpu-hal device-from-raw path, so a tiled VAAPI surface can be imported.
- dmabuf: import a VAAPI NV12 surface's tiled DMA-BUF as two aliasing wgpu
  textures (Y=R8, UV=RG8) at the plane offsets.
- zerocopy test: render values via Vulkan straight into the VAAPI surface
  and read them back 100% correct -- proving the GPU writes into the
  encoder surface with no CPU copy.

Not yet wired into the editor; real-frame render + encode-from-surface +
fallback wiring follow. Linux-only (libva); other platforms fall back.
2026-06-23 19:07:37 -04:00
Skyler Lehmkuhl da65b63bdf Repair test suite + fix sample key-range overlap bug
The suite had accumulated breakage from prior refactors:
- selection unification: rewrite the integration tests for the single
  unified clip_instances collection (shapes+clips are one set now).
- tempo-map: thread a TempoMap (constant 60 BPM = identity) into the
  clip remap_time tests so the second-based expectations hold.
- drop two dead rgba_to_yuv420p tests that asserted tight plane sizes
  incompatible with the function's 16-macroblock alignment.
- ignore the WIP theme var() cascade test (theme system not wired up).

Also a real bug the tests caught: auto_key_ranges produced overlapping
sample key ranges (the midpoint key mapped to both adjacent samples).
Start each range one past the previous midpoint.
2026-06-23 19:07:06 -04:00
Skyler Lehmkuhl 2564d807a0 Convert export frames RGBA->YUV420p on the GPU
The export read back 8MB RGBA per frame and ran swscale RGBA->YUV420p on
the UI thread (~6ms/frame). Add a tight GPU compute converter (gpu_yuv,
BT.709 full-range matching the encoder tags) and wire it into the
triple-buffered ReadbackPipeline: render to RGBA, convert on the GPU, read
back ~3MB of planar YUV, and skip the CPU pass. Gated on a runtime check
that the encoder's YUV420P plane strides are tight (no linesize padding),
with the swscale path as fallback for other dimensions; LB_DISABLE_GPU_YUV
forces the CPU path. Includes a CPU reference + unit tests for the packing.

Also guard render_next_video_frame against re-initializing/re-emitting
"Complete" every frame after the render finishes while the encoder/mux
drains (the completion nulled gpu_resources but left video_state set).
2026-06-23 19:06:47 -04:00
Skyler Lehmkuhl 70ac0cde00 Fix audio export hang when a video's audio is shorter than the video
Offline export blocks each streaming source until decoded frames are
available. For a video whose audio track ends before the video (or a
container like WebM/Opus with no exact stream duration), the tail chunks
wait for frames that never arrive — the 10s safety valve fires per chunk,
so the export appears to hang indefinitely.

Add a `finished` flag to ReadAheadBuffer, set when the disk reader hits
EOF (cleared on seek). The export-mode wait now breaks immediately once
the source is finished and the requested frames aren't present, rendering
silence for the missing tail instead of timing out chunk-by-chunk. Only
the export path reads the flag, so playback is unaffected.
2026-06-23 19:06:21 -04:00
Skyler Lehmkuhl 5844a0f070 Composite grouped/nested video on the GPU path
Imported video is a Group[Video, Audio] that rendered as a Vello-baked
Vector layer, re-uploading the full frame to Vello's image atlas every
frame (~17ms/frame at 1080p, hitting playback and export alike). Extract
video frames out of the Group/clip scene recursion into
VideoRenderInstances so they composite via the GPU Video path; mixed
video+vector containers fall back to Vello (correct, unaccelerated).

Also route video through hardware sRGB decode: upload raw sRGB bytes to an
Rgba8UnormSrgb texture and blit with a non-unpremultiplying shader variant
(blit_straight), removing the per-frame per-pixel CPU sRGB->linear pass.
Add an F3 GPU-timestamp timer and a per-frame video texture cache.

Drops the live composite of a 1080p video from ~17ms to ~2-3ms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 17:38:30 -04:00
Skyler Lehmkuhl ce151ffd61 Pack video into .beam and stream frames + audio from the blob
Video was the only media type always kept external (VideoClip.file_path),
so a project with video wasn't self-contained. Now video packs into the
SQLite container under the same large-media policy as audio (pack < 2 GB
unless the user chose Reference), and both the frames and the embedded audio
track decode by streaming directly from the blob — no temp files.

- New crate ffmpeg-blob-io: an AVIOContext-over-Read+Seek shim (BlobInput)
  that lets ffmpeg demux from an arbitrary byte source. Isolates all the
  unsafe FFI + ffmpeg ABI coupling (version-pinned =8.0.0/=8.0.1). Manual
  Drop teardown order; AVSEEK_SIZE restores the read position (FFmpeg assumes
  a size query doesn't move it — required for MP4 moov-at-end).
- Schema/save/load: VideoClip.media_id; save_beam packs/references video as
  MediaKind::Video (keyed by clip id); load resolves packed vs referenced and
  reports missing sources. A packed clip points its linked video-audio pool
  entry's media_id at the video row so the audio streams from the same blob.
- Frames: video.rs VideoSource{Path,Packed} threaded through new/seek/scan/
  probe/thumbnails (a fresh BlobReader per open); editor builds the source
  from current_file_path (now set before register_loaded_videos).
- Audio: VideoAudioReader::open_source via BlobInput; the disk_reader
  StreamSource block on packed video-audio is removed; the engine's existing
  factory activation routes it unchanged.

Tests: ffmpeg-blob-io AVIO unit tests (WAV via Cursor, seek, open/drop loop);
core packed_video_stream (blob->AVIO->Input) and beam_archive video round-trip;
daw-backend open_source test (compiles; links/runs only off-container).
Runtime-verified: a packed video plays frames + audio after the source file
is removed.
2026-06-22 09:17:10 -04:00
Skyler Lehmkuhl 34eee3a620 Fix .beam save quirks: real sample rate, drop dead fields
Address the code smells flagged in the .beam format spec:

- Write the project's actual sample rate on save instead of a hardcoded 48000
  (add AudioProject::sample_rate()).
- Remove the vestigial RasterKeyframe.media_path field (it was only used by the
  legacy ZIP loader, which now derives "media/raster/<id>.png" from the keyframe
  id) and the dead buffer_path_at_time accessor. Backward-compatible: older files
  carrying media_path deserialize fine (the field is ignored).
- Drop the unused SaveSettings fields auto_embed_threshold_bytes / force_embed_all
  / force_link_all; only large_media_mode was ever consulted. Un-prefix the now-used
  `settings` parameter.

Update BEAM_FILE_FORMAT.md to match. The remaining notes (reserved MediaKind::Video,
exact-match version check) are design choices, left as-is.
2026-06-21 23:21:48 -04:00
Skyler Lehmkuhl 5f0d5354ed Document the .beam SQLite format and port the inspector
The .beam container moved from ZIP to SQLite, but the docs/tooling still
described the old ZIP format.

- Rewrite BEAM_FILE_FORMAT.md as a normative spec of the current SQLite
  container: file-magic identification, the four-table schema, the two version
  numbers, the media model (kinds, packed/referenced storage, 4 MiB chunking),
  derived media-id formulas, project.json top-level + key entities, save/load
  semantics, the legacy-ZIP format + migration, large-media policy, conformance
  and security sections, and known quirks.
- Port beam_inspector.py to read SQLite (auto-detecting container by magic,
  with the legacy ZIP path preserved): a --media store view, packed/referenced
  storage detection, and chunk-reassembling media extraction.
- Update beam_inspector_README.md to match.
2026-06-21 23:08:15 -04:00
134 changed files with 18774 additions and 3186 deletions

View File

@ -71,6 +71,7 @@ jobs:
libx11-dev libxkbcommon-dev libxcb-shape0-dev libxcb-xfixes0-dev \
libxdo-dev libglib2.0-dev libgtk-3-dev libvulkan-dev \
yasm libx264-dev libx265-dev libvpx-dev libmp3lame-dev libopus-dev \
libva-dev libdrm-dev \
libpulse-dev squashfs-tools dpkg rpm
- name: Install cargo packaging tools (Linux)
@ -117,6 +118,19 @@ jobs:
run: |
sed -i.bak 's/ffmpeg-next = { version = "8.0", features = \["static"\] }/ffmpeg-next = { version = "8.0", features = ["build", "static"] }/' lightningbeam-ui/lightningbeam-editor/Cargo.toml
# ffmpeg-sys-next configures FFmpeg with --disable-autodetect and only passes --enable-vaapi
# when its build-vaapi feature is set; ffmpeg-next 8.0 doesn't forward it, so depend on the
# sys crate directly. Linux only (the feature is a no-op elsewhere and we only ship VAAPI on
# Linux). With it set, a missing libva-dev makes FFmpeg configure fail loudly instead of
# silently shipping a software-only build.
- name: Enable VAAPI in FFmpeg build (Linux)
if: matrix.platform == 'ubuntu-24.04'
shell: bash
run: |
sed -i '/^ffmpeg-next = /a ffmpeg-sys-next = { version = "8.0", features = ["build-vaapi"] }' \
lightningbeam-ui/lightningbeam-editor/Cargo.toml
grep -n 'ffmpeg-sys-next' lightningbeam-ui/lightningbeam-editor/Cargo.toml
- name: Setup FFmpeg (Windows)
if: matrix.platform == 'windows-latest'
shell: pwsh
@ -202,6 +216,20 @@ jobs:
mkdir -p lightningbeam-ui/target/release
cp lightningbeam-ui/target/${{ matrix.target }}/release/lightningbeam-editor lightningbeam-ui/target/release/
# Guard: the zero-copy export needs the static ffmpeg to have h264_vaapi, which only
# happens if libva headers were present at ffmpeg configure time. ffmpeg autodetects it,
# so a missing libva-dev silently ships a software-only build. A vaapi-enabled binary
# links libva — assert that, so the dep can never regress unnoticed.
- name: Verify VAAPI is compiled in (Linux)
if: matrix.platform == 'ubuntu-24.04'
shell: bash
run: |
if ! ldd lightningbeam-ui/target/release/lightningbeam-editor | grep -q 'libva\.so'; then
echo "::error::Release binary does not link libva — ffmpeg was built without VAAPI (is libva-dev installed?)."
exit 1
fi
echo "VAAPI OK: binary links libva."
# ── Stage presets next to binary for packaging ──
- name: Stage presets in target dir
shell: bash

484
BEAM_FILE_FORMAT.md Normal file
View File

@ -0,0 +1,484 @@
# `.beam` File Format Specification
**Status:** Normative.
**Container schema version:** `1` (`meta.schema_version`).
**Project payload version:** `1.0.0` (`BeamProject.version` / `meta.version`).
**Last updated:** 2026-06-21.
This document specifies the on-disk format of Lightningbeam `.beam` project files
precisely enough that an independent implementation can read and write them. It
describes the **current** format — a SQLite database — and the **legacy** format
(a ZIP archive) that current readers still accept and migrate.
> An earlier revision of this document described the ZIP container as the current
> format. That is now historical; see [§11 Legacy ZIP format](#11-legacy-zip-format-pre-sqlite).
## 1. Notational conventions
The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** are
to be interpreted as described in RFC 2119.
Byte sizes use binary units: `KiB = 1024`, `MiB = 1024²`, `GiB = 1024³`.
"UUID" means an RFC 4122 128-bit identifier. Unless stated otherwise, a UUID is
serialized in JSON as its canonical hyphenated lowercase string, and stored in
SQLite as its 16 raw bytes in **network (big-endian) byte order** — i.e. the byte
sequence returned by `Uuid::as_bytes`, most significant byte first.
## 2. Overview
A `.beam` file is a **single SQLite 3 database** (default rollback journal; no WAL,
no special pragmas) containing:
- one JSON document, `project.json`, holding all project state that is small and
structural (the document/scene tree, audio project, asset metadata); and
- zero or more **media** items (audio, raster frames, image assets, and derived
blobs such as waveforms and thumbnails), each either **packed** into the database
in chunks or **referenced** by an external file path.
The design goals are: a single beginner-friendly file (no project folder);
streaming reads of large media via SQLite blob I/O; transactional, crash-safe,
**in-place** saves that do not rewrite unchanged media; and inspectability with the
ordinary `sqlite3` CLI.
## 3. File identification
A reader MUST determine the container type from the first 16 bytes of the file:
- If the first 16 bytes equal the ASCII string `SQLite format 3\0` (i.e.
`53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00`), the file is a SQLite
`.beam` and MUST be read per [§4](#4-container-the-sqlite-schema)[§10](#10-load-semantics).
- Otherwise the file is treated as a [legacy ZIP `.beam`](#11-legacy-zip-format-pre-sqlite)
(ZIP local-file-header magic `50 4B 03 04`).
The `.beam` extension is used for both container types; the magic bytes — not the
extension — are authoritative.
## 4. Container: the SQLite schema
A conforming SQLite `.beam` MUST contain exactly these four tables (DDL as created;
column order and types are normative):
```sql
CREATE TABLE media (
id BLOB PRIMARY KEY, -- 16-byte UUID (big-endian)
kind INTEGER NOT NULL, -- MediaKind (§6.1)
codec TEXT NOT NULL, -- original codec/container, e.g. "flac","mp3","png"
storage INTEGER NOT NULL, -- MediaStorage (§6.2): 0=Packed, 1=Referenced
ext_path TEXT, -- external path, set iff storage=Referenced
total_len INTEGER NOT NULL DEFAULT 0, -- payload length in bytes (packed); 0 if referenced
channels INTEGER, -- audio: channel count (nullable)
sample_rate INTEGER, -- audio: sample rate in Hz (nullable)
width INTEGER, -- visual media: pixel width (nullable)
height INTEGER -- visual media: pixel height (nullable)
);
CREATE TABLE media_chunk (
id INTEGER PRIMARY KEY, -- rowid (used as the handle for blob streaming)
media_id BLOB NOT NULL, -- → media.id
chunk_index INTEGER NOT NULL, -- 0-based ordinal of this chunk
bytes BLOB NOT NULL, -- chunk payload
UNIQUE (media_id, chunk_index)
);
CREATE TABLE project_json (
id INTEGER PRIMARY KEY CHECK (id = 0), -- exactly one row, id 0
data TEXT NOT NULL -- the serialized BeamProject (§8)
);
CREATE TABLE meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
```
There is no foreign-key constraint between `media_chunk.media_id` and `media.id`;
referential integrity is maintained by the writer. A reader MUST NOT assume SQLite
enforces it.
### 4.1 The `meta` table
Defined keys (all values are strings):
| Key | Value | Meaning |
|------------------|--------------------|---------|
| `schema_version` | `"1"` | Container schema version. Set once at creation. |
| `version` | `"1.0.0"` | Project payload version (mirrors `project.json`'s `version`). |
| `created` | RFC 3339 timestamp | Creation time. Preserved across in-place re-saves. |
| `modified` | RFC 3339 timestamp | Last save time. Updated on every save. |
Writers MAY add additional keys; readers MUST ignore unknown keys.
### 4.2 Two independent version numbers
The format carries **two** version numbers with different compatibility rules:
- **`meta.schema_version`** (`INTEGER`, currently `1`) — versions the SQLite
container/table layout. A reader MUST reject a file whose `schema_version` is
**greater** than the highest it supports, and SHOULD accept any value less than
or equal to its maximum (forward-compatible for older files).
- **`BeamProject.version`** / `meta.version` (currently `"1.0.0"`) — versions the
`project.json` payload. The current implementation requires **exact string
equality** and rejects any other value. A future revision MAY relax this to a
semantic-version range; until then, writers MUST emit exactly `"1.0.0"` and
readers reject anything else.
These two numbers are orthogonal and MUST both be checked.
## 5. UUIDs and identity
Every media item is identified by a UUID. In `project.json` UUIDs appear as
canonical strings; in the `media`/`media_chunk` tables they appear as 16 raw
big-endian bytes (`media.id`, `media_chunk.media_id`). A reader MUST treat the two
representations as equal by their 128-bit value.
`media_chunk.id` is the SQLite rowid and is the handle a reader uses to open a blob
for streaming (`sqlite3_blob_open` on table `media_chunk`, column `bytes`). It has
no meaning beyond row identity and MUST NOT be relied upon to be stable across
re-saves.
## 6. Media model
### 6.1 `MediaKind` (`media.kind`)
| Value | Kind | `codec` examples | Notes |
|-------|---------------|------------------|-------|
| `0` | `Audio` | `flac`, `mp3`, `wav`, `ogg`, `opus`, `aac`, `m4a`, `alac`, `caf`, `aiff` | Source audio for a pool entry. |
| `1` | `Video` | `mp4`, `mov`, … | A video clip's container bytes, keyed by the clip id. Packed under the large-media policy (referenced above 2 GiB unless `Pack`). Frames **and** the embedded audio stream are decoded directly from this blob. |
| `2` | `Raster` | `png` | Full-resolution pixels of a raster keyframe (PNG-encoded RGBA). |
| `3` | `ImageAsset` | `png`, `jpg`, … | An imported image asset's original bytes. |
| `4` | `Waveform` | `lbwf` | Precomputed waveform LOD pyramid for an audio item. Opaque blob owned by `daw_backend::audio::waveform_pyramid`. |
| `5` | `Thumbnail` | `lbtn` | A pack of precomputed video thumbnails for a clip. Opaque blob owned by the editor. |
| `6` | `RasterProxy` | `png` | Low-resolution PNG proxy of a raster keyframe, shown during cold scrubs while full-res pages in. |
A reader MUST reject a `kind` value it does not recognise only if it needs that
item; unknown kinds MAY otherwise be ignored.
### 6.2 `MediaStorage` (`media.storage`)
| Value | Storage | Bytes location | `total_len` | `ext_path` |
|-------|--------------|----------------|-------------|------------|
| `0` | `Packed` | `media_chunk` rows in this DB | payload length | `NULL` |
| `1` | `Referenced` | external file at `ext_path` | `0` | path string |
### 6.3 Packed storage and chunking
Packed bytes are split into chunks of **`CHUNK_SIZE = 4 MiB`** and stored one chunk
per `media_chunk` row, ordered by `chunk_index` ascending starting at `0`. The
writer MUST:
1. set `media.total_len` to the exact total byte length of the payload;
2. write `ceil(total_len / CHUNK_SIZE)` chunk rows (zero rows iff `total_len == 0`);
3. make every chunk exactly `CHUNK_SIZE` bytes **except the last**, which holds the
remainder `total_len (n1)·CHUNK_SIZE`.
Because chunk lengths are fully determined by `total_len` and `CHUNK_SIZE`, a reader
performing random access MUST compute, for a byte offset `pos < total_len`:
```
chunk_index = pos / CHUNK_SIZE
offset_in_chunk = pos % CHUNK_SIZE
chunk_len = min(CHUNK_SIZE, total_len chunk_index·CHUNK_SIZE)
```
and read from the row with that `chunk_index`. A reader MAY stream the whole payload
by concatenating chunk `bytes` in `chunk_index` order. The chunk size MAY differ in
files written by other tools; a robust reader SHOULD derive chunk boundaries from
actual row lengths rather than assuming 4 MiB. (The reference reader assumes uniform
`CHUNK_SIZE` except for the last chunk; writers targeting it MUST keep chunks
uniform.)
### 6.4 Referenced storage
A referenced item stores only `ext_path` (with `total_len = 0` and no chunk rows).
The path is resolved relative to the directory containing the `.beam` file unless it
is absolute. If the file is absent at load time, the item is reported as a *missing
file* ([§10.4](#104-missing-referenced-files)) — this is non-fatal.
## 7. Derived media IDs
Three media kinds are keyed by UUIDs **derived** from another id rather than by an
independent random UUID. A reader/writer MUST compute them exactly as follows
(`from_u128` constructs a UUID from a 128-bit big-endian integer):
| Kind | Derived from | Formula |
|---------------|---------------------|---------|
| `Waveform` | audio pool **index** `i` (`usize`) | `from_u128((0x4C42_5746 << 96) \| (i as u128))` — high 32 bits = `0x4C425746` ("LBWF"), low 96 bits = the pool index. |
| `Thumbnail` | video clip UUID `c` | `from_u128(c.as_u128() ^ 0x4C42_544E_4C42_544E_4C42_544E_4C42_544E)` — full-width XOR with "LBTN" repeated 4×. |
| `RasterProxy` | raster keyframe UUID `k` | `from_u128(k.as_u128() ^ 0x4C42_5058_4C42_5058_4C42_5058_4C42_5058)` — full-width XOR with "LBPX" repeated 4×. |
The full-resolution `Raster` row is keyed by the keyframe's **own** UUID (no
derivation); an `ImageAsset` row is keyed by the asset's own UUID; a packed `Audio`
row is keyed by the pool entry's `media_id`.
> Note the asymmetry: the waveform id ORs the pool index into the high bits, while
> thumbnail/proxy ids XOR a 128-bit sentinel with a source UUID. This is intentional
> (waveforms are keyed by *index*, not by a UUID) but is a sharp edge for
> implementers.
## 8. `project.json`
`project_json.data` is a UTF-8 JSON document: the serde serialization of a
`BeamProject`. This section specifies the top-level structure and the entities a
reader needs to resolve media; the deep UI/scene tree is defined by the
corresponding Rust types (serde field names match struct field names unless a
`#[serde(rename)]` is noted) and is intentionally **not** enumerated exhaustively
here.
### 8.1 `BeamProject` (root)
| Field | Type | Notes |
|-----------------|---------------------------|-------|
| `version` | string | MUST be `"1.0.0"`. |
| `created` | string (RFC 3339) | |
| `modified` | string (RFC 3339) | |
| `ui_state` | `Document` | The scene/document tree (§8.2). |
| `audio_backend` | `SerializedAudioBackend` | Audio engine state (§8.3). |
### 8.2 `Document` (top-level fields)
Collections that carry media linkage are **bold**:
- `id: Uuid`, `name: string`, `background_color`, `width: f64`, `height: f64`,
`framerate: f64`, `duration: f64`
- `time_signature`, `master_layer`, `timeline_mode` — all `#[serde(default)]`
- `root: GraphicsObject` — the layer tree; raster keyframes live here and inside
nested group/clip layers
- **`image_assets: map<Uuid, ImageAsset>`** — keyed by asset UUID (= the media id)
- **`video_clips: map<Uuid, VideoClip>`** — when `VideoClip.media_id` is set the
video is packed (a `Video` media row at the clip id); otherwise `file_path` points
at the external video file. Thumbnails are derived media (§7)
- `vector_clips`, `audio_clips`, `instance_groups`, `effect_definitions`,
`script_definitions`, the various `*_folders` asset trees — structural, no direct
media-row linkage
- `ui_layout`, `ui_layout_base``Option`, skipped when `None`
- `current_time`, `layer_to_clip_map``#[serde(skip)]` (not persisted)
`RasterKeyframe` (within the layer tree) has `id: Uuid` (= its full-res `Raster`
media id and the seed for its `RasterProxy` id), `time`, `width`, `height`,
`tween_after`, and `stroke_log`. Pixel buffers are `#[serde(skip)]` and faulted in
from media rows. (The legacy ZIP entry path `media/raster/<id>.png` is derived from
`id`; older files may also carry a now-ignored `media_path` field.)
### 8.3 `SerializedAudioBackend`
| Field | Type | Notes |
|----------------------|---------------------|-------|
| `sample_rate` | `u32` | The project's system sample rate (mirrors `AudioProject.sample_rate`). Per-file rates are on each `AudioPoolEntry.sample_rate`. |
| `project` | `AudioProject` | Tracks + MIDI clip pool (§8.4). |
| `audio_pool_entries` | `[AudioPoolEntry]` | Audio source registry (§8.5). |
| `layer_to_track_map` | `map<Uuid, u32>` | UI layer UUID → engine track id. `#[serde(default)]`. |
### 8.4 `AudioProject` (top-level fields)
`tracks: map<TrackId, TrackNode>`, `root_tracks: [TrackId]`, `next_track_id`,
`sample_rate: u32`, `midi_clip_pool`, `next_midi_clip_instance_id`. DSP graphs are
not serialized; they are rebuilt on load.
### 8.5 `AudioPoolEntry` (media linkage)
| Field | Type | Role |
|-----------------|----------------------------|------|
| `pool_index` | `usize` | Stable index; seeds the `Waveform` media id (§7). |
| `name` | string | |
| `media_id` | `Option<string>` (UUID) | Set ⇔ audio bytes are **packed** in this DB under that id. `#[serde(default, skip_serializing_if=None)]`. |
| `relative_path` | `Option<string>` | Set ⇔ audio is **referenced** (external file) or missing. |
| `embedded_data` | `Option<EmbeddedAudioData>`| Legacy/inline: `{ data_base64, format }`. Used when bytes are neither packed nor referenced. |
| `sample_rate` | `u32` | Authoritative sample rate. |
| `channels` | `u32` | Channel count. |
| `duration` | `f64` | Seconds. |
| `is_video_audio`| `bool` | If set, the entry is the audio track of a video. Referenced when the video is external; when the video is **packed**, `media_id` points at the video's `Video` row and the audio streams from that same blob. `#[serde(default, skip_if=false)]`. |
| `waveform_blob` | (transient) | `#[serde(skip)]`; carries waveform bytes in memory only. |
### 8.6 Media linkage summary
| Media kind | Referenced from `project.json` by | Media-row id |
|--------------------|-----------------------------------|--------------|
| Audio (packed) | `AudioPoolEntry.media_id` | that UUID |
| Audio (referenced) | `AudioPoolEntry.relative_path` | — (external) |
| Audio (embedded) | `AudioPoolEntry.embedded_data` | — (inline base64) |
| Raster (full) | `RasterKeyframe.id` | that UUID |
| Raster proxy | derived from `RasterKeyframe.id` | §7 |
| Image asset | `Document.image_assets` key / `ImageAsset.id` | that UUID |
| Video (packed) | `VideoClip.media_id` (= clip id) | that UUID |
| Video (referenced) | `VideoClip.file_path` | — (external) |
| Video thumbnails | derived from `video_clips` key | §7 |
| Waveform | derived from `AudioPoolEntry.pool_index` | §7 |
## 9. Save semantics
A conforming writer MUST perform all media, `project.json`, and `meta` writes for a
save inside **one** SQLite transaction.
### 9.1 In-place vs. create
- If the target path exists **and** is already a SQLite `.beam`, the writer opens it
and writes in place. Unchanged packed media MUST NOT be rewritten (§9.3); this is
the central performance/crash-safety property, and writers MUST NOT use a
copy-to-temp-and-rename flow for in-place saves.
- Otherwise (new file, or migrating a legacy ZIP), the writer creates a fresh DB at
`<path>.beam.tmp`, writes everything, commits, then atomically `rename`s it over
the target.
### 9.2 Order of operations within the transaction
1. Audio pool entries → `Audio` (+ `Waveform`) media rows.
2. Raster keyframes → `Raster` (+ `RasterProxy`) media rows.
3. Video thumbnail packs → `Thumbnail` media rows.
4. Video clips → `Video` media rows (packed/referenced per the large-media policy);
when a clip is packed, its linked video-audio pool entry's `media_id` is set to
the clip's `Video` row so the audio streams from the same blob.
5. Image assets → `ImageAsset` media rows.
6. **Garbage-collect**: delete every `media` row (and its chunks) whose id is **not**
in the set of live ids accumulated in steps 15.
7. Write `project.json` and the `meta` keys (`version`, `created`, `modified`).
8. Commit; then (create path only) rename the temp file over the target.
### 9.3 Packed vs. referenced decision (audio)
For each audio pool entry, in order:
- If a packed row for its `media_id` already exists in the archive, keep it untouched
(do not rewrite the bytes) and keep the entry packed.
- Else if its external file resolves and exists: store **referenced** if the entry is
video audio, *or* the file is `≥ LARGE_MEDIA_THRESHOLD` (2 GiB) and the large-media
mode is not `Pack`; otherwise **pack** it (streamed from disk chunk-by-chunk).
- Else if it has `embedded_data`: base64-decode and pack it.
- Else: leave its references as-is (it will be reported missing on load).
Raster, image, and thumbnail media follow the same "write if present, else keep the
existing row" rule so that paged-out content survives a save without being held in
memory.
## 10. Load semantics
### 10.1 Dispatch and version checks
Open the file, read `project.json`, parse `BeamProject`. Reject unless
`version == "1.0.0"` (§4.2). The container's `schema_version` is verified on open
(`≤` the reader's maximum).
### 10.2 Audio resolution
Per pool entry with a `media_id`: look up the media row. If its `codec` is a
**streamable** audio codec (`mp3`, `flac`, `ogg`/`oga`, `wav`/`wave`, `aiff`/`aif`,
`aac`, `m4a`, `opus`, `alac`, `caf`), the bytes are **streamed lazily** from the blob
at playback time (the entry keeps `media_id`, with `embedded_data`/`path` cleared).
Otherwise the bytes are read eagerly into `embedded_data`. A precomputed `Waveform`
blob, if present, is read into the entry.
### 10.3 Visual media paging
- **Raster** full-res pixels are **not** eagerly decoded; each keyframe is flagged
for fault-in and its pixels are paged from its `Raster` row on demand. Its
`RasterProxy` PNG **is** read and decoded eagerly so cold scrubs show a low-res
frame immediately.
- **Image assets** are **not** eagerly read; they page from their `ImageAsset` rows
on demand.
- **Thumbnail** packs are read eagerly per video clip.
### 10.4 Missing referenced files
A pool entry that is neither packed nor embedded, whose `relative_path` resolves to a
non-existent file, MUST be reported as a missing file (non-fatal). The host
application prompts the user to relocate it.
## 11. Legacy ZIP format (pre-SQLite)
Files whose magic is not `SQLite format 3\0` are read as a ZIP archive with this
internal layout:
- `project.json` — the same serialized `BeamProject` (`version` must be `"1.0.0"`).
- `media/audio/<name>.<ext>` — audio source files. The codec is taken from the
extension. **FLAC entries are decoded to PCM `f32` and re-encoded in memory as
IEEE-float WAV (WAV format tag 3)** before being base64-embedded; other formats are
embedded as-is. The entry's `relative_path` is cleared after extraction.
- `media/raster/<uuid>.png` — raster keyframe pixels, named `media/raster/<id>.png`
after the keyframe's `id`.
A reader MUST NOT modify a legacy ZIP on open; it loads into memory only. **The next
save migrates the project to SQLite** (the save sees a non-SQLite target, so it takes
the create-and-rename path and writes a fresh `.beam` database).
> Known limitation of the legacy reader: raster pixels are loaded only for
> **top-level** layers; raster keyframes nested inside groups/clips are not populated
> from a ZIP. The SQLite path recurses correctly. Re-saving to SQLite is the remedy.
## 12. Large-media policy
- `LARGE_MEDIA_THRESHOLD = 2 GiB`. Files at or above it trigger the packed-vs-
referenced decision below; smaller files are always packed (unless video audio,
which is always referenced).
- `LargeMediaMode``{ Ask, Pack, Reference }`:
- `Pack` — pack large files into the database.
- `Reference` — store large files by external path.
- `Ask` (default) — prompt the user on the first large import, then persist their
choice; treated as `Reference` at save time until answered.
- **The chosen mode is an application preference, not part of the `.beam` file.** It
is stored in the editor's `config.json` (`AppConfig.large_media_default`) under the
platform config directory, not in the project. Readers/writers of the format need
not consult it except to drive the packed/referenced save decision.
## 13. Conformance summary
A conforming **reader** MUST:
1. dispatch on the 16-byte magic (§3);
2. reject `schema_version` greater than supported and `version != "1.0.0"` (§4.2);
3. resolve packed media via `media`/`media_chunk` (§6.3) and referenced media
relative to the file (§6.4), reporting missing referenced files non-fatally;
4. compute derived media ids exactly as in §7.
A conforming **writer** MUST:
1. produce the four tables of §4 with `schema_version="1"` and `version="1.0.0"`;
2. store UUIDs as 16 big-endian bytes and chunk packed media at a uniform size with a
remainder final chunk, setting `total_len` correctly (§6.3);
3. perform each save in a single transaction and garbage-collect orphaned media (§9),
and prefer in-place writes that do not rewrite unchanged packed media.
## 14. Security considerations
- **Path traversal:** referenced media and legacy ZIP entries carry filesystem paths.
Readers SHOULD resolve relative paths against the project directory and SHOULD
reject or sandbox paths that escape it (`..`, absolute paths) when opening
untrusted files.
- **Resource limits:** `project.json` and packed blobs are attacker-controllable in
an untrusted file. Readers SHOULD bound JSON size and avoid loading entire large
blobs into memory (stream via §6.3) to resist memory-exhaustion.
- **Decoder safety:** audio/image bytes are handed to media decoders (Symphonia,
image, FFmpeg); keep those dependencies current, as they parse untrusted input.
- **Legacy ZIP bombs:** the ZIP path SHOULD enforce sane decompression-ratio/size
limits.
## 15. Non-normative notes / known quirks
These reflect the current reference implementation and are flagged for implementers
and future spec revisions:
- The project `version` check is exact-match with no compatibility window; a future
revision should define semantic-version tolerance before bumping it.
(Earlier revisions of this section flagged a hardcoded save sample rate, a vestigial
`RasterKeyframe.media_path` field, and unused `SaveSettings` fields; these have since
been fixed/removed.)
## 16. Inspecting a `.beam` file
Because the container is plain SQLite, any `.beam` (SQLite variant) can be inspected
with standard tooling, e.g.:
```sh
sqlite3 project.beam '.tables'
sqlite3 project.beam 'SELECT key,value FROM meta;'
sqlite3 project.beam 'SELECT hex(id),kind,codec,storage,total_len FROM media;'
sqlite3 project.beam 'SELECT data FROM project_json;' | jq .
```
## 17. References
- Container: `lightningbeam-ui/lightningbeam-core/src/beam_archive.rs`
- Save/load orchestration & `BeamProject`: `lightningbeam-ui/lightningbeam-core/src/file_io.rs`
- Audio pool entry: `daw-backend/src/audio/pool.rs`
- Document / scene tree: `lightningbeam-ui/lightningbeam-core/src/document.rs`
- RFC 2119 (requirement keywords), RFC 4122 (UUID), RFC 3339 (timestamps)

View File

@ -1,3 +1,38 @@
# 1.0.7-alpha:
Changes:
- HDR video support: PQ/HLG/BT.2020 video is now read correctly (decoded to scene-linear), with a per-document output mode (clip vs highlight rolloff) and 10-bit HDR export (HEVC Main10, PQ or HLG)
- Hardware-accelerated video decode (VAAPI) for both playback and export, including a GPU NV12 preview path; the editor now runs on a shared VAAPI-capable GPU device so decode → composite → encode stay GPU-resident
- SVG import and export for vector layers: export the current frame to .svg, and import .svg as a new vector layer (Ctrl+I)
- Export fit modes: choose Stretch, Letterbox, or Crop when the export resolution's aspect ratio differs from the document (video and image export)
- Videos imported directly and dragged in from the asset library now use the same placement, so they no longer end up with different aspect ratios
- Resizing the document now leaves raster layers untouched; the Info Panel shows the active raster layer's size and a "Layer to document size" button (scale or expand/crop)
- The active raster layer now shows a dashed outline on the canvas
- H.264 export gained a color-range option (Limited/TV or Full/PC)
- Hide/Show Layer now works
Bugfixes:
- Fix sped-up and jerky 4K video playback (frame-index frame cache + request-based seeking)
- Fix washed-out HDR/10-bit video (propagate stream color tags to hardware frames, P010 import)
- Fix the final frame(s) of a clip occasionally failing to render at the end of the stream
- Fix the CPU export color path producing shifted colors on unusual resolutions (BT.709 + honor the chosen range)
- Fix fills occasionally vanishing after a paint-bucket or lasso cut
- Fix silent gaps in exported audio
- Fix black video thumbnails and decoder thrashing
- Fix file-descriptor and GPU-memory leaks on hardware-import error paths
- SVG export now omits hidden layers; SVG import respects gradient opacity
- Fix the macOS/Windows build (hardware export is Linux-only)
# 1.0.6-alpha:
Changes:
- Hardware-accelerated H.264 video export: each frame is rendered and encoded on the GPU (zero-copy VAAPI), roughly 2x faster, with automatic fallback to software encoding when hardware acceleration isn't available (Linux, Intel/AMD only for now)
- Video export now runs on a background thread, so the UI stays responsive during export and edits made while exporting no longer affect the output
- Grouped and nested video clips now composite on the GPU path
- Video is now packed into and streamed from the .beam project container
Bugfixes:
- Fix an export hang when a video's audio track is shorter than the video
- Fix a sample key-range overlap bug in instruments
# 1.0.5-alpha:
Changes:
- Add shape tweens (morph vector geometry between keyframes)

View File

@ -686,11 +686,17 @@ Phase 1a and Phase 2 can ship now; everything else waits on Phase 0 (the contain
`load_file_into_pool`'s full decode with the `do_import_audio` branching (PCM → mmap, compressed
`from_compressed` placeholder). Higher risk (touches the working referenced path); packed
covers the common <2GB case first.
- [ ] **Deferred (follow-up): packed video streaming.** Let small videos be packed into the `.beam`
(a `MediaKind::Video` blob, `VideoClip` referencing it by id) and stream **both frames and audio**
from the DB blob via FFmpeg. ffmpeg-next has no custom-I/O wrapper, so this needs an
`AVIOContext`-over-`BlobReader` shim via raw FFI. **Decision (user):** that FFI wrapper lives in
its **own crate, version-pinned to the ffmpeg version**, isolating the unsafe + the ABI coupling.
- [x] **DONE: packed video streaming.** Small videos pack into the `.beam`
(a `MediaKind::Video` blob at the clip id, `VideoClip.media_id` referencing it) and stream **both
frames and audio** from the DB blob via FFmpeg. The `AVIOContext`-over-`Read+Seek` shim lives in
the new `ffmpeg-blob-io` crate (`BlobInput`, version-pinned `=8.0.0`/`=8.0.1`), isolating the
unsafe + ABI coupling. Frames: `video.rs` `VideoSource{Path,Packed}` opens a fresh `BlobReader`
per decoder/seek/scan. Audio: `VideoAudioReader::open_source` over the same blob (the
`disk_reader.rs` `StreamSource` blocker is removed); save points the linked video-audio pool
entry's `media_id` at the video row so it streams from the same blob. Tests: ffmpeg-blob-io AVIO
unit tests (WAV via Cursor + seek + open/drop loop), core `packed_video_stream` (blob→AVIO→Input),
`beam_archive` packed-video round-trip, daw-backend `open_source` (compiles; can't link in the
container — user runtime-verifies actual A/V playback).
- [~] Phase 1c — video embedded-audio track ← **stopgap shipped; proper design next**
- [x] Stopgap: `extract_audio_from_video_to_wav` streams to a temp WAV → `import_audio_sync`
(mmap). Fixed the ~2.8GB-`Vec<f32>` OOM. But writes the whole WAV to `/tmp` (fills

701
beam_inspector.py Executable file
View File

@ -0,0 +1,701 @@
#!/usr/bin/env python3
"""
Lightningbeam .beam File Inspector
A command-line tool to inspect .beam project files.
"""
import argparse
import json
import sqlite3
import sys
import uuid as uuidlib
import zipfile
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional
# First 16 bytes of any SQLite 3 database.
SQLITE_MAGIC = b"SQLite format 3\x00"
# media.kind values (see BEAM_FILE_FORMAT.md §6.1).
MEDIA_KIND_NAMES = {
0: "Audio", 1: "Video", 2: "Raster", 3: "ImageAsset",
4: "Waveform", 5: "Thumbnail", 6: "RasterProxy",
}
# media.storage values (§6.2).
MEDIA_STORAGE_NAMES = {0: "Packed", 1: "Referenced"}
def _is_sqlite(path: Path) -> bool:
"""True if the file begins with the SQLite 3 header magic."""
try:
with open(path, "rb") as f:
return f.read(16) == SQLITE_MAGIC
except OSError:
return False
def _human_size(n: int) -> str:
f = float(n)
for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
if f < 1024 or unit == "TiB":
return f"{f:.0f} {unit}" if unit == "B" else f"{f:.1f} {unit}"
f /= 1024
return f"{n} B"
class BeamInspector:
"""Inspector for .beam project files (SQLite container, or legacy ZIP)."""
def __init__(self, beam_file: Path):
self.beam_file = beam_file
self.project_data: Optional[Dict[str, Any]] = None
# SQLite (current) vs ZIP (legacy) container.
self.is_sqlite = _is_sqlite(beam_file)
# Populated for SQLite files by load():
self.meta: Dict[str, str] = {}
self.media: List[Dict[str, Any]] = [] # rows from the media table
self.media_by_id: Dict[str, Dict[str, Any]] = {} # uuid-string -> row
def _connect(self) -> sqlite3.Connection:
"""Open the SQLite container read-only."""
uri = self.beam_file.resolve().as_uri() + "?mode=ro"
return sqlite3.connect(uri, uri=True)
def load(self) -> bool:
"""Load and parse the .beam file (container-agnostic)."""
try:
if self.is_sqlite:
return self._load_sqlite()
return self._load_zip()
except Exception as e:
print(f"Error loading .beam file: {e}", file=sys.stderr)
return False
def _load_zip(self) -> bool:
with zipfile.ZipFile(self.beam_file, 'r') as zip_ref:
with zip_ref.open('project.json') as f:
self.project_data = json.load(f)
return True
def _load_sqlite(self) -> bool:
con = self._connect()
try:
cur = con.cursor()
row = cur.execute("SELECT data FROM project_json WHERE id = 0").fetchone()
if not row:
raise ValueError("archive has no project.json row")
self.project_data = json.loads(row[0])
self.meta = {k: v for (k, v) in cur.execute("SELECT key, value FROM meta")}
for (idb, kind, codec, storage, ext_path, total_len,
channels, sample_rate, width, height) in cur.execute(
"SELECT id, kind, codec, storage, ext_path, total_len, "
"channels, sample_rate, width, height FROM media"
):
uid = str(uuidlib.UUID(bytes=bytes(idb)))
info = {
"uuid": uid,
"kind": kind,
"codec": codec,
"storage": storage,
"ext_path": ext_path,
"total_len": total_len or 0,
"channels": channels,
"sample_rate": sample_rate,
"width": width,
"height": height,
}
self.media.append(info)
self.media_by_id[uid] = info
finally:
con.close()
return True
def show_info(self):
"""Display basic project information."""
if not self.project_data:
return
print("=" * 60)
print("PROJECT INFORMATION")
print("=" * 60)
print(f"Container: {'SQLite' if self.is_sqlite else 'Legacy ZIP'}")
if self.is_sqlite:
print(f"Schema Ver: {self.meta.get('schema_version', 'Unknown')}")
print(f"Version: {self.project_data.get('version', 'Unknown')}")
print(f"Created: {self.project_data.get('created', 'Unknown')}")
print(f"Modified: {self.project_data.get('modified', 'Unknown')}")
ui_state = self.project_data.get('ui_state', {})
print(f"\nProject Name: {ui_state.get('name', 'Unnamed')}")
print(f"ID: {ui_state.get('id', 'Unknown')}")
print(f"Dimensions: {ui_state.get('width', 0):.0f} x {ui_state.get('height', 0):.0f}")
print(f"Framerate: {ui_state.get('framerate', 0):.1f} fps")
print(f"Duration: {ui_state.get('duration', 0):.2f} seconds")
bg = ui_state.get('background_color', {})
print(f"Background: rgba({bg.get('r', 0)}, {bg.get('g', 0)}, {bg.get('b', 0)}, {bg.get('a', 255)})")
audio_backend = self.project_data.get('audio_backend', {})
print(f"\nSample Rate: {audio_backend.get('sample_rate', 0)} Hz")
def show_clips(self):
"""Display clips information."""
if not self.project_data:
return
ui_state = self.project_data.get('ui_state', {})
vector_clips = ui_state.get('vector_clips', {})
video_clips = ui_state.get('video_clips', {})
audio_clips = ui_state.get('audio_clips', {})
print("\n" + "=" * 60)
print("CLIPS")
print("=" * 60)
print(f"Vector Clips: {len(vector_clips)}")
print(f"Video Clips: {len(video_clips)}")
print(f"Audio Clips: {len(audio_clips)}")
if vector_clips:
print("\nVector Clips:")
for clip_id, clip in vector_clips.items():
print(f" - {clip.get('name', 'Unnamed')} (ID: {clip_id[:8]}...)")
if video_clips:
print("\nVideo Clips:")
for clip_id, clip in video_clips.items():
print(f" - {clip.get('name', 'Unnamed')} (ID: {clip_id[:8]}...)")
if audio_clips:
print("\nAudio Clips:")
for clip_id, clip in audio_clips.items():
print(f" - {clip.get('name', 'Unnamed')} (ID: {clip_id[:8]}...)")
def show_layers(self):
"""Display layer hierarchy."""
if not self.project_data:
return
ui_state = self.project_data.get('ui_state', {})
root = ui_state.get('root', {})
print("\n" + "=" * 60)
print("LAYER HIERARCHY")
print("=" * 60)
def print_layer(layer: Dict[str, Any], indent: int = 0):
# Handle case where layer might not be a dictionary
if not isinstance(layer, dict):
prefix = " " * indent
print(f"{prefix}- ERROR: Layer is {type(layer).__name__}, not a dict: {repr(layer)[:50]}")
return
layer_type = layer.get('type', 'Unknown')
layer_name = layer.get('name', 'Unnamed')
layer_id = layer.get('id', 'Unknown')
prefix = " " * indent
# Handle layer_id that might not be a string
id_str = layer_id[:8] + "..." if isinstance(layer_id, str) and len(layer_id) > 8 else str(layer_id)
print(f"{prefix}- [{layer_type}] {layer_name} (ID: {id_str})")
# Recursively print children
children = layer.get('children', [])
for child in children:
print_layer(child, indent + 1)
print_layer(root)
def show_tracks(self):
"""Display audio tracks information."""
if not self.project_data:
return
audio_backend = self.project_data.get('audio_backend', {})
project = audio_backend.get('project', {})
tracks_dict = project.get('tracks', {})
root_tracks = project.get('root_tracks', [])
master_track = project.get('master_track', {})
print("\n" + "=" * 60)
print("AUDIO TRACKS")
print("=" * 60)
print(f"Total Tracks: {len(tracks_dict)}")
# Iterate through root_tracks in order
for track_id in root_tracks:
track_id_str = str(track_id)
if track_id_str not in tracks_dict:
print(f"\n Track {track_id}: ERROR - Track ID not found in tracks dict")
continue
track_node = tracks_dict[track_id_str]
# TrackNode is an enum with variants like {"Audio": {...}} or {"Midi": {...}}
if not isinstance(track_node, dict):
print(f"\n Track {track_id}: ERROR - Track node is {type(track_node).__name__}, not a dict")
continue
# Extract the variant (Audio, Midi, or Group)
if len(track_node) != 1:
print(f"\n Track {track_id}: ERROR - Track node has unexpected structure")
continue
track_type, track_data = list(track_node.items())[0]
if not isinstance(track_data, dict):
print(f"\n Track {track_id}: ERROR - Track data is {type(track_data).__name__}, not a dict")
continue
track_name = track_data.get('name', f'Track {track_id}')
muted = track_data.get('muted', False)
solo = track_data.get('solo', False)
volume = track_data.get('volume', 1.0)
pan = track_data.get('pan', 0.0)
status = []
if muted:
status.append('MUTED')
if solo:
status.append('SOLO')
status_str = f" [{', '.join(status)}]" if status else ""
print(f"\n Track {track_id}: {track_name}{status_str}")
print(f" Type: {track_type}")
print(f" Volume: {volume:.2f}")
print(f" Pan: {pan:.2f}")
if track_type == "Midi":
print(f" Instrument: {track_data.get('instrument', 'Unknown')}")
notes = track_data.get('notes', [])
print(f" Notes: {len(notes)}")
preset = track_data.get('instrument_graph_preset')
if preset:
nodes = preset.get('nodes', [])
conns = preset.get('connections', [])
print(f" Graph: {len(nodes)} nodes, {len(conns)} connections")
else:
print(f" Graph: (no preset saved)")
elif track_type == "Audio":
clips = track_data.get('clips', [])
print(f" Clips: {len(clips)}")
preset = track_data.get('effects_graph_preset')
if preset:
nodes = preset.get('nodes', [])
conns = preset.get('connections', [])
print(f" Graph: {len(nodes)} nodes, {len(conns)} connections")
else:
print(f" Graph: (no preset saved)")
print(f"\nMaster Track:")
print(f" Volume: {master_track.get('volume', 1.0):.2f}")
def show_graphs(self):
"""Display detailed node graph information for all tracks."""
if not self.project_data:
return
audio_backend = self.project_data.get('audio_backend', {})
project = audio_backend.get('project', {})
tracks_dict = project.get('tracks', {})
print("\n" + "=" * 60)
print("NODE GRAPHS")
print("=" * 60)
for track_id_str, track_node in tracks_dict.items():
if not isinstance(track_node, dict) or len(track_node) != 1:
continue
track_type, track_data = list(track_node.items())[0]
track_name = track_data.get('name', f'Track {track_id_str}')
# Get the appropriate preset
if track_type == "Audio":
preset = track_data.get('effects_graph_preset')
graph_label = "Effects Graph"
elif track_type == "Midi":
preset = track_data.get('instrument_graph_preset')
graph_label = "Instrument Graph"
else:
continue
print(f"\n Track {track_id_str}: {track_name} ({track_type}) - {graph_label}")
if not preset:
print(f" ** NO PRESET SAVED **")
continue
nodes = preset.get('nodes', [])
connections = preset.get('connections', [])
output_node = preset.get('output_node')
midi_targets = preset.get('midi_targets', [])
metadata = preset.get('metadata', {})
if metadata:
print(f" Preset Name: {metadata.get('name', 'Unknown')}")
print(f" Nodes ({len(nodes)}):")
for i, node in enumerate(nodes):
node_type = node.get('node_type', '?')
node_name = node.get('name', node_type)
params = node.get('parameters', {})
pos_x = node.get('position_x', 0)
pos_y = node.get('position_y', 0)
markers = []
if output_node is not None and i == output_node:
markers.append("OUTPUT")
if i in midi_targets:
markers.append("MIDI TARGET")
marker_str = f" [{', '.join(markers)}]" if markers else ""
print(f" [{i}] {node_type}{marker_str} (name={node_name}, pos={pos_x:.0f},{pos_y:.0f})")
if params:
for param_name, param_val in params.items():
print(f" {param_name} = {param_val}")
print(f" Connections ({len(connections)}):")
for conn in connections:
src = conn.get('from_node', '?')
src_port = conn.get('from_output', '?')
dst = conn.get('to_node', '?')
dst_port = conn.get('to_input', '?')
print(f" [{src}]:{src_port} -> [{dst}]:{dst_port}")
# Also show layer_to_track_map if present
track_map = audio_backend.get('layer_to_track_map', {})
if track_map:
print(f"\n Layer-to-Track Mapping ({len(track_map)} entries):")
for layer_id, track_id in track_map.items():
print(f" {layer_id[:8]}... -> Track {track_id}")
else:
print(f"\n Layer-to-Track Mapping: (none saved)")
def show_audio_pool(self):
"""Display audio pool entries."""
if not self.project_data:
return
audio_backend = self.project_data.get('audio_backend', {})
pool_entries = audio_backend.get('audio_pool_entries', [])
print("\n" + "=" * 60)
print("AUDIO POOL")
print("=" * 60)
print(f"Total Entries: {len(pool_entries)}")
for entry in pool_entries:
pool_index = entry.get('pool_index', '?')
name = entry.get('name', 'Unnamed')
relative_path = entry.get('relative_path')
media_id = entry.get('media_id')
channels = entry.get('channels', 0)
sample_rate = entry.get('sample_rate', 0)
has_embedded = entry.get('embedded_data') is not None
# Storage precedence matches the loader (§8.5/§9.3):
# packed (media_id) > external (relative_path) > embedded > unresolved.
row = self.media_by_id.get(media_id) if media_id else None
if media_id:
size = f", {_human_size(row['total_len'])}" if row else ""
codec = f", {row['codec']}" if row else ""
storage_type = f"Packed in DB{codec}{size}"
elif relative_path and not self.is_sqlite and relative_path.startswith("media/audio/"):
# Legacy ZIP: media/audio/* lives inside the archive, not external.
storage_type = "Embedded (in ZIP)"
elif relative_path:
storage_type = "External reference"
elif has_embedded:
storage_type = "Embedded (inline base64)"
else:
storage_type = "Unresolved (missing)"
print(f"\n [{pool_index}] {name}")
if media_id:
print(f" Media ID: {media_id}")
print(f" Path: {relative_path if relative_path else 'N/A'}")
print(f" Storage: {storage_type}")
print(f" Channels: {channels}")
print(f" Sample Rate: {sample_rate} Hz")
def show_media(self):
"""Display the SQLite media store (the `media` table)."""
if not self.is_sqlite:
print("\n(media table view applies to SQLite .beam files; "
"use --zip for legacy archives)", file=sys.stderr)
return
print("\n" + "=" * 60)
print("MEDIA STORE")
print("=" * 60)
print(f"Total media rows: {len(self.media)}")
# Summary by kind.
by_kind: Dict[str, List[Dict[str, Any]]] = {}
for m in self.media:
by_kind.setdefault(MEDIA_KIND_NAMES.get(m["kind"], f"Kind {m['kind']}"), []).append(m)
if by_kind:
print("\nBy kind:")
for kind_name, rows in sorted(by_kind.items()):
packed = sum(r["total_len"] for r in rows if r["storage"] == 0)
print(f" {kind_name:<12} {len(rows):>4} ({_human_size(packed)} packed)")
print(f"\n{'Kind':<12} {'Storage':<11} {'Codec':<6} {'Size':>10} UUID / details")
print("-" * 78)
for m in self.media:
kind = MEDIA_KIND_NAMES.get(m["kind"], f"Kind {m['kind']}")
storage = MEDIA_STORAGE_NAMES.get(m["storage"], f"Stor {m['storage']}")
size = _human_size(m["total_len"]) if m["storage"] == 0 else "-"
detail = m["uuid"]
extra = []
if m["channels"] is not None:
extra.append(f"{m['channels']}ch@{m['sample_rate']}Hz")
if m["width"] is not None:
extra.append(f"{m['width']}x{m['height']}")
if m["storage"] == 1 and m["ext_path"]:
extra.append(f"-> {m['ext_path']}")
if extra:
detail += " " + " ".join(extra)
print(f"{kind:<12} {storage:<11} {m['codec']:<6} {size:>10} {detail}")
def show_container(self):
"""Show whichever container structure applies to this file."""
if self.is_sqlite:
self.show_media()
else:
self.show_zip_structure()
def show_zip_structure(self):
"""Display the ZIP file structure."""
if self.is_sqlite:
print("\n(this is a SQLite .beam; use --media for the media store)",
file=sys.stderr)
return
print("\n" + "=" * 60)
print("ZIP ARCHIVE STRUCTURE")
print("=" * 60)
try:
with zipfile.ZipFile(self.beam_file, 'r') as zip_ref:
total_size = 0
compressed_size = 0
print(f"{'File':<40} {'Size':>12} {'Compressed':>12} {'Method':<10}")
print("-" * 80)
for info in zip_ref.infolist():
size = info.file_size
comp_size = info.compress_size
method = "DEFLATE" if info.compress_type == 8 else "STORED" if info.compress_type == 0 else f"Type {info.compress_type}"
total_size += size
compressed_size += comp_size
print(f"{info.filename:<40} {size:>12,} {comp_size:>12,} {method:<10}")
print("-" * 80)
print(f"{'TOTAL':<40} {total_size:>12,} {compressed_size:>12,}")
if total_size > 0:
ratio = (1 - compressed_size / total_size) * 100
print(f"\nCompression Ratio: {ratio:.1f}%")
except Exception as e:
print(f"Error reading ZIP structure: {e}", file=sys.stderr)
def extract_json(self, output_path: Optional[Path] = None):
"""Extract project.json to a file or stdout."""
try:
if self.is_sqlite:
con = self._connect()
try:
row = con.execute("SELECT data FROM project_json WHERE id = 0").fetchone()
finally:
con.close()
if not row:
raise ValueError("archive has no project.json row")
json_data = row[0].encode("utf-8") if isinstance(row[0], str) else row[0]
else:
with zipfile.ZipFile(self.beam_file, 'r') as zip_ref:
json_data = zip_ref.read('project.json')
if output_path:
output_path.write_bytes(json_data)
print(f"Extracted project.json to: {output_path}")
else:
data = json.loads(json_data)
print(json.dumps(data, indent=2))
except Exception as e:
print(f"Error extracting project.json: {e}", file=sys.stderr)
def extract_media(self, output_dir: Path):
"""Extract all media to a directory.
SQLite: each packed media row is reassembled from its chunks and written
as ``<uuid>.<codec>``; referenced rows are reported, not copied.
ZIP (legacy): the ``media/`` entries are extracted preserving paths.
"""
try:
if self.is_sqlite:
self._extract_media_sqlite(output_dir)
else:
self._extract_media_zip(output_dir)
except Exception as e:
print(f"Error extracting media: {e}", file=sys.stderr)
def _extract_media_sqlite(self, output_dir: Path):
con = self._connect()
try:
rows = con.execute(
"SELECT id, kind, codec, storage, ext_path, total_len FROM media"
).fetchall()
if not rows:
print("No media rows found in archive.")
return
output_dir.mkdir(parents=True, exist_ok=True)
written = 0
for (idb, kind, codec, storage, ext_path, total_len) in rows:
uid = str(uuidlib.UUID(bytes=bytes(idb)))
kind_name = MEDIA_KIND_NAMES.get(kind, f"kind{kind}")
if storage == 1: # Referenced
print(f"Referenced (not copied): {uid} [{kind_name}] -> {ext_path}")
continue
chunks = con.execute(
"SELECT bytes FROM media_chunk WHERE media_id = ? ORDER BY chunk_index",
(idb,),
).fetchall()
data = b"".join(bytes(c[0]) for c in chunks)
out = output_dir / f"{uid}.{codec}"
out.write_bytes(data)
print(f"Extracted: {out.name} [{kind_name}] {_human_size(len(data))}")
written += 1
print(f"\nExtracted {written} packed media item(s) to: {output_dir}")
finally:
con.close()
def _extract_media_zip(self, output_dir: Path):
with zipfile.ZipFile(self.beam_file, 'r') as zip_ref:
media_files = [f for f in zip_ref.namelist() if f.startswith('media/')]
if not media_files:
print("No media files found in archive.")
return
output_dir.mkdir(parents=True, exist_ok=True)
for media_file in media_files:
output_path = output_dir / media_file
output_path.parent.mkdir(parents=True, exist_ok=True)
with zip_ref.open(media_file) as source:
output_path.write_bytes(source.read())
print(f"Extracted: {media_file}")
print(f"\nExtracted {len(media_files)} media file(s) to: {output_dir}")
def main():
parser = argparse.ArgumentParser(
description="Inspect Lightningbeam .beam project files",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
%(prog)s project.beam # Show all information
%(prog)s project.beam --info # Show only basic info
%(prog)s project.beam --tracks # Show only tracks
%(prog)s project.beam --media # Show the SQLite media store
%(prog)s project.beam --zip # Show legacy ZIP structure
%(prog)s project.beam --extract-json # Print project.json to stdout
%(prog)s project.beam --extract-json out.json # Save project.json
%(prog)s project.beam --extract-media ./media # Extract media files
Handles both current SQLite .beam files and legacy ZIP .beam files
(detected automatically by the file's magic bytes).
"""
)
parser.add_argument('beam_file', type=Path, help='.beam file to inspect')
parser.add_argument('--info', action='store_true', help='Show project information')
parser.add_argument('--clips', action='store_true', help='Show clips')
parser.add_argument('--layers', action='store_true', help='Show layer hierarchy')
parser.add_argument('--tracks', action='store_true', help='Show audio tracks')
parser.add_argument('--graphs', action='store_true', help='Show node graph details for all tracks')
parser.add_argument('--pool', action='store_true', help='Show audio pool')
parser.add_argument('--media', action='store_true', help='Show the SQLite media store')
parser.add_argument('--zip', action='store_true', help='Show legacy ZIP structure')
parser.add_argument('--extract-json', nargs='?', const=True, metavar='OUTPUT',
help='Extract project.json (to file or stdout)')
parser.add_argument('--extract-media', type=Path, metavar='DIR',
help='Extract media files to directory')
args = parser.parse_args()
# Validate input file
if not args.beam_file.exists():
print(f"Error: File not found: {args.beam_file}", file=sys.stderr)
sys.exit(1)
if not args.beam_file.suffix == '.beam':
print(f"Warning: File does not have .beam extension: {args.beam_file}", file=sys.stderr)
# Create inspector
inspector = BeamInspector(args.beam_file)
# Handle extract operations (don't need to load project.json)
if args.extract_json:
if args.extract_json is True:
inspector.extract_json()
else:
inspector.extract_json(Path(args.extract_json))
return
if args.extract_media:
inspector.extract_media(args.extract_media)
return
# Load the beam file
if not inspector.load():
sys.exit(1)
# If no specific flags, show everything
show_all = not any([args.info, args.clips, args.layers, args.tracks,
args.graphs, args.pool, args.media, args.zip])
if show_all or args.info:
inspector.show_info()
if show_all or args.clips:
inspector.show_clips()
if show_all or args.layers:
inspector.show_layers()
if show_all or args.tracks:
inspector.show_tracks()
if show_all or args.graphs:
inspector.show_graphs()
if show_all or args.pool:
inspector.show_audio_pool()
if show_all:
# Show whichever container structure applies to this file.
inspector.show_container()
elif args.media:
inspector.show_media()
elif args.zip:
inspector.show_zip_structure()
if __name__ == '__main__':
main()

210
beam_inspector_README.md Normal file
View File

@ -0,0 +1,210 @@
# Lightningbeam .beam File Inspector
A Python command-line tool to inspect and analyze `.beam` project files.
Handles **both** container formats automatically (detected by the file's magic
bytes): the current **SQLite** `.beam` and the **legacy ZIP** `.beam`. See
[BEAM_FILE_FORMAT.md](./BEAM_FILE_FORMAT.md) for the format specification.
## Features
- **Project Information**: Container type, schema version, metadata, dimensions, framerate, duration, background color
- **Clips Analysis**: List all vector, video, and audio clips
- **Layer Hierarchy**: Display the complete layer tree structure
- **Audio Tracks**: Show all audio/MIDI tracks with their settings
- **Audio Pool**: List all audio files and their storage details
- **Media Store** (SQLite): List the `media` table — kinds, storage, codecs, sizes
- **ZIP Structure** (legacy): Examine the internal ZIP archive structure and compression
- **Extraction**: Extract `project.json` or media files
## Installation
The tool requires Python 3.6+ with no external dependencies (uses only the standard
library; `sqlite3` ships with Python).
```bash
chmod +x beam_inspector.py
```
## Usage
### Show All Information
```bash
./beam_inspector.py project.beam
```
### Show Specific Sections
```bash
# Basic project info only
./beam_inspector.py project.beam --info
# Audio tracks only
./beam_inspector.py project.beam --tracks
# Audio pool entries
./beam_inspector.py project.beam --pool
# Layer hierarchy
./beam_inspector.py project.beam --layers
# Clips summary
./beam_inspector.py project.beam --clips
# Media store (SQLite files): the media table — kinds, storage, codecs, sizes
./beam_inspector.py project.beam --media
# ZIP archive structure (legacy ZIP files only)
./beam_inspector.py project.beam --zip
```
When run with no section flags, the tool shows everything and automatically
picks the media-store view (SQLite) or ZIP-structure view (legacy) for the file.
### Extract Files
```bash
# Print project.json to stdout (pretty-printed)
./beam_inspector.py project.beam --extract-json
# Save project.json to a file
./beam_inspector.py project.beam --extract-json output.json
# Extract all media files to a directory
./beam_inspector.py project.beam --extract-media ./extracted_media
```
## Example Output
```
============================================================
PROJECT INFORMATION
============================================================
Container: SQLite
Schema Ver: 1
Version: 1.0.0
Created: 2025-12-01T12:00:00Z
Modified: 2025-12-01T12:30:00Z
Project Name: My Animation
ID: 550e8400-e29b-41d4-a716-446655440000
Dimensions: 1920 x 1080
Framerate: 60.0 fps
Duration: 10.00 seconds
Background: rgba(255, 255, 255, 255)
Sample Rate: 48000 Hz
============================================================
AUDIO TRACKS
============================================================
Total Tracks: 2
Track 0: Piano [SOLO]
Type: Midi
Volume: 0.80
Pan: 0.00
Instrument: Piano
Notes: 24
Track 1: Background Music
Type: Audio
Volume: 0.60
Pan: 0.00
Clips: 3
Master Track:
Volume: 1.00
============================================================
AUDIO POOL
============================================================
Total Entries: 2
[0] C2.mp3
Media ID: e7e555a6-85f1-4faa-bfd4-c6e4b790986a
Path: N/A
Storage: Packed in DB, mp3, 412.0 KiB
Channels: 2
Sample Rate: 44100 Hz
[1] Background.flac
Media ID: a18c0e22-1d3b-4c77-9f0a-2b5e6c4d8e10
Path: N/A
Storage: Packed in DB, flac, 6.4 MiB
Channels: 2
Sample Rate: 48000 Hz
```
## Understanding the Output
### Storage Types
- **Packed in DB**: Audio bytes are chunked into the SQLite `media` table (the entry's `media_id` resolves to a packed row). Current default for most audio.
- **External reference**: File is referenced from the filesystem by `relative_path` (e.g. large media or video audio).
- **Embedded (inline base64)**: Bytes stored directly in `project.json` (`embedded_data`) — legacy/fallback.
- **Embedded (in ZIP)**: Legacy ZIP files only — bytes stored inside the archive under `media/audio/`.
- **Unresolved (missing)**: No packed row, external file, or embedded data — reported as a missing file on load.
### Track Types
- **Audio**: Traditional audio track with clips from the audio pool
- **Midi**: MIDI track with note events and virtual instrument
### Layer Types
Based on the `AnyLayer` enum:
- **Group**: Container for other layers
- **Vector**: Vector graphics layer
- **Video**: Video clip layer
- **Audio**: Audio waveform layer
- **Image**: Raster image layer
- **Text**: Text layer
## Advanced Usage
### Combine with Other Tools
```bash
# Pretty-print and page through JSON
./beam_inspector.py project.beam --extract-json | less
# Search for specific content in JSON
./beam_inspector.py project.beam --extract-json | grep "sample_rate"
# Count total audio files
./beam_inspector.py project.beam --pool | grep "^\[" | wc -l
# Extract and process media files
# SQLite: writes flat <uuid>.<codec> files; ZIP: preserves media/ paths
./beam_inspector.py project.beam --extract-media /tmp/media
ls -lh /tmp/media/
```
### Scripting
```python
from beam_inspector import BeamInspector
from pathlib import Path
inspector = BeamInspector(Path("project.beam"))
if inspector.load():
# Access parsed data
version = inspector.project_data['version']
tracks = inspector.project_data['audio_backend']['project']['tracks']
print(f"Found {len(tracks)} tracks in version {version}")
```
## Troubleshooting
### "Error loading .beam file"
- Ensure the file is a valid SQLite database (current) or ZIP archive (legacy)
- Check that `project.json` exists (the `project_json` table for SQLite, or a top-level entry for ZIP)
- Verify the JSON is well-formed
### "File not found"
- Provide the full path to the .beam file
- Check file permissions
### Missing media files
- External references may point to files that don't exist
- Use `--extract-media` to see what's actually in the archive
- Check the `relative_path` values in `--pool` output
## See Also
- [BEAM_FILE_FORMAT.md](./BEAM_FILE_FORMAT.md) - Complete .beam file format specification
- [Lightningbeam Documentation](./README.md) - Main project documentation

View File

@ -23,6 +23,8 @@ memmap2 = "0.9"
# Audio export
hound = "3.5"
ffmpeg-next = "8.0" # For MP3/AAC encoding
# AVIO-over-Read+Seek shim: stream a packed video's audio track from the SQLite blob.
ffmpeg-blob-io = { path = "../ffmpeg-blob-io" }
# Node-based audio graph dependencies
dasp_graph = "0.11"

View File

@ -77,6 +77,12 @@ pub struct ReadAheadBuffer {
/// When true, `render_from_file` will block-wait for frames instead of
/// returning silence on buffer miss. Used during offline export.
export_mode: AtomicBool,
/// Set by the disk reader when the source reaches EOF (no more frames will
/// ever be decoded past the current valid range). Lets export-mode block-waits
/// give up immediately for frames past the end of the audio (e.g. a video whose
/// audio track is shorter than the video) instead of timing out per chunk.
/// Cleared on `reset` (a seek may decode fresh data again).
finished: AtomicBool,
}
// SAFETY: See the doc comment on ReadAheadBuffer for the full safety argument.
@ -112,6 +118,7 @@ impl ReadAheadBuffer {
sample_rate,
target_frame: AtomicU64::new(0),
export_mode: AtomicBool::new(false),
finished: AtomicBool::new(false),
}
}
@ -216,11 +223,24 @@ impl ReadAheadBuffer {
self.export_mode.load(Ordering::Acquire)
}
/// Mark that the source has reached EOF (set by the disk reader).
pub fn set_finished(&self, finished: bool) {
self.finished.store(finished, Ordering::Release);
}
/// True once the source hit EOF: no frames past the current valid range
/// will ever be decoded (until a `reset`/seek).
pub fn is_finished(&self) -> bool {
self.finished.load(Ordering::Acquire)
}
/// Reset the buffer to start at `new_start` with zero valid frames.
/// Called by the **disk reader thread** (producer) after a seek.
pub fn reset(&self, new_start: u64) {
self.valid_frames.store(0, Ordering::Release);
self.start_frame.store(new_start, Ordering::Release);
// A seek may decode fresh data past the previous EOF position.
self.finished.store(false, Ordering::Release);
}
/// Write interleaved samples into the buffer, extending the valid range.
@ -571,8 +591,37 @@ impl CompressedReader {
///
/// Public (vs. the private `CompressedReader`) only so integration tests can
/// exercise it directly; treat it as crate-internal.
/// Adapts a host `MediaByteSource` (Read+Seek+Send+Sync) to the `ffmpeg-blob-io`
/// `BlobSource` (Read+Seek+Send) so a packed video can be demuxed from a blob.
struct ByteSourceAdapter(Box<dyn MediaByteSource>);
impl std::io::Read for ByteSourceAdapter {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.0.read(buf)
}
}
impl std::io::Seek for ByteSourceAdapter {
fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
self.0.seek(pos)
}
}
/// A demuxer input for video-audio: a video file path, or a container blob streamed
/// via the AVIO shim. Both expose the same ffmpeg `Input`.
enum AudioInput {
Path(ffmpeg_next::format::context::Input),
Blob(ffmpeg_blob_io::BlobInput),
}
impl AudioInput {
fn get(&mut self) -> &mut ffmpeg_next::format::context::Input {
match self {
AudioInput::Path(i) => i,
AudioInput::Blob(b) => b.input_mut(),
}
}
}
pub struct VideoAudioReader {
input: ffmpeg_next::format::context::Input,
input: AudioInput,
decoder: ffmpeg_next::decoder::Audio,
/// Built lazily from the first decoded frame's format/layout → interleaved f32.
resampler: Option<ffmpeg_next::software::resampling::Context>,
@ -605,15 +654,30 @@ impl VideoAudioReader {
self.total_frames
}
/// Open the audio track of a video **file**.
pub fn open(path: &Path) -> Result<Self, String> {
ffmpeg_next::init().map_err(|e| e.to_string())?;
let input = ffmpeg_next::format::input(&path)
.map_err(|e| format!("Failed to open media: {}", e))?;
Self::from_input(AudioInput::Path(input))
}
/// Open the audio track of a video streamed from a **byte source** (a packed
/// `.beam` video blob), via the `ffmpeg-blob-io` AVIO shim. `ext` is a container
/// hint (e.g. `"mp4"`).
pub fn open_source(src: Box<dyn MediaByteSource>, ext: Option<&str>) -> Result<Self, String> {
ffmpeg_next::init().map_err(|e| e.to_string())?;
let blob = ffmpeg_blob_io::BlobInput::open(Box::new(ByteSourceAdapter(src)), ext)
.map_err(|e| format!("Failed to open packed video audio: {}", e))?;
Self::from_input(AudioInput::Blob(blob))
}
fn from_input(mut input: AudioInput) -> Result<Self, String> {
// Pull stream scalars + build the decoder inside a scope so the stream
// borrow of `input` ends before we use `input` again.
let (stream_index, time_base, stream_duration, decoder) = {
let stream = input
.get()
.streams()
.best(ffmpeg_next::media::Type::Audio)
.ok_or_else(|| "No audio stream found".to_string())?;
@ -631,8 +695,8 @@ impl VideoAudioReader {
let duration_secs = if stream_duration > 0 {
stream_duration as f64 * time_base
} else if input.duration() > 0 {
input.duration() as f64 / f64::from(ffmpeg_next::ffi::AV_TIME_BASE)
} else if input.get().duration() > 0 {
input.get().duration() as f64 / f64::from(ffmpeg_next::ffi::AV_TIME_BASE)
} else {
0.0
};
@ -659,6 +723,7 @@ impl VideoAudioReader {
// Seek to at-or-before the target (max_ts = ts_av) so we can decode
// forward to it exactly. ffmpeg-next's `seek` wants a bounded range.
self.input
.get()
.seek(ts_av, 0..(ts_av + 1))
.map_err(|e| format!("Seek failed: {}", e))?;
self.decoder.flush();
@ -684,7 +749,7 @@ impl VideoAudioReader {
}
// Read one packet (owned), releasing the `input` borrow before decoding.
let packet = self.input.packets().next().map(|(_, p)| p);
let packet = self.input.get().packets().next().map(|(_, p)| p);
match packet {
Some(packet) => {
if packet.stream() == self.stream_index {
@ -827,8 +892,8 @@ impl StreamSource {
(SourceKind::VideoAudio, StreamOpen::Path(p)) => {
Ok(StreamSource::Video(VideoAudioReader::open(&p)?))
}
(SourceKind::VideoAudio, StreamOpen::Source { .. }) => {
Err("VideoAudio cannot be opened from a packed byte source".to_string())
(SourceKind::VideoAudio, StreamOpen::Source { src, ext }) => {
Ok(StreamSource::Video(VideoAudioReader::open_source(src, ext.as_deref())?))
}
}
}
@ -1093,7 +1158,12 @@ impl DiskReader {
// Decode more data into the buffer.
match reader.decode_next(&mut decode_buf) {
Ok(0) => {} // EOF
Ok(0) => {
// EOF: no more frames will be decoded for this buffer until a
// seek. Tell export-mode waiters so they stop blocking on
// past-the-end frames (e.g. video longer than its audio).
buffer.set_finished(true);
}
Ok(frames) => {
let was_empty = buffer.valid_frames_count() == 0;
buffer.write_samples(&decode_buf, frames);

View File

@ -2432,6 +2432,7 @@ impl Engine {
channels,
sample_rate,
total_frames,
None, // imported from an external video file; packing happens on save
);
let pool_index = self.audio_pool.add_file(audio_file);

View File

@ -164,11 +164,11 @@ impl AudioNode for OscillatorNode {
output[frame * 2] = sample; // Left
output[frame * 2 + 1] = sample; // Right
// Update phase once per frame
self.phase += freq_mod / sample_rate_f32;
if self.phase >= 1.0 {
self.phase -= 1.0;
}
// Update phase once per frame. `rem_euclid(1.0)` gives a numerically stable
// wraparound into [0, 1): repeated conditional subtraction accumulates f32
// rounding error over long-held notes (timbre drift), and FM can drive
// `freq_mod` negative, which a one-sided `if >= 1.0` wouldn't wrap at all.
self.phase = (self.phase + freq_mod / sample_rate_f32).rem_euclid(1.0);
}
}

View File

@ -244,12 +244,15 @@ impl AudioFile {
/// Create a placeholder AudioFile for a video's audio track. `path` is the
/// source video file; the audio is streamed on demand by the disk reader's
/// FFmpeg-backed `VideoAudioReader`.
/// FFmpeg-backed `VideoAudioReader`. `packed_media_id` is `Some` when the video
/// bytes are packed in the `.beam` container (audio streams from the same blob
/// via the host factory); `None` when the audio comes from an external video file.
pub fn from_video_audio(
path: PathBuf,
channels: u32,
sample_rate: u32,
total_frames: u64,
packed_media_id: Option<String>,
) -> Self {
Self {
path,
@ -263,7 +266,7 @@ impl AudioFile {
frames: total_frames,
original_format: None,
original_bytes: None,
packed_media_id: None,
packed_media_id,
}
}
@ -574,6 +577,27 @@ impl AudioClipPool {
// Spin-wait with small sleeps until the disk reader fills the buffer
let mut wait_iters = 0u64;
while !ra.has_range(src_start, frames_needed) {
// The source reached EOF and these frames are past the end of the
// audio (e.g. a video whose audio track is shorter than the video,
// or a container without exact stream duration). They will never
// arrive — stop waiting and let the render fill silence, instead of
// burning the 10s safety valve on every remaining chunk.
//
// BUT `finished` may be stale: we just moved the target with
// `set_target_frame`, and on a forward jump past the buffer the reader
// will reset()+seek() (clearing `finished`) on its next poll. The reader
// re-seeks when `target < buf_start || target > buf_end + sample_rate`
// (see disk_reader.rs), so only trust EOF when the target is inside the
// current window — otherwise a stale flag from the previous region would
// make us render silence over audio that's about to be decoded.
if ra.is_finished() {
let (buf_start, buf_end) = ra.snapshot();
let sr = audio_file.sample_rate as u64;
let seek_pending = src_start < buf_start || src_start > buf_end + sr;
if !seek_pending {
break;
}
}
std::thread::sleep(std::time::Duration::from_micros(100));
wait_iters += 1;
if wait_iters > 100_000 {
@ -1168,7 +1192,27 @@ impl AudioClipPool {
let entry_start = std::time::Instant::now();
eprintln!("📊 [LOAD_SERIALIZED] Processing entry {}/{}: '{}'", i + 1, entries.len(), entry.name);
let success = if entry.is_video_audio {
let success = if entry.is_video_audio && entry.media_id.is_some() {
// Packed video: the audio track lives in the same container blob as
// the video. Build a streaming VideoAudio entry from the saved
// metadata (no probe needed); playback opens the blob via the host
// factory at clip-activation time → VideoAudioReader::open_source.
let media_id = entry.media_id.clone().unwrap();
let total_frames = (entry.duration * entry.sample_rate as f64).ceil() as u64;
let file = AudioFile::from_video_audio(
PathBuf::new(),
entry.channels,
entry.sample_rate,
total_frames,
Some(media_id),
);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;
true
} else {
false
}
} else if entry.is_video_audio {
// Re-probe the video's audio track via FFmpeg → a streaming
// VideoAudio entry (keeps full 5.1/7.1; no decode-to-RAM).
match entry.relative_path.as_ref() {
@ -1186,6 +1230,7 @@ impl AudioClipPool {
reader.channels(),
reader.sample_rate(),
reader.total_frames(),
None,
);
if entry.pool_index < self.files.len() {
self.files[entry.pool_index] = file;

View File

@ -41,6 +41,11 @@ impl Project {
}
}
/// The project's system sample rate in Hz.
pub fn sample_rate(&self) -> u32 {
self.sample_rate
}
/// Generate a new unique track ID
fn next_id(&mut self) -> TrackId {
let id = self.next_track_id;

View File

@ -113,11 +113,10 @@ impl SynthVoice {
// Simple sine wave
let sample = (self.phase * 2.0 * PI).sin() * (self.velocity as f32 / 127.0) * 0.3;
// Update phase
self.phase += self.frequency / sample_rate;
if self.phase >= 1.0 {
self.phase -= 1.0;
}
// Update phase. Use `.fract()` for a numerically stable wraparound: repeated
// conditional subtraction accumulates f32 rounding error over long-held notes,
// which drifts the timbre.
self.phase = (self.phase + self.frequency / sample_rate).fract();
self.age += 1;

View File

@ -251,3 +251,69 @@ fn seek_is_sample_accurate() {
}
let _ = std::fs::remove_file(&path);
}
// ── Stage 3: stream video-audio from a byte source (packed .beam blob) ──────────
/// A `MediaByteSource` over an in-memory buffer (stands in for a SQLite BlobReader).
struct VecSource(std::io::Cursor<Vec<u8>>, u64);
impl std::io::Read for VecSource {
fn read(&mut self, b: &mut [u8]) -> std::io::Result<usize> { self.0.read(b) }
}
impl std::io::Seek for VecSource {
fn seek(&mut self, p: std::io::SeekFrom) -> std::io::Result<u64> { self.0.seek(p) }
}
impl daw_backend::audio::disk_reader::MediaByteSource for VecSource {
fn byte_len(&self) -> u64 { self.1 }
}
fn ramp_wav_bytes(n: u32, sample_rate: u32) -> Vec<u8> {
let channels = 1u16;
let bps = 4u32;
let data_size = n * bps;
let mut buf = Vec::with_capacity(44 + data_size as usize);
buf.extend_from_slice(b"RIFF");
buf.extend_from_slice(&(36 + data_size).to_le_bytes());
buf.extend_from_slice(b"WAVE");
buf.extend_from_slice(b"fmt ");
buf.extend_from_slice(&16u32.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes()); // IEEE float
buf.extend_from_slice(&channels.to_le_bytes());
buf.extend_from_slice(&sample_rate.to_le_bytes());
buf.extend_from_slice(&(sample_rate * channels as u32 * bps).to_le_bytes());
buf.extend_from_slice(&((channels as u32 * bps) as u16).to_le_bytes());
buf.extend_from_slice(&32u16.to_le_bytes());
buf.extend_from_slice(b"data");
buf.extend_from_slice(&data_size.to_le_bytes());
for i in 0..n {
buf.extend_from_slice(&((i as f32) / (n as f32)).to_le_bytes());
}
buf
}
#[test]
fn video_audio_open_source_streams_from_bytes() {
let sr = 8000;
let n = 4000;
let bytes = ramp_wav_bytes(n, sr);
let len = bytes.len() as u64;
let src = Box::new(VecSource(std::io::Cursor::new(bytes), len));
// Open the audio track by streaming from the byte source (no file path).
let mut reader = VideoAudioReader::open_source(src, Some("wav")).unwrap();
assert_eq!(reader.channels(), 1);
assert_eq!(reader.sample_rate(), sr);
let mut all: Vec<f32> = Vec::new();
let mut buf: Vec<f32> = Vec::new();
loop {
let frames = reader.decode_next(&mut buf).unwrap();
if frames == 0 {
break;
}
all.extend_from_slice(&buf);
}
assert!(all.len() as u32 >= n - 4, "decoded most of the ramp: {} of {}", all.len(), n);
// The ramp rises monotonically; sample 0 ≈ 0.0 and the last is near 1.0.
assert!(all[0].abs() < 1e-3, "first sample ~0, got {}", all[0]);
assert!(*all.last().unwrap() > 0.9, "last sample ~1.0, got {}", all.last().unwrap());
}

287
ffmpeg-blob-io/Cargo.lock generated Normal file
View File

@ -0,0 +1,287 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "aho-corasick"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
dependencies = [
"memchr",
]
[[package]]
name = "bindgen"
version = "0.72.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
dependencies = [
"bitflags",
"cexpr",
"clang-sys",
"itertools",
"proc-macro2",
"quote",
"regex",
"rustc-hash",
"shlex 1.3.0",
"syn",
]
[[package]]
name = "bitflags"
version = "2.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
[[package]]
name = "cc"
version = "1.2.63"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f"
dependencies = [
"find-msvc-tools",
"shlex 2.0.1",
]
[[package]]
name = "cexpr"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
dependencies = [
"nom",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "clang-sys"
version = "1.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
dependencies = [
"glob",
"libc",
"libloading",
]
[[package]]
name = "either"
version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
[[package]]
name = "ffmpeg-blob-io"
version = "0.1.0"
dependencies = [
"ffmpeg-next",
"ffmpeg-sys-next",
"libc",
]
[[package]]
name = "ffmpeg-next"
version = "8.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d658424d233cbd993a972dd73a66ca733acd12a494c68995c9ac32ae1fe65b40"
dependencies = [
"bitflags",
"ffmpeg-sys-next",
"libc",
]
[[package]]
name = "ffmpeg-sys-next"
version = "8.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9bca20aa4ee774fe384c2490096c122b0b23cf524a9910add0686691003d797b"
dependencies = [
"bindgen",
"cc",
"libc",
"num_cpus",
"pkg-config",
"vcpkg",
]
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "glob"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
[[package]]
name = "hermit-abi"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
[[package]]
name = "itertools"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
dependencies = [
"either",
]
[[package]]
name = "libc"
version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "libloading"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
dependencies = [
"cfg-if",
"windows-link",
]
[[package]]
name = "memchr"
version = "2.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b947ae49db0d222b1dbc6b113ce7248a3fc3a6ca21b696717bfc000ba4484d8"
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
[[package]]
name = "num_cpus"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
dependencies = [
"hermit-abi",
"libc",
]
[[package]]
name = "pkg-config"
version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "regex"
version = "1.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276"
dependencies = [
"aho-corasick",
"memchr",
"regex-automata",
"regex-syntax",
]
[[package]]
name = "regex-automata"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax",
]
[[package]]
name = "regex-syntax"
version = "0.8.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "shlex"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "vcpkg"
version = "0.2.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"

22
ffmpeg-blob-io/Cargo.toml Normal file
View File

@ -0,0 +1,22 @@
# Standalone crate (own workspace root) — depended on by daw-backend and
# lightningbeam-core via relative path, like daw-backend itself.
[workspace]
[package]
name = "ffmpeg-blob-io"
version = "0.1.0"
edition = "2021"
description = "Custom AVIOContext-over-Read+Seek shim for ffmpeg-next: lets a demuxer stream from an arbitrary byte source (e.g. a SQLite blob) instead of a file path. Isolates the unsafe FFI + ffmpeg ABI coupling."
# ABI-PINNED. This crate dereferences raw AVFormatContext / AVIOContext fields
# (`pb`, `flags`, `buffer`, `opaque`) and wraps the result back into ffmpeg-next's
# `Input`, so it MUST link the exact same libavformat ABI as the rest of the
# workspace. The `=` pins match what lightningbeam-core / daw-backend resolve;
# bump all three together when upgrading ffmpeg.
[dependencies]
ffmpeg-next = "=8.0.0"
ffmpeg-sys-next = "=8.0.1"
libc = "0.2"
[dev-dependencies]
ffmpeg-next = "=8.0.0"

240
ffmpeg-blob-io/src/lib.rs Normal file
View File

@ -0,0 +1,240 @@
//! Stream a demuxer from an arbitrary `Read + Seek` byte source via a custom
//! FFmpeg `AVIOContext`.
//!
//! `ffmpeg-next`'s high-level API can only open inputs by filesystem path. This
//! crate builds an `AVFormatContext` whose I/O is backed by a Rust reader (e.g. a
//! SQLite `BlobReader`), then hands back a normal [`ffmpeg_next::format::context::Input`]
//! that the rest of the codebase decodes exactly as a file-backed input.
//!
//! All of the `unsafe` FFI and ffmpeg ABI coupling lives here, isolated behind the
//! safe [`BlobInput`] type. See `BEAM_FILE_FORMAT.md` / `STREAMING_TO_DISK_PLAN.md`.
//!
//! # Safety model
//! - The reader is boxed and leaked to FFmpeg as the AVIO `opaque` pointer; the
//! read/seek callbacks reconstitute it as `&mut`. It is never aliased — only the
//! owning [`BlobInput`] (on one thread) drives it.
//! - [`BlobInput`] is `Send` but **not** `Sync`: the typical reader (`BlobReader`)
//! owns a `!Sync` SQLite connection. Each decoder / seek-reopen / off-thread scan
//! must construct its **own** `BlobInput` from a **fresh** reader.
//! - `Drop` tears down in the one correct order (Input first, then the AVIO buffer +
//! context, then the reader) so there is no use-after-free, double-free, or leak.
use std::io::{Read, Seek, SeekFrom};
use std::os::raw::{c_char, c_int, c_void};
use std::ptr;
use ffmpeg_next::format::context::Input;
use ffmpeg_sys_next as sys;
/// A byte source FFmpeg can stream from. Blanket-implemented for any
/// `Read + Seek + Send` (e.g. `std::io::Cursor`, a SQLite `BlobReader`).
pub trait BlobSource: Read + Seek + Send {}
impl<T: Read + Seek + Send> BlobSource for T {}
// Stable FFmpeg ABI constants (avoid depending on whether bindgen exported the
// corresponding `#define`s). These are fixed across the libavformat 5861 ABIs.
const AVSEEK_SIZE: c_int = 0x10000;
const AVSEEK_FORCE: c_int = 0x20000;
const AVFMT_FLAG_CUSTOM_IO: c_int = 0x0080;
/// Size of the AVIO read buffer handed to FFmpeg. FFmpeg may grow/replace it.
const IO_BUFFER_SIZE: usize = 32 * 1024;
/// A demuxer input streaming from a boxed `Read + Seek` source.
///
/// Deref to [`Input`] for the usual ffmpeg-next decode API.
pub struct BlobInput {
// Dropped FIRST in `Drop` (avformat_close_input). `Option` so we can drop it
// explicitly before freeing the AVIO resources it points at.
input: Option<Input>,
// The custom AVIOContext. With AVFMT_FLAG_CUSTOM_IO set, avformat_close_input
// does NOT free this — we own it. Its `opaque` field is the boxed reader.
avio: *mut sys::AVIOContext,
}
// The reader is `Send` and never aliased; we manage its lifetime manually.
unsafe impl Send for BlobInput {}
// Intentionally NOT `Sync`: see the module-level safety notes.
impl BlobInput {
/// Open a demuxer over `src`.
///
/// `format_hint` is an optional container short-name / extension (e.g. `"mp4"`,
/// `"mov"`, `"matroska"`) passed to `av_find_input_format` to skip probe
/// ambiguity; `None` lets FFmpeg probe from the stream.
pub fn open(src: Box<dyn BlobSource>, format_hint: Option<&str>) -> Result<Self, String> {
ffmpeg_next::init().map_err(|e| format!("ffmpeg init failed: {e}"))?;
unsafe {
// 1. IO buffer, allocated with FFmpeg's allocator (freed with av_freep).
let buffer = sys::av_malloc(IO_BUFFER_SIZE) as *mut u8;
if buffer.is_null() {
return Err("av_malloc failed for AVIO buffer".into());
}
// 2. Box the reader and leak it as the opaque pointer. Double-box so the
// raw pointer is thin (a `*mut dyn` would be a fat pointer).
let opaque = Box::into_raw(Box::new(src)) as *mut c_void;
// 3. AVIOContext over our callbacks (read-only: write_flag = 0).
let avio = sys::avio_alloc_context(
buffer,
IO_BUFFER_SIZE as c_int,
0,
opaque,
Some(read_cb),
None,
Some(seek_cb),
);
if avio.is_null() {
sys::av_free(buffer as *mut c_void);
drop(Box::from_raw(opaque as *mut Box<dyn BlobSource>));
return Err("avio_alloc_context failed".into());
}
// 4. Format context wired to the custom IO.
let mut fmt = sys::avformat_alloc_context();
if fmt.is_null() {
destroy_io(avio);
return Err("avformat_alloc_context failed".into());
}
(*fmt).pb = avio;
(*fmt).flags |= AVFMT_FLAG_CUSTOM_IO;
// 5. Optional format hint.
let hint_cstr = format_hint.and_then(|s| std::ffi::CString::new(s).ok());
let infmt: *const sys::AVInputFormat = match &hint_cstr {
Some(c) => sys::av_find_input_format(c.as_ptr() as *const c_char),
None => ptr::null(),
};
// 6. Open. On failure avformat_open_input frees `fmt` itself (and, with
// CUSTOM_IO, leaves our pb), so we still own avio+buffer+reader.
let ret = sys::avformat_open_input(&mut fmt, ptr::null(), infmt, ptr::null_mut());
if ret < 0 {
destroy_io(avio);
return Err(format!("avformat_open_input failed (error {ret})"));
}
// 7. Probe streams. On failure close the now-open input, then free IO.
let ret = sys::avformat_find_stream_info(fmt, ptr::null_mut());
if ret < 0 {
sys::avformat_close_input(&mut fmt);
destroy_io(avio);
return Err(format!("avformat_find_stream_info failed (error {ret})"));
}
// 8. Hand ownership of `fmt` to ffmpeg-next's Input (closes on drop).
let input = Input::wrap(fmt);
Ok(BlobInput { input: Some(input), avio })
}
}
/// The underlying demuxer input.
pub fn input(&self) -> &Input {
self.input.as_ref().expect("BlobInput input present until drop")
}
/// The underlying demuxer input, mutably (for `seek`, `packets`, …).
pub fn input_mut(&mut self) -> &mut Input {
self.input.as_mut().expect("BlobInput input present until drop")
}
}
impl std::ops::Deref for BlobInput {
type Target = Input;
fn deref(&self) -> &Input {
self.input()
}
}
impl std::ops::DerefMut for BlobInput {
fn deref_mut(&mut self) -> &mut Input {
self.input_mut()
}
}
impl Drop for BlobInput {
fn drop(&mut self) {
unsafe {
// 1. Drop the Input first: avformat_close_input. With CUSTOM_IO this does
// NOT touch self.avio or its buffer.
self.input = None;
// 2..4. Free the AVIO buffer + context and reclaim the boxed reader.
if !self.avio.is_null() {
destroy_io(self.avio);
self.avio = ptr::null_mut();
}
}
}
}
/// Free a standalone custom AVIOContext: its (possibly reallocated) buffer, the
/// boxed reader behind `opaque`, then the context itself — in that order. Only
/// call when the owning AVFormatContext has already been closed (or never opened).
unsafe fn destroy_io(avio: *mut sys::AVIOContext) {
if avio.is_null() {
return;
}
// Reclaim the reader box *before* freeing the context (we need `opaque`).
let opaque = (*avio).opaque;
// Free the current IO buffer (FFmpeg may have replaced the original).
sys::av_freep(&mut (*avio).buffer as *mut _ as *mut c_void);
// Free the AVIOContext struct (nulls the local).
let mut avio = avio;
sys::avio_context_free(&mut avio);
// Drop the reader last — no callback can fire now.
if !opaque.is_null() {
drop(Box::from_raw(opaque as *mut Box<dyn BlobSource>));
}
}
/// FFmpeg read callback: fill `buf` from the Rust reader. Returns bytes read,
/// `AVERROR_EOF` at end of stream, or a negative error.
unsafe extern "C" fn read_cb(opaque: *mut c_void, buf: *mut u8, buf_size: c_int) -> c_int {
if opaque.is_null() || buf.is_null() || buf_size <= 0 {
return sys::AVERROR_EOF;
}
let reader = &mut *(opaque as *mut Box<dyn BlobSource>);
let slice = std::slice::from_raw_parts_mut(buf, buf_size as usize);
match reader.read(slice) {
Ok(0) => sys::AVERROR_EOF,
Ok(n) => n as c_int,
// AVERROR(EIO) == -EIO on Unix; a negative value signals a read error.
Err(_) => -(libc::EIO as c_int),
}
}
/// FFmpeg seek callback. Handles `SEEK_SET/CUR/END` and `AVSEEK_SIZE` (report total
/// length). Returns the new position, the size, or `-1` on error.
unsafe extern "C" fn seek_cb(opaque: *mut c_void, offset: i64, whence: c_int) -> i64 {
if opaque.is_null() {
return -1;
}
let reader = &mut *(opaque as *mut Box<dyn BlobSource>);
let whence = whence & !AVSEEK_FORCE;
if whence & AVSEEK_SIZE != 0 {
// Report total length WITHOUT moving the logical position. FFmpeg does not
// restore the position after an AVSEEK_SIZE query, so we must: measure by
// seeking to the end, then seek back. (Failing to restore corrupts reads of
// containers whose index lives at the end of the file, e.g. MP4 `moov`.)
let cur = match reader.stream_position() {
Ok(p) => p,
Err(_) => return -1,
};
let size = match reader.seek(SeekFrom::End(0)) {
Ok(s) => s,
Err(_) => return -1,
};
if reader.seek(SeekFrom::Start(cur)).is_err() {
return -1;
}
return size as i64;
}
let from = match whence {
libc::SEEK_SET => SeekFrom::Start(offset as u64),
libc::SEEK_CUR => SeekFrom::Current(offset),
libc::SEEK_END => SeekFrom::End(offset),
_ => return -1,
};
reader.seek(from).map(|n| n as i64).unwrap_or(-1)
}

View File

@ -0,0 +1,100 @@
//! Exercises the custom AVIO shim end to end against a hand-built WAV, fed through
//! an in-memory `Cursor` (a `Read + Seek` source). FFmpeg can only demux it by
//! calling our `read_cb`/`seek_cb`, so a successful open proves the shim works.
use ffmpeg_blob_io::BlobInput;
use std::io::Cursor;
/// Build a minimal 16-bit PCM WAV in memory.
fn make_wav(sample_rate: u32, channels: u16, samples: &[i16]) -> Vec<u8> {
let bits: u16 = 16;
let block_align: u16 = channels * (bits / 8);
let byte_rate: u32 = sample_rate * block_align as u32;
let data_len: u32 = (samples.len() * 2) as u32;
let mut v = Vec::new();
v.extend_from_slice(b"RIFF");
v.extend_from_slice(&(36 + data_len).to_le_bytes());
v.extend_from_slice(b"WAVE");
v.extend_from_slice(b"fmt ");
v.extend_from_slice(&16u32.to_le_bytes()); // fmt chunk size
v.extend_from_slice(&1u16.to_le_bytes()); // PCM
v.extend_from_slice(&channels.to_le_bytes());
v.extend_from_slice(&sample_rate.to_le_bytes());
v.extend_from_slice(&byte_rate.to_le_bytes());
v.extend_from_slice(&block_align.to_le_bytes());
v.extend_from_slice(&bits.to_le_bytes());
v.extend_from_slice(b"data");
v.extend_from_slice(&data_len.to_le_bytes());
for s in samples {
v.extend_from_slice(&s.to_le_bytes());
}
v
}
fn sample_wav() -> Vec<u8> {
let samples: Vec<i16> = (0..1600).map(|i| ((i % 100) as i16 - 50) * 200).collect();
make_wav(8000, 1, &samples)
}
#[test]
fn opens_from_reader_and_reports_stream() {
let input = BlobInput::open(Box::new(Cursor::new(sample_wav())), Some("wav"))
.expect("open WAV through the blob shim");
let stream = input
.streams()
.best(ffmpeg_next::media::Type::Audio)
.expect("an audio stream");
let ctx = ffmpeg_next::codec::context::Context::from_parameters(stream.parameters())
.expect("codec context");
let decoder = ctx.decoder().audio().expect("audio decoder");
assert_eq!(decoder.rate(), 8000, "sample rate read via AVIO");
assert_eq!(decoder.channels(), 1, "channel count read via AVIO");
}
#[test]
fn reads_packets_through_callbacks() {
let mut input = BlobInput::open(Box::new(Cursor::new(sample_wav())), Some("wav"))
.expect("open WAV");
let mut packets = 0usize;
let mut bytes = 0usize;
for (_stream, packet) in input.packets() {
packets += 1;
bytes += packet.size();
}
assert!(packets > 0, "demuxer produced packets via read_cb");
assert!(bytes > 0, "packets carried PCM payload");
}
#[test]
fn seek_then_read() {
let mut input = BlobInput::open(Box::new(Cursor::new(sample_wav())), Some("wav"))
.expect("open WAV");
// Seek back to the start (exercises seek_cb SEEK_SET + AVSEEK_SIZE).
input.seek(0, ..).expect("seek to start");
let got_packet = input.packets().next().is_some();
assert!(got_packet, "can still read after seek");
}
#[test]
fn open_drop_loop_is_clean() {
// Repeated open+drop surfaces double-free / leak in the Drop teardown
// (run under `RUSTFLAGS=-Zsanitizer=address` on nightly for full coverage).
for _ in 0..200 {
let input = BlobInput::open(Box::new(Cursor::new(sample_wav())), Some("wav"))
.expect("open WAV");
assert!(input.streams().count() >= 1);
drop(input);
}
}
#[test]
fn bad_format_hint_errors_without_leak() {
// Garbage bytes with a real hint: open should fail cleanly (error path frees
// buffer + avio + reader), not panic or leak.
let garbage = vec![0u8; 4096];
let res = BlobInput::open(Box::new(Cursor::new(garbage)), Some("wav"));
assert!(res.is_err(), "non-WAV bytes should fail to open as WAV");
}

View File

@ -15,3 +15,8 @@
# OS
.DS_Store
Thumbs.db
# Local, machine-specific cargo config (e.g. PKG_CONFIG_PATH for a local ffmpeg).
# CI/packaging build ffmpeg from source (see .github/workflows/build.yml), so this
# must never leak into a commit.
.cargo/config.toml

File diff suppressed because it is too large Load Diff

View File

@ -4,6 +4,7 @@ members = [
"lightningbeam-editor",
"lightningbeam-core",
"beamdsp",
"gpu-video-encoder",
]
[workspace.dependencies]
@ -23,6 +24,11 @@ vello = { git = "https://github.com/linebender/vello", branch = "main" }
wgpu = { version = "27", features = ["vulkan", "metal", "gles"] }
kurbo = { version = "0.12", features = ["serde"] }
peniko = "0.5"
# Text layout/shaping for text layers. Pinned to git main to match the peniko 0.5
# / skrifa 0.37 that vello's git-main resolves to — a released parley still pins
# peniko 0.4, which would split `peniko::Font` into two incompatible types at the
# vello `draw_glyphs` boundary.
parley = { git = "https://github.com/linebender/parley", branch = "main" }
# Windowing
winit = "0.30"

View File

@ -0,0 +1,102 @@
# GPU-resident video decode + dynamic decode resolution
## Context
Profiling the zero-copy H.264 export (single Group[Video, Audio] clip, `LB_RENDER_PROFILE=1`)
broke the per-frame CPU "render" bucket down as:
| Cost (ms/frame) | 1080p | 4K | What it is |
|-------------------------|-------|-------|-----------------------------------------------------|
| decode | 3.1 | 19.0 | software ffmpeg decode (`video.rs::get_frame`) |
| background re-render | 3.6 | 7.5 | static background pushed through Vello *every frame* |
| video upload + blit | 4.1 | 4.2 | per-frame transient texture alloc + `write_texture` |
| srgb | 0.4 | 0.4 | linear→sRGB pass |
The video correctly takes the GPU Video-instance path (not Vello-baked) — `LB_LAYER_DEBUG=1`
shows `Video (1 instance)`. So the cost is **the video frame itself**: software decode, then an
8 MB `write_texture` upload of the decoded RGBA every frame. At 4K, software decode (19 ms)
dominates everything.
### Two correctness problems found alongside the perf issue
1. **Decode resolution is frozen to document size at import.** `load_video(clip, src, doc_w, doc_h)`
(`main.rs:4302`) sizes the decoder's swscale output to the document, capped to never upscale
(`video.rs:149`). Export *reuses that decoder*, so exporting **above** document resolution yields
video that was decoded to ≤document res and then GPU-**up**scaled — real source detail thrown away.
2. **It can't follow the consumer or a document resize.** Preview wants small/fast frames; export
wants full res; changing the document size should re-target the decode. None of that works with a
size frozen at import.
## Goal
Decouple **decode resolution** from import/document size: the renderer requests a frame *at a target
resolution*, and the decode path produces it. Hardware-decode H.264 (and later HEVC/AV1) into a GPU
surface and keep it GPU-resident through composite into the encoder — no CPU frame copy in either
direction. Software decode stays a **first-class** path (codecs/platforms without HW support), decoding
at the requested target res. This fixes the 4K decode wall, the 8 MB upload, *and* the resolution bugs.
## Design principles
- **Decode native, scale to the consumer's target.**
- *Hardware path:* decode into a native VAAPI surface → import as a wgpu texture (reuse the
`gpu-video-encoder` `dmabuf.rs` / `vk_device.rs` plumbing, read direction) → the GPU blit that
already composites the Video instance scales native→target for free. Handles any target res and
document resizes inherently; the cached frame is a native GPU texture.
- *Software path:* decode native → `swscale` to the requested target (the reusable scaler is keyed
on input format/size **and** output size — rebuilt when the target changes). Preview requests
preview res (cheap); export requests export res (full quality).
- **`VideoManager::get_frame` takes a target `(w, h)`** instead of relying on a frozen output size.
The frame cache is keyed to handle multiple live targets (preview + export) — either cache native
frames and scale on demand, or key by `(clip, ts, target)`; decide in Stage 2 by measuring cache
hit/scale tradeoff.
- **Software is not optional.** Hardware decode is an acceleration of the same `get_frame` contract,
selected per source when the codec/driver supports it; everything falls back to software cleanly.
## Approach (staged; each stage compiles + is independently useful)
### Stage 0 — independent quick wins (not blocked on decode)
- **Cache the static background** (`composite_document_to_hdr`): render once, reuse via a persistent
HDR texture (copy-in each frame) instead of a full Vello render + 2 passes/submits every frame.
Recovers ~3.6 ms (1080p) / ~7.5 ms (4K) per frame on *every* export. (In flight.)
### Stage 1 — software: decode at the requested target res (testable; fixes the quality bug now)
- Change `VideoManager::get_frame(clip, ts)``get_frame(clip, ts, target_w, target_h)`; thread the
target from the renderer (preview = current doc/preview res, export = export res). Cap at native.
- Rework `VideoDecoder` so output size is per-request, not frozen at construction; cache the swscale
context per output size (already cached per stream — extend the key). Adjust the frame cache key.
- Result: software exports are full-quality at any export res, and document resizes re-target decode.
No hardware needed; this is the correctness fix for the codecs HW can't handle anyway.
### Stage 2 — hardware decode primitive (headless-testable here, like the 8 encode tests)
- In `gpu-video-encoder` (rename → `gpu-video-codec`): `h264_vaapi`-style **decode** → VAAPI surface →
export DMA-BUF → import as a wgpu texture. Hardware test: decode a known file, verify dims/contents.
### Stage 3 — wire hardware decode into `get_frame` (blind; user-verifies)
- When the source codec/driver is HW-decodable, `get_frame` returns a **GPU texture** (native res)
instead of `Arc<Vec<u8>>`; the compositor uses it directly (no `write_texture`), GPU-scaling to the
target. For the zero-copy export the frame never leaves the GPU: **decode → composite → encode** on
one device. Software path is the fallback for everything else.
## Critical files
- `lightningbeam-core/src/video.rs``VideoDecoder` (per-request output size, scaler cache),
`VideoManager::get_frame` (target param, cache key).
- `lightningbeam-core/src/renderer.rs` — pass the render target res into the video-instance build.
- `lightningbeam-editor/src/export/video_exporter.rs` — background cache (Stage 0); consume a GPU
texture instead of uploading RGBA (Stage 3).
- `gpu-video-encoder/` (→ `gpu-video-codec`) — `dmabuf.rs`/`vk_device.rs` reused for the decode import.
## Risks
- **Codec coverage** — only some codecs are HW-decodable per GPU/driver; software must stay correct
and well-tested. Selection must probe support per source, not assume.
- **Cache memory** — native-res GPU textures (esp. 4K) are large; the frame cache budget needs revisiting.
- **Colorspace/format** — VAAPI decode surfaces are NV12/tiled; the existing import handles NV12, but
10-bit/HDR sources (P010) need format handling.
- **Preview vs export sharing** — two live targets (preview res + export res) from the same source; the
cache/scaler design must serve both without thrashing.
## Verification
- Stage 0/1: visual — export above document res is now full-quality (not upscaled); profile shows
background ≈ 0 and (Stage 1) software export correct at the chosen res.
- Stage 2: headless hardware test in `gpu-video-codec` (decode → wgpu texture, ffprobe/byte checks).
- Stage 3 (user): 1080p + 4K H.264 export — decode/upload buckets collapse; software fallback for a
non-HW codec (e.g. ProRes) still produces correct full-res output.

View File

@ -0,0 +1,20 @@
[package]
name = "gpu-video-encoder"
version = "0.1.0"
edition = "2021"
description = "Zero-copy GPU video encoding (RGBA->NV12 compute + hardware encoder interop). Unsafe FFI isolated here."
[dependencies]
wgpu = { workspace = true }
# Raw Vulkan access for the DMA-BUF import. Versions MUST match what wgpu links
# (wgpu-hal 27.0.4 / ash 0.38) so the hal/ash types unify across the boundary.
wgpu-hal = { version = "27", features = ["vulkan"] }
wgpu-types = "27"
ash = "0.38"
# Raw FFmpeg FFI for VAAPI hwcontext + hardware encode. Matches the editor's
# ffmpeg-next 8.0 / static link so cargo unifies to one libav* across the build.
ffmpeg-sys-next = { version = "8.0", features = ["static"] }
libc = "0.2"
[dev-dependencies]
pollster = "0.4"

View File

@ -0,0 +1,217 @@
//! VAAPI hardware video decode → wgpu textures. The mirror of [`crate::encoder`]: the codec
//! decodes into a VAAPI NV12 surface, which is mapped to a DRM-PRIME DMA-BUF and imported as two
//! wgpu plane textures via [`crate::dmabuf::import_raw`] — the exact same path the encoder uses,
//! in the read direction. Stays GPU-resident: no CPU frame copy.
use crate::dmabuf::{self, ImportedNv12, Nv12DmaBuf};
use crate::vk_device::{self, DrmDevice};
use ffmpeg_sys_next as ff;
use std::ffi::CString;
use std::path::Path;
use std::ptr;
#[inline]
fn averror(e: i32) -> i32 {
-e
}
/// `get_format` callback: pick VAAPI surfaces so the decoder outputs hardware frames. With
/// `hw_device_ctx` set, FFmpeg auto-allocates the matching frames context.
unsafe extern "C" fn get_vaapi_format(
_ctx: *mut ff::AVCodecContext,
mut fmts: *const ff::AVPixelFormat,
) -> ff::AVPixelFormat {
while *fmts != ff::AVPixelFormat::AV_PIX_FMT_NONE {
if *fmts == ff::AVPixelFormat::AV_PIX_FMT_VAAPI {
return ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
}
fmts = fmts.add(1);
}
ff::AVPixelFormat::AV_PIX_FMT_NONE
}
/// Hardware decoder for a single video file/stream. Frames come back as importable NV12 textures
/// on [`Self::device`].
pub struct VaapiDecoder {
drm: DrmDevice,
hw_device: *mut ff::AVBufferRef,
fmt: *mut ff::AVFormatContext,
dec: *mut ff::AVCodecContext,
pkt: *mut ff::AVPacket,
frame: *mut ff::AVFrame,
stream_index: i32,
flushing: bool,
}
// Owns its FFmpeg/Vulkan handles exclusively; only moved, never shared (same as the encoder).
unsafe impl Send for VaapiDecoder {}
impl VaapiDecoder {
/// Open `input_path` and set up VAAPI hardware decoding of its best video stream.
pub fn new(input_path: &Path) -> Result<Self, String> {
let drm = vk_device::create()?;
unsafe {
let mut hw_device = crate::vaapi::create_device()?;
let cleanup_hw = |hw: *mut ff::AVBufferRef| {
let mut h = hw;
ff::av_buffer_unref(&mut h);
};
let path_c = CString::new(input_path.to_string_lossy().as_ref()).unwrap();
let mut fmt: *mut ff::AVFormatContext = ptr::null_mut();
if ff::avformat_open_input(&mut fmt, path_c.as_ptr(), ptr::null_mut(), ptr::null_mut()) < 0 {
cleanup_hw(hw_device);
return Err(format!("avformat_open_input {input_path:?} failed"));
}
if ff::avformat_find_stream_info(fmt, ptr::null_mut()) < 0 {
ff::avformat_close_input(&mut fmt);
cleanup_hw(hw_device);
return Err("avformat_find_stream_info failed".into());
}
let mut decoder: *const ff::AVCodec = ptr::null();
let stream_index = ff::av_find_best_stream(
fmt,
ff::AVMediaType::AVMEDIA_TYPE_VIDEO,
-1,
-1,
&mut decoder,
0,
);
if stream_index < 0 || decoder.is_null() {
ff::avformat_close_input(&mut fmt);
cleanup_hw(hw_device);
return Err("no decodable video stream".into());
}
let dec = ff::avcodec_alloc_context3(decoder);
let stream = *(*fmt).streams.add(stream_index as usize);
if ff::avcodec_parameters_to_context(dec, (*stream).codecpar) < 0 {
ff::avcodec_free_context(&mut (dec as *mut _));
ff::avformat_close_input(&mut fmt);
cleanup_hw(hw_device);
return Err("avcodec_parameters_to_context failed".into());
}
(*dec).hw_device_ctx = ff::av_buffer_ref(hw_device);
(*dec).get_format = Some(get_vaapi_format);
if ff::avcodec_open2(dec, decoder, ptr::null_mut()) < 0 {
ff::avcodec_free_context(&mut (dec as *mut _));
ff::avformat_close_input(&mut fmt);
cleanup_hw(hw_device);
return Err("avcodec_open2 (vaapi decode) failed".into());
}
// `mut` only to satisfy the move into the struct; the binding above is consumed.
let _ = &mut hw_device;
Ok(Self {
drm,
hw_device,
fmt,
dec,
pkt: ff::av_packet_alloc(),
frame: ff::av_frame_alloc(),
stream_index,
flushing: false,
})
}
}
/// The wgpu device the decoded textures live on (the DMA-BUF-import device).
pub fn device(&self) -> &wgpu::Device {
&self.drm.device
}
pub fn queue(&self) -> &wgpu::Queue {
&self.drm.queue
}
/// Decode the next frame and import it as NV12 plane textures, or `Ok(None)` at end of stream.
pub fn next_frame(&mut self) -> Result<Option<ImportedNv12>, String> {
unsafe {
loop {
let r = ff::avcodec_receive_frame(self.dec, self.frame);
if r == 0 {
let imported = self.map_current();
ff::av_frame_unref(self.frame);
return imported.map(Some);
}
if r == ff::AVERROR_EOF {
return Ok(None);
}
if r != averror(libc::EAGAIN) {
return Err(format!("avcodec_receive_frame failed: {r}"));
}
if self.flushing {
return Ok(None); // already drained the flush
}
// Decoder wants more input: pump one packet (or signal EOF to flush).
let rp = ff::av_read_frame(self.fmt, self.pkt);
if rp < 0 {
self.flushing = true;
ff::avcodec_send_packet(self.dec, ptr::null());
continue;
}
if (*self.pkt).stream_index == self.stream_index {
let rs = ff::avcodec_send_packet(self.dec, self.pkt);
ff::av_packet_unref(self.pkt);
if rs < 0 && rs != averror(libc::EAGAIN) {
return Err(format!("avcodec_send_packet failed: {rs}"));
}
} else {
ff::av_packet_unref(self.pkt);
}
}
}
}
/// Map the just-decoded VAAPI surface (`self.frame`) to a DRM-PRIME DMA-BUF and import it.
unsafe fn map_current(&self) -> Result<ImportedNv12, String> {
let drm_f = ff::av_frame_alloc();
(*drm_f).format = ff::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let flags = ff::AV_HWFRAME_MAP_DIRECT as i32 | ff::AV_HWFRAME_MAP_READ as i32;
if ff::av_hwframe_map(drm_f, self.frame, flags) < 0 {
ff::av_frame_free(&mut (drm_f as *mut _));
return Err("av_hwframe_map failed".into());
}
let desc = (*drm_f).data[0] as *const ff::AVDRMFrameDescriptor;
let obj = &(*desc).objects[0];
let width = (*self.frame).width as u32;
let height = (*self.frame).height as u32;
// NV12: Y then UV — either as two layers (one plane each) or one layer with two planes.
let (y_pl, uv_pl) = if (*desc).nb_layers >= 2 {
(&(*desc).layers[0].planes[0], &(*desc).layers[1].planes[0])
} else {
(&(*desc).layers[0].planes[0], &(*desc).layers[0].planes[1])
};
let buf = Nv12DmaBuf {
fd: obj.fd,
size: obj.size as u64,
modifier: obj.format_modifier,
width,
height,
y_offset: y_pl.offset as u64,
y_pitch: y_pl.pitch as u64,
uv_offset: uv_pl.offset as u64,
uv_pitch: uv_pl.pitch as u64,
ten_bit: false,
};
let imported = dmabuf::import_raw(&self.drm.device, &self.drm.adapter, &buf);
ff::av_frame_free(&mut (drm_f as *mut _)); // the fd was dup'd into Vulkan
imported
}
}
impl Drop for VaapiDecoder {
fn drop(&mut self) {
unsafe {
ff::av_frame_free(&mut (self.frame as *mut _));
ff::av_packet_free(&mut (self.pkt as *mut _));
ff::avcodec_free_context(&mut (self.dec as *mut _));
if !self.fmt.is_null() {
ff::avformat_close_input(&mut self.fmt);
}
ff::av_buffer_unref(&mut self.hw_device);
}
}
}

View File

@ -0,0 +1,309 @@
//! Import a tiled VAAPI NV12 DMA-BUF as two wgpu textures (Y = R8, UV = RG8), aliasing
//! the one imported `VkDeviceMemory` at the plane offsets. Two single-format images are
//! used instead of one multi-planar image so each is an ordinary wgpu render target.
use crate::vaapi::MappedSurface;
use crate::vk_device::DrmDevice;
use ash::vk;
/// Plane layout for a single-object NV12/P010 DMA-BUF (the common VAAPI case).
#[derive(Clone, Copy)]
pub struct Nv12DmaBuf {
pub fd: i32,
pub size: u64,
pub modifier: u64,
pub width: u32,
pub height: u32,
pub y_offset: u64,
pub y_pitch: u64,
pub uv_offset: u64,
pub uv_pitch: u64,
/// True for 10/12/16-bit content (P010 etc.): planes are 16-bit (R16/Rg16) rather than 8-bit
/// (R8/Rg8). The sampled float is normalized either way, so the consumer needs no change.
pub ten_bit: bool,
}
/// Frees the shared imported `VkDeviceMemory` once both plane images are gone. Held by
/// both textures' drop callbacks (via `Arc`); the last one to run frees the memory —
/// after wgpu has destroyed the images, in its wait-idle'd deferred-destruction pass.
struct MemoryGuard {
device: ash::Device,
memory: vk::DeviceMemory,
}
impl Drop for MemoryGuard {
fn drop(&mut self) {
unsafe { self.device.free_memory(self.memory, None) };
}
}
/// Frees the duplicated dma-buf fd and any partially-created Vulkan objects when an import
/// errors out before ownership has been handed to wgpu/`MemoryGuard`. `commit()` disarms it on
/// the success path; `fd_consumed()` is called once `vkAllocateMemory` has taken the fd.
struct ImportGuard {
device: ash::Device,
fd: libc::c_int,
img_y: vk::Image,
img_uv: vk::Image,
memory: vk::DeviceMemory,
armed: bool,
}
impl ImportGuard {
fn fd_consumed(&mut self) {
self.fd = -1; // vkAllocateMemory owns the fd now; don't close it ourselves
}
fn commit(&mut self) {
self.armed = false;
}
}
impl Drop for ImportGuard {
fn drop(&mut self) {
if !self.armed {
return;
}
unsafe {
if self.img_uv != vk::Image::null() {
self.device.destroy_image(self.img_uv, None);
}
if self.img_y != vk::Image::null() {
self.device.destroy_image(self.img_y, None);
}
if self.memory != vk::DeviceMemory::null() {
self.device.free_memory(self.memory, None);
}
if self.fd >= 0 {
libc::close(self.fd);
}
}
}
}
/// A VAAPI surface imported as two wgpu plane textures. The underlying Vulkan image/
/// memory are destroyed by wgpu (via drop callbacks) when these textures drop.
pub struct ImportedNv12 {
y: wgpu::Texture,
uv: wgpu::Texture,
}
impl ImportedNv12 {
pub fn y(&self) -> &wgpu::Texture {
&self.y
}
pub fn uv(&self) -> &wgpu::Texture {
&self.uv
}
/// Consume into the `(Y, UV)` plane textures (for handing to a longer-lived owner).
pub fn into_planes(self) -> (wgpu::Texture, wgpu::Texture) {
(self.y, self.uv)
}
}
/// Convenience: map a freshly-allocated `MappedSurface` and import it onto `drm`.
pub fn import(drm: &DrmDevice, surf: &MappedSurface) -> Result<ImportedNv12, String> {
import_raw(
&drm.device,
&drm.adapter,
&Nv12DmaBuf {
fd: surf.fd,
size: surf.size,
modifier: surf.modifier,
width: surf.width,
height: surf.height,
y_offset: surf.y_offset,
y_pitch: surf.y_pitch,
uv_offset: surf.uv_offset,
uv_pitch: surf.uv_pitch,
ten_bit: false,
},
)
}
/// Import an NV12 DMA-BUF (described by `buf`) as two wgpu plane textures **on `device`**. The raw
/// Vulkan handles are extracted from `device`/`adapter` via `as_hal`, so this works with any
/// DMA-BUF-import-capable wgpu device — the encoder/decoder's own `DrmDevice` *or* the editor's
/// shared device. The fd is duplicated, so the caller keeps ownership of theirs.
pub fn import_raw(
device: &wgpu::Device,
adapter: &wgpu::Adapter,
buf: &Nv12DmaBuf,
) -> Result<ImportedNv12, String> {
use wgpu_hal::vulkan::Api as Vk;
unsafe {
let hal_device = device
.as_hal::<Vk>()
.ok_or("device is not Vulkan")?;
let raw_device = hal_device.raw_device().clone();
let raw_instance = adapter
.as_hal::<Vk>()
.ok_or("adapter is not Vulkan")?
.shared_instance()
.raw_instance()
.clone();
let instance = &raw_instance;
let dup_fd = libc::dup(buf.fd);
if dup_fd < 0 {
return Err("dup(dma-buf fd) failed".into());
}
// Owns the fd + any Vk objects created below until ownership transfers to wgpu; on any
// early `?`/return before that, its Drop frees them (was leaking on every failed import).
let mut guard = ImportGuard {
device: raw_device.clone(),
fd: dup_fd,
img_y: vk::Image::null(),
img_uv: vk::Image::null(),
memory: vk::DeviceMemory::null(),
armed: true,
};
// 16-bit-norm plane formats (P010) are NOT renderable, so the import is sample-only for
// those (decode path). 8-bit planes keep COLOR_ATTACHMENT for the encoder's RGBA→NV12 write.
let vk_usage = if buf.ten_bit {
vk::ImageUsageFlags::SAMPLED
| vk::ImageUsageFlags::TRANSFER_SRC
| vk::ImageUsageFlags::TRANSFER_DST
} else {
vk::ImageUsageFlags::COLOR_ATTACHMENT
| vk::ImageUsageFlags::SAMPLED
| vk::ImageUsageFlags::TRANSFER_SRC
| vk::ImageUsageFlags::TRANSFER_DST
};
let make_image = |format: vk::Format, w: u32, h: u32, pitch: u64| -> Result<vk::Image, String> {
let mut ext = vk::ExternalMemoryImageCreateInfo::default()
.handle_types(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT);
let plane_layouts = [vk::SubresourceLayout::default().offset(0).row_pitch(pitch)];
let mut drm_info = vk::ImageDrmFormatModifierExplicitCreateInfoEXT::default()
.drm_format_modifier(buf.modifier)
.plane_layouts(&plane_layouts);
let info = vk::ImageCreateInfo::default()
.image_type(vk::ImageType::TYPE_2D)
.format(format)
.extent(vk::Extent3D { width: w, height: h, depth: 1 })
.mip_levels(1)
.array_layers(1)
.samples(vk::SampleCountFlags::TYPE_1)
.tiling(vk::ImageTiling::DRM_FORMAT_MODIFIER_EXT)
.usage(vk_usage)
.sharing_mode(vk::SharingMode::EXCLUSIVE)
.initial_layout(vk::ImageLayout::UNDEFINED)
.push_next(&mut ext)
.push_next(&mut drm_info);
raw_device
.create_image(&info, None)
.map_err(|e| format!("vkCreateImage(modifier) failed: {e:?}"))
};
// 8-bit NV12 → R8/Rg8 planes; 10/12/16-bit P010-style → R16/Rg16 (sampled value is
// normalized either way, so the NV12→RGB consumer is unchanged).
let (vk_y, vk_uv) = if buf.ten_bit {
(vk::Format::R16_UNORM, vk::Format::R16G16_UNORM)
} else {
(vk::Format::R8_UNORM, vk::Format::R8G8_UNORM)
};
let (wgpu_y, wgpu_uv) = if buf.ten_bit {
(wgpu::TextureFormat::R16Unorm, wgpu::TextureFormat::Rg16Unorm)
} else {
(wgpu::TextureFormat::R8Unorm, wgpu::TextureFormat::Rg8Unorm)
};
let img_y = make_image(vk_y, buf.width, buf.height, buf.y_pitch)?;
guard.img_y = img_y;
let img_uv = make_image(vk_uv, buf.width / 2, buf.height / 2, buf.uv_pitch)?;
guard.img_uv = img_uv;
let fd_dev = ash::khr::external_memory_fd::Device::new(instance, &raw_device);
let mut fd_props = vk::MemoryFdPropertiesKHR::default();
fd_dev
.get_memory_fd_properties(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT, dup_fd, &mut fd_props)
.map_err(|e| format!("vkGetMemoryFdPropertiesKHR failed: {e:?}"))?;
let req_y = raw_device.get_image_memory_requirements(img_y);
let req_uv = raw_device.get_image_memory_requirements(img_uv);
let type_bits = fd_props.memory_type_bits & req_y.memory_type_bits & req_uv.memory_type_bits;
if type_bits == 0 {
return Err("no memory type compatible with dma-buf + both plane images".into());
}
let mem_type = type_bits.trailing_zeros();
let mut import_info = vk::ImportMemoryFdInfoKHR::default()
.handle_type(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT)
.fd(dup_fd);
let alloc = vk::MemoryAllocateInfo::default()
.allocation_size(buf.size)
.memory_type_index(mem_type)
.push_next(&mut import_info);
let memory = raw_device
.allocate_memory(&alloc, None)
.map_err(|e| format!("vkAllocateMemory(import dma-buf) failed: {e:?}"))?;
guard.fd_consumed(); // the import transferred fd ownership to Vulkan
guard.memory = memory;
raw_device
.bind_image_memory(img_y, memory, buf.y_offset)
.map_err(|e| format!("bind Y plane: {e:?}"))?;
raw_device
.bind_image_memory(img_uv, memory, buf.uv_offset)
.map_err(|e| format!("bind UV plane: {e:?}"))?;
// Shared guard: frees `memory` once both images' drop callbacks have run.
let mem_guard = std::sync::Arc::new(MemoryGuard { device: raw_device.clone(), memory });
// Match the Vulkan usage: 16-bit-norm planes (P010) are sample-only (not renderable).
let (hal_usage, wgpu_usage) = if buf.ten_bit {
(
wgpu_types::TextureUses::RESOURCE | wgpu_types::TextureUses::COPY_SRC,
wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_SRC,
)
} else {
(
wgpu_types::TextureUses::COLOR_TARGET
| wgpu_types::TextureUses::RESOURCE
| wgpu_types::TextureUses::COPY_SRC,
wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::COPY_SRC,
)
};
let wrap = |img: vk::Image, format: wgpu::TextureFormat, w: u32, h: u32| -> wgpu::Texture {
// wgpu destroys the image (after wait-idle) when the texture drops; the
// captured Arc<MemoryGuard> frees the shared memory once both have run.
let dev = raw_device.clone();
let guard = mem_guard.clone();
let cb: wgpu_hal::DropCallback = Box::new(move || {
dev.destroy_image(img, None);
drop(guard);
});
let hal_desc = wgpu_hal::TextureDescriptor {
label: Some("vaapi-plane"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format,
usage: hal_usage,
memory_flags: wgpu_hal::MemoryFlags::empty(),
view_formats: vec![],
};
let hal_tex = hal_device.texture_from_raw(img, &hal_desc, Some(cb));
device.create_texture_from_hal::<wgpu_hal::vulkan::Api>(
hal_tex,
&wgpu::TextureDescriptor {
label: Some("vaapi-plane"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format,
usage: wgpu_usage,
view_formats: &[],
},
)
};
// Ownership of the images (→ texture drop callbacks) and memory (→ MemoryGuard) has now
// transferred to wgpu; disarm the cleanup guard so it doesn't double-free them.
guard.commit();
let y = wrap(img_y, wgpu_y, buf.width, buf.height);
let uv = wrap(img_uv, wgpu_uv, buf.width / 2, buf.height / 2);
drop(hal_device);
Ok(ImportedNv12 { y, uv })
}
}

View File

@ -0,0 +1,334 @@
//! End-to-end zero-copy H.264 encoder: render an RGBA wgpu texture straight into a VAAPI
//! NV12 surface (no CPU copy) and encode it with `h264_vaapi`. The caller renders frames
//! on [`ZeroCopyEncoder::device`] (the custom Vulkan device with DMA-BUF import enabled).
//!
//! Imports are cached by VASurface id, so the pooled surfaces are imported once each.
use crate::dmabuf::{self, ImportedNv12, Nv12DmaBuf};
use crate::render_nv12::Rgba2Nv12;
use crate::vk_device;
use ffmpeg_sys_next as ff;
use std::collections::HashMap;
use std::ffi::CString;
use std::path::Path;
use std::ptr;
#[inline]
fn averror(e: i32) -> i32 {
-e
}
pub struct ZeroCopyEncoder {
/// wgpu handles the NV12 render runs on — either an own `DrmDevice`'s (via `new`) or the
/// editor's shared device (via `new_on_device`). Cloned (Arc-backed) so the source can drop.
device: wgpu::Device,
queue: wgpu::Queue,
adapter: wgpu::Adapter,
renderer: Rgba2Nv12,
hw_device: *mut ff::AVBufferRef,
frames_ref: *mut ff::AVBufferRef,
enc: *mut ff::AVCodecContext,
pkt: *mut ff::AVPacket,
/// Output container (e.g. `.mp4`); packets are muxed into it directly.
oc: *mut ff::AVFormatContext,
enc_tb: ff::AVRational,
stream_tb: ff::AVRational,
width: u32,
height: u32,
pts: i64,
cache: HashMap<usize, ImportedNv12>,
}
// The encoder owns its FFmpeg contexts (raw `*mut`) and Vulkan/wgpu handles exclusively; it is
// never shared, only moved. Sending it to a dedicated export thread is sound.
unsafe impl Send for ZeroCopyEncoder {}
impl ZeroCopyEncoder {
/// Build a zero-copy `h264_vaapi` encoder writing to `output_path` (container inferred
/// from the extension, e.g. `.mp4`). `Err` if VAAPI/the device is unavailable.
#[allow(clippy::too_many_arguments)]
pub fn new(
width: u32,
height: u32,
framerate: i32,
bitrate_kbps: u32,
output_path: &Path,
full_range: bool,
) -> Result<Self, String> {
// Build a dedicated DMA-BUF-import device and run the encoder on it.
let drm = vk_device::create()?;
Self::new_on_device(
drm.device, drm.queue, drm.adapter,
width, height, framerate, bitrate_kbps, output_path, full_range,
)
}
/// Build the encoder running its NV12 render + DMA-BUF import on an existing wgpu device (the
/// editor's shared device), so decode→composite→encode stay GPU-resident on one device. The
/// device must have the DMA-BUF import extensions (a `DrmDevice` / `vk_device::create_windowed`).
#[allow(clippy::too_many_arguments)]
pub fn new_on_device(
device: wgpu::Device,
queue: wgpu::Queue,
adapter: wgpu::Adapter,
width: u32,
height: u32,
framerate: i32,
bitrate_kbps: u32,
output_path: &Path,
full_range: bool,
) -> Result<Self, String> {
let renderer = Rgba2Nv12::new(&device, full_range);
unsafe {
let mut hw_device = crate::vaapi::create_device()?;
let name = CString::new("h264_vaapi").unwrap();
let codec = ff::avcodec_find_encoder_by_name(name.as_ptr());
if codec.is_null() {
ff::av_buffer_unref(&mut hw_device);
return Err("h264_vaapi not found".into());
}
let enc = ff::avcodec_alloc_context3(codec);
(*enc).width = width as i32;
(*enc).height = height as i32;
(*enc).time_base = ff::AVRational { num: 1, den: framerate };
(*enc).framerate = ff::AVRational { num: framerate, den: 1 };
(*enc).pix_fmt = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
(*enc).bit_rate = (bitrate_kbps as i64) * 1000;
// Color signalling for the H.264 VUI. The Rgba2Nv12 shader produces BT.709 luma/chroma
// in the matching range; without these tags players assume limited range and a
// full-range stream looks dark + oversaturated.
(*enc).color_range = if full_range {
ff::AVColorRange::AVCOL_RANGE_JPEG
} else {
ff::AVColorRange::AVCOL_RANGE_MPEG
};
(*enc).colorspace = ff::AVColorSpace::AVCOL_SPC_BT709;
(*enc).color_primaries = ff::AVColorPrimaries::AVCOL_PRI_BT709;
(*enc).color_trc = ff::AVColorTransferCharacteristic::AVCOL_TRC_BT709;
let frames_ref = ff::av_hwframe_ctx_alloc(hw_device);
{
let fctx = (*frames_ref).data as *mut ff::AVHWFramesContext;
(*fctx).format = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
(*fctx).sw_format = ff::AVPixelFormat::AV_PIX_FMT_NV12;
(*fctx).width = width as i32;
(*fctx).height = height as i32;
(*fctx).initial_pool_size = 16;
}
if ff::av_hwframe_ctx_init(frames_ref) < 0 {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::avcodec_free_context(&mut (enc as *mut _));
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_ctx_init failed".into());
}
(*enc).hw_frames_ctx = ff::av_buffer_ref(frames_ref);
// Output container (format inferred from the path's extension).
let cleanup = |frames_ref: *mut ff::AVBufferRef, enc: *mut ff::AVCodecContext, hw: *mut ff::AVBufferRef| {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::avcodec_free_context(&mut (enc as *mut _));
let mut h = hw;
ff::av_buffer_unref(&mut h);
};
let path_c = CString::new(output_path.to_string_lossy().as_ref()).unwrap();
let mut oc: *mut ff::AVFormatContext = ptr::null_mut();
if ff::avformat_alloc_output_context2(&mut oc, ptr::null(), ptr::null(), path_c.as_ptr()) < 0
|| oc.is_null()
{
cleanup(frames_ref, enc, hw_device);
return Err(format!("avformat_alloc_output_context2 for {output_path:?} failed"));
}
// mp4/mov want SPS/PPS in extradata, not inline — set before opening the encoder.
if (*(*oc).oformat).flags & ff::AVFMT_GLOBALHEADER as i32 != 0 {
(*enc).flags |= ff::AV_CODEC_FLAG_GLOBAL_HEADER as i32;
}
if ff::avcodec_open2(enc, codec, ptr::null_mut()) < 0 {
ff::avformat_free_context(oc);
cleanup(frames_ref, enc, hw_device);
return Err("avcodec_open2(h264_vaapi) failed".into());
}
let stream = ff::avformat_new_stream(oc, codec);
if stream.is_null() {
ff::avformat_free_context(oc);
cleanup(frames_ref, enc, hw_device);
return Err("avformat_new_stream failed".into());
}
if ff::avcodec_parameters_from_context((*stream).codecpar, enc) < 0 {
ff::avformat_free_context(oc);
cleanup(frames_ref, enc, hw_device);
return Err("avcodec_parameters_from_context failed".into());
}
(*stream).time_base = (*enc).time_base;
if ff::avio_open(&mut (*oc).pb, path_c.as_ptr(), ff::AVIO_FLAG_WRITE as i32) < 0 {
ff::avformat_free_context(oc);
cleanup(frames_ref, enc, hw_device);
return Err(format!("avio_open {output_path:?} failed"));
}
if ff::avformat_write_header(oc, ptr::null_mut()) < 0 {
ff::avio_closep(&mut (*oc).pb);
ff::avformat_free_context(oc);
cleanup(frames_ref, enc, hw_device);
return Err("avformat_write_header failed".into());
}
// The muxer may rewrite the stream time_base in write_header.
let stream_tb = (*stream).time_base;
Ok(Self {
device,
queue,
adapter,
renderer,
hw_device,
frames_ref,
enc,
pkt: ff::av_packet_alloc(),
oc,
enc_tb: (*enc).time_base,
stream_tb,
width,
height,
pts: 0,
cache: HashMap::new(),
})
}
}
/// The wgpu device frames must be rendered on (so the RGBA texture is importable).
pub fn device(&self) -> &wgpu::Device {
&self.device
}
pub fn queue(&self) -> &wgpu::Queue {
&self.queue
}
/// Render `rgba` (an `Rgba8Unorm` texture on [`Self::device`], `TEXTURE_BINDING`)
/// into a VAAPI surface and encode it. Appends any produced packets internally.
pub fn encode_rgba(&mut self, rgba: &wgpu::Texture) -> Result<(), String> {
unsafe {
let surf = ff::av_frame_alloc();
if ff::av_hwframe_get_buffer(self.frames_ref, surf, 0) < 0 {
ff::av_frame_free(&mut (surf as *mut _));
return Err("av_hwframe_get_buffer failed".into());
}
let id = (*surf).data[3] as usize; // VASurfaceID
if !self.cache.contains_key(&id) {
let drm_f = ff::av_frame_alloc();
(*drm_f).format = ff::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let flags = ff::AV_HWFRAME_MAP_DIRECT as i32
| ff::AV_HWFRAME_MAP_READ as i32
| ff::AV_HWFRAME_MAP_WRITE as i32;
if ff::av_hwframe_map(drm_f, surf, flags) < 0 {
ff::av_frame_free(&mut (drm_f as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
return Err("av_hwframe_map failed".into());
}
let desc = (*drm_f).data[0] as *const ff::AVDRMFrameDescriptor;
let obj = &(*desc).objects[0];
let y = &(*desc).layers[0].planes[0];
let uv = &(*desc).layers[1].planes[0];
let buf = Nv12DmaBuf {
fd: obj.fd,
size: obj.size as u64,
modifier: obj.format_modifier,
width: self.width,
height: self.height,
y_offset: y.offset as u64,
y_pitch: y.pitch as u64,
uv_offset: uv.offset as u64,
uv_pitch: uv.pitch as u64,
ten_bit: false,
};
let imported = match dmabuf::import_raw(&self.device, &self.adapter, &buf) {
Ok(i) => i,
Err(e) => {
ff::av_frame_free(&mut (drm_f as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
return Err(e);
}
};
ff::av_frame_free(&mut (drm_f as *mut _)); // fd was dup'd into Vulkan
self.cache.insert(id, imported);
}
// Render RGBA -> NV12 directly into the surface planes.
let imp = self.cache.get(&id).unwrap();
let rgba_view = rgba.create_view(&Default::default());
let y_view = imp.y().create_view(&Default::default());
let uv_view = imp.uv().create_view(&Default::default());
let mut cmd = self.device.create_command_encoder(&Default::default());
self.renderer.convert(&self.device, &mut cmd, &rgba_view, &y_view, &uv_view);
self.queue.submit(Some(cmd.finish()));
let _ = self.device.poll(wgpu::PollType::wait_indefinitely());
// Encode the surface.
(*surf).pts = self.pts;
self.pts += 1;
let r = ff::avcodec_send_frame(self.enc, surf);
ff::av_frame_free(&mut (surf as *mut _));
if r < 0 {
return Err(format!("avcodec_send_frame failed: {r}"));
}
self.drain()
}
}
unsafe fn drain(&mut self) -> Result<(), String> {
loop {
let r = ff::avcodec_receive_packet(self.enc, self.pkt);
if r == averror(libc::EAGAIN) || r == ff::AVERROR_EOF {
break;
}
if r < 0 {
return Err(format!("avcodec_receive_packet failed: {r}"));
}
ff::av_packet_rescale_ts(self.pkt, self.enc_tb, self.stream_tb);
(*self.pkt).stream_index = 0;
// Takes ownership of the packet's buffer (unrefs it for us).
let w = ff::av_interleaved_write_frame(self.oc, self.pkt);
if w < 0 {
return Err(format!("av_interleaved_write_frame failed: {w}"));
}
}
Ok(())
}
/// Flush the encoder, write the container trailer, and close the output file.
pub fn finish(mut self) -> Result<(), String> {
unsafe {
ff::avcodec_send_frame(self.enc, ptr::null_mut());
self.drain()?;
if ff::av_write_trailer(self.oc) < 0 {
return Err("av_write_trailer failed".into());
}
ff::avio_closep(&mut (*self.oc).pb);
}
Ok(())
}
}
impl Drop for ZeroCopyEncoder {
fn drop(&mut self) {
unsafe {
self.cache.clear(); // frees imported Vulkan resources first
ff::av_packet_free(&mut (self.pkt as *mut _));
ff::avcodec_free_context(&mut (self.enc as *mut _));
let mut fr = self.frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut self.hw_device);
if !self.oc.is_null() {
// `finish` nulls pb via avio_closep; close here too if it wasn't called.
if !(*self.oc).pb.is_null() {
ff::avio_closep(&mut (*self.oc).pb);
}
ff::avformat_free_context(self.oc);
self.oc = ptr::null_mut();
}
}
}
}

View File

@ -0,0 +1,65 @@
//! Zero-copy GPU video encoding.
//!
//! Converts a rendered RGBA texture to the encoder's pixel format (NV12) on the GPU
//! and feeds it to a hardware video encoder without a CPU round-trip. All the unsafe
//! GPU↔encoder interop (Vulkan external memory / DMA-BUF → VAAPI on Linux, etc.) is
//! isolated in this crate.
//!
//! Status: scaffolding. Headless GPU probe + (next) NV12 compute live here first so
//! the GPU-side conversion can be validated against a CPU reference before any unsafe
//! interop is written. See `lightningbeam-ui/ZEROCOPY_GPU_ENCODE_PLAN.md`.
pub mod nv12;
/// Fragment-shader RGBA→NV12 conversion that renders into plane textures.
pub mod render_nv12;
/// VAAPI hardware encode (Linux-only; libva).
#[cfg(target_os = "linux")]
pub mod vaapi;
/// Custom Vulkan device with DMA-BUF import extensions (Linux).
#[cfg(target_os = "linux")]
pub mod vk_device;
/// Import a VAAPI NV12 DMA-BUF as wgpu textures (Linux).
#[cfg(target_os = "linux")]
pub mod dmabuf;
/// End-to-end zero-copy `h264_vaapi` encoder (Linux).
#[cfg(target_os = "linux")]
pub mod encoder;
/// VAAPI hardware decode → wgpu textures (Linux).
#[cfg(target_os = "linux")]
pub mod decoder;
#[cfg(test)]
mod probe_tests {
/// Confirm a headless GPU adapter is reachable (Vulkan on Linux/Intel). This gates
/// whether the GPU-side conversion can be tested on real hardware in this env.
#[test]
fn headless_adapter_available() {
let instance = wgpu::Instance::new(&wgpu::InstanceDescriptor {
backends: wgpu::Backends::VULKAN | wgpu::Backends::GL,
..Default::default()
});
let adapter = pollster::block_on(instance.request_adapter(
&wgpu::RequestAdapterOptions {
power_preference: wgpu::PowerPreference::HighPerformance,
force_fallback_adapter: false,
compatible_surface: None,
},
));
match adapter {
Ok(a) => {
let info = a.get_info();
eprintln!(
"[gpu-probe] adapter: {} | backend={:?} | type={:?} | driver={}",
info.name, info.backend, info.device_type, info.driver
);
}
Err(e) => panic!("no GPU adapter available headless: {e:?}"),
}
}
}

View File

@ -0,0 +1,207 @@
//! GPU RGBA→NV12 conversion (BT.709 full-range), the pixel format hardware video
//! encoders (VAAPI/QSV/NVENC/VideoToolbox) consume.
//!
//! NV12 layout (what this writes, tight-packed into a storage buffer):
//! - `[0, W*H)` Y plane, one byte/pixel, row stride `W`
//! - `[W*H, W*H + W*H/2)` UV plane, interleaved `U,V` at 4:2:0, row stride `W`
//! (`W/2` chroma columns × 2 bytes), `H/2` rows
//!
//! Same BT.709 full-range matrix as the editor's planar YUV420p path, so colors match.
//! Requires `W % 8 == 0 && H % 2 == 0` (the shader packs 4 bytes per `u32`).
/// `true` when [`Nv12Converter`] can handle these dimensions (else caller pads/falls back).
pub fn supports(width: u32, height: u32) -> bool {
width % 8 == 0 && height % 2 == 0 && width > 0 && height > 0
}
/// Tight NV12 byte length for `width`×`height`.
pub fn nv12_len(width: u32, height: u32) -> usize {
(width * height + width * (height / 2)) as usize
}
/// Compute pipeline: `Rgba8Unorm` texture → tight NV12 storage buffer.
pub struct Nv12Converter {
y_pipeline: wgpu::ComputePipeline,
uv_pipeline: wgpu::ComputePipeline,
bind_group_layout: wgpu::BindGroupLayout,
}
impl Nv12Converter {
pub fn new(device: &wgpu::Device) -> Self {
let bind_group_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("nv12_bgl"),
entries: &[
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: false },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
},
wgpu::BindGroupLayoutEntry {
binding: 1,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Buffer {
ty: wgpu::BufferBindingType::Storage { read_only: false },
has_dynamic_offset: false,
min_binding_size: None,
},
count: None,
},
],
});
let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("nv12_pl"),
bind_group_layouts: &[&bind_group_layout],
push_constant_ranges: &[],
});
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("nv12_shader"),
source: wgpu::ShaderSource::Wgsl(SHADER.into()),
});
let mk = |entry: &str| {
device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor {
label: Some("nv12_pipeline"),
layout: Some(&pipeline_layout),
module: &shader,
entry_point: Some(entry),
compilation_options: wgpu::PipelineCompilationOptions::default(),
cache: None,
})
};
Self {
y_pipeline: mk("y_main"),
uv_pipeline: mk("uv_main"),
bind_group_layout,
}
}
/// Record RGBA→NV12 into `encoder`. `out_buffer` must be `STORAGE | COPY_SRC` of at
/// least [`nv12_len`] bytes. Caller must ensure [`supports`]`(width, height)`.
pub fn convert(
&self,
device: &wgpu::Device,
encoder: &mut wgpu::CommandEncoder,
rgba_view: &wgpu::TextureView,
out_buffer: &wgpu::Buffer,
width: u32,
height: u32,
) {
debug_assert!(supports(width, height));
let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("nv12_bg"),
layout: &self.bind_group_layout,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: wgpu::BindingResource::TextureView(rgba_view) },
wgpu::BindGroupEntry { binding: 1, resource: out_buffer.as_entire_binding() },
],
});
let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor {
label: Some("nv12_pass"),
timestamp_writes: None,
});
pass.set_bind_group(0, &bind_group, &[]);
let wg = 8u32;
// Y: one thread per 4 horizontal luma samples.
pass.set_pipeline(&self.y_pipeline);
pass.dispatch_workgroups(((width / 4) + wg - 1) / wg, (height + wg - 1) / wg, 1);
// UV: one thread per 4 interleaved UV bytes = 2 chroma columns; (W/4)×(H/2) threads.
pass.set_pipeline(&self.uv_pipeline);
pass.dispatch_workgroups(((width / 4) + wg - 1) / wg, ((height / 2) + wg - 1) / wg, 1);
}
}
/// CPU reference producing the exact bytes the shader should — used by tests to verify
/// the GPU output on real hardware.
pub fn cpu_reference(rgba: &[u8], width: u32, height: u32) -> Vec<u8> {
let w = width as usize;
let h = height as usize;
let mut out = vec![0u8; nv12_len(width, height)];
let to_byte = |v: f32| (v.clamp(0.0, 1.0) * 255.0 + 0.5) as u8;
let px = |x: usize, y: usize| {
let i = (y * w + x) * 4;
[rgba[i] as f32 / 255.0, rgba[i + 1] as f32 / 255.0, rgba[i + 2] as f32 / 255.0]
};
// Y
for y in 0..h {
for x in 0..w {
let p = px(x, y);
out[y * w + x] = to_byte(0.2126 * p[0] + 0.7152 * p[1] + 0.0722 * p[2]);
}
}
// Interleaved UV (2x2 box average)
let y_size = w * h;
for cy in 0..h / 2 {
for cx in 0..w / 2 {
let mut acc = [0.0f32; 3];
for (dx, dy) in [(0, 0), (1, 0), (0, 1), (1, 1)] {
let p = px(2 * cx + dx, 2 * cy + dy);
acc[0] += p[0]; acc[1] += p[1]; acc[2] += p[2];
}
let a = [acc[0] / 4.0, acc[1] / 4.0, acc[2] / 4.0];
let u = -0.1146 * a[0] - 0.3854 * a[1] + 0.5000 * a[2] + 0.5;
let v = 0.5000 * a[0] - 0.4542 * a[1] - 0.0458 * a[2] + 0.5;
out[y_size + cy * w + 2 * cx] = to_byte(u);
out[y_size + cy * w + 2 * cx + 1] = to_byte(v);
}
}
out
}
const SHADER: &str = r#"
@group(0) @binding(0) var input_rgba: texture_2d<f32>;
@group(0) @binding(1) var<storage, read_write> out_buf: array<u32>;
fn to_byte(v: f32) -> u32 { return u32(clamp(v, 0.0, 1.0) * 255.0 + 0.5); }
// Y plane: pack 4 horizontal luma bytes.
@compute @workgroup_size(8, 8, 1)
fn y_main(@builtin(global_invocation_id) gid: vec3<u32>) {
let dims = textureDimensions(input_rgba);
let w = dims.x;
let h = dims.y;
let x4 = gid.x * 4u;
let y = gid.y;
if (x4 >= w || y >= h) { return; }
var packed: u32 = 0u;
for (var i = 0u; i < 4u; i = i + 1u) {
let c = textureLoad(input_rgba, vec2<u32>(x4 + i, y), 0).rgb;
let yy = 0.2126 * c.r + 0.7152 * c.g + 0.0722 * c.b;
packed = packed | (to_byte(yy) << (8u * i));
}
out_buf[(y * w + x4) / 4u] = packed;
}
// UV plane: each thread writes 4 interleaved bytes = U0 V0 U1 V1 for 2 chroma columns.
@compute @workgroup_size(8, 8, 1)
fn uv_main(@builtin(global_invocation_id) gid: vec3<u32>) {
let dims = textureDimensions(input_rgba);
let w = dims.x;
let h = dims.y;
let k = gid.x; // chroma-column pair index: covers columns 2k, 2k+1
let cy = gid.y;
if (k * 2u >= w / 2u || cy >= h / 2u) { return; }
let y_size = w * h;
var packed: u32 = 0u;
for (var j = 0u; j < 2u; j = j + 1u) {
let cx = 2u * k + j; // chroma column
let sx = 2u * cx;
let sy = 2u * cy;
let p00 = textureLoad(input_rgba, vec2<u32>(sx, sy), 0).rgb;
let p10 = textureLoad(input_rgba, vec2<u32>(sx + 1u, sy), 0).rgb;
let p01 = textureLoad(input_rgba, vec2<u32>(sx, sy + 1u), 0).rgb;
let p11 = textureLoad(input_rgba, vec2<u32>(sx + 1u, sy + 1u), 0).rgb;
let a = (p00 + p10 + p01 + p11) * 0.25;
let u = -0.1146 * a.r - 0.3854 * a.g + 0.5000 * a.b + 0.5;
let v = 0.5000 * a.r - 0.4542 * a.g - 0.0458 * a.b + 0.5;
packed = packed | (to_byte(u) << (16u * j)); // byte 0 or 2
packed = packed | (to_byte(v) << (16u * j + 8u)); // byte 1 or 3
}
// UV row stride is w bytes; this thread writes 4 bytes at column 4k.
out_buf[(y_size + cy * w + 4u * k) / 4u] = packed;
}
"#;

View File

@ -0,0 +1,189 @@
//! Fragment-shader RGBA→NV12 conversion that **renders** luma/chroma into the encoder
//! surface's plane textures (R8 Y, RG8 UV). Render targets (not compute storage) so it
//! works with the DMA-BUF-imported plane images, which aren't storage-writable.
//!
//! BT.709 matrix; the Y/chroma scale+offset are chosen by `full_range` — limited (TV, 16235)
//! is the H.264 convention most players assume, full (PC, 0255) needs the matching color tag.
/// Converts a bound RGBA texture into a Y plane (R8) and a UV plane (RG8) via two passes.
pub struct Rgba2Nv12 {
y_pipeline: wgpu::RenderPipeline,
uv_pipeline: wgpu::RenderPipeline,
bgl: wgpu::BindGroupLayout,
params_buf: wgpu::Buffer,
}
impl Rgba2Nv12 {
/// `full_range`: true → full/PC range (Y 0255, no scaling); false → limited/TV range
/// (Y 16235, chroma 16240), the H.264 default. The encoder must tag the stream to match.
pub fn new(device: &wgpu::Device, full_range: bool) -> Self {
// (y_offset, y_scale, chroma_offset, chroma_scale). The shader applies these to the
// BT.709 luma/chroma dot products.
let params: [f32; 4] = if full_range {
[0.0, 1.0, 128.0 / 255.0, 1.0]
} else {
[16.0 / 255.0, 219.0 / 255.0, 128.0 / 255.0, 224.0 / 255.0]
};
let params_bytes: Vec<u8> = params.iter().flat_map(|f| f.to_le_bytes()).collect();
let params_buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("rgba2nv12_params"),
size: params_bytes.len() as u64,
usage: wgpu::BufferUsages::UNIFORM,
mapped_at_creation: true,
});
params_buf.slice(..).get_mapped_range_mut().copy_from_slice(&params_bytes);
params_buf.unmap();
let bgl = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("rgba2nv12_bgl"),
entries: &[
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::FRAGMENT,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: false },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
},
wgpu::BindGroupLayoutEntry {
binding: 1,
visibility: wgpu::ShaderStages::FRAGMENT,
ty: wgpu::BindingType::Buffer {
ty: wgpu::BufferBindingType::Uniform,
has_dynamic_offset: false,
min_binding_size: None,
},
count: None,
},
],
});
let layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("rgba2nv12_pl"),
bind_group_layouts: &[&bgl],
push_constant_ranges: &[],
});
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("rgba2nv12_shader"),
source: wgpu::ShaderSource::Wgsl(SHADER.into()),
});
let mk = |fs: &str, fmt: wgpu::TextureFormat| {
device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
label: Some("rgba2nv12_pipeline"),
layout: Some(&layout),
vertex: wgpu::VertexState {
module: &shader,
entry_point: Some("vs_main"),
buffers: &[],
compilation_options: Default::default(),
},
fragment: Some(wgpu::FragmentState {
module: &shader,
entry_point: Some(fs),
targets: &[Some(fmt.into())],
compilation_options: Default::default(),
}),
primitive: wgpu::PrimitiveState {
topology: wgpu::PrimitiveTopology::TriangleList,
..Default::default()
},
depth_stencil: None,
multisample: Default::default(),
multiview: None,
cache: None,
})
};
Self {
y_pipeline: mk("y_fs", wgpu::TextureFormat::R8Unorm),
uv_pipeline: mk("uv_fs", wgpu::TextureFormat::Rg8Unorm),
bgl,
params_buf,
}
}
/// Record both plane passes. `y_view`/`uv_view` are the R8/RG8 plane render targets.
pub fn convert(
&self,
device: &wgpu::Device,
encoder: &mut wgpu::CommandEncoder,
rgba_view: &wgpu::TextureView,
y_view: &wgpu::TextureView,
uv_view: &wgpu::TextureView,
) {
let bg = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("rgba2nv12_bg"),
layout: &self.bgl,
entries: &[
wgpu::BindGroupEntry {
binding: 0,
resource: wgpu::BindingResource::TextureView(rgba_view),
},
wgpu::BindGroupEntry {
binding: 1,
resource: self.params_buf.as_entire_binding(),
},
],
});
for (pipeline, view) in [(&self.y_pipeline, y_view), (&self.uv_pipeline, uv_view)] {
let mut pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("rgba2nv12_pass"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view,
resolve_target: None,
depth_slice: None,
ops: wgpu::Operations {
load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT),
store: wgpu::StoreOp::Store,
},
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
});
pass.set_pipeline(pipeline);
pass.set_bind_group(0, &bg, &[]);
pass.draw(0..3, 0..1);
}
}
}
const SHADER: &str = r#"
@group(0) @binding(0) var input_rgba: texture_2d<f32>;
// (y_offset, y_scale, chroma_offset, chroma_scale) — selects limited vs full range.
@group(0) @binding(1) var<uniform> params: vec4<f32>;
// Fullscreen triangle.
@vertex
fn vs_main(@builtin(vertex_index) vi: u32) -> @builtin(position) vec4<f32> {
let x = f32((vi << 1u) & 2u);
let y = f32(vi & 2u);
return vec4<f32>(x * 2.0 - 1.0, 1.0 - y * 2.0, 0.0, 1.0);
}
fn load(p: vec2<i32>) -> vec3<f32> {
return textureLoad(input_rgba, p, 0).rgb;
}
// Y plane (full res): one luma byte per pixel.
@fragment
fn y_fs(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> {
let c = load(vec2<i32>(i32(pos.x), i32(pos.y)));
let luma = 0.2126 * c.r + 0.7152 * c.g + 0.0722 * c.b;
let y = params.x + params.y * luma;
return vec4<f32>(y, 0.0, 0.0, 1.0);
}
// UV plane (half res): 2x2 box-averaged chroma, interleaved into RG.
@fragment
fn uv_fs(@builtin(position) pos: vec4<f32>) -> @location(0) vec4<f32> {
let sx = 2 * i32(pos.x);
let sy = 2 * i32(pos.y);
let a = (load(vec2<i32>(sx, sy)) + load(vec2<i32>(sx + 1, sy))
+ load(vec2<i32>(sx, sy + 1)) + load(vec2<i32>(sx + 1, sy + 1))) * 0.25;
let cb = -0.1146 * a.r - 0.3854 * a.g + 0.5000 * a.b;
let cr = 0.5000 * a.r - 0.4542 * a.g - 0.0458 * a.b;
let u = params.z + params.w * cb;
let v = params.z + params.w * cr;
return vec4<f32>(u, v, 0.0, 1.0);
}
"#;

View File

@ -0,0 +1,521 @@
//! VAAPI hardware H.264 encoding (Linux/Intel/AMD).
//!
//! Level 1 (this module first): a CPU-fed encoder — upload NV12 frames to VAAPI
//! surfaces (`av_hwframe_transfer_data`) and encode with `h264_vaapi`. This proves the
//! encoder works and establishes the FFI scaffolding. Level 2 (zero-copy: GPU writes
//! NV12 straight into the VAAPI surface via DMA-BUF) builds on this.
//!
//! All `unsafe` FFmpeg FFI is contained here.
use ffmpeg_sys_next as ff;
use std::ffi::CString;
use std::ptr;
#[inline]
fn averror(e: i32) -> i32 {
-e
}
/// Create a VAAPI hwdevice on `/dev/dri/renderD128`, trying driver names in turn.
///
/// libva's auto-selection can pick a driver that doesn't support the GPU — notably it
/// chooses the legacy `i965` driver on newer Intel parts (Gen 11+) where the modern `iHD`
/// driver is required. Each `av_hwdevice_ctx_create` opens a fresh VADisplay, so
/// `LIBVA_DRIVER_NAME` is re-read per attempt. We try `iHD` first (modern Intel), then the
/// caller's original setting, then `i965` (older Intel) and `radeonsi` (AMD). On success the
/// working driver name is left in the env; on total failure the original value is restored.
pub fn create_device() -> Result<*mut ff::AVBufferRef, String> {
unsafe {
let node = CString::new("/dev/dri/renderD128").unwrap();
let original = std::env::var_os("LIBVA_DRIVER_NAME");
let attempts: [Option<&str>; 4] = [Some("iHD"), None, Some("i965"), Some("radeonsi")];
for drv in attempts {
match drv {
Some(d) => std::env::set_var("LIBVA_DRIVER_NAME", d),
// `None` = the caller's original setting (or libva auto if unset).
None => match &original {
Some(v) => std::env::set_var("LIBVA_DRIVER_NAME", v),
None => std::env::remove_var("LIBVA_DRIVER_NAME"),
},
}
let mut hw: *mut ff::AVBufferRef = ptr::null_mut();
if ff::av_hwdevice_ctx_create(
&mut hw,
ff::AVHWDeviceType::AV_HWDEVICE_TYPE_VAAPI,
node.as_ptr(),
ptr::null_mut(),
0,
) >= 0
{
return Ok(hw);
}
}
match &original {
Some(v) => std::env::set_var("LIBVA_DRIVER_NAME", v),
None => std::env::remove_var("LIBVA_DRIVER_NAME"),
}
Err("av_hwdevice_ctx_create(VAAPI) failed for all drivers (iHD/i965/radeonsi)".into())
}
}
/// Copy tight NV12 (`Y` then interleaved `UV`) into an AVFrame's planes, respecting
/// each plane's linesize (which FFmpeg may pad).
unsafe fn fill_nv12(frame: *mut ff::AVFrame, nv12: &[u8], width: u32, height: u32) {
let w = width as usize;
let h = height as usize;
// Y plane: h rows of w bytes.
let dst_y = (*frame).data[0];
let ls_y = (*frame).linesize[0] as usize;
for row in 0..h {
let src = &nv12[row * w..row * w + w];
ptr::copy_nonoverlapping(src.as_ptr(), dst_y.add(row * ls_y), w);
}
// UV plane: h/2 rows of w bytes (interleaved U,V), source offset starts at w*h.
let dst_uv = (*frame).data[1];
let ls_uv = (*frame).linesize[1] as usize;
let uv_off = w * h;
for row in 0..h / 2 {
let src = &nv12[uv_off + row * w..uv_off + row * w + w];
ptr::copy_nonoverlapping(src.as_ptr(), dst_uv.add(row * ls_uv), w);
}
}
/// A VAAPI NV12 surface mapped to a DMA-BUF, with its layout extracted for Vulkan import.
/// Keeps the FFmpeg handles alive; the `fd` stays valid until drop (dup it for Vulkan).
pub struct MappedSurface {
hw_device: *mut ff::AVBufferRef,
frames_ref: *mut ff::AVBufferRef,
surf: *mut ff::AVFrame,
drm: *mut ff::AVFrame,
pub width: u32,
pub height: u32,
pub fd: i32,
pub size: u64,
pub modifier: u64,
pub y_offset: u64,
pub y_pitch: u64,
pub uv_offset: u64,
pub uv_pitch: u64,
}
impl MappedSurface {
/// Allocate a VAAPI NV12 surface and map it to DRM-PRIME.
pub fn alloc(width: u32, height: u32) -> Result<Self, String> {
unsafe {
let mut hw_device: *mut ff::AVBufferRef = ptr::null_mut();
let node = CString::new("/dev/dri/renderD128").unwrap();
if ff::av_hwdevice_ctx_create(
&mut hw_device,
ff::AVHWDeviceType::AV_HWDEVICE_TYPE_VAAPI,
node.as_ptr(),
ptr::null_mut(),
0,
) < 0
{
return Err("av_hwdevice_ctx_create failed".into());
}
let frames_ref = ff::av_hwframe_ctx_alloc(hw_device);
if frames_ref.is_null() {
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_ctx_alloc failed".into());
}
{
let fctx = (*frames_ref).data as *mut ff::AVHWFramesContext;
(*fctx).format = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
(*fctx).sw_format = ff::AVPixelFormat::AV_PIX_FMT_NV12;
(*fctx).width = width as i32;
(*fctx).height = height as i32;
(*fctx).initial_pool_size = 4;
}
if ff::av_hwframe_ctx_init(frames_ref) < 0 {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_ctx_init failed".into());
}
let surf = ff::av_frame_alloc();
if ff::av_hwframe_get_buffer(frames_ref, surf, 0) < 0 {
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_get_buffer failed".into());
}
let drm = ff::av_frame_alloc();
(*drm).format = ff::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let flags = ff::AV_HWFRAME_MAP_DIRECT as i32
| ff::AV_HWFRAME_MAP_READ as i32
| ff::AV_HWFRAME_MAP_WRITE as i32;
if ff::av_hwframe_map(drm, surf, flags) < 0 {
ff::av_frame_free(&mut (drm as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_map failed".into());
}
let desc = (*drm).data[0] as *const ff::AVDRMFrameDescriptor;
// Expect 1 object, 2 layers (Y=R8, UV=GR88).
if (*desc).nb_objects != 1 || (*desc).nb_layers != 2 {
let msg = format!(
"unexpected DRM layout: {} objects, {} layers",
(*desc).nb_objects, (*desc).nb_layers
);
// Free everything mapped/allocated above (this path was leaking the device,
// frames context, and both AVFrames on every odd-layout surface).
ff::av_frame_free(&mut (drm as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err(msg);
}
let obj = &(*desc).objects[0];
let y = &(*desc).layers[0].planes[0];
let uv = &(*desc).layers[1].planes[0];
Ok(MappedSurface {
hw_device,
frames_ref,
surf,
drm,
width,
height,
fd: obj.fd,
size: obj.size as u64,
modifier: obj.format_modifier,
y_offset: y.offset as u64,
y_pitch: y.pitch as u64,
uv_offset: uv.offset as u64,
uv_pitch: uv.pitch as u64,
})
}
}
/// The underlying VASurface AVFrame (to hand to the encoder).
pub fn av_frame(&self) -> *mut ff::AVFrame {
self.surf
}
/// Read the surface back to tight CPU NV12 (for verifying what the GPU wrote).
pub fn readback_nv12(&self) -> Result<Vec<u8>, String> {
unsafe {
let sw = ff::av_frame_alloc();
(*sw).format = ff::AVPixelFormat::AV_PIX_FMT_NV12 as i32;
(*sw).width = self.width as i32;
(*sw).height = self.height as i32;
if ff::av_frame_get_buffer(sw, 0) < 0 {
ff::av_frame_free(&mut (sw as *mut _));
return Err("av_frame_get_buffer failed".into());
}
if ff::av_hwframe_transfer_data(sw, self.surf, 0) < 0 {
ff::av_frame_free(&mut (sw as *mut _));
return Err("av_hwframe_transfer_data (download) failed".into());
}
let w = self.width as usize;
let h = self.height as usize;
let mut out = vec![0u8; w * h + w * (h / 2)];
let ls_y = (*sw).linesize[0] as usize;
for row in 0..h {
let src = (*sw).data[0].add(row * ls_y);
ptr::copy_nonoverlapping(src, out.as_mut_ptr().add(row * w), w);
}
let ls_uv = (*sw).linesize[1] as usize;
let uv_off = w * h;
for row in 0..h / 2 {
let src = (*sw).data[1].add(row * ls_uv);
ptr::copy_nonoverlapping(src, out.as_mut_ptr().add(uv_off + row * w), w);
}
ff::av_frame_free(&mut (sw as *mut _));
Ok(out)
}
}
}
impl Drop for MappedSurface {
fn drop(&mut self) {
unsafe {
ff::av_frame_free(&mut (self.drm as *mut _));
ff::av_frame_free(&mut (self.surf as *mut _));
let mut fr = self.frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut self.hw_device);
}
}
}
/// Allocate one VAAPI NV12 surface, map it to a DRM-PRIME descriptor, and return a
/// human-readable dump of its DMA-BUF layout (object fds/size/modifier; layer fourcc;
/// per-plane object/offset/pitch). The format **modifier** decides the zero-copy path:
/// `0` = LINEAR (compute can write a linear NV12 buffer/image), anything else = tiled
/// (needs a GPU copy into the tiled surface, or a linear import VAAPI accepts).
pub fn probe_surface_drm(width: u32, height: u32) -> Result<String, String> {
unsafe {
let mut hw_device: *mut ff::AVBufferRef = ptr::null_mut();
let node = CString::new("/dev/dri/renderD128").unwrap();
if ff::av_hwdevice_ctx_create(
&mut hw_device,
ff::AVHWDeviceType::AV_HWDEVICE_TYPE_VAAPI,
node.as_ptr(),
ptr::null_mut(),
0,
) < 0
{
return Err("av_hwdevice_ctx_create(VAAPI) failed".into());
}
let frames_ref = ff::av_hwframe_ctx_alloc(hw_device);
if frames_ref.is_null() {
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_ctx_alloc failed".into());
}
{
let fctx = (*frames_ref).data as *mut ff::AVHWFramesContext;
(*fctx).format = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
(*fctx).sw_format = ff::AVPixelFormat::AV_PIX_FMT_NV12;
(*fctx).width = width as i32;
(*fctx).height = height as i32;
(*fctx).initial_pool_size = 2;
}
if ff::av_hwframe_ctx_init(frames_ref) < 0 {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_ctx_init failed".into());
}
let surf = ff::av_frame_alloc();
if ff::av_hwframe_get_buffer(frames_ref, surf, 0) < 0 {
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err("av_hwframe_get_buffer failed".into());
}
let drm = ff::av_frame_alloc();
(*drm).format = ff::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let flags = ff::AV_HWFRAME_MAP_DIRECT as i32
| ff::AV_HWFRAME_MAP_READ as i32
| ff::AV_HWFRAME_MAP_WRITE as i32;
let r = ff::av_hwframe_map(drm, surf, flags);
if r < 0 {
ff::av_frame_free(&mut (drm as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
return Err(format!("av_hwframe_map(DRM_PRIME) failed: {r}"));
}
let desc = (*drm).data[0] as *const ff::AVDRMFrameDescriptor;
let mut s = format!("VAAPI NV12 {width}x{height} surface as DRM-PRIME:\n");
s += &format!(" nb_objects = {}\n", (*desc).nb_objects);
for o in 0..(*desc).nb_objects as usize {
let obj = &(*desc).objects[o];
s += &format!(
" object[{o}]: fd={} size={} format_modifier=0x{:016x}{}\n",
obj.fd,
obj.size,
obj.format_modifier,
if obj.format_modifier == 0 { " (LINEAR)" } else { " (tiled)" },
);
}
s += &format!(" nb_layers = {}\n", (*desc).nb_layers);
for l in 0..(*desc).nb_layers as usize {
let lay = &(*desc).layers[l];
let f = lay.format;
let fourcc = [(f & 0xff) as u8, ((f >> 8) & 0xff) as u8, ((f >> 16) & 0xff) as u8, ((f >> 24) & 0xff) as u8];
s += &format!(
" layer[{l}]: format='{}' (0x{:08x}) nb_planes={}\n",
String::from_utf8_lossy(&fourcc),
f,
lay.nb_planes,
);
for p in 0..lay.nb_planes as usize {
let pl = &lay.planes[p];
s += &format!(
" plane[{p}]: object_index={} offset={} pitch={}\n",
pl.object_index, pl.offset, pl.pitch,
);
}
}
ff::av_frame_free(&mut (drm as *mut _));
ff::av_frame_free(&mut (surf as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::av_buffer_unref(&mut hw_device);
Ok(s)
}
}
/// Encode NV12 frames with `h264_vaapi` and write the raw Annex-B H.264 to `out_path`.
/// Returns the number of encoded packets. `Err` (rather than panic) when VAAPI/the
/// encoder is unavailable, so callers can fall back.
pub fn encode_nv12_to_file(
width: u32,
height: u32,
frames: &[Vec<u8>],
framerate: i32,
out_path: &str,
) -> Result<usize, String> {
unsafe {
// 1. VAAPI device.
let mut hw_device: *mut ff::AVBufferRef = ptr::null_mut();
let node = CString::new("/dev/dri/renderD128").unwrap();
let r = ff::av_hwdevice_ctx_create(
&mut hw_device,
ff::AVHWDeviceType::AV_HWDEVICE_TYPE_VAAPI,
node.as_ptr(),
ptr::null_mut(),
0,
);
if r < 0 {
return Err(format!("av_hwdevice_ctx_create(VAAPI) failed: {r}"));
}
let cleanup_dev = |dev: *mut ff::AVBufferRef| {
let mut d = dev;
ff::av_buffer_unref(&mut d);
};
// 2. Encoder.
let name = CString::new("h264_vaapi").unwrap();
let codec = ff::avcodec_find_encoder_by_name(name.as_ptr());
if codec.is_null() {
cleanup_dev(hw_device);
return Err("encoder h264_vaapi not found in this FFmpeg build".into());
}
let enc = ff::avcodec_alloc_context3(codec);
if enc.is_null() {
cleanup_dev(hw_device);
return Err("avcodec_alloc_context3 failed".into());
}
(*enc).width = width as i32;
(*enc).height = height as i32;
(*enc).time_base = ff::AVRational { num: 1, den: framerate };
(*enc).framerate = ff::AVRational { num: framerate, den: 1 };
(*enc).pix_fmt = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
// 3. HW frames context (VAAPI surfaces with NV12 sw layout).
let frames_ref = ff::av_hwframe_ctx_alloc(hw_device);
if frames_ref.is_null() {
ff::avcodec_free_context(&mut (enc as *mut _));
cleanup_dev(hw_device);
return Err("av_hwframe_ctx_alloc failed".into());
}
{
let fctx = (*frames_ref).data as *mut ff::AVHWFramesContext;
(*fctx).format = ff::AVPixelFormat::AV_PIX_FMT_VAAPI;
(*fctx).sw_format = ff::AVPixelFormat::AV_PIX_FMT_NV12;
(*fctx).width = width as i32;
(*fctx).height = height as i32;
(*fctx).initial_pool_size = 8;
}
let r = ff::av_hwframe_ctx_init(frames_ref);
if r < 0 {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::avcodec_free_context(&mut (enc as *mut _));
cleanup_dev(hw_device);
return Err(format!("av_hwframe_ctx_init failed: {r}"));
}
(*enc).hw_frames_ctx = ff::av_buffer_ref(frames_ref);
// 4. Open.
let r = ff::avcodec_open2(enc, codec, ptr::null_mut());
if r < 0 {
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::avcodec_free_context(&mut (enc as *mut _));
cleanup_dev(hw_device);
return Err(format!("avcodec_open2(h264_vaapi) failed: {r}"));
}
let mut out: Vec<u8> = Vec::new();
let pkt = ff::av_packet_alloc();
let mut count = 0usize;
// Drain helper: pull packets and append to `out`.
let drain = |enc: *mut ff::AVCodecContext, out: &mut Vec<u8>, count: &mut usize| -> Result<(), String> {
loop {
let r = ff::avcodec_receive_packet(enc, pkt);
if r == averror(libc::EAGAIN) || r == ff::AVERROR_EOF {
break;
}
if r < 0 {
return Err(format!("avcodec_receive_packet failed: {r}"));
}
let data = std::slice::from_raw_parts((*pkt).data, (*pkt).size as usize);
out.extend_from_slice(data);
*count += 1;
ff::av_packet_unref(pkt);
}
Ok(())
};
let mut err: Option<String> = None;
for (i, nv12) in frames.iter().enumerate() {
// Software NV12 frame.
let sw = ff::av_frame_alloc();
(*sw).format = ff::AVPixelFormat::AV_PIX_FMT_NV12 as i32;
(*sw).width = width as i32;
(*sw).height = height as i32;
if ff::av_frame_get_buffer(sw, 0) < 0 {
err = Some("av_frame_get_buffer(sw) failed".into());
ff::av_frame_free(&mut (sw as *mut _));
break;
}
fill_nv12(sw, nv12, width, height);
// VAAPI surface frame + upload.
let hw = ff::av_frame_alloc();
if ff::av_hwframe_get_buffer(frames_ref, hw, 0) < 0 {
err = Some("av_hwframe_get_buffer failed".into());
ff::av_frame_free(&mut (sw as *mut _));
ff::av_frame_free(&mut (hw as *mut _));
break;
}
if ff::av_hwframe_transfer_data(hw, sw, 0) < 0 {
err = Some("av_hwframe_transfer_data failed".into());
ff::av_frame_free(&mut (sw as *mut _));
ff::av_frame_free(&mut (hw as *mut _));
break;
}
(*hw).pts = i as i64;
let r = ff::avcodec_send_frame(enc, hw);
ff::av_frame_free(&mut (sw as *mut _));
ff::av_frame_free(&mut (hw as *mut _));
if r < 0 {
err = Some(format!("avcodec_send_frame failed: {r}"));
break;
}
if let Err(e) = drain(enc, &mut out, &mut count) {
err = Some(e);
break;
}
}
// Flush.
if err.is_none() {
ff::avcodec_send_frame(enc, ptr::null_mut());
if let Err(e) = drain(enc, &mut out, &mut count) {
err = Some(e);
}
}
// Cleanup.
ff::av_packet_free(&mut (pkt as *mut _));
let mut fr = frames_ref;
ff::av_buffer_unref(&mut fr);
ff::avcodec_free_context(&mut (enc as *mut _));
cleanup_dev(hw_device);
if let Some(e) = err {
return Err(e);
}
std::fs::write(out_path, &out).map_err(|e| format!("write {out_path}: {e}"))?;
Ok(count)
}
}

View File

@ -0,0 +1,168 @@
//! Custom wgpu Vulkan device that additionally enables `VK_EXT_image_drm_format_modifier`
//! (plus the external-memory extensions wgpu-hal already turns on), so we can import a
//! tiled VAAPI NV12 DMA-BUF as a Vulkan image. wgpu's safe API can't add arbitrary device
//! extensions, so we build the `VkDevice` ourselves and wrap it via `device_from_raw`.
//!
//! All `unsafe` is contained here. Returns owned handles the caller must keep alive
//! together (instance → adapter → device/queue).
use ash::vk;
use std::ffi::CStr;
/// A wgpu device/queue backed by a hand-built Vulkan device with DMA-BUF import enabled.
pub struct DrmDevice {
// Order matters for drop; wgpu handles refcount internally but we keep these owned.
pub device: wgpu::Device,
pub queue: wgpu::Queue,
pub adapter: wgpu::Adapter,
pub instance: wgpu::Instance,
/// The raw VkDevice (for the ash image-import calls in `dmabuf.rs`).
pub raw_device: ash::Device,
pub raw_physical_device: vk::PhysicalDevice,
pub raw_instance: ash::Instance,
}
/// Create a headless DMA-BUF-import device (encoder/decoder), or `Err` if Vulkan/the extension
/// isn't available (caller falls back).
pub fn create() -> Result<DrmDevice, String> {
unsafe { create_inner(false) }
}
/// Like [`create`] but also enables `VK_KHR_swapchain` so the device can present to a window —
/// for use as the editor's **shared** wgpu device (eframe + compositor + decode + encode all on
/// one device, so hardware-decoded DMA-BUF textures are usable by the preview compositor).
pub fn create_windowed() -> Result<DrmDevice, String> {
unsafe { create_inner(true) }
}
unsafe fn create_inner(windowed: bool) -> Result<DrmDevice, String> {
use wgpu_hal::vulkan::Api as Vk;
// Bring the HAL Instance trait into scope for `init` / `enumerate_adapters`.
use wgpu_hal::Instance as _;
// 1. HAL instance.
let hal_instance = wgpu_hal::vulkan::Instance::init(&wgpu_hal::InstanceDescriptor {
name: "gpu-video-encoder",
flags: wgpu::InstanceFlags::empty(),
memory_budget_thresholds: Default::default(),
backend_options: Default::default(),
})
.map_err(|e| format!("vulkan instance init failed: {e:?}"))?;
let ash_instance = hal_instance.shared_instance().raw_instance().clone();
// 2. Pick an adapter (prefer the integrated/discrete GPU).
let mut exposed_adapters = hal_instance.enumerate_adapters(None);
if exposed_adapters.is_empty() {
return Err("no Vulkan adapters".into());
}
// Prefer a real GPU over CPU/llvmpipe.
exposed_adapters.sort_by_key(|a| match a.info.device_type {
wgpu::DeviceType::DiscreteGpu => 0,
wgpu::DeviceType::IntegratedGpu => 1,
_ => 2,
});
let exposed = exposed_adapters.into_iter().next().unwrap();
let phys = exposed.adapter.raw_physical_device();
// 3. Queue family with graphics + compute.
let qf_props = ash_instance.get_physical_device_queue_family_properties(phys);
let family_index = qf_props
.iter()
.position(|p| {
p.queue_flags
.contains(vk::QueueFlags::GRAPHICS | vk::QueueFlags::COMPUTE)
})
.ok_or("no graphics+compute queue family")? as u32;
// 4. Extensions: what wgpu-hal wants + DRM modifier import set.
let mut ext_names: Vec<&'static CStr> =
exposed.adapter.required_device_extensions(exposed.features);
// Only the genuine extensions; external_memory / bind_memory2 / ycbcr / format_list
// are core in Vulkan 1.1+ (this device is 1.3) so they need no enabling.
let mut extra: Vec<&'static CStr> = vec![
ash::ext::image_drm_format_modifier::NAME,
ash::khr::external_memory_fd::NAME,
ash::ext::external_memory_dma_buf::NAME,
ash::ext::queue_family_foreign::NAME,
];
// Presentation (windowed shared device only): the WSI surface instance extensions are already
// enabled by `Instance::init`; the device needs the swapchain extension to present.
if windowed {
extra.push(ash::khr::swapchain::NAME);
}
for e in extra {
if !ext_names.contains(&e) {
ext_names.push(e);
}
}
let ext_ptrs: Vec<*const i8> = ext_names.iter().map(|c| c.as_ptr()).collect();
// 5. Enable all supported physical-device features (so wgpu has what it needs) plus
// sampler YCbCr conversion (required for the NV12 multi-planar image).
let supported = ash_instance.get_physical_device_features(phys);
let mut ycbcr =
vk::PhysicalDeviceSamplerYcbcrConversionFeatures::default().sampler_ycbcr_conversion(true);
let priorities = [1.0f32];
let queue_info = vk::DeviceQueueCreateInfo::default()
.queue_family_index(family_index)
.queue_priorities(&priorities);
let queue_infos = [queue_info];
let create_info = vk::DeviceCreateInfo::default()
.queue_create_infos(&queue_infos)
.enabled_extension_names(&ext_ptrs)
.enabled_features(&supported)
.push_next(&mut ycbcr);
let ash_device = ash_instance
.create_device(phys, &create_info, None)
.map_err(|e| format!("vkCreateDevice failed: {e:?}"))?;
// 6. Wrap the raw device into a hal OpenDevice, then a wgpu device.
let open_device = exposed
.adapter
.device_from_raw(
ash_device.clone(),
None,
&ext_names,
exposed.features,
&wgpu::MemoryHints::default(),
family_index,
0,
)
.map_err(|e| format!("device_from_raw failed: {e:?}"))?;
let raw_physical_device = phys;
let wgpu_instance = wgpu::Instance::from_hal::<Vk>(hal_instance);
let wgpu_adapter = wgpu_instance.create_adapter_from_hal::<Vk>(exposed);
let (device, queue) = wgpu_adapter
.create_device_from_hal::<Vk>(
open_device,
&wgpu::DeviceDescriptor {
label: Some("drm-import-device"),
// R16/Rg16 plane textures for P010 (10-bit HDR) import need this; request it only
// when the adapter supports it (else 10-bit falls back to software decode).
required_features: wgpu_adapter.features()
& wgpu::Features::TEXTURE_FORMAT_16BIT_NORM,
// Vello's compute pipelines need more than downlevel limits (e.g.
// max_storage_buffers_per_shader_stage >= 5). This device only ever runs on a
// real VAAPI-capable GPU, so request the adapter's full limits.
required_limits: wgpu_adapter.limits(),
..Default::default()
},
)
.map_err(|e| format!("create_device_from_hal failed: {e:?}"))?;
Ok(DrmDevice {
device,
queue,
adapter: wgpu_adapter,
instance: wgpu_instance,
raw_device: ash_device,
raw_physical_device,
raw_instance: ash_instance,
})
}

View File

@ -0,0 +1,91 @@
//! Round-trip: encode solid frames with the zero-copy encoder, then hardware-decode them back
//! into a wgpu texture and read the Y plane. Verifies the VAAPI decode → DMA-BUF → wgpu import
//! path produces real pixels on the GPU. Skips when VAAPI is unavailable.
#![cfg(target_os = "linux")]
use gpu_video_encoder::decoder::VaapiDecoder;
use gpu_video_encoder::encoder::ZeroCopyEncoder;
#[test]
fn vaapi_decode_roundtrip() {
// 256-wide so the R8 Y readback row (256 B) is already 256-aligned.
let (w, h) = (256u32, 256u32);
let out = std::env::temp_dir().join("gpu_video_encoder_decode_rt.mp4");
let _ = std::fs::remove_file(&out);
// --- Encode 10 frames of solid mid-gray. Full range → Y == luma ≈ 128. ---
{
let mut enc = match ZeroCopyEncoder::new(w, h, 30, 4000, &out, true) {
Ok(e) => e,
Err(e) => {
eprintln!("[decode-rt] encode unavailable, skipping: {e}");
return;
}
};
let device = enc.device();
let src = device.create_texture(&wgpu::TextureDescriptor {
label: Some("gray"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
let gray = vec![128u8; (w * h * 4) as usize];
enc.queue().write_texture(
wgpu::TexelCopyTextureInfo { texture: &src, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
&gray,
wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w * 4), rows_per_image: Some(h) },
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
for _ in 0..10 {
enc.encode_rgba(&src).expect("encode_rgba");
}
enc.finish().expect("finish");
}
// --- Decode it back on the GPU. ---
let mut dec = match VaapiDecoder::new(&out) {
Ok(d) => d,
Err(e) => {
eprintln!("[decode-rt] decode unavailable, skipping: {e}");
return;
}
};
let frame = dec.next_frame().expect("next_frame").expect("expected at least one frame");
assert_eq!(frame.y().width(), w, "decoded Y width");
assert_eq!(frame.y().height(), h, "decoded Y height");
// Read back the Y plane (R8) and check it's ≈ the gray we encoded.
let device = dec.device();
let buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("y_readback"),
size: (w * h) as u64,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let mut cmd = device.create_command_encoder(&Default::default());
cmd.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo { texture: frame.y(), mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyBufferInfo {
buffer: &buf,
layout: wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w), rows_per_image: Some(h) },
},
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
dec.queue().submit(Some(cmd.finish()));
buf.slice(..).map_async(wgpu::MapMode::Read, |_| {});
let _ = device.poll(wgpu::PollType::wait_indefinitely());
let data = buf.slice(..).get_mapped_range();
let mean = data.iter().map(|&b| b as f64).sum::<f64>() / data.len() as f64;
eprintln!("[decode-rt] decoded {w}x{h}, mean Y = {mean:.1}");
assert!(
(mean - 128.0).abs() < 12.0,
"mean Y {mean} not ≈ 128 — decode produced wrong pixels"
);
eprintln!("[decode-rt] ✅ VAAPI decode → wgpu texture verified");
}

View File

@ -0,0 +1,42 @@
//! Step 1 of zero-copy: the custom Vulkan device with DMA-BUF import extensions builds
//! and can do a trivial GPU op. Skips (passes) when Vulkan is unavailable.
#![cfg(target_os = "linux")]
#[test]
fn drm_device_creates_and_works() {
let dev = match gpu_video_encoder::vk_device::create() {
Ok(d) => d,
Err(e) => {
eprintln!("[drm-device] unavailable, skipping: {e}");
return;
}
};
eprintln!("[drm-device] created custom Vulkan device OK");
// Trivial sanity op: write+read a small buffer, proving the wrapped device is usable.
let data: Vec<u8> = (0..256u32).map(|i| i as u8).collect();
let src = wgpu::util::DeviceExt::create_buffer_init(
&dev.device,
&wgpu::util::BufferInitDescriptor {
label: Some("src"),
contents: &data,
usage: wgpu::BufferUsages::COPY_SRC,
},
);
let dst = dev.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("dst"),
size: 256,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let mut enc = dev.device.create_command_encoder(&Default::default());
enc.copy_buffer_to_buffer(&src, 0, &dst, 0, 256);
dev.queue.submit(Some(enc.finish()));
let slice = dst.slice(..);
slice.map_async(wgpu::MapMode::Read, |_| {});
let _ = dev.device.poll(wgpu::PollType::wait_indefinitely());
let got = slice.get_mapped_range().to_vec();
assert_eq!(got, data, "round-trip through custom device failed");
eprintln!("[drm-device] buffer round-trip OK on custom device");
}

View File

@ -0,0 +1,117 @@
//! Real-hardware test: run the RGBA→NV12 compute on the GPU and check it byte-matches
//! the CPU reference. Skips (passes) if no GPU adapter is available.
use gpu_video_encoder::nv12::{cpu_reference, nv12_len, Nv12Converter};
fn device_queue() -> Option<(wgpu::Device, wgpu::Queue)> {
let instance = wgpu::Instance::new(&wgpu::InstanceDescriptor {
backends: wgpu::Backends::VULKAN | wgpu::Backends::GL,
..Default::default()
});
let adapter = pollster::block_on(instance.request_adapter(&wgpu::RequestAdapterOptions {
power_preference: wgpu::PowerPreference::HighPerformance,
force_fallback_adapter: false,
compatible_surface: None,
}))
.ok()?;
pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
label: Some("nv12-test"),
required_features: wgpu::Features::empty(),
required_limits: wgpu::Limits::downlevel_defaults(),
..Default::default()
}))
.ok()
}
/// A deterministic, varied RGBA pattern so luma and 2x2 chroma subsampling are exercised.
fn pattern(w: u32, h: u32) -> Vec<u8> {
let mut v = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
v.push(((x * 37 + y * 11) % 256) as u8); // R
v.push(((x * 5 + y * 53) % 256) as u8); // G
v.push(((x * 97 + y * 17) % 256) as u8); // B
v.push(255);
}
}
v
}
#[test]
fn gpu_nv12_matches_cpu_reference() {
let Some((device, queue)) = device_queue() else {
eprintln!("[gpu_nv12] no GPU adapter; skipping");
return;
};
let (w, h) = (64u32, 16u32);
let rgba = pattern(w, h);
// Source RGBA texture.
let tex = device.create_texture(&wgpu::TextureDescriptor {
label: Some("src_rgba"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
queue.write_texture(
wgpu::TexelCopyTextureInfo {
texture: &tex,
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
&rgba,
wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w * 4), rows_per_image: Some(h) },
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
let view = tex.create_view(&Default::default());
let len = nv12_len(w, h) as u64;
let out = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("nv12_out"),
size: len,
usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC,
mapped_at_creation: false,
});
let staging = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("nv12_staging"),
size: len,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let conv = Nv12Converter::new(&device);
let mut enc = device.create_command_encoder(&Default::default());
conv.convert(&device, &mut enc, &view, &out, w, h);
enc.copy_buffer_to_buffer(&out, 0, &staging, 0, len);
queue.submit(Some(enc.finish()));
let slice = staging.slice(..);
slice.map_async(wgpu::MapMode::Read, |_| {});
let _ = device.poll(wgpu::PollType::wait_indefinitely());
let gpu = slice.get_mapped_range().to_vec();
let cpu = cpu_reference(&rgba, w, h);
assert_eq!(gpu.len(), cpu.len(), "length mismatch");
// Allow ±1 for rounding differences between GPU and CPU float paths.
let mut max_diff = 0i32;
let mut nbad = 0;
for (i, (g, c)) in gpu.iter().zip(cpu.iter()).enumerate() {
let d = (*g as i32 - *c as i32).abs();
max_diff = max_diff.max(d);
if d > 1 {
nbad += 1;
if nbad <= 8 {
eprintln!("[gpu_nv12] byte {i}: gpu={g} cpu={c} (diff {d})");
}
}
}
eprintln!("[gpu_nv12] {}x{} NV12, max byte diff = {max_diff}", w, h);
assert_eq!(nbad, 0, "{nbad} bytes differ from CPU reference by >1");
}

View File

@ -0,0 +1,66 @@
//! Level-1 spike: prove `h264_vaapi` encodes NV12 in this environment. Skips (passes)
//! when VAAPI isn't available so it's a no-op on CI/macOS/Windows.
#![cfg(target_os = "linux")]
use gpu_video_encoder::nv12::{cpu_reference, nv12_len};
use gpu_video_encoder::vaapi::encode_nv12_to_file;
/// A moving-gradient RGBA pattern → NV12 via the CPU reference, so we feed valid frames.
fn nv12_frames(w: u32, h: u32, n: usize) -> Vec<Vec<u8>> {
(0..n)
.map(|f| {
let mut rgba = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
rgba.push(((x + f as u32 * 4) % 256) as u8);
rgba.push(((y + f as u32 * 2) % 256) as u8);
rgba.push(((x + y) % 256) as u8);
rgba.push(255);
}
}
let v = cpu_reference(&rgba, w, h);
assert_eq!(v.len(), nv12_len(w, h));
v
})
.collect()
}
#[test]
fn vaapi_surface_drm_layout() {
match gpu_video_encoder::vaapi::probe_surface_drm(1920, 1088) {
Ok(s) => eprintln!("[vaapi-drm]\n{s}"),
Err(e) => eprintln!("[vaapi-drm] unavailable, skipping: {e}"),
}
}
#[test]
fn vaapi_h264_encode_smoke() {
let (w, h) = (320u32, 240u32);
let frames = nv12_frames(w, h, 30);
let out = std::env::temp_dir().join("gpu_video_encoder_vaapi_smoke.h264");
let out_str = out.to_str().unwrap();
match encode_nv12_to_file(w, h, &frames, 30, out_str) {
Ok(packets) => {
let meta = std::fs::metadata(&out).expect("output file missing");
eprintln!(
"[vaapi] encoded {} packets, {} bytes -> {}",
packets,
meta.len(),
out_str
);
assert!(packets > 0, "no packets produced");
assert!(meta.len() > 0, "empty output file");
// First frame should be an IDR; Annex-B starts with a start code.
let head = std::fs::read(&out).unwrap();
assert!(
head.starts_with(&[0, 0, 0, 1]) || head.starts_with(&[0, 0, 1]),
"output is not Annex-B H.264 (no start code)"
);
}
Err(e) => {
eprintln!("[vaapi] unavailable, skipping: {e}");
}
}
}

View File

@ -0,0 +1,169 @@
//! End-to-end zero-copy proof: import a VAAPI NV12 surface as wgpu textures, render
//! known values into them via Vulkan, read the surface back, and verify the bytes —
//! proving the GPU wrote straight into the encoder's surface with no CPU upload.
#![cfg(target_os = "linux")]
use gpu_video_encoder::{dmabuf, nv12, render_nv12, vaapi, vk_device};
/// Render a real RGBA frame into the VAAPI surface (zero-copy) and verify the surface's
/// NV12 matches the CPU reference for that frame.
#[test]
fn zerocopy_real_frame_render() {
let drm = match vk_device::create() {
Ok(d) => d,
Err(e) => {
eprintln!("[zerocopy-real] no Vulkan, skipping: {e}");
return;
}
};
let (w, h) = (640u32, 480u32);
let surf = match vaapi::MappedSurface::alloc(w, h) {
Ok(s) => s,
Err(e) => {
eprintln!("[zerocopy-real] no VAAPI, skipping: {e}");
return;
}
};
let imported = dmabuf::import(&drm, &surf).expect("import");
// A varied RGBA pattern.
let mut rgba = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
rgba.push(((x * 3 + y) % 256) as u8);
rgba.push(((x + y * 2) % 256) as u8);
rgba.push(((x * 2 + y * 3) % 256) as u8);
rgba.push(255);
}
}
let src = drm.device.create_texture(&wgpu::TextureDescriptor {
label: Some("rgba_src"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
drm.queue.write_texture(
wgpu::TexelCopyTextureInfo { texture: &src, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
&rgba,
wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w * 4), rows_per_image: Some(h) },
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
let conv = render_nv12::Rgba2Nv12::new(&drm.device, true);
let src_view = src.create_view(&Default::default());
let y_view = imported.y().create_view(&Default::default());
let uv_view = imported.uv().create_view(&Default::default());
let mut enc = drm.device.create_command_encoder(&Default::default());
conv.convert(&drm.device, &mut enc, &src_view, &y_view, &uv_view);
drm.queue.submit(Some(enc.finish()));
let _ = drm.device.poll(wgpu::PollType::wait_indefinitely());
let got = surf.readback_nv12().expect("readback");
let want = nv12::cpu_reference(&rgba, w, h);
assert_eq!(got.len(), want.len());
let mut max_diff = 0i32;
let mut nbad = 0;
for (g, c) in got.iter().zip(want.iter()) {
let d = (*g as i32 - *c as i32).abs();
max_diff = max_diff.max(d);
if d > 2 {
nbad += 1;
}
}
eprintln!("[zerocopy-real] {}x{} real-frame render, max diff={max_diff}, bad={nbad}/{}", w, h, got.len());
assert!(nbad * 100 < got.len(), "too many bytes differ from CPU NV12 reference");
eprintln!("[zerocopy-real] ✅ real RGBA frame rendered into VAAPI surface, NV12 matches reference");
}
#[test]
fn zerocopy_render_into_vaapi_surface() {
let drm = match vk_device::create() {
Ok(d) => d,
Err(e) => {
eprintln!("[zerocopy] no Vulkan device, skipping: {e}");
return;
}
};
let surf = match vaapi::MappedSurface::alloc(640, 480) {
Ok(s) => s,
Err(e) => {
eprintln!("[zerocopy] no VAAPI surface, skipping: {e}");
return;
}
};
eprintln!(
"[zerocopy] surface: modifier=0x{:016x} y(off={},pitch={}) uv(off={},pitch={}) size={}",
surf.modifier, surf.y_offset, surf.y_pitch, surf.uv_offset, surf.uv_pitch, surf.size
);
let imported = match dmabuf::import(&drm, &surf) {
Ok(i) => i,
Err(e) => panic!("dma-buf import failed: {e}"),
};
eprintln!("[zerocopy] imported surface as wgpu Y(R8) + UV(RG8) textures");
// Render known constants via clear: Y=0.5(->128), U=0.25(->64), V=0.75(->191).
let y_view = imported.y().create_view(&Default::default());
let uv_view = imported.uv().create_view(&Default::default());
let mut enc = drm.device.create_command_encoder(&Default::default());
{
enc.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("clear-y"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &y_view,
resolve_target: None,
depth_slice: None,
ops: wgpu::Operations {
load: wgpu::LoadOp::Clear(wgpu::Color { r: 0.5, g: 0.0, b: 0.0, a: 0.0 }),
store: wgpu::StoreOp::Store,
},
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
});
enc.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("clear-uv"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &uv_view,
resolve_target: None,
depth_slice: None,
ops: wgpu::Operations {
load: wgpu::LoadOp::Clear(wgpu::Color { r: 0.25, g: 0.75, b: 0.0, a: 0.0 }),
store: wgpu::StoreOp::Store,
},
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
});
}
drm.queue.submit(Some(enc.finish()));
let _ = drm.device.poll(wgpu::PollType::wait_indefinitely());
// Read the VAAPI surface back and check what the GPU wrote.
let nv12 = surf.readback_nv12().expect("readback");
let (w, h) = (640usize, 480usize);
let y_plane = &nv12[..w * h];
let uv_plane = &nv12[w * h..];
let near = |v: u8, t: i32| (v as i32 - t).abs() <= 3;
let y_ok = y_plane.iter().filter(|&&v| near(v, 128)).count();
let u_ok = uv_plane.iter().step_by(2).filter(|&&v| near(v, 64)).count();
let v_ok = uv_plane.iter().skip(1).step_by(2).filter(|&&v| near(v, 191)).count();
eprintln!(
"[zerocopy] Y~128: {}/{}, U~64: {}/{}, V~191: {}/{}",
y_ok, w * h, u_ok, uv_plane.len() / 2, v_ok, uv_plane.len() / 2
);
let frac = |ok: usize, n: usize| ok as f64 / n as f64;
assert!(frac(y_ok, w * h) > 0.98, "Y plane not the rendered value (sample {:?})", &y_plane[..8]);
assert!(frac(u_ok, uv_plane.len() / 2) > 0.98, "U not rendered value");
assert!(frac(v_ok, uv_plane.len() / 2) > 0.98, "V not rendered value");
eprintln!("[zerocopy] ✅ GPU rendered straight into the VAAPI surface (verified via readback)");
}

View File

@ -0,0 +1,76 @@
//! Capstone: encode RGBA frames fully zero-copy (GPU render → VAAPI surface → h264_vaapi)
//! and verify the output is real H.264. Skips when VAAPI is unavailable.
#![cfg(target_os = "linux")]
use gpu_video_encoder::encoder::ZeroCopyEncoder;
#[test]
fn zerocopy_encode_h264() {
let (w, h) = (640u32, 480u32);
let out = std::env::temp_dir().join("gpu_video_encoder_zerocopy.mp4");
let _ = std::fs::remove_file(&out);
let mut enc = match ZeroCopyEncoder::new(w, h, 30, 4000, &out, false) {
Ok(e) => e,
Err(e) => {
eprintln!("[zc-encode] unavailable, skipping: {e}");
return;
}
};
// Build one reusable RGBA source texture; update it per frame with a moving pattern.
let device = enc.device();
let src = device.create_texture(&wgpu::TextureDescriptor {
label: Some("rgba"),
size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
let n = 30;
for f in 0..n {
let mut rgba = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
rgba.push(((x + f * 8) % 256) as u8);
rgba.push(((y + f * 4) % 256) as u8);
rgba.push(((x + y) % 256) as u8);
rgba.push(255);
}
}
enc.queue().write_texture(
wgpu::TexelCopyTextureInfo { texture: &src, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
&rgba,
wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w * 4), rows_per_image: Some(h) },
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
enc.encode_rgba(&src).expect("encode_rgba");
}
enc.finish().expect("finish");
let meta = std::fs::metadata(&out).expect("output .mp4 missing");
eprintln!("[zc-encode] {} frames -> {} bytes mp4 at {}", n, meta.len(), out.display());
assert!(meta.len() > 1000, "implausibly small output");
// ffprobe-verify the container: H.264 stream, right dims, ~n frames.
let o = std::process::Command::new("ffprobe")
.args([
"-hide_banner", "-v", "error", "-count_frames",
"-show_entries", "stream=codec_name,width,height,nb_read_frames",
"-of", "default=noprint_wrappers=1",
])
.arg(&out)
.output()
.expect("run ffprobe");
let s = String::from_utf8_lossy(&o.stdout);
eprintln!("[zc-encode] ffprobe:\n{s}");
assert!(s.contains("codec_name=h264"), "ffprobe didn't see H.264");
assert!(s.contains(&format!("width={w}")), "wrong width");
assert!(s.contains(&format!("height={h}")), "wrong height");
assert!(s.contains(&format!("nb_read_frames={n}")), "expected {n} frames");
eprintln!("[zc-encode] ✅ zero-copy H.264 mp4 encode verified");
}

View File

@ -18,6 +18,9 @@ bytemuck = { version = "1.14", features = ["derive"] }
kurbo = { workspace = true }
vello = { workspace = true }
# Text layout/shaping (text layers)
parley = { workspace = true }
# Image decoding for image fills
image = { workspace = true }
@ -29,6 +32,8 @@ daw-backend = { path = "../../daw-backend" }
# Video decoding
ffmpeg-next = "8.0"
# AVIO-over-Read+Seek shim: decode packed video by streaming from the SQLite blob.
ffmpeg-blob-io = { path = "../../ffmpeg-blob-io" }
lru = "0.12"
# File I/O
@ -73,3 +78,6 @@ windows-sys = { version = "0.60", features = [
version = "0.11"
[dev-dependencies]
# For the packed-video streaming integration test (blob -> AVIO -> ffmpeg Input).
ffmpeg-blob-io = { path = "../../ffmpeg-blob-io" }
ffmpeg-next = "8.0"

View File

@ -0,0 +1,14 @@
Bundled fonts for Lightningbeam text layers
===========================================
These are the Liberation fonts, used as the built-in serif / sans-serif / monospaced
families so text renders deterministically without depending on system fonts:
LiberationSans-Regular.ttf
LiberationSerif-Regular.ttf
LiberationMono-Regular.ttf
License: SIL Open Font License, Version 1.1.
Copyright (c) Red Hat, Inc., with Reserved Font Name "Liberation".
Full license text: https://openfontlicense.org/ (also distributed with the Liberation
fonts package as its OFL.txt / LICENSE).

View File

@ -104,7 +104,7 @@ impl Action for AddClipInstanceAction {
AnyLayer::Group(_) => {
return Err("Cannot add clip instances directly to group layers".to_string());
}
AnyLayer::Raster(_) => {
AnyLayer::Raster(_) | AnyLayer::Text(_) => {
return Err("Cannot add clip instances directly to group layers".to_string());
}
}
@ -145,8 +145,8 @@ impl Action for AddClipInstanceAction {
AnyLayer::Group(_) => {
// Group layers don't have clip instances, nothing to rollback
}
AnyLayer::Raster(_) => {
// Raster layers don't have clip instances, nothing to rollback
AnyLayer::Raster(_) | AnyLayer::Text(_) => {
// Raster/text layers don't have clip instances, nothing to rollback
}
}
self.executed = false;

View File

@ -138,6 +138,7 @@ impl Action for AddLayerAction {
AnyLayer::Effect(_) => "Add effect layer",
AnyLayer::Group(_) => "Add group layer",
AnyLayer::Raster(_) => "Add raster layer",
AnyLayer::Text(_) => "Add text layer",
}
.to_string()
}

View File

@ -0,0 +1,133 @@
//! Create-text-clip action
//!
//! Used by the text tool when the active layer is a **vector** layer: it creates a
//! VectorClip containing a single text layer, registers it, and places a clip
//! instance in the parent vector layer. The editor then enters that clip so the
//! text is directly editable. All of this is one undoable step so undo never
//! leaves an orphan clip.
use crate::action::Action;
use crate::clip::{ClipInstance, VectorClip};
use crate::document::Document;
use crate::layer::AnyLayer;
use crate::text_layer::TextLayer;
use uuid::Uuid;
/// Minimum clip duration (seconds) when the document has no duration set yet.
const FALLBACK_DURATION: f64 = 10.0;
pub struct CreateTextClipAction {
/// The vector layer to place the clip instance in.
parent_layer_id: Uuid,
/// The text layer to embed (its id is stable across redo).
text_layer: TextLayer,
/// Instance position in the parent layer's space.
position: (f64, f64),
// Assigned on first execute; reused on redo so ids are stable.
clip_id: Option<Uuid>,
instance_id: Option<Uuid>,
executed: bool,
}
impl CreateTextClipAction {
pub fn new(parent_layer_id: Uuid, text_layer: TextLayer, position: (f64, f64)) -> Self {
Self {
parent_layer_id,
text_layer,
position,
clip_id: None,
instance_id: None,
executed: false,
}
}
/// Preset the clip + instance ids (so the caller knows them up-front for
/// entering the clip immediately after execute). Ids stay stable across redo.
pub fn with_ids(mut self, clip_id: Uuid, instance_id: Uuid) -> Self {
self.clip_id = Some(clip_id);
self.instance_id = Some(instance_id);
self
}
/// The vector clip id (after execute).
pub fn clip_id(&self) -> Option<Uuid> {
self.clip_id
}
/// The clip instance id placed in the parent layer (after execute).
pub fn instance_id(&self) -> Option<Uuid> {
self.instance_id
}
/// The embedded text layer's id.
pub fn text_layer_id(&self) -> Uuid {
self.text_layer.layer.id
}
}
impl Action for CreateTextClipAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let clip_id = self.clip_id.unwrap_or_else(Uuid::new_v4);
self.clip_id = Some(clip_id);
let instance_id = self.instance_id.unwrap_or_else(Uuid::new_v4);
self.instance_id = Some(instance_id);
let duration = if document.duration > 0.0 { document.duration } else { FALLBACK_DURATION };
// Build the clip with the text layer as its single root layer.
let mut clip = VectorClip::with_id(
clip_id,
"Text",
self.text_layer.box_width.max(1.0),
self.text_layer.box_height.max(1.0),
duration,
);
// A movie clip (not keyframe-gated) so the text persists across the timeline.
clip.is_group = false;
clip.layers.add_root(AnyLayer::Text(self.text_layer.clone()));
// Registers the clip and its layers in layer_to_clip_map for O(1) lookup.
document.add_vector_clip(clip);
// Place an instance of the clip in the parent vector layer.
let layer = document
.get_layer_mut(&self.parent_layer_id)
.ok_or_else(|| format!("Parent layer {} not found", self.parent_layer_id))?;
let AnyLayer::Vector(vector_layer) = layer else {
return Err("Text clip can only be created inside a vector layer".to_string());
};
let mut instance = ClipInstance::with_id(instance_id, clip_id);
instance.transform.x = self.position.0;
instance.transform.y = self.position.1;
vector_layer.clip_instances.push(instance);
self.executed = true;
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
if !self.executed {
return Ok(());
}
// Remove the clip instance from the parent layer.
if let (Some(instance_id), Some(AnyLayer::Vector(vector_layer))) =
(self.instance_id, document.get_layer_mut(&self.parent_layer_id))
{
vector_layer.clip_instances.retain(|ci| ci.id != instance_id);
}
// Remove the clip and its layer_to_clip_map registrations.
if let Some(clip_id) = self.clip_id {
if let Some(clip) = document.vector_clips.remove(&clip_id) {
for node in &clip.layers.roots {
document.layer_to_clip_map.remove(&node.data.id());
}
}
}
self.executed = false;
Ok(())
}
fn description(&self) -> String {
"Add text".to_string()
}
}

View File

@ -37,6 +37,7 @@ impl Action for LoopClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
for (instance_id, _old_dur, new_dur, _old_lb, new_lb) in loops {
@ -61,6 +62,7 @@ impl Action for LoopClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
for (instance_id, old_dur, _new_dur, old_lb, _new_lb) in loops {

View File

@ -11,6 +11,7 @@ pub mod add_layer;
pub mod add_shape;
pub mod modify_shape_path;
pub mod move_clip_instances;
pub mod reorder_clip_instances;
pub mod paint_bucket;
pub mod remove_effect;
pub mod set_document_properties;
@ -40,9 +41,13 @@ pub mod raster_diff;
pub mod raster_stroke;
pub mod raster_fill;
pub mod add_raster_keyframe;
pub mod resize_raster_layer;
pub mod move_layer;
pub mod set_fill_paint;
pub mod set_image_fill;
pub mod create_text_clip;
pub mod set_text_content;
pub mod resize_text_box;
pub use add_clip_instance::AddClipInstanceAction;
pub use add_effect::AddEffectAction;
@ -50,6 +55,7 @@ pub use add_layer::AddLayerAction;
pub use add_shape::AddShapeAction;
pub use modify_shape_path::ModifyGraphAction;
pub use move_clip_instances::MoveClipInstancesAction;
pub use reorder_clip_instances::ReorderClipInstancesAction;
pub use paint_bucket::PaintBucketAction;
pub use remove_effect::RemoveEffectAction;
pub use set_document_properties::SetDocumentPropertiesAction;
@ -77,8 +83,12 @@ pub use group_layers::GroupLayersAction;
pub use raster_stroke::RasterStrokeAction;
pub use raster_fill::RasterFillAction;
pub use add_raster_keyframe::AddRasterKeyframeAction;
pub use resize_raster_layer::ResizeRasterLayerAction;
pub use move_layer::MoveLayerAction;
pub use set_fill_paint::SetFillPaintAction;
pub use set_image_fill::SetImageFillAction;
pub use change_bpm::ChangeBpmAction;
pub use change_fps::ChangeFpsAction;
pub use create_text_clip::CreateTextClipAction;
pub use set_text_content::SetTextContentAction;
pub use resize_text_box::ResizeTextBoxAction;

View File

@ -58,6 +58,7 @@ impl Action for MoveClipInstancesAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
if let Some(instance) = clip_instances.iter().find(|ci| ci.id == *member_instance_id) {
@ -97,6 +98,7 @@ impl Action for MoveClipInstancesAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
let group: Vec<(Uuid, f64, f64)> = moves.iter().filter_map(|(id, old_start, _)| {
@ -132,6 +134,7 @@ impl Action for MoveClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
// Update timeline_start for each clip instance
@ -159,6 +162,7 @@ impl Action for MoveClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
// Restore original timeline_start for each clip instance

View File

@ -98,11 +98,16 @@ impl RasterDiff {
self.before_region.len() + self.after_region.len()
}
/// Restore the pre-edit pixels into `raw` (undo / first-execute rollback).
pub fn apply_before(&self, raw: &mut Vec<u8>) {
/// Restore the pre-edit pixels into `raw` (undo / first-execute rollback). `cur_w`/`cur_h`
/// are the keyframe's current dimensions; if they differ from the diff's captured size the
/// diff predates a resize and is skipped rather than applied at mismatched dims.
pub fn apply_before(&self, raw: &mut Vec<u8>, cur_w: u32, cur_h: u32) {
if self.bbox.is_none() {
return; // no change
}
if cur_w != self.full_width || cur_h != self.full_height {
return; // diff predates a keyframe resize
}
if self.before_blank {
// The frame was blank before this edit (it was the first stroke); undoing
// it returns to blank regardless of the current buffer.
@ -112,11 +117,16 @@ impl RasterDiff {
self.stamp_resident(&self.before_region, raw);
}
/// Apply the post-edit pixels into `raw` (commit / redo).
pub fn apply_after(&self, raw: &mut Vec<u8>) {
/// Apply the post-edit pixels into `raw` (commit / redo). `cur_w`/`cur_h` are the keyframe's
/// current dimensions; a mismatch with the captured size means the diff predates a resize, so
/// it's skipped rather than rebuilding the buffer at stale dimensions.
pub fn apply_after(&self, raw: &mut Vec<u8>, cur_w: u32, cur_h: u32) {
if self.bbox.is_none() {
return; // no change
}
if cur_w != self.full_width || cur_h != self.full_height {
return; // diff predates a keyframe resize
}
if self.before_blank {
// Base was blank: build a full transparent buffer then stamp the bbox. The
// commit/redo path frequently starts from empty `raw_pixels` here.
@ -175,9 +185,9 @@ mod tests {
assert_eq!(diff.bbox, Some((3, 2, 2, 2)));
let mut buf = after.clone();
diff.apply_before(&mut buf);
diff.apply_before(&mut buf, w, h);
assert_eq!(buf, before, "undo must reproduce the pre-edit buffer exactly");
diff.apply_after(&mut buf);
diff.apply_after(&mut buf, w, h);
assert_eq!(buf, after, "redo must reproduce the post-edit buffer exactly");
}
@ -195,15 +205,15 @@ mod tests {
// First execute / redo from EMPTY raw_pixels (the real commit path): builds
// the full buffer from transparent + the stroke.
let mut buf: Vec<u8> = Vec::new();
diff.apply_after(&mut buf);
diff.apply_after(&mut buf, w, h);
assert_eq!(buf, after, "commit/redo must build the frame from a blank base");
// Undo the first stroke → back to blank (empty).
diff.apply_before(&mut buf);
diff.apply_before(&mut buf, w, h);
assert!(buf.is_empty(), "undoing the first stroke restores the blank keyframe");
// Redo again from the now-empty buffer.
diff.apply_after(&mut buf);
diff.apply_after(&mut buf, w, h);
assert_eq!(buf, after);
}
@ -215,7 +225,7 @@ mod tests {
assert_eq!(diff.bbox, None);
assert_eq!(diff.byte_size(), 0);
let mut b = buf.clone();
diff.apply_before(&mut b);
diff.apply_before(&mut b, w, h);
assert_eq!(b, buf);
}
@ -226,7 +236,7 @@ mod tests {
let after = solid(w, h, [1, 2, 3, 255]);
let diff = RasterDiff::compute(&before, &after, w, h);
let mut empty: Vec<u8> = Vec::new();
diff.apply_before(&mut empty); // base not resident
diff.apply_before(&mut empty, w, h); // base not resident
assert!(empty.is_empty(), "must not resize/corrupt a non-resident base");
}
}

View File

@ -53,7 +53,7 @@ impl Action for RasterFillAction {
if let Some(full) = self.full_after.take() {
kf.raw_pixels = full;
} else {
self.diff.apply_after(&mut kf.raw_pixels);
self.diff.apply_after(&mut kf.raw_pixels, kf.width, kf.height);
}
kf.texture_dirty = true;
kf.dirty = true;
@ -71,7 +71,7 @@ impl Action for RasterFillAction {
let kf = raster
.keyframe_at_mut(self.time)
.ok_or_else(|| format!("No raster keyframe at/before t={}", self.time))?;
self.diff.apply_before(&mut kf.raw_pixels);
self.diff.apply_before(&mut kf.raw_pixels, kf.width, kf.height);
kf.texture_dirty = true;
kf.dirty = true;
Ok(())

View File

@ -61,7 +61,7 @@ impl Action for RasterStrokeAction {
kf.raw_pixels = full;
} else {
// Redo: replay via the diff onto the (resident) base.
self.diff.apply_after(&mut kf.raw_pixels);
self.diff.apply_after(&mut kf.raw_pixels, kf.width, kf.height);
}
kf.texture_dirty = true;
kf.dirty = true;
@ -70,7 +70,7 @@ impl Action for RasterStrokeAction {
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let kf = get_keyframe_mut(document, &self.layer_id, self.time, self.width, self.height)?;
self.diff.apply_before(&mut kf.raw_pixels);
self.diff.apply_before(&mut kf.raw_pixels, kf.width, kf.height);
kf.texture_dirty = true;
kf.dirty = true;
Ok(())

View File

@ -46,6 +46,7 @@ impl Action for RemoveClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
// Find and remove the instance, saving it for rollback
@ -72,6 +73,7 @@ impl Action for RemoveClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
clip_instances.push(instance);

View File

@ -0,0 +1,78 @@
//! Reorder clip instances within a layer's stacking order (Send to Back / Bring to Front).
//!
//! A layer's `clip_instances` Vec order *is* the stacking order — the last element renders on top
//! (hit-testing walks it in reverse). Geometry renders underneath and is unaffected. This action
//! moves the selected instances to the front (end) or back (start) of that Vec.
use crate::action::Action;
use crate::clip::ClipInstance;
use crate::document::Document;
use crate::layer::AnyLayer;
use uuid::Uuid;
pub struct ReorderClipInstancesAction {
layer_id: Uuid,
instance_ids: Vec<Uuid>,
/// `true` = bring to front (top), `false` = send to back (bottom).
to_front: bool,
/// Full instance order captured on `execute`, for `rollback`.
old_order: Option<Vec<Uuid>>,
}
impl ReorderClipInstancesAction {
pub fn new(layer_id: Uuid, instance_ids: Vec<Uuid>, to_front: bool) -> Self {
Self { layer_id, instance_ids, to_front, old_order: None }
}
}
/// The clip-instance stack for a layer, if it has one (Group/Raster/Text don't).
fn clip_instances_mut<'a>(document: &'a mut Document, layer_id: &Uuid) -> Option<&'a mut Vec<ClipInstance>> {
match document.get_layer_mut(layer_id)? {
AnyLayer::Vector(l) => Some(&mut l.clip_instances),
AnyLayer::Audio(l) => Some(&mut l.clip_instances),
AnyLayer::Video(l) => Some(&mut l.clip_instances),
AnyLayer::Effect(l) => Some(&mut l.clip_instances),
_ => None,
}
}
impl Action for ReorderClipInstancesAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let instances = clip_instances_mut(document, &self.layer_id)
.ok_or_else(|| "Layer has no clip-instance stack".to_string())?;
self.old_order = Some(instances.iter().map(|c| c.id).collect());
let mut selected = Vec::new();
let mut rest = Vec::new();
for ci in instances.drain(..) {
if self.instance_ids.contains(&ci.id) {
selected.push(ci);
} else {
rest.push(ci);
}
}
if self.to_front {
rest.extend(selected); // selected last → rendered on top
*instances = rest;
} else {
selected.extend(rest); // selected first → rendered at the bottom (of the stack)
*instances = selected;
}
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let Some(order) = self.old_order.clone() else {
return Ok(());
};
let instances = clip_instances_mut(document, &self.layer_id)
.ok_or_else(|| "Layer has no clip-instance stack".to_string())?;
let rank = |id: &Uuid| order.iter().position(|o| o == id).unwrap_or(usize::MAX);
instances.sort_by(|a, b| rank(&a.id).cmp(&rank(&b.id)));
Ok(())
}
fn description(&self) -> String {
if self.to_front { "Bring to Front".to_string() } else { "Send to Back".to_string() }
}
}

View File

@ -0,0 +1,90 @@
//! Resize every keyframe of a raster layer to a new canvas size (Scale or Canvas mode), with undo.
//!
//! Used by the info panel's "Layer to document size" button. Pixels must be resident (the editor
//! faults them in first) so the resample/copy is exact and a later page-in won't mismatch the size.
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use crate::raster_layer::RasterResizeMode;
use crate::raster_store::RasterStore;
use uuid::Uuid;
/// Per-keyframe state captured for undo.
struct OldKeyframe {
id: Uuid,
width: u32,
height: u32,
pixels: Vec<u8>,
}
pub struct ResizeRasterLayerAction {
layer_id: Uuid,
new_w: u32,
new_h: u32,
mode: RasterResizeMode,
/// Read-only page-in for keyframes whose pixels aren't resident. (No incremental write exists, so
/// resized keyframes stay resident + dirty and persist on the next full save.)
store: RasterStore,
/// Captured on first execute for rollback.
old: Option<Vec<OldKeyframe>>,
}
impl ResizeRasterLayerAction {
pub fn new(layer_id: Uuid, new_w: u32, new_h: u32, mode: RasterResizeMode, store: RasterStore) -> Self {
Self { layer_id, new_w, new_h, mode, store, old: None }
}
}
impl Action for ResizeRasterLayerAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let Some(AnyLayer::Raster(rl)) = document.get_layer_mut(&self.layer_id) else {
return Err("ResizeRasterLayerAction: layer is not a raster layer".into());
};
let capture = self.old.is_none();
let mut old = Vec::new();
for kf in rl.keyframes.iter_mut() {
// Page the keyframe in one at a time so we never hold the whole layer in memory at once.
if kf.raw_pixels.is_empty() {
if let Some(px) = self.store.load_pixels(kf.id) {
kf.raw_pixels = px;
kf.needs_fault_in = false;
}
}
if capture {
old.push(OldKeyframe { id: kf.id, width: kf.width, height: kf.height, pixels: kf.raw_pixels.clone() });
}
kf.resize_to(self.new_w, self.new_h, self.mode);
}
if capture {
self.old = Some(old);
}
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let Some(AnyLayer::Raster(rl)) = document.get_layer_mut(&self.layer_id) else {
return Err("ResizeRasterLayerAction: layer is not a raster layer".into());
};
if let Some(old) = &self.old {
for o in old {
if let Some(kf) = rl.keyframes.iter_mut().find(|kf| kf.id == o.id) {
kf.width = o.width;
kf.height = o.height;
kf.raw_pixels = o.pixels.clone();
kf.proxy = None;
kf.texture_dirty = true;
kf.dirty = true;
}
}
}
Ok(())
}
fn description(&self) -> String {
match self.mode {
RasterResizeMode::Scale => "Scale raster layer".into(),
RasterResizeMode::Canvas => "Resize raster canvas".into(),
}
}
}

View File

@ -0,0 +1,59 @@
//! Resize-text-box action
//!
//! Updates a text layer's box origin and dimensions (which changes the wrap
//! width, so parley re-wraps the text). One undoable step.
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use kurbo::Point;
use uuid::Uuid;
pub struct ResizeTextBoxAction {
layer_id: Uuid,
new_origin: Point,
new_width: f64,
new_height: f64,
old: Option<(Point, f64, f64)>,
}
impl ResizeTextBoxAction {
pub fn new(layer_id: Uuid, new_origin: Point, new_width: f64, new_height: f64) -> Self {
Self { layer_id, new_origin, new_width, new_height, old: None }
}
}
impl Action for ResizeTextBoxAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
let AnyLayer::Text(text_layer) = layer else {
return Err("ResizeTextBoxAction target is not a text layer".to_string());
};
if self.old.is_none() {
self.old = Some((text_layer.box_origin, text_layer.box_width, text_layer.box_height));
}
text_layer.box_origin = self.new_origin;
text_layer.box_width = self.new_width.max(1.0);
text_layer.box_height = self.new_height.max(1.0);
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let Some((origin, w, h)) = self.old else { return Ok(()) };
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
if let AnyLayer::Text(text_layer) = layer {
text_layer.box_origin = origin;
text_layer.box_width = w;
text_layer.box_height = h;
}
Ok(())
}
fn description(&self) -> String {
"Resize text box".to_string()
}
}

View File

@ -4,7 +4,7 @@
//! with undo/redo support.
use crate::action::Action;
use crate::document::Document;
use crate::document::{Document, HdrOutputMode};
use crate::shape::ShapeColor;
/// Individual property change for a document
@ -15,6 +15,7 @@ pub enum DocumentPropertyChange {
Duration(f64),
Framerate(f64),
BackgroundColor(ShapeColor),
HdrOutputMode(HdrOutputMode),
}
/// Stored old value for undo (either f64 or color)
@ -22,6 +23,7 @@ pub enum DocumentPropertyChange {
enum OldValue {
F64(f64),
Color(ShapeColor),
Hdr(HdrOutputMode),
}
/// Action that sets a property on the document
@ -72,6 +74,14 @@ impl SetDocumentPropertiesAction {
old_value: None,
}
}
/// Create a new action to set the HDR→SDR output mode
pub fn set_hdr_output_mode(mode: HdrOutputMode) -> Self {
Self {
property: DocumentPropertyChange::HdrOutputMode(mode),
old_value: None,
}
}
}
impl Action for SetDocumentPropertiesAction {
@ -83,6 +93,7 @@ impl Action for SetDocumentPropertiesAction {
DocumentPropertyChange::Duration(_) => OldValue::F64(document.duration),
DocumentPropertyChange::Framerate(_) => OldValue::F64(document.framerate),
DocumentPropertyChange::BackgroundColor(_) => OldValue::Color(document.background_color),
DocumentPropertyChange::HdrOutputMode(_) => OldValue::Hdr(document.hdr_output_mode),
});
}
@ -92,6 +103,7 @@ impl Action for SetDocumentPropertiesAction {
DocumentPropertyChange::Duration(v) => document.duration = *v,
DocumentPropertyChange::Framerate(v) => document.framerate = *v,
DocumentPropertyChange::BackgroundColor(c) => document.background_color = *c,
DocumentPropertyChange::HdrOutputMode(m) => document.hdr_output_mode = *m,
}
Ok(())
}
@ -106,11 +118,15 @@ impl Action for SetDocumentPropertiesAction {
DocumentPropertyChange::Duration(_) => document.duration = v,
DocumentPropertyChange::Framerate(_) => document.framerate = v,
DocumentPropertyChange::BackgroundColor(_) => {}
DocumentPropertyChange::HdrOutputMode(_) => {}
}
}
Some(OldValue::Color(c)) => {
document.background_color = *c;
}
Some(OldValue::Hdr(m)) => {
document.hdr_output_mode = *m;
}
None => {}
}
Ok(())
@ -123,6 +139,7 @@ impl Action for SetDocumentPropertiesAction {
DocumentPropertyChange::Duration(_) => "duration",
DocumentPropertyChange::Framerate(_) => "framerate",
DocumentPropertyChange::BackgroundColor(_) => "background color",
DocumentPropertyChange::HdrOutputMode(_) => "HDR output mode",
};
format!("Set {}", property_name)
}

View File

@ -0,0 +1,61 @@
//! Set-text-content action
//!
//! Replaces a text layer's [`TextContent`] (text string, font size, color, family,
//! alignment) as one undoable step. Used both by in-place editing (text changes)
//! and the info panel (style changes).
use crate::action::Action;
use crate::document::Document;
use crate::layer::AnyLayer;
use crate::text_layer::TextContent;
use uuid::Uuid;
pub struct SetTextContentAction {
layer_id: Uuid,
new: TextContent,
old: Option<TextContent>,
}
impl SetTextContentAction {
pub fn new(layer_id: Uuid, new: TextContent) -> Self {
Self { layer_id, new, old: None }
}
/// Construct with an explicit `old` value. Used by in-place editing, which mutates
/// the document live for preview and then records one undoable step capturing the
/// content as it was *before* editing began.
pub fn with_old(layer_id: Uuid, old: TextContent, new: TextContent) -> Self {
Self { layer_id, new, old: Some(old) }
}
}
impl Action for SetTextContentAction {
fn execute(&mut self, document: &mut Document) -> Result<(), String> {
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
let AnyLayer::Text(text_layer) = layer else {
return Err("SetTextContentAction target is not a text layer".to_string());
};
if self.old.is_none() {
self.old = Some(text_layer.content.clone());
}
text_layer.content = self.new.clone();
Ok(())
}
fn rollback(&mut self, document: &mut Document) -> Result<(), String> {
let Some(old) = self.old.clone() else { return Ok(()) };
let layer = document
.get_layer_mut(&self.layer_id)
.ok_or_else(|| format!("Layer {} not found", self.layer_id))?;
if let AnyLayer::Text(text_layer) = layer {
text_layer.content = old;
}
Ok(())
}
fn description(&self) -> String {
"Edit text".to_string()
}
}

View File

@ -114,6 +114,7 @@ impl Action for SplitClipInstanceAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => return Err("Cannot split clip instances on group layers".to_string()),
AnyLayer::Raster(_) => return Err("Cannot split clip instances on group layers".to_string()),
AnyLayer::Text(_) => return Err("Cannot split clip instances on group layers".to_string()),
};
let instance = clip_instances
@ -233,7 +234,7 @@ impl Action for SplitClipInstanceAction {
AnyLayer::Group(_) => {
return Err("Cannot split clip instances on group layers".to_string());
}
AnyLayer::Raster(_) => {
AnyLayer::Raster(_) | AnyLayer::Text(_) => {
return Err("Cannot split clip instances on group layers".to_string());
}
}
@ -294,8 +295,8 @@ impl Action for SplitClipInstanceAction {
AnyLayer::Group(_) => {
// Group layers don't have clip instances, nothing to rollback
}
AnyLayer::Raster(_) => {
// Raster layers don't have clip instances, nothing to rollback
AnyLayer::Raster(_) | AnyLayer::Text(_) => {
// Raster/text layers don't have clip instances, nothing to rollback
}
}

View File

@ -101,6 +101,7 @@ impl Action for TransformClipInstancesAction {
AnyLayer::Effect(_) => {}
AnyLayer::Group(_) => {}
AnyLayer::Raster(_) => {}
AnyLayer::Text(_) => {}
}
Ok(())
}
@ -140,6 +141,7 @@ impl Action for TransformClipInstancesAction {
AnyLayer::Effect(_) => {}
AnyLayer::Group(_) => {}
AnyLayer::Raster(_) => {}
AnyLayer::Text(_) => {}
}
Ok(())
}

View File

@ -101,6 +101,7 @@ impl Action for TrimClipInstancesAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
if let Some(instance) = clip_instances.iter().find(|ci| ci.id == *member_instance_id) {
@ -138,6 +139,7 @@ impl Action for TrimClipInstancesAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
if let Some(instance) = clip_instances.iter().find(|ci| ci.id == *member_instance_id) {
@ -182,6 +184,7 @@ impl Action for TrimClipInstancesAction {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
let instance = clip_instances.iter()
@ -275,6 +278,7 @@ impl Action for TrimClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
// Apply trims
@ -315,6 +319,7 @@ impl Action for TrimClipInstancesAction {
AnyLayer::Effect(el) => &mut el.clip_instances,
AnyLayer::Group(_) => continue,
AnyLayer::Raster(_) => continue,
AnyLayer::Text(_) => continue,
};
// Restore original trim values

View File

@ -62,6 +62,10 @@ pub enum MediaKind {
/// the keyframe id). Decoded eagerly on load and shown while the full-res pixels
/// page in, so cold scrubs don't flash blank. See `raster_proxy_media_id`.
RasterProxy = 6,
/// An embedded font file (TTF/OTF) used by a text layer (keyed by a content hash
/// of the bytes so identical fonts dedupe). Registered into the font collection
/// on load so documents render faithfully on machines lacking the font.
Font = 7,
}
impl MediaKind {
@ -74,6 +78,7 @@ impl MediaKind {
4 => Some(Self::Waveform),
5 => Some(Self::Thumbnail),
6 => Some(Self::RasterProxy),
7 => Some(Self::Font),
_ => None,
}
}
@ -374,7 +379,11 @@ impl BeamArchive {
let rows = stmt
.query_map([&id_bytes], |r| r.get::<_, Vec<u8>>(0))
.map_err(map_sql)?;
let mut out = Vec::with_capacity(info.total_len as usize);
// `total_len` is an untrusted DB field; cap the preallocation so a corrupt/oversized
// value can't trigger a multi-GB eager allocation (or, on 32-bit, a truncated `as usize`).
// The Vec still grows to fit the actual chunk bytes regardless.
const PREALLOC_CAP: u64 = 64 * 1024 * 1024;
let mut out = Vec::with_capacity(info.total_len.min(PREALLOC_CAP) as usize);
for row in rows {
out.extend_from_slice(&row.map_err(map_sql)?);
}

View File

@ -122,6 +122,7 @@ impl VectorClip {
AnyLayer::Effect(el) => &el.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
for ci in clip_instances {
// Compute end position of this clip instance in beats
@ -223,6 +224,17 @@ impl VectorClip {
Some(existing) => existing.union(transformed_bounds),
});
}
} else if let AnyLayer::Text(text_layer) = &layer_node.data {
// Text layers contribute their box bounds (so a text-only clip is
// selectable/draggable, not a degenerate point).
let r = Rect::from_origin_size(
text_layer.box_origin,
(text_layer.box_width, text_layer.box_height),
);
combined_bounds = Some(match combined_bounds {
None => r,
Some(existing) => existing.union(r),
});
}
}
@ -343,6 +355,13 @@ pub struct VideoClip {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub linked_audio_clip_id: Option<Uuid>,
/// When set, the video bytes are packed into the `.beam` container under this
/// media id (== the clip id) and decoded by streaming from the SQLite blob.
/// `None` means the video is referenced externally via [`Self::file_path`].
/// Reconstructed from the archive on load (see `load_beam_sqlite`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub media_id: Option<Uuid>,
/// Folder this clip belongs to (None = root of category)
#[serde(default)]
pub folder_id: Option<Uuid>,
@ -367,6 +386,7 @@ impl VideoClip {
duration,
frame_rate,
linked_audio_clip_id: None,
media_id: None,
folder_id: None,
}
}

View File

@ -62,6 +62,7 @@ impl ClipboardLayerType {
AnyLayer::Effect(_) => ClipboardLayerType::Effect,
AnyLayer::Group(_) => ClipboardLayerType::Vector,
AnyLayer::Raster(_) => ClipboardLayerType::Vector,
AnyLayer::Text(_) => ClipboardLayerType::Vector,
}
}
@ -449,6 +450,13 @@ fn regen_any_layer(layer: &AnyLayer, id_map: &mut HashMap<Uuid, Uuid>) -> AnyLay
nl.layer.id = new_layer_id;
AnyLayer::Raster(nl)
}
AnyLayer::Text(tl) => {
let new_layer_id = Uuid::new_v4();
id_map.insert(tl.layer.id, new_layer_id);
let mut nl = tl.clone();
nl.layer.id = new_layer_id;
AnyLayer::Text(nl)
}
AnyLayer::Group(gl) => {
let new_layer_id = Uuid::new_v4();
id_map.insert(gl.layer.id, new_layer_id);

View File

@ -145,6 +145,26 @@ pub enum TimelineMode {
Frames,
}
/// How super-white (HDR) values are mapped to the SDR display/export output at the final
/// linear→sRGB encode. SDR content sits in [0,1] and is unaffected by either mode.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, Default)]
pub enum HdrOutputMode {
/// Hard-clip values above 1.0 (the historical behaviour). SDR-exact; HDR highlights blow out.
#[default]
Clip,
/// Roll highlights above a knee smoothly toward 1.0 to recover detail. Slightly dims near-white.
HighlightRolloff,
}
impl HdrOutputMode {
pub fn name(&self) -> &'static str {
match self {
HdrOutputMode::Clip => "Clip",
HdrOutputMode::HighlightRolloff => "Highlight rolloff",
}
}
}
/// Asset category for folder tree access
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AssetCategory {
@ -244,6 +264,10 @@ pub struct Document {
#[serde(default)]
pub timeline_mode: TimelineMode,
/// How HDR (super-white) values are mapped to the SDR output at the final encode.
#[serde(default)]
pub hdr_output_mode: HdrOutputMode,
/// Current UI layout state (serialized for save/load)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub ui_layout: Option<LayoutNode>,
@ -278,6 +302,7 @@ impl Default for Document {
ml
},
duration: 10.0,
hdr_output_mode: HdrOutputMode::default(),
root: GraphicsObject::default(),
vector_clips: HashMap::new(),
video_clips: HashMap::new(),
@ -464,8 +489,8 @@ impl Document {
}
}
}
crate::layer::AnyLayer::Raster(_) => {
// Raster layers don't have clip instances
crate::layer::AnyLayer::Raster(_) | crate::layer::AnyLayer::Text(_) => {
// Raster and text layers don't have clip instances
}
crate::layer::AnyLayer::Group(group) => {
// Recurse into group children to find their clip instance endpoints
@ -505,8 +530,8 @@ impl Document {
}
}
}
crate::layer::AnyLayer::Raster(_) => {
// Raster layers don't have clip instances
crate::layer::AnyLayer::Raster(_) | crate::layer::AnyLayer::Text(_) => {
// Raster and text layers don't have clip instances
}
crate::layer::AnyLayer::Group(g) => {
process_group_children(&g.children, doc, max_end, calc_end);
@ -912,6 +937,7 @@ impl Document {
AnyLayer::Effect(effect) => &effect.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
let instance = instances.iter().find(|inst| &inst.id == instance_id)?;
@ -952,6 +978,7 @@ impl Document {
AnyLayer::Effect(effect) => &effect.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
for instance in instances {
@ -1014,6 +1041,7 @@ impl Document {
AnyLayer::Vector(_) => return Some(desired_start), // Shouldn't reach here
AnyLayer::Group(_) => return Some(desired_start), // Groups don't have own clips
AnyLayer::Raster(_) => return Some(desired_start), // Raster layers don't have own clips
AnyLayer::Text(_) => return Some(desired_start), // Text layers don't have own clips
};
let mut occupied_ranges: Vec<(f64, f64, Uuid)> = Vec::new();
@ -1109,6 +1137,7 @@ impl Document {
AnyLayer::Vector(v) => &v.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
// Collect non-group clip ranges
@ -1180,6 +1209,7 @@ impl Document {
AnyLayer::Vector(vector) => &vector.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
for other in instances {
@ -1228,6 +1258,7 @@ impl Document {
AnyLayer::Vector(vector) => &vector.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
let mut nearest_start = f64::MAX;
@ -1275,6 +1306,7 @@ impl Document {
AnyLayer::Vector(vector) => &vector.clip_instances,
AnyLayer::Group(_) => &[],
AnyLayer::Raster(_) => &[],
AnyLayer::Text(_) => &[],
};
let mut nearest_end = 0.0;

View File

@ -265,6 +265,84 @@ impl VideoCodec {
}
}
/// YUV color range for the encoded video (currently H.264). Limited/TV (16235) is what nearly
/// every player assumes; Full/PC (0255) keeps a bit more precision but only looks right in players
/// that honor the full-range tag — so Limited is the safe default.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum ColorRange {
Limited,
Full,
}
impl Default for ColorRange {
fn default() -> Self { ColorRange::Limited }
}
impl ColorRange {
pub fn is_full(&self) -> bool { matches!(self, ColorRange::Full) }
pub fn name(&self) -> &'static str {
match self {
ColorRange::Limited => "Limited (TV, 16235)",
ColorRange::Full => "Full (PC, 0255)",
}
}
}
/// HDR output mode for video export. SDR encodes BT.709 8-bit as before; the HDR modes encode
/// 10-bit BT.2020 with the PQ (HDR10) or HLG transfer, preserving super-white highlights from the
/// linear compositor. HDR requires a 10-bit codec (HEVC Main10) — the exporter forces H.265.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
pub enum HdrExportMode {
#[default]
Sdr,
/// PQ (SMPTE ST 2084) — HDR10. Graphics white at 203 nits (matches the compositor convention).
Pq,
/// HLG (ARIB STD-B67) — broadcast HDR, also displayable as SDR.
Hlg,
}
impl HdrExportMode {
pub fn is_hdr(&self) -> bool { !matches!(self, HdrExportMode::Sdr) }
pub fn name(&self) -> &'static str {
match self {
HdrExportMode::Sdr => "SDR (BT.709, 8-bit)",
HdrExportMode::Pq => "HDR10 / PQ (BT.2020, 10-bit)",
HdrExportMode::Hlg => "HLG (BT.2020, 10-bit)",
}
}
/// FFmpeg transfer-characteristic name for the color tags.
pub fn transfer_name(&self) -> &'static str {
match self {
HdrExportMode::Sdr => "bt709",
HdrExportMode::Pq => "smpte2084",
HdrExportMode::Hlg => "arib-std-b67",
}
}
}
/// How the document is fit into the export frame when the export resolution's aspect ratio differs
/// from the document's. Applied as the export `base_transform` (document space → export pixels).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
pub enum ExportFitMode {
/// Scale each axis independently to fill the frame — distorts when aspects differ.
Stretch,
/// Scale uniformly to fit, centered, with black bars (letterbox/pillarbox). Preserves aspect.
#[default]
Letterbox,
/// Scale uniformly to fill, centered, cropping the overflow. Preserves aspect, no bars.
Crop,
}
impl ExportFitMode {
pub fn name(&self) -> &'static str {
match self {
ExportFitMode::Stretch => "Stretch (distort to fill)",
ExportFitMode::Letterbox => "Letterbox (fit, black bars)",
ExportFitMode::Crop => "Crop (fill, trim edges)",
}
}
}
/// Video quality presets
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum VideoQuality {
@ -322,6 +400,18 @@ pub struct VideoExportSettings {
/// Video quality
pub quality: VideoQuality,
/// YUV color range (H.264 only; ignored by other codecs).
#[serde(default)]
pub color_range: ColorRange,
/// HDR output mode. HDR forces 10-bit HEVC (BT.2020 + PQ/HLG); SDR is the default.
#[serde(default)]
pub hdr: HdrExportMode,
/// How the document is fit into the export frame when aspect ratios differ (default Letterbox).
#[serde(default)]
pub fit: ExportFitMode,
/// Audio settings (None = no audio)
pub audio: Option<AudioExportSettings>,
@ -340,6 +430,9 @@ impl Default for VideoExportSettings {
height: None,
framerate: 60.0,
quality: VideoQuality::High,
color_range: ColorRange::Limited,
hdr: HdrExportMode::Sdr,
fit: ExportFitMode::Letterbox,
audio: Some(AudioExportSettings::high_quality_aac()),
start_time: 0.0,
end_time: 60.0,
@ -431,11 +524,14 @@ pub struct ImageExportSettings {
/// When false, the image is composited onto an opaque background before encoding.
/// Only meaningful for formats that support alpha (PNG, WebP).
pub allow_transparency: bool,
/// How the document is fit into the output frame when aspect ratios differ (default Letterbox).
#[serde(default)]
pub fit: ExportFitMode,
}
impl Default for ImageExportSettings {
fn default() -> Self {
Self { format: ImageFormat::Png, time: 0.0, width: None, height: None, quality: 90, allow_transparency: false }
Self { format: ImageFormat::Png, time: 0.0, width: None, height: None, quality: 90, allow_transparency: false, fit: ExportFitMode::Letterbox }
}
}

View File

@ -9,7 +9,7 @@
//! too (via [`load_beam_zip_legacy`]). Saving always writes the SQLite form, so
//! opening a legacy file and saving migrates it.
use crate::beam_archive::{BeamArchive, MediaKind, MediaMeta, LARGE_MEDIA_THRESHOLD};
use crate::beam_archive::{BeamArchive, MediaKind, MediaMeta, MediaStorage, LARGE_MEDIA_THRESHOLD};
use crate::document::Document;
use daw_backend::audio::pool::AudioPoolEntry;
use daw_backend::audio::project::Project as AudioProject;
@ -88,15 +88,6 @@ impl Default for LargeMediaMode {
/// Settings for saving a project
#[derive(Debug, Clone)]
pub struct SaveSettings {
/// Automatically embed files smaller than this size (in bytes)
pub auto_embed_threshold_bytes: u64,
/// Force embedding all media files
pub force_embed_all: bool,
/// Force linking all media files (don't embed any)
pub force_link_all: bool,
/// How to store files at/above [`LARGE_MEDIA_THRESHOLD`] (pack vs reference).
/// `Ask` behaves as `Reference` here (safe default: don't bloat the DB).
pub large_media_mode: LargeMediaMode,
@ -105,9 +96,6 @@ pub struct SaveSettings {
impl Default for SaveSettings {
fn default() -> Self {
Self {
auto_embed_threshold_bytes: 10_000_000, // 10 MB
force_embed_all: false,
force_link_all: false,
large_media_mode: LargeMediaMode::Ask,
}
}
@ -254,6 +242,21 @@ fn thumbnail_media_id(clip_id: Uuid) -> Uuid {
Uuid::from_u128(clip_id.as_u128() ^ SENTINEL)
}
/// Content-hash id for an embedded font row, so identical font files dedupe to one
/// row regardless of which / how many text layers use them. A 128-bit id is built
/// from two salted 64-bit hashes of the bytes.
fn font_media_id(bytes: &[u8]) -> Uuid {
use std::hash::{Hash, Hasher};
let mut h1 = std::collections::hash_map::DefaultHasher::new();
bytes.hash(&mut h1);
let mut h2 = std::collections::hash_map::DefaultHasher::new();
0xF0E1_D2C3_B4A5_9687u64.hash(&mut h2);
bytes.hash(&mut h2);
let hi = h1.finish() as u128;
let lo = h2.finish() as u128;
Uuid::from_u128((hi << 64) | lo)
}
/// Derived id for a raster keyframe's low-res proxy row (distinct from the keyframe's
/// own full-res `Raster` row, which is keyed by the raw keyframe id).
fn raster_proxy_media_id(kf_id: Uuid) -> Uuid {
@ -269,7 +272,7 @@ pub fn save_beam(
audio_pool_entries: Vec<AudioPoolEntry>,
layer_to_track_map: &std::collections::HashMap<uuid::Uuid, u32>,
thumbnail_blobs: &std::collections::HashMap<uuid::Uuid, Vec<u8>>,
_settings: &SaveSettings,
settings: &SaveSettings,
) -> Result<(), String> {
let fn_start = std::time::Instant::now();
eprintln!("📊 [SAVE_BEAM] Starting save_beam() (SQLite container)...");
@ -349,7 +352,7 @@ pub fn save_beam(
// (`Ask` == reference); smaller files are always packed.
let reference_it = entry.is_video_audio
|| (size >= LARGE_MEDIA_THRESHOLD
&& _settings.large_media_mode != LargeMediaMode::Pack);
&& settings.large_media_mode != LargeMediaMode::Pack);
if reference_it {
referenced = Some(rel.clone());
} else {
@ -462,6 +465,65 @@ pub fn save_beam(
}
}
// --- video clips -> media rows (Video bytes), keyed by the clip id ---
// Mirror the audio pack/reference decision: pack unless the file is
// >= LARGE_MEDIA_THRESHOLD and the user didn't choose Pack. Packed video is
// decoded by streaming from the blob (see video.rs `VideoSource` + ffmpeg-blob-io).
for (clip_id, clip) in &document.video_clips {
// In-place re-save: an unchanged packed video keeps its row, never re-streamed.
if clip.media_id == Some(*clip_id) && txn.media_exists(*clip_id)? {
live_media.insert(*clip_id);
continue;
}
if clip.file_path.is_empty() {
continue;
}
let full = if Path::new(&clip.file_path).is_absolute() {
PathBuf::from(&clip.file_path)
} else {
project_dir.join(&clip.file_path)
};
if !full.is_file() {
continue; // source gone — reported missing on load
}
let size = std::fs::metadata(&full).map(|m| m.len()).unwrap_or(0);
let codec = full
.extension()
.and_then(|x| x.to_str())
.unwrap_or("mp4")
.to_lowercase();
let meta = MediaMeta {
width: Some(clip.width as u32),
height: Some(clip.height as u32),
..Default::default()
};
let reference_it =
size >= LARGE_MEDIA_THRESHOLD && settings.large_media_mode != LargeMediaMode::Pack;
if reference_it {
txn.put_media_referenced(*clip_id, MediaKind::Video, &codec, &clip.file_path, meta)?;
} else {
txn.put_media_packed_from_path(*clip_id, MediaKind::Video, &codec, &full, meta)?;
// The video's audio track lives in this same blob. Point the linked
// video-audio pool entry at the video's media row so its audio streams
// from the blob too (instead of re-probing the external file on load).
if let Some(aid) = clip.linked_audio_clip_id {
if let Some(crate::clip::AudioClipType::Sampled { audio_pool_index }) =
document.audio_clips.get(&aid).map(|c| &c.clip_type)
{
if let Some(e) = modified_entries
.iter_mut()
.find(|e| e.pool_index == *audio_pool_index)
{
e.media_id = Some(clip_id.to_string());
e.relative_path = None;
e.embedded_data = None;
}
}
}
}
live_media.insert(*clip_id);
}
// --- image assets -> media rows (original file bytes), keyed by asset id ---
let mut image_count = 0usize;
for (id, asset) in &document.image_assets {
@ -488,6 +550,32 @@ pub fn save_beam(
}
let _ = image_count;
// --- embedded fonts -> media rows (Font), keyed by content hash so identical
// fonts dedupe. Embed every non-bundled family used by a text layer so the
// document renders faithfully on machines lacking the font. The bundled
// three ship with the app and are never embedded. ---
{
use crate::layer::AnyLayer;
let mut families: HashSet<String> = HashSet::new();
for layer in document.all_layers() {
if let AnyLayer::Text(tl) = layer {
let fam = &tl.content.font_family;
if !fam.is_empty() && !crate::fonts::is_bundled(fam) {
families.insert(fam.clone());
}
}
}
for fam in families {
if let Some(bytes) = crate::fonts::family_font_bytes(&fam) {
let id = font_media_id(&bytes);
if !txn.media_exists(id)? {
txn.put_media_packed(id, MediaKind::Font, &fam, &bytes, MediaMeta::default())?;
}
live_media.insert(id);
}
}
}
// --- orphan cleanup: drop media for removed clips/keyframes ---
let removed = txn.retain_media(&live_media)?;
@ -498,7 +586,7 @@ pub fn save_beam(
modified: now.clone(),
ui_state: document.clone(),
audio_backend: SerializedAudioBackend {
sample_rate: 48000, // TODO: Get from audio engine
sample_rate: audio_project.sample_rate(),
project: audio_project.clone(),
audio_pool_entries: modified_entries,
layer_to_track_map: layer_to_track_map.clone(),
@ -551,6 +639,17 @@ fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
eprintln!("📊 [LOAD_BEAM] Starting load_beam() (SQLite container)...");
let archive = BeamArchive::open(path)?;
// Register document-embedded fonts into the runtime font collection so text
// layers referencing them render faithfully even if the host lacks the font.
if let Ok(font_ids) = archive.media_ids_of_kind(MediaKind::Font) {
for id in font_ids {
if let Ok(bytes) = archive.read_media_full(id) {
crate::fonts::register_embedded(bytes);
}
}
}
let json = archive.get_project_json()?;
let beam_project: BeamProject = serde_json::from_str(&json)
.map_err(|e| format!("Failed to deserialize project.json: {}", e))?;
@ -658,7 +757,7 @@ fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
// Missing external files (referenced entries whose file no longer exists).
let project_dir = path.parent().unwrap_or_else(|| Path::new("."));
let missing_files: Vec<MissingFileInfo> = restored_entries
let mut missing_files: Vec<MissingFileInfo> = restored_entries
.iter()
.enumerate()
.filter_map(|(idx, entry)| {
@ -682,6 +781,42 @@ fn load_beam_sqlite(path: &Path) -> Result<LoadedProject, String> {
})
.collect();
// Resolve packed/referenced video bytes per clip (mirrors audio resolution).
// A `MediaKind::Video` row keyed by the clip id means the bytes are in the
// container; the decoder streams from the blob. Referenced rows update the
// external path and report a missing source. Old projects (no Video row) stay
// referenced via their existing `file_path`.
for (clip_id, clip) in document.video_clips.iter_mut() {
match archive.media_info(*clip_id) {
Ok(Some(info)) if info.kind == MediaKind::Video => match info.storage {
MediaStorage::Packed => {
clip.media_id = Some(*clip_id);
}
MediaStorage::Referenced => {
clip.media_id = None;
if let Some(p) = info.ext_path.clone() {
clip.file_path = p;
}
let full = if Path::new(&clip.file_path).is_absolute() {
PathBuf::from(&clip.file_path)
} else {
project_dir.join(&clip.file_path)
};
if !clip.file_path.is_empty() && !full.exists() {
missing_files.push(MissingFileInfo {
pool_index: 0,
original_path: full,
file_type: MediaFileType::Video,
});
}
}
},
_ => {
clip.media_id = None;
}
}
}
// Persisted video thumbnail packs (opaque LBTN blobs), keyed by clip id. The
// editor decodes + inserts them and skips regeneration for these clips.
let mut thumbnail_blobs = std::collections::HashMap::new();
@ -879,8 +1014,11 @@ fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
for layer in document.root.children.iter_mut() {
if let crate::layer::AnyLayer::Raster(rl) = layer {
for kf in &mut rl.keyframes {
if !kf.media_path.is_empty() {
match zip.by_name(&kf.media_path) {
// Legacy ZIP raster entries are named "media/raster/<uuid>.png",
// derivable from the keyframe id (the old `media_path` field).
let entry_path = format!("media/raster/{}.png", kf.id);
{
match zip.by_name(&entry_path) {
Ok(mut png_file) => {
let mut png_bytes = Vec::new();
let _ = png_file.read_to_end(&mut png_bytes);
@ -890,7 +1028,7 @@ fn load_beam_zip_legacy(path: &Path) -> Result<LoadedProject, String> {
kf.raw_pixels = rgba.into_raw();
raster_load_count += 1;
}
Err(e) => eprintln!("⚠️ [LOAD_BEAM] Failed to decode raster PNG {}: {}", kf.media_path, e),
Err(e) => eprintln!("⚠️ [LOAD_BEAM] Failed to decode raster PNG {}: {}", entry_path, e),
}
}
Err(_) => {

View File

@ -15,7 +15,9 @@ pub const VIDEO_EXTENSIONS: &[&str] = &["mp4", "mov", "avi", "mkv", "webm", "m4v
/// Supported MIDI file extensions
pub const MIDI_EXTENSIONS: &[&str] = &["mid", "midi"];
// Note: SVG import deferred to future task
/// Supported vector file extensions (imported as a new vector layer, not an asset)
pub const VECTOR_EXTENSIONS: &[&str] = &["svg"];
// Note: .beam project files handled separately in file save/load feature
/// File type categories for import routing
@ -25,6 +27,7 @@ pub enum FileType {
Audio,
Video,
Midi,
Vector,
}
/// Detect file type from extension string
@ -53,6 +56,9 @@ pub fn get_file_type(extension: &str) -> Option<FileType> {
if MIDI_EXTENSIONS.contains(&ext.as_str()) {
return Some(FileType::Midi);
}
if VECTOR_EXTENSIONS.contains(&ext.as_str()) {
return Some(FileType::Vector);
}
None
}
@ -65,6 +71,7 @@ pub fn all_supported_extensions() -> Vec<&'static str> {
all.extend_from_slice(AUDIO_EXTENSIONS);
all.extend_from_slice(VIDEO_EXTENSIONS);
all.extend_from_slice(MIDI_EXTENSIONS);
all.extend_from_slice(VECTOR_EXTENSIONS);
all
}
@ -90,7 +97,8 @@ mod tests {
assert_eq!(get_file_type("midi"), Some(FileType::Midi));
assert_eq!(get_file_type("unknown"), None);
assert_eq!(get_file_type("svg"), None); // SVG deferred
assert_eq!(get_file_type("svg"), Some(FileType::Vector));
assert_eq!(get_file_type("SVG"), Some(FileType::Vector));
}
#[test]

View File

@ -0,0 +1,296 @@
//! Font registry + text layout for text layers.
//!
//! Wraps a thread-local parley [`FontContext`]/[`LayoutContext`]. Three fonts are
//! bundled (serif, sans-serif, monospaced) so text renders deterministically even
//! offline; system fonts are also enumerated (parley's fontique source) and
//! document-embedded fonts can be registered at load time (see `file_io`).
//!
//! The same `linebender_resource_handle` crate (0.1.1) backs both vello's
//! `peniko::FontData` and parley's `FontData`, so a glyph run's `run.font()` can be
//! handed straight to `vello::Scene::draw_glyphs` with no conversion.
use std::borrow::Cow;
use std::cell::RefCell;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, OnceLock};
use parley::{
Alignment, AlignmentOptions, FontContext, FontFamily, FontFamilyName, Layout, LayoutContext,
PositionedLayoutItem, StyleProperty,
};
use vello::peniko::Blob;
use crate::text_layer::{TextAlign, TextContent};
// ── Bundled fonts (SIL OFL; vendored under assets/fonts) ─────────────────────
static BUNDLED_SANS: &[u8] = include_bytes!("../assets/fonts/LiberationSans-Regular.ttf");
static BUNDLED_SERIF: &[u8] = include_bytes!("../assets/fonts/LiberationSerif-Regular.ttf");
static BUNDLED_MONO: &[u8] = include_bytes!("../assets/fonts/LiberationMono-Regular.ttf");
/// Parley requires a brush type, but glyph color is applied by vello at draw time,
/// so we use a zero-sized placeholder.
#[derive(Clone, Copy, PartialEq, Default, Debug)]
pub struct NoBrush;
struct FontStore {
fcx: FontContext,
lcx: LayoutContext<NoBrush>,
/// Registered family names of the three bundled fonts (sans, serif, mono).
bundled: Vec<String>,
/// The default family (bundled sans-serif), used when `font_family` is empty.
default_family: String,
}
impl FontStore {
fn new() -> Self {
let mut fcx = FontContext::new();
let mut bundled = Vec::new();
for bytes in [BUNDLED_SANS, BUNDLED_SERIF, BUNDLED_MONO] {
let blob = Blob::new(Arc::new(bytes) as Arc<dyn AsRef<[u8]> + Send + Sync>);
for (family_id, _) in fcx.collection.register_fonts(blob, None) {
if let Some(name) = fcx.collection.family_name(family_id) {
let name = name.to_string();
if !bundled.contains(&name) {
bundled.push(name);
}
}
}
}
let default_family = bundled.first().cloned().unwrap_or_else(|| "sans-serif".to_string());
Self { fcx, lcx: LayoutContext::new(), bundled, default_family }
}
}
thread_local! {
static STORE: RefCell<FontStore> = RefCell::new(FontStore::new());
}
/// The default family name (bundled sans-serif), used when a text layer's
/// `font_family` is empty.
pub fn default_family() -> String {
STORE.with(|s| s.borrow().default_family.clone())
}
/// Whether `family` is one of the three bundled families (which therefore must
/// never be embedded into a `.beam`).
pub fn is_bundled(family: &str) -> bool {
STORE.with(|s| s.borrow().bundled.iter().any(|f| f == family))
}
/// True if a shorter word-prefix of `name` is itself a family in `set` — i.e. `name`
/// is a variant of a base family (e.g. "Noto Sans Arabic" when "Noto Sans" exists).
/// Used to collapse the many script/region-specific fonts (mostly Noto) into their
/// base family in the picker; parley's fallback still resolves the right script font
/// automatically when rendering non-Latin text.
fn is_variant_of_listed_base(name: &str, set: &std::collections::HashSet<&str>) -> bool {
let words: Vec<&str> = name.split(' ').collect();
for k in 1..words.len() {
if set.contains(words[..k].join(" ").as_str()) {
return true;
}
}
false
}
/// Available family names for the info-panel picker: the three bundled families first,
/// then a consolidated, alphabetical list of system families — script/region variants
/// whose base family is present are dropped (monospace variants are kept).
pub fn families() -> Vec<String> {
STORE.with(|s| {
let s = &mut *s.borrow_mut();
let mut system: Vec<String> =
s.fcx.collection.family_names().map(|n| n.to_string()).collect();
system.sort();
system.dedup();
let set: std::collections::HashSet<&str> = system.iter().map(|x| x.as_str()).collect();
let mut out = s.bundled.clone();
for name in &system {
if out.iter().any(|b| b == name) {
continue;
}
// Keep base families and monospace variants; drop script/region variants
// whose base family is already in the list.
if name.contains("Mono") || !is_variant_of_listed_base(name, &set) {
out.push(name.clone());
}
}
out
})
}
/// Register a document-embedded font (raw TTF/OTF bytes) into the runtime font
/// collection so text layers referencing its family resolve to it. Returns the
/// registered family names.
pub fn register_embedded(bytes: Vec<u8>) -> Vec<String> {
STORE.with(|s| {
let s = &mut *s.borrow_mut();
let blob = Blob::new(Arc::new(bytes) as Arc<dyn AsRef<[u8]> + Send + Sync>);
let mut names = Vec::new();
for (family_id, _) in s.fcx.collection.register_fonts(blob, None) {
if let Some(name) = s.fcx.collection.family_name(family_id) {
names.push(name.to_string());
}
}
names
})
}
/// Whether `family` currently resolves to a registered family (bundled, system,
/// or embedded). Used to flag "missing font" on load.
pub fn family_available(family: &str) -> bool {
if family.is_empty() {
return true;
}
STORE.with(|s| {
let s = &mut *s.borrow_mut();
s.fcx.collection.family_id(family).is_some()
})
}
/// The raw bytes of the font file that `family` resolves to (for embedding into a
/// `.beam`). Returns `None` if the family resolves to no glyphs. Lays out a single
/// glyph and reads the resolved run's font blob (the same `linebender_resource_handle`
/// blob vello uses).
pub fn family_font_bytes(family: &str) -> Option<Vec<u8>> {
let content = TextContent {
text: "A".to_string(),
font_family: family.to_string(),
..TextContent::default()
};
with_layout(&content, 10_000.0, |layout| {
for line in layout.lines() {
for item in line.items() {
if let PositionedLayoutItem::GlyphRun(run) = item {
return Some(run.run().font().data.data().to_vec());
}
}
}
None
})
}
// ── Background font preloading ───────────────────────────────────────────────
//
// Loading a family's bytes (`family_font_bytes`) is the expensive part (font file IO +
// shaping). Doing it for the whole picker on the UI thread causes hitches, so a
// background thread (started at app launch) loads every picker family's bytes ahead of
// time into a shared map. The UI then only has to *register* them with its renderer
// (cheap), and synchronously loads the few stragglers only if needed before they're ready.
struct PreloadState {
loaded: HashMap<String, Vec<u8>>,
started: bool,
done: bool,
}
fn preload() -> &'static Mutex<PreloadState> {
static P: OnceLock<Mutex<PreloadState>> = OnceLock::new();
P.get_or_init(|| Mutex::new(PreloadState {
loaded: HashMap::new(),
started: false,
done: false,
}))
}
/// Start (once) a background thread that loads every picker font's bytes. Idempotent;
/// safe to call every frame or from app startup.
pub fn start_preload() {
{
let mut p = preload().lock().unwrap();
if p.started {
return;
}
p.started = true;
}
let _ = std::thread::Builder::new()
.name("font-preload".into())
.spawn(|| {
// This thread gets its own thread-local FontContext (enumerates system fonts
// off the UI thread). Bytes are plain `Vec<u8>` (Send) handed back via the map.
for fam in families() {
if let Some(bytes) = family_font_bytes(&fam) {
preload().lock().unwrap().loaded.insert(fam, bytes);
}
}
preload().lock().unwrap().done = true;
});
}
/// Remove and return a preloaded font's bytes if the background thread has them ready.
pub fn take_preloaded(family: &str) -> Option<Vec<u8>> {
preload().lock().unwrap().loaded.remove(family)
}
/// Whether the background preloader has finished loading every family.
pub fn preload_done() -> bool {
preload().lock().unwrap().done
}
/// Caret rectangle (x0, y0, x1, y1, in layout space relative to the box origin) for
/// `byte_index` into `content.text`, wrapped to `max_width`.
pub fn caret_geometry(content: &TextContent, max_width: f32, byte_index: usize) -> Option<(f64, f64, f64, f64)> {
let caret_w = (content.font_size as f32 * 0.06).max(1.0);
with_layout(content, max_width, |layout| {
let cur = parley::Cursor::from_byte_index(layout, byte_index, parley::Affinity::Downstream);
let bb = cur.geometry(layout, caret_w);
Some((bb.x0, bb.y0, bb.x1, bb.y1))
})
}
/// Selection highlight rectangles (each x0, y0, x1, y1 in layout space) for the byte
/// range `[start, end)` into `content.text`, wrapped to `max_width`.
pub fn selection_geometry(content: &TextContent, max_width: f32, start: usize, end: usize) -> Vec<(f64, f64, f64, f64)> {
if start == end {
return Vec::new();
}
with_layout(content, max_width, |layout| {
let anchor = parley::Cursor::from_byte_index(layout, start, parley::Affinity::Downstream);
let focus = parley::Cursor::from_byte_index(layout, end, parley::Affinity::Downstream);
let sel = parley::Selection::new(anchor, focus);
sel.geometry(layout)
.into_iter()
.map(|(bb, _)| (bb.x0, bb.y0, bb.x1, bb.y1))
.collect()
})
}
fn alignment_of(a: TextAlign) -> Alignment {
match a {
TextAlign::Left => Alignment::Left,
TextAlign::Center => Alignment::Center,
TextAlign::Right => Alignment::Right,
TextAlign::Justify => Alignment::Justify,
}
}
/// Build a parley layout for `content` wrapped to `max_width` (document units),
/// then invoke `f` with it. The layout is rebuilt on each call (v1: no cache).
pub fn with_layout<R>(
content: &TextContent,
max_width: f32,
f: impl FnOnce(&Layout<NoBrush>) -> R,
) -> R {
STORE.with(|s| {
let s = &mut *s.borrow_mut();
let default_family = s.default_family.clone();
let FontStore { fcx, lcx, .. } = s;
let mut builder = lcx.ranged_builder(fcx, &content.text, 1.0, true);
builder.push_default(StyleProperty::FontSize(content.font_size as f32));
let family_name = if content.font_family.is_empty() {
default_family
} else {
content.font_family.clone()
};
let family = FontFamily::Single(FontFamilyName::Named(Cow::Owned(family_name)));
builder.push_default(StyleProperty::FontFamily(family));
let mut layout = builder.build(&content.text);
layout.break_all_lines(Some(max_width));
layout.align(alignment_of(content.align), AlignmentOptions::default());
f(&layout)
})
}

View File

@ -9,6 +9,7 @@ use crate::vector_graph::VectorGraph;
use crate::effect_layer::EffectLayer;
use crate::object::ShapeInstance;
use crate::raster_layer::RasterLayer;
use crate::text_layer::TextLayer;
use crate::shape::Shape;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
@ -31,6 +32,8 @@ pub enum LayerType {
Group,
/// Raster pixel-buffer painting layer
Raster,
/// Text layer (a single editable text box)
Text,
}
/// Common trait for all layer types
@ -866,6 +869,7 @@ impl GroupLayer {
AnyLayer::Effect(l) => &l.clip_instances,
AnyLayer::Group(_) => &[], // no nested groups
AnyLayer::Raster(_) => &[], // raster layers have no clip instances
AnyLayer::Text(_) => &[], // raster layers have no clip instances
};
for ci in instances {
result.push((child_id, ci));
@ -884,6 +888,7 @@ pub enum AnyLayer {
Effect(EffectLayer),
Group(GroupLayer),
Raster(RasterLayer),
Text(TextLayer),
}
impl LayerTrait for AnyLayer {
@ -895,6 +900,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.id(),
AnyLayer::Group(l) => l.id(),
AnyLayer::Raster(l) => l.id(),
AnyLayer::Text(l) => l.id(),
}
}
@ -906,6 +912,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.name(),
AnyLayer::Group(l) => l.name(),
AnyLayer::Raster(l) => l.name(),
AnyLayer::Text(l) => l.name(),
}
}
@ -917,6 +924,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_name(name),
AnyLayer::Group(l) => l.set_name(name),
AnyLayer::Raster(l) => l.set_name(name),
AnyLayer::Text(l) => l.set_name(name),
}
}
@ -928,6 +936,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.has_custom_name(),
AnyLayer::Group(l) => l.has_custom_name(),
AnyLayer::Raster(l) => l.has_custom_name(),
AnyLayer::Text(l) => l.has_custom_name(),
}
}
@ -939,6 +948,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_has_custom_name(custom),
AnyLayer::Group(l) => l.set_has_custom_name(custom),
AnyLayer::Raster(l) => l.set_has_custom_name(custom),
AnyLayer::Text(l) => l.set_has_custom_name(custom),
}
}
@ -950,6 +960,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.visible(),
AnyLayer::Group(l) => l.visible(),
AnyLayer::Raster(l) => l.visible(),
AnyLayer::Text(l) => l.visible(),
}
}
@ -961,6 +972,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_visible(visible),
AnyLayer::Group(l) => l.set_visible(visible),
AnyLayer::Raster(l) => l.set_visible(visible),
AnyLayer::Text(l) => l.set_visible(visible),
}
}
@ -972,6 +984,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.opacity(),
AnyLayer::Group(l) => l.opacity(),
AnyLayer::Raster(l) => l.opacity(),
AnyLayer::Text(l) => l.opacity(),
}
}
@ -983,6 +996,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_opacity(opacity),
AnyLayer::Group(l) => l.set_opacity(opacity),
AnyLayer::Raster(l) => l.set_opacity(opacity),
AnyLayer::Text(l) => l.set_opacity(opacity),
}
}
@ -994,6 +1008,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.volume(),
AnyLayer::Group(l) => l.volume(),
AnyLayer::Raster(l) => l.volume(),
AnyLayer::Text(l) => l.volume(),
}
}
@ -1005,6 +1020,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_volume(volume),
AnyLayer::Group(l) => l.set_volume(volume),
AnyLayer::Raster(l) => l.set_volume(volume),
AnyLayer::Text(l) => l.set_volume(volume),
}
}
@ -1016,6 +1032,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.muted(),
AnyLayer::Group(l) => l.muted(),
AnyLayer::Raster(l) => l.muted(),
AnyLayer::Text(l) => l.muted(),
}
}
@ -1027,6 +1044,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_muted(muted),
AnyLayer::Group(l) => l.set_muted(muted),
AnyLayer::Raster(l) => l.set_muted(muted),
AnyLayer::Text(l) => l.set_muted(muted),
}
}
@ -1038,6 +1056,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.soloed(),
AnyLayer::Group(l) => l.soloed(),
AnyLayer::Raster(l) => l.soloed(),
AnyLayer::Text(l) => l.soloed(),
}
}
@ -1049,6 +1068,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_soloed(soloed),
AnyLayer::Group(l) => l.set_soloed(soloed),
AnyLayer::Raster(l) => l.set_soloed(soloed),
AnyLayer::Text(l) => l.set_soloed(soloed),
}
}
@ -1060,6 +1080,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.locked(),
AnyLayer::Group(l) => l.locked(),
AnyLayer::Raster(l) => l.locked(),
AnyLayer::Text(l) => l.locked(),
}
}
@ -1071,6 +1092,7 @@ impl LayerTrait for AnyLayer {
AnyLayer::Effect(l) => l.set_locked(locked),
AnyLayer::Group(l) => l.set_locked(locked),
AnyLayer::Raster(l) => l.set_locked(locked),
AnyLayer::Text(l) => l.set_locked(locked),
}
}
}
@ -1085,6 +1107,7 @@ impl AnyLayer {
AnyLayer::Effect(l) => &l.layer,
AnyLayer::Group(l) => &l.layer,
AnyLayer::Raster(l) => &l.layer,
AnyLayer::Text(l) => &l.layer,
}
}
@ -1097,6 +1120,7 @@ impl AnyLayer {
AnyLayer::Effect(l) => &mut l.layer,
AnyLayer::Group(l) => &mut l.layer,
AnyLayer::Raster(l) => &mut l.layer,
AnyLayer::Text(l) => &mut l.layer,
}
}

View File

@ -54,6 +54,8 @@ pub mod svg_export;
pub mod snap;
pub mod webcam;
pub mod raster_layer;
pub mod text_layer;
pub mod fonts;
pub mod raster_store;
pub mod brush_settings;
pub mod brush_engine;

View File

@ -47,6 +47,21 @@ impl Transform {
Self::default()
}
/// Set scale + position so `content_w × content_h` is fit **uniformly and centered** within
/// `doc_w × doc_h`, preserving aspect ratio (letterbox/pillarbox). The single shared way to
/// place a media clip (video/image) on the document — both import paths use this so a clip
/// looks identical however it was added. No-op for non-positive content dimensions.
pub fn fit_centered(&mut self, content_w: f64, content_h: f64, doc_w: f64, doc_h: f64) {
if content_w <= 0.0 || content_h <= 0.0 {
return;
}
let scale = (doc_w / content_w).min(doc_h / content_h);
self.scale_x = scale;
self.scale_y = scale;
self.x = (doc_w - content_w * scale) / 2.0;
self.y = (doc_h - content_h * scale) / 2.0;
}
/// Create a transform with position
pub fn with_position(x: f64, y: f64) -> Self {
Self {

View File

@ -111,8 +111,6 @@ pub struct RasterKeyframe {
pub time: f64,
pub width: u32,
pub height: u32,
/// ZIP-relative path: `"media/raster/<uuid>.png"`
pub media_path: String,
/// Stroke history (for potential replay / future non-destructive editing)
pub stroke_log: Vec<StrokeRecord>,
pub tween_after: TweenType,
@ -157,13 +155,11 @@ impl RasterKeyframe {
pub fn new(time: f64, width: u32, height: u32) -> Self {
let id = Uuid::new_v4();
let media_path = format!("media/raster/{}.png", id);
Self {
id,
time,
width,
height,
media_path,
stroke_log: Vec::new(),
tween_after: TweenType::Hold,
raw_pixels: Vec::new(),
@ -173,6 +169,57 @@ impl RasterKeyframe {
proxy: None,
}
}
/// Change the canvas to `(new_w, new_h)`. `Scale` resamples the content (Lanczos3) to fill the
/// new size; `Canvas` keeps the content at native resolution anchored top-left, padding with
/// transparent (expand) or trimming (crop). No-op if the size is unchanged. If pixels aren't
/// resident the buffer is left empty and only the declared size changes — the caller must fault
/// pixels in first (a later load would otherwise mismatch the new size).
pub fn resize_to(&mut self, new_w: u32, new_h: u32, mode: RasterResizeMode) {
if new_w == 0 || new_h == 0 || (self.width == new_w && self.height == new_h) {
return;
}
// Resample/recanvas the buffer only when pixels are resident. A blank keyframe (no content
// and no store row) just takes the new declared size — there's nothing to corrupt. Paged-out
// keyframes are loaded by the caller (ResizeRasterLayerAction) before this runs.
if !self.raw_pixels.is_empty() {
let old = std::mem::take(&mut self.raw_pixels);
self.raw_pixels = match mode {
RasterResizeMode::Scale => match image::RgbaImage::from_raw(self.width, self.height, old) {
Some(img) => image::imageops::resize(&img, new_w, new_h, image::imageops::FilterType::Lanczos3).into_raw(),
None => vec![0u8; (new_w as usize) * (new_h as usize) * 4],
},
RasterResizeMode::Canvas => {
// Copy the old pixels into a transparent new buffer, anchored top-left (matching
// the raster's (0,0) document anchor); right/bottom is padded or trimmed.
let mut buf = vec![0u8; (new_w as usize) * (new_h as usize) * 4];
let copy_w = self.width.min(new_w) as usize;
let copy_h = self.height.min(new_h) as usize;
let (ow, nw) = (self.width as usize, new_w as usize);
for y in 0..copy_h {
let src = y * ow * 4;
let dst = y * nw * 4;
buf[dst..dst + copy_w * 4].copy_from_slice(&old[src..src + copy_w * 4]);
}
buf
}
};
self.proxy = None; // invalidate any downsampled proxy
}
self.width = new_w;
self.height = new_h;
self.texture_dirty = true;
self.dirty = true;
}
}
/// How a raster canvas resize treats existing pixels.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum RasterResizeMode {
/// Resample the content to fill the new size (changes pixel resolution).
Scale,
/// Keep content at native resolution, anchored top-left; pad/trim the canvas.
Canvas,
}
/// A pixel-buffer painting layer
@ -264,10 +311,6 @@ impl RasterLayer {
.map(|pos| self.keyframes.remove(pos))
}
/// Return the ZIP-relative PNG path for the active keyframe at `time`, or `None`.
pub fn buffer_path_at_time(&self, time: f64) -> Option<&str> {
self.keyframe_at(time).map(|kf| kf.media_path.as_str())
}
}
// Delegate all LayerTrait methods to self.layer

View File

@ -221,8 +221,11 @@ fn decode_image_brush(data: &[u8]) -> Option<ImageBrush> {
/// A single decoded video frame ready for GPU upload, with its document-space transform.
pub struct VideoRenderInstance {
/// sRGB RGBA8 pixel data (straight alpha — as decoded by ffmpeg).
/// sRGB RGBA8 pixel data (straight alpha — as decoded by ffmpeg). Empty when `gpu` is `Some`.
pub rgba_data: Arc<Vec<u8>>,
/// Hardware-decoded frame as GPU NV12 textures (preview path). When `Some`, the compositor
/// samples it directly (NV12→RGB) instead of uploading `rgba_data`.
pub gpu: Option<crate::video::GpuVideoFrame>,
pub width: u32,
pub height: u32,
/// Affine transform that maps from video-pixel space to document space.
@ -232,6 +235,21 @@ pub struct VideoRenderInstance {
pub opacity: f32,
}
/// Sink for pulling video frames out of a container layer's scene recursion, so
/// they composite via the fast GPU Video path instead of baking into Vello.
///
/// Threaded as `Option<&mut VideoExtract>` through the isolated-render scene
/// functions. When present, [`render_video_layer`] pushes a [`VideoRenderInstance`]
/// instead of drawing into the Vello scene. `drew_other` is set whenever any
/// non-video primitive (vector graph, image asset, raster) is emitted; that forces
/// the safe Vello fallback because the container mixes video with other content
/// and we can't preserve z-order by extraction alone.
#[derive(Default)]
struct VideoExtract {
instances: Vec<VideoRenderInstance>,
drew_other: bool,
}
/// Type of rendered layer for compositor handling
pub enum RenderedLayerType {
/// Vector / group layer — Vello scene in `RenderedLayer::scene` is used.
@ -351,6 +369,20 @@ pub struct CompositeRenderResult {
/// and effects in the GPU compositor.
///
/// Layers are returned in bottom-to-top order for compositing.
/// Decode-target resolution for video clips = the output (export/preview) resolution, derived from
/// the document→output `base_transform` scale. The decoder caps this to the source's native size,
/// so it means "decode at the size we'll actually display, never upscaling": full detail when
/// exporting above document res (instead of upscaling a document-res frame), and cheap small frames
/// for the canvas. Stable per render pass, so it doesn't thrash the decoder's scaler/cache.
fn video_decode_target(document: &Document, base_transform: Affine) -> (u32, u32) {
let c = base_transform.as_coeffs(); // [a, b, c, d, e, f]
let sx = (c[0] * c[0] + c[1] * c[1]).sqrt();
let sy = (c[2] * c[2] + c[3] * c[3]).sqrt();
let w = (document.width * sx).ceil().max(1.0) as u32;
let h = (document.height * sy).ceil().max(1.0) as u32;
(w, h)
}
pub fn render_document_for_compositing(
document: &Document,
base_transform: Affine,
@ -428,6 +460,30 @@ pub fn render_document_for_compositing(
}
}
// One-shot diagnostic: dump what the compositor actually receives. Set
// LB_LAYER_DEBUG=1 to print a single snapshot of each layer's resolved type.
if std::env::var("LB_LAYER_DEBUG").is_ok() {
static ONCE: std::sync::Once = std::sync::Once::new();
ONCE.call_once(|| {
eprintln!("[LB_LAYER_DEBUG] composite layers = {}", rendered_layers.len());
for (i, l) in rendered_layers.iter().enumerate() {
let desc = match &l.layer_type {
RenderedLayerType::Vector =>
format!("Vector (has_content={}, scene_empty={})", l.has_content, l.cpu_pixmap.is_none()),
RenderedLayerType::Raster { width, height, dirty, .. } =>
format!("Raster {width}x{height} dirty={dirty}"),
RenderedLayerType::Video { instances } =>
format!("Video ({} instance(s))", instances.len()),
RenderedLayerType::Float { width, height, .. } =>
format!("Float {width}x{height}"),
RenderedLayerType::Effect { effect_instances } =>
format!("Effect ({} instance(s))", effect_instances.len()),
};
eprintln!("[LB_LAYER_DEBUG] layer[{i}] id={} type={desc}", l.layer_id);
}
});
}
CompositeRenderResult {
background,
background_cpu: None,
@ -462,6 +518,9 @@ pub fn render_layer_isolated(
// Render layer content with full opacity (1.0) - opacity applied during compositing
match layer {
AnyLayer::Vector(vector_layer) => {
// Render into the scene with an extraction sink so a clip that is purely
// video (no vector geometry) composites via the fast GPU Video path.
let mut ex = VideoExtract::default();
render_vector_layer_to_scene(
document,
time,
@ -471,11 +530,29 @@ pub fn render_layer_isolated(
1.0, // Full opacity - layer opacity handled in compositing
image_cache,
video_manager,
Some(&mut ex),
);
if !ex.instances.is_empty() && !ex.drew_other {
// Pure video: discard the (now-empty) scene and emit GPU instances.
rendered.scene = Scene::new();
rendered.has_content = true;
rendered.layer_type = RenderedLayerType::Video { instances: ex.instances };
} else {
if !ex.instances.is_empty() {
// Mixed video + vector: the first pass diverted the video out of
// the scene, so re-render with no sink to bake it back in (correct
// z-order; Vello path). Rare — fast-splitting is deferred.
rendered.scene = Scene::new();
render_vector_layer_to_scene(
document, time, vector_layer, &mut rendered.scene,
base_transform, 1.0, image_cache, video_manager, None,
);
}
rendered.has_content = vector_layer.graph_at_time(time)
.map_or(false, |graph| !graph.edges.iter().all(|e| e.deleted) || !graph.fills.iter().all(|f| f.deleted))
|| !vector_layer.clip_instances.is_empty();
}
}
AnyLayer::Audio(_) => {
// Audio layers don't render visually
rendered.has_content = false;
@ -485,12 +562,13 @@ pub fn render_layer_isolated(
let layer_opacity = layer.opacity();
let mut video_mgr = video_manager.lock().unwrap();
let mut instances = Vec::new();
let (target_w, target_h) = video_decode_target(document, base_transform);
let tempo_map = document.tempo_map();
for clip_instance in &video_layer.clip_instances {
let Some(video_clip) = document.video_clips.get(&clip_instance.clip_id) else { continue };
let Some(clip_time) = clip_instance.remap_time(time, video_clip.duration, tempo_map) else { continue };
let Some(frame) = video_mgr.get_frame(&clip_instance.clip_id, clip_time) else { continue };
let Some(frame) = video_mgr.get_frame(&clip_instance.clip_id, clip_time, target_w, target_h) else { continue };
// Evaluate animated transform properties.
let anim = &video_layer.layer.animation_data;
@ -534,6 +612,7 @@ pub fn render_layer_isolated(
};
instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(),
gpu: frame.gpu.clone(),
width: frame.width,
height: frame.height,
transform: base_transform * clip_transform * frame_to_clip,
@ -554,6 +633,7 @@ pub fn render_layer_isolated(
* Affine::scale(scale);
instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(),
gpu: None, // webcam frames are always CPU
width: frame.width,
height: frame.height,
transform: cam_transform,
@ -577,16 +657,35 @@ pub fn render_layer_isolated(
return RenderedLayer::effect_layer(layer_id, opacity, active_effects);
}
AnyLayer::Group(group_layer) => {
// Render each child layer's content into the group's scene
// Render each child into the group's scene with an extraction sink. The
// common imported-video case is a Group[Video, Audio] — audio draws
// nothing, so it's pure video and composites via the GPU Video path.
let mut ex = VideoExtract::default();
for child in &group_layer.children {
render_layer(
document, time, child, &mut rendered.scene, base_transform,
1.0, // Full opacity - layer opacity handled in compositing
image_cache, video_manager, camera_frame,
image_cache, video_manager, camera_frame, Some(&mut ex),
);
}
if !ex.instances.is_empty() && !ex.drew_other {
rendered.scene = Scene::new();
rendered.has_content = true;
rendered.layer_type = RenderedLayerType::Video { instances: ex.instances };
} else {
if !ex.instances.is_empty() {
// Mixed: re-render with no sink to bake the video back into the scene.
rendered.scene = Scene::new();
for child in &group_layer.children {
render_layer(
document, time, child, &mut rendered.scene, base_transform,
1.0, image_cache, video_manager, camera_frame, None,
);
}
}
rendered.has_content = !group_layer.children.is_empty();
}
}
AnyLayer::Raster(raster_layer) => {
if let Some(kf) = raster_layer.keyframe_at(time) {
rendered.has_content = kf.has_pixels();
@ -599,6 +698,11 @@ pub fn render_layer_isolated(
};
}
}
AnyLayer::Text(text_layer) => {
// Text composites as vector geometry (glyphs in the Vello scene).
rendered.has_content =
render_text_layer_to_scene(text_layer, time, &mut rendered.scene, base_transform);
}
}
rendered
@ -614,6 +718,7 @@ fn render_vector_layer_to_scene(
parent_opacity: f64,
image_cache: &mut ImageCache,
video_manager: &std::sync::Arc<std::sync::Mutex<crate::video::VideoManager>>,
extract: Option<&mut VideoExtract>,
) {
render_vector_layer(
document,
@ -624,6 +729,7 @@ fn render_vector_layer_to_scene(
parent_opacity,
image_cache,
video_manager,
extract,
);
}
@ -656,6 +762,57 @@ fn render_raster_layer_to_scene(
scene.fill(Fill::NonZero, base_transform, &brush, None, &canvas_rect);
}
/// Render a text layer's glyphs into a Vello scene.
///
/// Text is laid out with parley (wrapped to the box width) and drawn via
/// `Scene::draw_glyphs`. The box origin offsets the whole layout; `base_transform`
/// carries the layer/clip-instance transform and camera. Returns whether anything
/// was drawn.
fn render_text_layer_to_scene(
layer: &crate::text_layer::TextLayer,
time: f64,
scene: &mut Scene,
base_transform: Affine,
) -> bool {
let content = layer.content_at(time);
if content.text.is_empty() {
return false;
}
let color = vello::peniko::Color::new(content.color);
let origin = Affine::translate((layer.box_origin.x, layer.box_origin.y));
let mut drew = false;
crate::fonts::with_layout(content, layer.box_width as f32, |layout| {
for line in layout.lines() {
for item in line.items() {
let parley::PositionedLayoutItem::GlyphRun(glyph_run) = item else { continue };
let run = glyph_run.run();
let font = run.font();
let font_size = run.font_size();
let synthesis = run.synthesis();
let glyph_xform = synthesis
.skew()
.map(|angle| Affine::skew((angle as f64).to_radians().tan(), 0.0));
drew = true;
scene
.draw_glyphs(font)
.font_size(font_size)
.brush(color)
.transform(base_transform * origin)
.glyph_transform(glyph_xform)
.draw(
Fill::NonZero,
glyph_run.positioned_glyphs().map(|g| vello::Glyph {
id: g.id as u32,
x: g.x,
y: g.y,
}),
);
}
}
});
drew
}
// ============================================================================
// Legacy Single-Scene Rendering (kept for backwards compatibility)
// ============================================================================
@ -691,10 +848,10 @@ pub fn render_document_with_transform(
for layer in document.visible_layers() {
if any_soloed {
if layer.soloed() {
render_layer(document, time, layer, scene, base_transform, 1.0, image_cache, video_manager, None);
render_layer(document, time, layer, scene, base_transform, 1.0, image_cache, video_manager, None, None);
}
} else {
render_layer(document, time, layer, scene, base_transform, 1.0, image_cache, video_manager, None);
render_layer(document, time, layer, scene, base_transform, 1.0, image_cache, video_manager, None, None);
}
}
}
@ -755,10 +912,11 @@ fn render_layer(
image_cache: &mut ImageCache,
video_manager: &std::sync::Arc<std::sync::Mutex<crate::video::VideoManager>>,
camera_frame: Option<&crate::webcam::CaptureFrame>,
mut extract: Option<&mut VideoExtract>,
) {
match layer {
AnyLayer::Vector(vector_layer) => {
render_vector_layer(document, time, vector_layer, scene, base_transform, parent_opacity, image_cache, video_manager)
render_vector_layer(document, time, vector_layer, scene, base_transform, parent_opacity, image_cache, video_manager, extract)
}
AnyLayer::Audio(_) => {
// Audio layers don't render visually
@ -766,20 +924,27 @@ fn render_layer(
AnyLayer::Video(video_layer) => {
let mut video_mgr = video_manager.lock().unwrap();
let layer_camera_frame = if video_layer.camera_enabled { camera_frame } else { None };
render_video_layer(document, time, video_layer, scene, base_transform, parent_opacity, &mut video_mgr, layer_camera_frame);
render_video_layer(document, time, video_layer, scene, base_transform, parent_opacity, &mut video_mgr, layer_camera_frame, extract);
}
AnyLayer::Effect(_) => {
// Effect layers are processed during GPU compositing, not rendered to scene
}
AnyLayer::Group(group_layer) => {
// Render each child layer in the group
// Render each child layer in the group, passing the extract sink down.
for child in &group_layer.children {
render_layer(document, time, child, scene, base_transform, parent_opacity, image_cache, video_manager, camera_frame);
render_layer(document, time, child, scene, base_transform, parent_opacity, image_cache, video_manager, camera_frame, extract.as_deref_mut());
}
}
AnyLayer::Raster(raster_layer) => {
// Raster is non-video content — force the Vello fallback if extracting.
if let Some(ex) = extract.as_deref_mut() { ex.drew_other = true; }
render_raster_layer_to_scene(raster_layer, time, scene, base_transform);
}
AnyLayer::Text(text_layer) => {
// Text is non-video content — force the Vello fallback if extracting.
if let Some(ex) = extract.as_deref_mut() { ex.drew_other = true; }
render_text_layer_to_scene(text_layer, time, scene, base_transform);
}
}
}
@ -816,6 +981,7 @@ pub fn render_single_clip_instance(
render_clip_instance(
document, time, clip_instance, layer_opacity, scene, base_transform,
&vector_layer.layer.animation_data, image_cache, video_manager, group_end_time,
None, // edit-inside-clip overlay keeps the Vello path
);
}
@ -831,6 +997,7 @@ fn render_clip_instance(
image_cache: &mut ImageCache,
video_manager: &std::sync::Arc<std::sync::Mutex<crate::video::VideoManager>>,
group_end_time: Option<f64>,
mut extract: Option<&mut VideoExtract>,
) {
// Try to find the clip in the document's clip libraries
// For now, only handle VectorClips (VideoClip and AudioClip rendering not yet implemented)
@ -972,7 +1139,7 @@ fn render_clip_instance(
if !layer_node.data.visible() {
continue;
}
render_layer(document, clip_time, &layer_node.data, scene, instance_transform, clip_opacity, image_cache, video_manager, None);
render_layer(document, clip_time, &layer_node.data, scene, instance_transform, clip_opacity, image_cache, video_manager, None, extract.as_deref_mut());
}
}
@ -986,6 +1153,7 @@ fn render_video_layer(
parent_opacity: f64,
video_manager: &mut crate::video::VideoManager,
camera_frame: Option<&crate::webcam::CaptureFrame>,
mut extract: Option<&mut VideoExtract>,
) {
use crate::animation::TransformProperty;
@ -1008,8 +1176,9 @@ fn render_video_layer(
continue; // Clip instance not active at this time
};
// Get video frame from VideoManager
let Some(frame) = video_manager.get_frame(&clip_instance.clip_id, clip_time) else {
// Get video frame from VideoManager at the output (export/preview) resolution.
let (target_w, target_h) = video_decode_target(document, base_transform);
let Some(frame) = video_manager.get_frame(&clip_instance.clip_id, clip_time, target_w, target_h) else {
continue; // Frame not available
};
@ -1151,7 +1320,19 @@ fn render_video_layer(
Affine::IDENTITY
};
// Render video frame as image fill
// Extract to the GPU Video path when a sink is present; otherwise bake into
// the Vello scene. The combined frame-pixel → document transform is
// instance_transform * brush_transform (matching the top-level Video path).
if let Some(ex) = extract.as_deref_mut() {
ex.instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(),
gpu: frame.gpu.clone(),
width: frame.width,
height: frame.height,
transform: instance_transform * brush_transform,
opacity: final_opacity,
});
} else {
scene.fill(
Fill::NonZero,
instance_transform,
@ -1159,6 +1340,7 @@ fn render_video_layer(
Some(brush_transform),
&video_rect,
);
}
clip_rendered = true;
}
@ -1194,6 +1376,18 @@ fn render_video_layer(
* Affine::translate((offset_x, offset_y))
* Affine::scale(uniform_scale);
// preview_transform maps frame-pixel space → document directly, so it
// is exactly the instance transform for the GPU path.
if let Some(ex) = extract.as_deref_mut() {
ex.instances.push(VideoRenderInstance {
rgba_data: frame.rgba_data.clone(),
gpu: None, // webcam frames are always CPU
width: frame.width,
height: frame.height,
transform: preview_transform,
opacity: final_opacity,
});
} else {
scene.fill(
Fill::NonZero,
preview_transform,
@ -1203,6 +1397,7 @@ fn render_video_layer(
);
}
}
}
}
/// Compute start/end canvas points for a linear gradient across a bounding box.
@ -1210,7 +1405,7 @@ fn render_video_layer(
/// The axis is centred on the bbox midpoint and oriented at `angle_deg` degrees
/// (0 = left→right, 90 = top→bottom). The axis extends ± half the bbox diagonal
/// so the gradient covers the entire shape regardless of angle.
fn gradient_bbox_endpoints(angle_deg: f32, bbox: kurbo::Rect) -> (kurbo::Point, kurbo::Point) {
pub(crate) fn gradient_bbox_endpoints(angle_deg: f32, bbox: kurbo::Rect) -> (kurbo::Point, kurbo::Point) {
let cx = bbox.center().x;
let cy = bbox.center().y;
let dx = bbox.width();
@ -1352,6 +1547,7 @@ fn render_vector_layer(
parent_opacity: f64,
image_cache: &mut ImageCache,
video_manager: &std::sync::Arc<std::sync::Mutex<crate::video::VideoManager>>,
mut extract: Option<&mut VideoExtract>,
) {
// Cascade opacity: parent_opacity × layer.opacity
let layer_opacity = parent_opacity * layer.layer.opacity;
@ -1359,6 +1555,10 @@ fn render_vector_layer(
// Render the layer's own VectorGraph (loose shapes) first, then clip instances
// (groups / movie clips) on top. Shape tweens are applied here.
if let Some(graph) = layer.tweened_graph_at(time) {
// Loose vector geometry (and any image-asset fills) is non-video content —
// force the Vello fallback. Conservative: a present-but-empty graph still
// trips this, which only costs the fallback, never correctness.
if let Some(ex) = extract.as_deref_mut() { ex.drew_other = true; }
render_vector_graph(&graph, scene, base_transform, layer_opacity, document, image_cache);
}
@ -1370,7 +1570,7 @@ fn render_vector_layer(
let frame_duration = 1.0 / document.framerate;
layer.group_visibility_end(&clip_instance.id, clip_instance.timeline_start, frame_duration)
});
render_clip_instance(document, time, clip_instance, layer_opacity, scene, base_transform, &layer.layer.animation_data, image_cache, video_manager, group_end_time);
render_clip_instance(document, time, clip_instance, layer_opacity, scene, base_transform, &layer.layer.animation_data, image_cache, video_manager, group_end_time, extract.as_deref_mut());
}
}
@ -1761,7 +1961,9 @@ fn render_vector_content_cpu(
render_vector_content_cpu(document, time, child, pixmap, base_transform, parent_opacity, image_cache);
}
}
AnyLayer::Audio(_) | AnyLayer::Video(_) | AnyLayer::Effect(_) | AnyLayer::Raster(_) => {}
// Text is not rendered in the tiny-skia CPU fallback (GPU path only) for v1.
AnyLayer::Audio(_) | AnyLayer::Video(_) | AnyLayer::Effect(_) | AnyLayer::Raster(_)
| AnyLayer::Text(_) => {}
}
}

View File

@ -218,3 +218,274 @@ fn cubic_to_svg_path(curve: &CubicBez) -> String {
curve.p3.x, curve.p3.y,
)
}
// ===========================================================================
// Document / VectorGraph → SVG (the current model). The functions above target
// the legacy DCEL and are kept only for the clipboard stub.
// ===========================================================================
use crate::document::Document;
use crate::gradient::{GradientExtend, GradientType, ShapeGradient};
use crate::layer::AnyLayer;
use crate::shape::{Cap, FillRule, Join, ShapeColor};
use crate::vector_graph::{FillId, VectorGraph};
use kurbo::{BezPath, PathEl, Rect, Shape};
/// Serialize the document's **vector** content to a standalone SVG string, at document time `time`.
/// Vector layers (and groups of them) only — raster/video/audio/effect layers are skipped (a later
/// pass can rasterize them to `<image>`). Animation is a single static frame at `time`.
pub fn document_to_svg(document: &Document, time: f64) -> String {
let (w, h) = (document.width, document.height);
let mut defs = String::new();
let mut body = String::new();
let mut grad_n = 0usize;
// Opaque background rect (skip if the document background is transparent).
let bg = document.background_color;
if bg.a > 0 {
body.push_str(&format!(
r#"<rect x="0" y="0" width="{w:.3}" height="{h:.3}" {}/>"#,
fill_attrs(&bg)
));
}
for layer in &document.root.children {
layer_to_svg(layer, time, 1.0, &mut body, &mut defs, &mut grad_n);
}
format!(
concat!(
r#"<svg xmlns="http://www.w3.org/2000/svg" width="{:.0}" height="{:.0}" "#,
r#"viewBox="0 0 {:.3} {:.3}"><defs>{}</defs>{}</svg>"#
),
w, h, w, h, defs, body
)
}
/// Append one layer's SVG. Recurses into groups (`<g>`); other non-vector layer types are skipped.
fn layer_to_svg(layer: &AnyLayer, time: f64, parent_opacity: f64, body: &mut String, defs: &mut String, grad_n: &mut usize) {
match layer {
AnyLayer::Vector(vl) => {
if !vl.layer.visible {
return; // hidden layers are not rendered, so don't export them
}
let opacity = parent_opacity * vl.layer.opacity;
if let Some(graph) = vl.tweened_graph_at(time) {
let wrap = opacity < 0.999;
if wrap {
body.push_str(&format!(r#"<g opacity="{opacity:.4}">"#));
}
vector_graph_to_svg(&graph, body, defs, grad_n);
if wrap {
body.push_str("</g>");
}
}
// NOTE: placed clip instances (nested clips with their own transform) are not yet
// exported — a refinement once loose-geometry export is verified.
}
AnyLayer::Group(g) => {
if !g.layer.visible {
return;
}
// Render children first; only emit the <g> wrapper if it has exportable content
// (avoids empty groups when every child is a non-vector/hidden layer).
let mut inner = String::new();
for child in &g.children {
layer_to_svg(child, time, 1.0, &mut inner, defs, grad_n);
}
if !inner.is_empty() {
let opacity = parent_opacity * g.layer.opacity;
body.push_str(&format!(r#"<g opacity="{opacity:.4}">"#));
body.push_str(&inner);
body.push_str("</g>");
}
}
// Raster/Video/Audio/Effect have no lossless vector representation — skipped this pass.
_ => {}
}
}
/// Emit a vector graph's fills (`<path fill>`) and stroked edges (`<path stroke>`) into `body`,
/// accumulating any gradients into `defs`. Geometry is in document space (no per-layer transform).
fn vector_graph_to_svg(graph: &VectorGraph, body: &mut String, defs: &mut String, grad_n: &mut usize) {
// Fills first (drawn under strokes, matching the renderer).
for (i, fill) in graph.fills.iter().enumerate() {
if fill.deleted {
continue;
}
let path = graph.fill_to_bezpath(FillId(i as u32));
let d = bezpath_to_d(&path);
if d.is_empty() {
continue;
}
let rule = match fill.fill_rule {
FillRule::NonZero => "nonzero",
FillRule::EvenOdd => "evenodd",
};
if let Some(grad) = &fill.gradient_fill {
let id = format!("grad{}", *grad_n);
*grad_n += 1;
defs.push_str(&gradient_to_svg(grad, &id, path.bounding_box()));
body.push_str(&format!(r#"<path fill="url(#{id})" fill-rule="{rule}" d="{d}"/>"#));
} else if fill.image_fill.is_some() {
// Image fills need <image>/<pattern> + asset embedding — skipped this (vector-only) pass.
continue;
} else if let Some(c) = &fill.color {
body.push_str(&format!(r#"<path {} fill-rule="{rule}" d="{d}"/>"#, fill_attrs(c)));
}
}
// Strokes: one <path> per stroked edge (each edge may carry its own style).
for edge in &graph.edges {
if edge.deleted {
continue;
}
if let (Some(style), Some(color)) = (&edge.stroke_style, &edge.stroke_color) {
let d = cubic_to_svg_path(&edge.curve);
body.push_str(&format!(
r#"<path fill="none" {} stroke-width="{:.3}" stroke-linecap="{}" stroke-linejoin="{}" stroke-miterlimit="{:.3}" d="{d}"/>"#,
stroke_attrs(color), style.width, cap_str(style.cap), join_str(style.join), style.miter_limit
));
}
}
}
/// `<linearGradient>` / `<radialGradient>` definition matching the renderer's start/end semantics.
fn gradient_to_svg(grad: &ShapeGradient, id: &str, bbox: Rect) -> String {
use kurbo::Point;
// Mirror renderer.rs: explicit world endpoints if present (radial reflects the edge through the
// center so midpoint(start,end) == center), else derive from angle + bbox.
let (start, end) = match (grad.start_world, grad.end_world) {
(Some((sx, sy)), Some((ex, ey))) => match grad.kind {
GradientType::Linear => (Point::new(sx, sy), Point::new(ex, ey)),
GradientType::Radial => (Point::new(2.0 * sx - ex, 2.0 * sy - ey), Point::new(ex, ey)),
},
_ => crate::renderer::gradient_bbox_endpoints(grad.angle, bbox),
};
let stops: String = grad
.stops
.iter()
.map(|s| {
format!(
r##"<stop offset="{:.4}" stop-color="#{:02x}{:02x}{:02x}" stop-opacity="{:.4}"/>"##,
s.position, s.color.r, s.color.g, s.color.b, s.color.a as f32 / 255.0
)
})
.collect();
let spread = match grad.extend {
GradientExtend::Pad => "pad",
GradientExtend::Reflect => "reflect",
GradientExtend::Repeat => "repeat",
};
match grad.kind {
GradientType::Linear => format!(
r#"<linearGradient id="{id}" gradientUnits="userSpaceOnUse" x1="{:.3}" y1="{:.3}" x2="{:.3}" y2="{:.3}" spreadMethod="{spread}">{stops}</linearGradient>"#,
start.x, start.y, end.x, end.y
),
GradientType::Radial => {
let (cx, cy) = ((start.x + end.x) * 0.5, (start.y + end.y) * 0.5);
let r = (((end.x - start.x).powi(2) + (end.y - start.y).powi(2)).sqrt()) * 0.5;
format!(
r#"<radialGradient id="{id}" gradientUnits="userSpaceOnUse" cx="{cx:.3}" cy="{cy:.3}" r="{r:.3}" spreadMethod="{spread}">{stops}</radialGradient>"#
)
}
}
}
/// kurbo `BezPath` → SVG path-data string (`M/L/Q/C/Z`).
fn bezpath_to_d(path: &BezPath) -> String {
let mut d = String::new();
for el in path.elements() {
match el {
PathEl::MoveTo(p) => d.push_str(&format!("M{:.3} {:.3} ", p.x, p.y)),
PathEl::LineTo(p) => d.push_str(&format!("L{:.3} {:.3} ", p.x, p.y)),
PathEl::QuadTo(p1, p) => d.push_str(&format!("Q{:.3} {:.3} {:.3} {:.3} ", p1.x, p1.y, p.x, p.y)),
PathEl::CurveTo(p1, p2, p) => d.push_str(&format!(
"C{:.3} {:.3} {:.3} {:.3} {:.3} {:.3} ",
p1.x, p1.y, p2.x, p2.y, p.x, p.y
)),
PathEl::ClosePath => d.push_str("Z "),
}
}
d.trim_end().to_string()
}
// sRGB color → SVG attributes. Hex color + a separate `*-opacity` for max compatibility (Inkscape).
fn fill_attrs(c: &ShapeColor) -> String {
if c.a == 255 {
format!(r##"fill="#{:02x}{:02x}{:02x}""##, c.r, c.g, c.b)
} else {
format!(r##"fill="#{:02x}{:02x}{:02x}" fill-opacity="{:.4}""##, c.r, c.g, c.b, c.a as f32 / 255.0)
}
}
fn stroke_attrs(c: &ShapeColor) -> String {
if c.a == 255 {
format!(r##"stroke="#{:02x}{:02x}{:02x}""##, c.r, c.g, c.b)
} else {
format!(r##"stroke="#{:02x}{:02x}{:02x}" stroke-opacity="{:.4}""##, c.r, c.g, c.b, c.a as f32 / 255.0)
}
}
fn cap_str(cap: Cap) -> &'static str {
match cap {
Cap::Butt => "butt",
Cap::Round => "round",
Cap::Square => "square",
}
}
fn join_str(join: Join) -> &'static str {
match join {
Join::Miter => "miter",
Join::Round => "round",
Join::Bevel => "bevel",
}
}
#[cfg(test)]
mod export_tests {
use super::*;
use crate::shape::{ShapeColor, StrokeStyle};
use crate::vector_graph::{Direction, VectorGraph};
use kurbo::{CubicBez, Point};
fn line(a: Point, b: Point) -> CubicBez {
// Degenerate cubic representing a straight segment (matches our model).
CubicBez::new(a, a.lerp(b, 1.0 / 3.0), a.lerp(b, 2.0 / 3.0), b)
}
#[test]
fn solid_triangle_fill_and_stroke() {
let mut g = VectorGraph::new();
let p0 = Point::new(10.0, 10.0);
let p1 = Point::new(90.0, 10.0);
let p2 = Point::new(50.0, 80.0);
let v0 = g.alloc_vertex(p0);
let v1 = g.alloc_vertex(p1);
let v2 = g.alloc_vertex(p2);
let stroke = Some(StrokeStyle { width: 2.0, ..Default::default() });
let scol = Some(ShapeColor::rgb(0, 0, 0));
let e0 = g.alloc_edge(line(p0, p1), v0, v1, stroke.clone(), scol);
let e1 = g.alloc_edge(line(p1, p2), v1, v2, stroke.clone(), scol);
let e2 = g.alloc_edge(line(p2, p0), v2, v0, stroke.clone(), scol);
g.alloc_fill(
vec![(e0, Direction::Forward), (e1, Direction::Forward), (e2, Direction::Forward)],
ShapeColor::rgb(255, 0, 0),
crate::shape::FillRule::NonZero,
);
let mut body = String::new();
let mut defs = String::new();
let mut n = 0;
vector_graph_to_svg(&g, &mut body, &mut defs, &mut n);
assert!(body.contains(r##"fill="#ff0000""##), "fill color missing: {body}");
assert!(body.contains(r#"fill-rule="nonzero""#), "fill-rule missing: {body}");
assert!(body.contains(r#"fill="none""#), "stroke path missing: {body}");
assert!(body.contains(r#"stroke-width="2.000""#), "stroke width missing: {body}");
assert!(defs.is_empty(), "no gradients expected: {defs}");
// 1 fill path + 3 stroked edges = 4 <path> elements.
assert_eq!(body.matches("<path").count(), 4, "{body}");
}
}

View File

@ -0,0 +1,145 @@
//! Text layer for Lightningbeam
//!
//! A text layer holds a single editable text field inside a resizable box,
//! with editable size, color, and font. Text is rendered as vector glyphs
//! (via parley + Vello) so it composites through the same path as vector art.
//!
//! The text/style fields are grouped in [`TextContent`] so they can move to a
//! per-keyframe model later without touching call sites; v1 stores a single
//! static instance and reads it through [`TextLayer::content_at`].
use crate::layer::{Layer, LayerTrait, LayerType};
use kurbo::Point;
use serde::{Deserialize, Serialize};
use std::hash::{Hash, Hasher};
use uuid::Uuid;
/// Horizontal alignment of text within the box.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum TextAlign {
Left,
Center,
Right,
Justify,
}
impl Default for TextAlign {
fn default() -> Self {
TextAlign::Left
}
}
/// The text content + styling for a text layer.
///
/// Grouped as its own struct so a future keyframed model can store
/// `Vec<TextKeyframe>` of these without changing the layer's public shape.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct TextContent {
/// The text string.
pub text: String,
/// Font size in pixels (document space).
pub font_size: f64,
/// Fill color, linear RGBA in 0..=1.
pub color: [f32; 4],
/// Logical font family name. Empty string = the bundled default font.
pub font_family: String,
/// Horizontal alignment within the box.
pub align: TextAlign,
}
impl Default for TextContent {
fn default() -> Self {
Self {
text: String::new(),
font_size: 48.0,
color: [1.0, 1.0, 1.0, 1.0],
font_family: String::new(),
align: TextAlign::Left,
}
}
}
impl TextContent {
/// A stable hash of everything that affects shaped layout (everything except
/// `color`, which only affects the brush, not glyph positions). Used to key
/// the renderer's external parley-layout cache, including the wrap width.
pub fn layout_hash(&self, box_width: f64) -> u64 {
let mut h = std::collections::hash_map::DefaultHasher::new();
self.text.hash(&mut h);
self.font_size.to_bits().hash(&mut h);
self.font_family.hash(&mut h);
self.align.hash(&mut h);
box_width.to_bits().hash(&mut h);
h.finish()
}
}
impl Hash for TextAlign {
fn hash<H: Hasher>(&self, state: &mut H) {
(*self as u8).hash(state);
}
}
/// A text layer: a single resizable text box.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct TextLayer {
/// Base layer properties (id, name, opacity, visibility, animation data, …).
pub layer: Layer,
/// Top-left of the text box, in layer/clip-local space.
pub box_origin: Point,
/// Box width in document units (drives text wrapping).
pub box_width: f64,
/// Box height in document units.
pub box_height: f64,
/// The text content + styling (single static instance for v1).
pub content: TextContent,
}
impl TextLayer {
/// Default text-box dimensions for a freshly created layer.
pub const DEFAULT_WIDTH: f64 = 300.0;
pub const DEFAULT_HEIGHT: f64 = 100.0;
/// Create a new, empty text layer with the given name, positioned at `origin`.
pub fn new(name: impl Into<String>, origin: Point) -> Self {
Self {
layer: Layer::new(LayerType::Text, name),
box_origin: origin,
box_width: Self::DEFAULT_WIDTH,
box_height: Self::DEFAULT_HEIGHT,
content: TextContent::default(),
}
}
/// Read the active text content at `time`. v1 is static, so `time` is unused;
/// a future keyframed model will select the active keyframe here.
pub fn content_at(&self, _time: f64) -> &TextContent {
&self.content
}
/// Mutable access to the active text content at `time`.
pub fn content_at_mut(&mut self, _time: f64) -> &mut TextContent {
&mut self.content
}
}
// Delegate all LayerTrait methods to self.layer (mirrors RasterLayer).
impl LayerTrait for TextLayer {
fn id(&self) -> Uuid { self.layer.id }
fn name(&self) -> &str { &self.layer.name }
fn set_name(&mut self, name: String) { self.layer.name = name; }
fn has_custom_name(&self) -> bool { self.layer.has_custom_name }
fn set_has_custom_name(&mut self, custom: bool) { self.layer.has_custom_name = custom; }
fn visible(&self) -> bool { self.layer.visible }
fn set_visible(&mut self, visible: bool) { self.layer.visible = visible; }
fn opacity(&self) -> f64 { self.layer.opacity }
fn set_opacity(&mut self, opacity: f64) { self.layer.opacity = opacity; }
fn volume(&self) -> f64 { self.layer.volume }
fn set_volume(&mut self, volume: f64) { self.layer.volume = volume; }
fn muted(&self) -> bool { self.layer.muted }
fn set_muted(&mut self, muted: bool) { self.layer.muted = muted; }
fn soloed(&self) -> bool { self.layer.soloed }
fn set_soloed(&mut self, soloed: bool) { self.layer.soloed = soloed; }
fn locked(&self) -> bool { self.layer.locked }
fn set_locked(&mut self, locked: bool) { self.layer.locked = locked; }
}

View File

@ -381,7 +381,9 @@ impl Tool {
use crate::layer::LayerType;
match layer_type {
None | Some(LayerType::Vector) => Tool::all(),
Some(LayerType::Audio) | Some(LayerType::Video) => &[Tool::Select, Tool::Split],
// The Text tool is available on audio/video/raster too: clicking the
// stage with it there creates a new top-level text layer.
Some(LayerType::Audio) | Some(LayerType::Video) => &[Tool::Select, Tool::Split, Tool::Text],
Some(LayerType::Raster) => &[
// Brush tools
Tool::Draw, Tool::Pencil, Tool::Pen, Tool::Airbrush,
@ -397,9 +399,10 @@ impl Tool {
// Transform
Tool::Transform, Tool::Warp, Tool::Liquify,
// Utility
Tool::Eyedropper,
Tool::Eyedropper, Tool::Text,
],
_ => &[Tool::Select],
Some(LayerType::Text) => &[Tool::Select, Tool::Text, Tool::Transform],
_ => &[Tool::Select, Tool::Text],
}
}

View File

@ -351,6 +351,29 @@ impl VectorGraph {
}
}
/// Free any vertex no longer referenced by a non-deleted edge (e.g. after deleting a shape's
/// edges), so stale vertices don't linger as snap targets.
pub fn gc_isolated_vertices(&mut self) {
let mut referenced = vec![false; self.vertices.len()];
for e in &self.edges {
if e.deleted {
continue;
}
for v in e.vertices.iter() {
if !v.is_none() {
referenced[v.idx()] = true;
}
}
}
let to_free: Vec<VertexId> = (0..self.vertices.len())
.filter(|&i| !self.vertices[i].deleted && !referenced[i])
.map(|i| VertexId(i as u32))
.collect();
for vid in to_free {
self.free_vertex(vid);
}
}
// -------------------------------------------------------------------
// Fill / hit-test queries
// -------------------------------------------------------------------
@ -1302,21 +1325,26 @@ impl VectorGraph {
/// bouncing across near-coincident duplicate edges). These are zero-area and would make
/// `boundary_to_bezpath` render a stray hair; collapsing them yields a simple loop.
fn collapse_boundary_spikes(&self, face: &mut Vec<(EdgeId, Direction)>) {
let dstart = |entry: &(EdgeId, Direction)| -> Point {
// The four control points of an entry's curve in its traversal order.
let traversed = |entry: &(EdgeId, Direction)| -> [Point; 4] {
let c = self.edges[entry.0.idx()].curve;
match entry.1 {
Direction::Forward => c.p0,
Direction::Backward => c.p3,
}
};
let dend = |entry: &(EdgeId, Direction)| -> Point {
let c = self.edges[entry.0.idx()].curve;
match entry.1 {
Direction::Forward => c.p3,
Direction::Backward => c.p0,
Direction::Forward => [c.p0, c.p1, c.p2, c.p3],
Direction::Backward => [c.p3, c.p2, c.p1, c.p0],
}
};
const EPS: f64 = 0.5;
// Entries i and j cancel only when j is the *exact reverse* of i — every control point of
// j matches the mirror of i. Testing endpoints alone would also collapse a genuine
// lens/sliver (two distinct edges that merely share near-coincident endpoints), silently
// deleting real boundary geometry and dropping the fill.
let reverses = |a: &(EdgeId, Direction), b: &(EdgeId, Direction)| -> bool {
let (ca, cb) = (traversed(a), traversed(b));
(0..4).all(|k| {
let (p, q) = (ca[k], cb[3 - k]);
(p.x - q.x).hypot(p.y - q.y) < EPS
})
};
loop {
let n = face.len();
if n < 2 {
@ -1325,10 +1353,7 @@ impl VectorGraph {
let mut collapsed = false;
for i in 0..n {
let j = (i + 1) % n;
// entries i and j cancel when j ends back at i's start.
let si = dstart(&face[i]);
let ej = dend(&face[j]);
if (si.x - ej.x).hypot(si.y - ej.y) < EPS {
if reverses(&face[i], &face[j]) {
let (hi, lo) = if i > j { (i, j) } else { (j, i) };
face.remove(hi);
face.remove(lo);

File diff suppressed because it is too large Load Diff

View File

@ -205,3 +205,51 @@ fn overwrite_media_replaces_chunks() {
assert_eq!(archive.read_media_full(id).unwrap(), vec![2u8; 50]);
let _ = std::fs::remove_file(&path);
}
#[test]
fn packed_video_from_path_roundtrips() {
// Simulates the save path: stream a video file into a MediaKind::Video blob,
// then read it back (frames/audio decode would open this via the AVIO shim).
let path = temp_db_path("video_pack");
let mut src = std::env::temp_dir();
src.push(format!("beam_video_src_{}.mp4", std::process::id()));
let bytes: Vec<u8> = (0..(9 * 1024 * 1024u32)).map(|i| (i % 251) as u8).collect();
std::fs::write(&src, &bytes).unwrap();
let id = Uuid::new_v4();
let mut archive = BeamArchive::create(&path).unwrap();
{
let txn = archive.transaction().unwrap();
txn.put_media_packed_from_path(
id,
MediaKind::Video,
"mp4",
&src,
MediaMeta { width: Some(1920), height: Some(1080), ..Default::default() },
)
.unwrap();
txn.commit().unwrap();
}
let info = archive.media_info(id).unwrap().expect("video media row");
assert_eq!(info.kind, MediaKind::Video);
assert_eq!(info.storage, MediaStorage::Packed);
assert_eq!(info.codec, "mp4");
assert_eq!(info.total_len, bytes.len() as u64);
assert_eq!(info.width, Some(1920));
assert_eq!(info.height, Some(1080));
assert_eq!(archive.read_media_full(id).unwrap(), bytes);
// Streaming read mirrors how the decoder pulls bytes via its BlobReader.
let mut reader = archive.open_blob_reader(&path, id).unwrap();
let mut streamed = Vec::new();
reader.read_to_end(&mut streamed).unwrap();
assert_eq!(streamed, bytes);
reader.seek(SeekFrom::Start(5 * 1024 * 1024)).unwrap();
let mut buf = [0u8; 4];
reader.read_exact(&mut buf).unwrap();
assert_eq!(buf[0], ((5 * 1024 * 1024u32) % 251) as u8);
let _ = std::fs::remove_file(&src);
let _ = std::fs::remove_file(&path);
}

View File

@ -309,30 +309,34 @@ fn test_clip_time_remapping() {
document.root.add_child(AnyLayer::Vector(layer));
// timeline_start is in beats; at 60 BPM 1 beat == 1 second, so the tempo map
// is identity and these timeline values read directly as seconds.
let tempo_map = lightningbeam_core::tempo_map::TempoMap::constant(60.0);
// Test time remapping
// At timeline time 5.0, clip internal time should be 2.0 (trim_start)
let clip_time = instance.remap_time(5.0, clip_duration);
let clip_time = instance.remap_time(5.0, clip_duration, &tempo_map);
assert_eq!(clip_time, Some(2.0));
// At timeline time 6.0, clip internal time should be 3.0
let clip_time = instance.remap_time(6.0, clip_duration);
let clip_time = instance.remap_time(6.0, clip_duration, &tempo_map);
assert_eq!(clip_time, Some(3.0));
// At timeline time 10.999, clip internal time should be just under 8.0
// The clip plays from timeline 5.0 to 11.0 (exclusive end)
// At timeline 10.999: relative_time = 5.999, content_time = 5.999
// Since content_window = 6.0, we get: trim_start + 5.999 = 7.999
let clip_time = instance.remap_time(10.999, clip_duration);
let clip_time = instance.remap_time(10.999, clip_duration, &tempo_map);
assert!(clip_time.is_some());
let time = clip_time.unwrap();
assert!(time > 7.9 && time < 8.0, "Expected ~7.999, got {}", time);
// At timeline time 11.0 (exact end), clip should be past its end (None)
// because the range is [timeline_start, timeline_start + effective_duration)
let clip_time = instance.remap_time(11.0, clip_duration);
let clip_time = instance.remap_time(11.0, clip_duration, &tempo_map);
assert_eq!(clip_time, None);
// At timeline time 4.0, clip hasn't started yet (None)
let clip_time = instance.remap_time(4.0, clip_duration);
let clip_time = instance.remap_time(4.0, clip_duration, &tempo_map);
assert_eq!(clip_time, None);
}

View File

@ -0,0 +1,81 @@
//! Integration test for the packed-video streaming path:
//! SQLite `MediaKind::Video` blob -> `BlobReader` -> `ffmpeg-blob-io` AVIO shim ->
//! ffmpeg `Input`. This is exactly what `video.rs`'s `VideoSource::Packed` does to
//! decode frames (and what `daw-backend` does for the embedded audio), minus the
//! thin wrapper. We use a hand-built WAV as the packed container — FFmpeg must
//! demux it through our blob reader, proving the production byte path end to end.
//!
//! (Decoding real video frames is left to user runtime verification; here we prove
//! the container streams from the blob and exposes its streams.)
use ffmpeg_blob_io::BlobInput;
use lightningbeam_core::beam_archive::{BeamArchive, MediaKind, MediaMeta, MediaStorage};
use std::sync::atomic::{AtomicU64, Ordering};
use uuid::Uuid;
fn temp_db_path(tag: &str) -> std::path::PathBuf {
static N: AtomicU64 = AtomicU64::new(0);
let n = N.fetch_add(1, Ordering::Relaxed);
let mut p = std::env::temp_dir();
p.push(format!("packed_video_it_{}_{}_{}.beam", std::process::id(), tag, n));
let _ = std::fs::remove_file(&p);
p
}
/// Minimal 16-bit PCM WAV (a real, demuxable container).
fn make_wav(sample_rate: u32, channels: u16, samples: &[i16]) -> Vec<u8> {
let bits: u16 = 16;
let block_align: u16 = channels * (bits / 8);
let byte_rate: u32 = sample_rate * block_align as u32;
let data_len: u32 = (samples.len() * 2) as u32;
let mut v = Vec::new();
v.extend_from_slice(b"RIFF");
v.extend_from_slice(&(36 + data_len).to_le_bytes());
v.extend_from_slice(b"WAVE");
v.extend_from_slice(b"fmt ");
v.extend_from_slice(&16u32.to_le_bytes());
v.extend_from_slice(&1u16.to_le_bytes());
v.extend_from_slice(&channels.to_le_bytes());
v.extend_from_slice(&sample_rate.to_le_bytes());
v.extend_from_slice(&byte_rate.to_le_bytes());
v.extend_from_slice(&block_align.to_le_bytes());
v.extend_from_slice(&bits.to_le_bytes());
v.extend_from_slice(b"data");
v.extend_from_slice(&data_len.to_le_bytes());
for s in samples {
v.extend_from_slice(&s.to_le_bytes());
}
v
}
#[test]
fn packed_media_streams_through_avio_to_ffmpeg() {
let path = temp_db_path("stream");
let id = Uuid::new_v4();
let samples: Vec<i16> = (0..4000).map(|i| ((i % 200) as i16 - 100) * 100).collect();
let container = make_wav(8000, 1, &samples);
// Pack as a Video media row (the save path does this for real videos).
let mut archive = BeamArchive::create(&path).unwrap();
archive
.put_media_packed(id, MediaKind::Video, "wav", &container, MediaMeta::default())
.unwrap();
let info = archive.media_info(id).unwrap().unwrap();
assert_eq!(info.kind, MediaKind::Video);
assert_eq!(info.storage, MediaStorage::Packed);
drop(archive);
// Reproduce VideoSource::Packed::open(): fresh read-only archive + blob reader.
let archive = BeamArchive::open(&path).unwrap();
let hint = archive.media_info(id).unwrap().map(|i| i.codec);
let reader = archive.open_blob_reader(&path, id).unwrap();
let input = BlobInput::open(Box::new(reader), hint.as_deref())
.expect("open the packed container by streaming from the SQLite blob");
assert!(
input.streams().count() >= 1,
"demuxer found streams via the blob-backed AVIO shim"
);
let _ = std::fs::remove_file(&path);
}

View File

@ -62,22 +62,22 @@ fn test_selection_of_shape_instances() {
let mut selection = Selection::new();
// Select first shape instance
selection.add_shape_instance(shape_ids[0]);
assert!(selection.contains_shape_instance(&shape_ids[0]));
assert!(!selection.contains_shape_instance(&shape_ids[1]));
assert_eq!(selection.shape_instances().len(), 1);
selection.add_clip_instance(shape_ids[0]);
assert!(selection.contains_clip_instance(&shape_ids[0]));
assert!(!selection.contains_clip_instance(&shape_ids[1]));
assert_eq!(selection.clip_instances().len(), 1);
// Add second shape instance
selection.add_shape_instance(shape_ids[1]);
assert!(selection.contains_shape_instance(&shape_ids[0]));
assert!(selection.contains_shape_instance(&shape_ids[1]));
assert_eq!(selection.shape_instances().len(), 2);
selection.add_clip_instance(shape_ids[1]);
assert!(selection.contains_clip_instance(&shape_ids[0]));
assert!(selection.contains_clip_instance(&shape_ids[1]));
assert_eq!(selection.clip_instances().len(), 2);
// Toggle first shape instance (deselect)
selection.toggle_shape_instance(shape_ids[0]);
assert!(!selection.contains_shape_instance(&shape_ids[0]));
assert!(selection.contains_shape_instance(&shape_ids[1]));
assert_eq!(selection.shape_instances().len(), 1);
selection.toggle_clip_instance(shape_ids[0]);
assert!(!selection.contains_clip_instance(&shape_ids[0]));
assert!(selection.contains_clip_instance(&shape_ids[1]));
assert_eq!(selection.clip_instances().len(), 1);
}
#[test]
@ -108,51 +108,25 @@ fn test_mixed_selection() {
let mut selection = Selection::new();
// Select both shapes and clips
selection.add_shape_instance(shape_ids[0]);
selection.add_shape_instance(shape_ids[1]);
// Selection is unified: shapes and clips are both clip instances.
selection.add_clip_instance(shape_ids[0]);
selection.add_clip_instance(shape_ids[1]);
selection.add_clip_instance(clip_ids[0]);
selection.add_clip_instance(clip_ids[1]);
assert_eq!(selection.shape_instances().len(), 2);
assert_eq!(selection.clip_instances().len(), 2);
assert_eq!(selection.clip_instances().len(), 4);
// Clear only clip instances
// Clearing clip instances clears them all.
selection.clear_clip_instances();
assert_eq!(selection.shape_instances().len(), 2);
assert_eq!(selection.clip_instances().len(), 0);
// Re-add clip
// Re-add one, then full clear.
selection.add_clip_instance(clip_ids[0]);
// Full clear
assert_eq!(selection.clip_instances().len(), 1);
selection.clear();
assert_eq!(selection.shape_instances().len(), 0);
assert_eq!(selection.clip_instances().len(), 0);
}
#[test]
fn test_select_only_shape_instance() {
let (_document, _layer_id, shape_ids, clip_ids) = setup_mixed_content_document();
let mut selection = Selection::new();
// Select multiple items
selection.add_shape_instance(shape_ids[0]);
selection.add_shape_instance(shape_ids[1]);
selection.add_clip_instance(clip_ids[0]);
// Select only shape_ids[0] - this clears ALL selections first
selection.select_only_shape_instance(shape_ids[0]);
assert!(selection.contains_shape_instance(&shape_ids[0]));
assert!(!selection.contains_shape_instance(&shape_ids[1]));
// select_only_shape_instance calls clear() so clip instances are also cleared
assert!(!selection.contains_clip_instance(&clip_ids[0]));
}
#[test]
fn test_select_only_clip_instance() {
let (_document, _layer_id, shape_ids, clip_ids) = setup_mixed_content_document();
@ -160,7 +134,27 @@ fn test_select_only_clip_instance() {
let mut selection = Selection::new();
// Select multiple items
selection.add_shape_instance(shape_ids[0]);
selection.add_clip_instance(shape_ids[0]);
selection.add_clip_instance(shape_ids[1]);
selection.add_clip_instance(clip_ids[0]);
// Select only shape_ids[0] - this clears ALL selections first
selection.select_only_clip_instance(shape_ids[0]);
assert!(selection.contains_clip_instance(&shape_ids[0]));
assert!(!selection.contains_clip_instance(&shape_ids[1]));
// select_only_shape_instance calls clear() so clip instances are also cleared
assert!(!selection.contains_clip_instance(&clip_ids[0]));
}
#[test]
fn test_select_only_clip_instance_clears_others() {
let (_document, _layer_id, shape_ids, clip_ids) = setup_mixed_content_document();
let mut selection = Selection::new();
// Select multiple items
selection.add_clip_instance(shape_ids[0]);
selection.add_clip_instance(clip_ids[0]);
selection.add_clip_instance(clip_ids[1]);
@ -170,7 +164,7 @@ fn test_select_only_clip_instance() {
assert!(selection.contains_clip_instance(&clip_ids[0]));
assert!(!selection.contains_clip_instance(&clip_ids[1]));
// select_only_clip_instance calls clear() so shape instances are also cleared
assert!(!selection.contains_shape_instance(&shape_ids[0]));
assert!(!selection.contains_clip_instance(&shape_ids[0]));
}
#[test]
@ -223,7 +217,7 @@ fn test_selection_is_empty() {
assert!(selection.is_empty());
let mut selection2 = Selection::new();
selection2.add_shape_instance(Uuid::new_v4());
selection2.add_clip_instance(Uuid::new_v4());
assert!(!selection2.is_empty());
let mut selection3 = Selection::new();
@ -239,20 +233,18 @@ fn test_selection_count() {
let id2 = Uuid::new_v4();
let clip_id = Uuid::new_v4();
selection.add_shape_instance(id1);
selection.add_shape_instance(id2);
selection.add_clip_instance(id1);
selection.add_clip_instance(id2);
selection.add_clip_instance(clip_id);
assert_eq!(selection.shape_instances().len(), 2);
assert_eq!(selection.clip_instances().len(), 1);
assert_eq!(selection.clip_instances().len(), 3);
// Remove one
selection.remove_shape_instance(&id1);
assert_eq!(selection.shape_instances().len(), 1);
// Remove one at a time.
selection.remove_clip_instance(&id1);
assert_eq!(selection.clip_instances().len(), 2);
// Remove clip
selection.remove_clip_instance(&clip_id);
assert_eq!(selection.clip_instances().len(), 0);
assert_eq!(selection.clip_instances().len(), 1);
}
#[test]
@ -261,17 +253,17 @@ fn test_duplicate_selection_handling() {
let id = Uuid::new_v4();
// Add same ID multiple times
selection.add_shape_instance(id);
selection.add_shape_instance(id);
selection.add_shape_instance(id);
selection.add_clip_instance(id);
selection.add_clip_instance(id);
selection.add_clip_instance(id);
// Should only contain one instance (dedup behavior)
assert_eq!(selection.shape_instances().len(), 1);
assert_eq!(selection.clip_instances().len(), 1);
// Same for clip instances
// A second distinct ID, also added twice, dedups to one more (total 2).
let clip_id = Uuid::new_v4();
selection.add_clip_instance(clip_id);
selection.add_clip_instance(clip_id);
assert_eq!(selection.clip_instances().len(), 1);
assert_eq!(selection.clip_instances().len(), 2);
}

View File

@ -1,6 +1,6 @@
[package]
name = "lightningbeam-editor"
version = "1.0.5-alpha"
version = "1.0.7-alpha"
edition = "2021"
description = "Multimedia editor for audio, video and 2D animation"
license = "GPL-3.0-or-later"
@ -9,6 +9,7 @@ license = "GPL-3.0-or-later"
lightningbeam-core = { path = "../lightningbeam-core" }
daw-backend = { path = "../../daw-backend" }
beamdsp = { path = "../beamdsp" }
gpu-video-encoder = { path = "../gpu-video-encoder" }
rtrb = "0.3"
cpal = "0.17"
ffmpeg-next = { version = "8.0", features = ["static"] }
@ -78,7 +79,11 @@ extended-description = "GPU-accelerated multimedia editor for audio, video and 2
license-file = ["../../LICENSE", "0"]
section = "video"
priority = "optional"
depends = "libasound2, libwayland-client0, libx11-6, libvulkan1"
# libva2/libva-drm2 are hard deps: the vaapi-enabled static ffmpeg links them (DT_NEEDED), so the
# app won't launch without them. The VA *driver* is a soft dep — absent, the export falls back to
# software — so it's a recommends (va-driver-all pulls intel-media + i965 + mesa drivers).
depends = "libasound2, libwayland-client0, libx11-6, libvulkan1, libva2, libva-drm2, libdrm2"
recommends = "va-driver-all"
assets = [
["target/release/lightningbeam-editor", "usr/bin/", "755"],
["assets/com.lightningbeam.editor.desktop", "usr/share/applications/", "644"],
@ -100,6 +105,15 @@ alsa-lib = "*"
wayland = "*"
libX11 = "*"
vulkan-loader = "*"
# Hard dep: the vaapi-enabled static ffmpeg links libva (libva.so.2 + libva-drm.so.2 + libdrm).
libva = "*"
libdrm = "*"
# Soft deps: the VA driver is only needed to actually use hardware encode; absent it, the editor
# falls back to software. (Requires a cargo-generate-rpm new enough to support weak-dep tables.)
[package.metadata.generate-rpm.recommends]
intel-media-driver = "*"
mesa-va-drivers = "*"
[[package.metadata.generate-rpm.assets]]
source = "target/release/lightningbeam-editor"

View File

@ -0,0 +1,43 @@
ISC License
Copyright (c) 2026 Lucide Icons and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
---
The following Lucide icons are derived from the Feather project:
airplay, alert-circle, alert-octagon, alert-triangle, aperture, arrow-down-circle, arrow-down-left, arrow-down-right, arrow-down, arrow-left-circle, arrow-left, arrow-right-circle, arrow-right, arrow-up-circle, arrow-up-left, arrow-up-right, arrow-up, at-sign, calendar, cast, check, chevron-down, chevron-left, chevron-right, chevron-up, chevrons-down, chevrons-left, chevrons-right, chevrons-up, circle, clipboard, clock, code, columns, command, compass, corner-down-left, corner-down-right, corner-left-down, corner-left-up, corner-right-down, corner-right-up, corner-up-left, corner-up-right, crosshair, database, divide-circle, divide-square, dollar-sign, download, external-link, feather, frown, hash, headphones, help-circle, info, italic, key, layout, life-buoy, link-2, link, loader, lock, log-in, log-out, maximize, meh, minimize, minimize-2, minus-circle, minus-square, minus, monitor, moon, more-horizontal, more-vertical, move, music, navigation-2, navigation, octagon, pause-circle, percent, plus-circle, plus-square, plus, power, radio, rss, search, server, share, shopping-bag, sidebar, smartphone, smile, square, table-2, tablet, target, terminal, trash-2, trash, triangle, tv, type, upload, x-circle, x-octagon, x-square, x, zoom-in, zoom-out
The MIT License (MIT) (for the icons listed above)
Copyright (c) 2013-present Cole Bemis
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@ -99,6 +99,14 @@
--font-size-small: 11px;
--font-size-default: 13px;
--pane-border-width: 1px;
/* Modal backdrop (translucent) + intent/category accents (shared by light & dark) */
--scrim: #10141ab0;
--accent-coral: #e8826b;
--accent-cyan: #54c3e8;
--accent-amber: #f4a340;
--accent-pink: #c75b8a;
--accent-violet: #8a6ec0;
}
/* ============================================

View File

@ -39,6 +39,77 @@ pub fn update_prepare_timing(
t.composite_ms = composite_ms;
}
}
/// GPU-measured composite cost (from timestamp queries; see `gpu_timer.rs`).
#[derive(Debug, Clone, Default)]
pub struct GpuCompositeTiming {
/// True when the adapter supports timestamp queries (else the ms is meaningless).
pub supported: bool,
/// GPU time of the whole composite section (Vello render + sRGB→linear +
/// compositor + tonemap), in milliseconds. Read back asynchronously, so it
/// lags the displayed frame by a frame or two.
pub composite_gpu_ms: f64,
/// Layers composited this frame.
pub layers: u32,
/// `queue.submit()` calls in the composite section this frame.
pub submits: u32,
}
static GPU_COMPOSITE: OnceLock<Mutex<GpuCompositeTiming>> = OnceLock::new();
/// Called from `VelloCallback::prepare()` with the GPU composite measurement.
pub fn update_gpu_composite(supported: bool, composite_gpu_ms: f64, layers: u32, submits: u32) {
let cell = GPU_COMPOSITE.get_or_init(|| Mutex::new(GpuCompositeTiming::default()));
if let Ok(mut t) = cell.lock() {
t.supported = supported;
t.composite_gpu_ms = composite_gpu_ms;
t.layers = layers;
t.submits = submits;
}
}
fn get_gpu_composite() -> GpuCompositeTiming {
GPU_COMPOSITE
.get_or_init(|| Mutex::new(GpuCompositeTiming::default()))
.lock()
.map(|t| t.clone())
.unwrap_or_default()
}
/// CPU-side breakdown of the composite section (wall-clock `Instant` deltas). Since
/// the GPU idles waiting on these CPU operations, this is where the per-frame cost
/// actually lives. Sums should ≈ the CPU `composite_ms` for the doc's active paths.
#[derive(Debug, Clone, Default)]
pub struct CompositeCpuBreakdown {
/// `renderer.render_to_texture` — Vello scene encode + its internal submit.
pub vello_ms: f64,
/// `srgb_to_linear.convert` — recording the conversion pass.
pub convert_ms: f64,
/// `canvas_blit.blit` — recording + its internal submit.
pub blit_ms: f64,
/// `compositor.composite` — recording + per-call uniforms buffer / bind group alloc.
pub composite_ms: f64,
/// Explicit `queue.submit()` calls.
pub submit_ms: f64,
}
static COMPOSITE_CPU: OnceLock<Mutex<CompositeCpuBreakdown>> = OnceLock::new();
/// Called from `VelloCallback::prepare()` with the composite CPU breakdown.
pub fn update_composite_cpu(b: CompositeCpuBreakdown) {
let cell = COMPOSITE_CPU.get_or_init(|| Mutex::new(CompositeCpuBreakdown::default()));
if let Ok(mut t) = cell.lock() {
*t = b;
}
}
fn get_composite_cpu() -> CompositeCpuBreakdown {
COMPOSITE_CPU
.get_or_init(|| Mutex::new(CompositeCpuBreakdown::default()))
.lock()
.map(|t| t.clone())
.unwrap_or_default()
}
/// GPU memory the editor tracks itself (wgpu has no allocator query). Currently the
/// raster-layer texture cache — the only unbounded-by-default VRAM consumer.
#[derive(Debug, Clone, Default)]
@ -90,6 +161,12 @@ pub struct DebugStats {
// GPU prepare() timing breakdown (from render thread)
pub prepare_timing: PrepareTiming,
// GPU-measured composite cost (timestamp queries)
pub gpu_composite: GpuCompositeTiming,
// CPU breakdown of the composite section
pub composite_cpu: CompositeCpuBreakdown,
// Performance metrics for each section
pub timing_memory_us: u64,
pub timing_gpu_us: u64,
@ -254,6 +331,8 @@ impl DebugStatsCollector {
audio_input_devices,
has_pointer,
prepare_timing,
gpu_composite: get_gpu_composite(),
composite_cpu: get_composite_cpu(),
timing_memory_us,
timing_gpu_us,
timing_midi_us,
@ -306,8 +385,33 @@ pub fn render_debug_overlay(ctx: &egui::Context, stats: &DebugStats) {
ui.colored_label(egui::Color32::YELLOW, format!("GPU prepare: {:.2} ms", pt.total_ms));
ui.label(format!(" removals: {:.2} ms", pt.removals_ms));
ui.label(format!(" gpu_dispatch: {:.2} ms", pt.gpu_dispatches_ms));
ui.label(format!(" scene_build: {:.2} ms", pt.scene_build_ms));
ui.label(format!(" composite: {:.2} ms", pt.composite_ms));
ui.label(format!(" scene_build: {:.2} ms (CPU)", pt.scene_build_ms));
ui.label(format!(" composite: {:.2} ms (CPU)", pt.composite_ms));
// GPU-measured composite cost (timestamp queries).
let gc = &stats.gpu_composite;
if gc.supported {
ui.colored_label(
egui::Color32::LIGHT_GREEN,
format!("GPU composite: {:.2} ms (GPU)", gc.composite_gpu_ms),
);
ui.label(format!(" layers: {} submits: {}", gc.layers, gc.submits));
} else {
ui.label(format!(
"GPU composite: n/a (no timestamp support) layers: {} submits: {}",
gc.layers, gc.submits
));
}
// CPU breakdown of the composite (where the GPU is actually waiting).
let cc = &stats.composite_cpu;
let cc_sum = cc.vello_ms + cc.convert_ms + cc.blit_ms + cc.composite_ms + cc.submit_ms;
ui.colored_label(egui::Color32::LIGHT_BLUE, format!("Composite CPU breakdown: {:.2} ms", cc_sum));
ui.label(format!(" vello(render): {:.2} ms", cc.vello_ms));
ui.label(format!(" srgb→linear: {:.2} ms", cc.convert_ms));
ui.label(format!(" blit: {:.2} ms", cc.blit_ms));
ui.label(format!(" compositor: {:.2} ms", cc.composite_ms));
ui.label(format!(" queue.submit: {:.2} ms", cc.submit_ms));
ui.add_space(8.0);

View File

@ -26,9 +26,9 @@ impl CpuYuvConverter {
/// # Arguments
/// * `width` - Frame width in pixels
/// * `height` - Frame height in pixels
pub fn new(width: u32, height: u32) -> Result<Self, String> {
pub fn new(width: u32, height: u32, full_range: bool) -> Result<Self, String> {
// BT.709 (HD) RGBA→YUV420p context, created once.
let scaler = ffmpeg::software::scaling::Context::get(
let mut scaler = ffmpeg::software::scaling::Context::get(
ffmpeg::format::Pixel::RGBA,
width,
height,
@ -38,6 +38,23 @@ impl CpuYuvConverter {
ffmpeg::software::scaling::Flags::BILINEAR,
)
.map_err(|e| format!("Failed to create swscale context: {}", e))?;
// swscale defaults to BT.601 + limited range; force BT.709 with the requested output
// range so this fallback matches the GPU path and the encoder's color tags
// (otherwise non-%8-width exports come out with shifted hue / wrong levels). There is
// no safe ffmpeg-next wrapper for sws_setColorspaceDetails, so this is the raw call.
unsafe {
let coeffs = ffmpeg::ffi::sws_getCoefficients(ffmpeg::ffi::SWS_CS_ITU709 as i32);
let dst_range = if full_range { 1 } else { 0 };
let one = 1 << 16; // 16.16 fixed-point 1.0
ffmpeg::ffi::sws_setColorspaceDetails(
scaler.as_mut_ptr(),
coeffs, 1, // source table (RGB input is full-range)
coeffs, dst_range, // dest table = BT.709, dest range = requested
0, one, one, // brightness, contrast, saturation (neutral)
);
}
let rgba_frame = ffmpeg::frame::Video::new(ffmpeg::format::Pixel::RGBA, width, height);
let yuv_frame = ffmpeg::frame::Video::new(ffmpeg::format::Pixel::YUV420P, width, height);
Ok(Self { width, height, scaler, rgba_frame, yuv_frame })
@ -90,13 +107,13 @@ mod tests {
#[test]
fn test_converter_creation() {
let converter = CpuYuvConverter::new(1920, 1080);
let converter = CpuYuvConverter::new(1920, 1080, true);
assert!(converter.is_ok());
}
#[test]
fn test_conversion_output_sizes() {
let mut converter = CpuYuvConverter::new(1920, 1080).unwrap();
let mut converter = CpuYuvConverter::new(1920, 1080, true).unwrap();
// Create dummy RGBA data (all black)
let rgba_data = vec![0u8; 1920 * 1080 * 4];
@ -117,7 +134,7 @@ mod tests {
#[test]
#[should_panic(expected = "RGBA data size mismatch")]
fn test_wrong_input_size_panics() {
let mut converter = CpuYuvConverter::new(1920, 1080).unwrap();
let mut converter = CpuYuvConverter::new(1920, 1080, true).unwrap();
// Wrong size input
let rgba_data = vec![0u8; 1000];

View File

@ -6,7 +6,7 @@ use eframe::egui;
use lightningbeam_core::export::{
AudioExportSettings, AudioFormat,
ImageExportSettings, ImageFormat,
VideoExportSettings, VideoCodec, VideoQuality,
VideoExportSettings, VideoCodec, VideoQuality, ColorRange,
};
use std::path::PathBuf;
@ -25,6 +25,8 @@ pub enum ExportType {
Audio,
Image,
Video,
/// Vector-only SVG of the current frame (lossless; raster/video layers skipped).
Svg,
}
/// Export result from dialog
@ -34,6 +36,8 @@ pub enum ExportResult {
Image(ImageExportSettings, PathBuf),
VideoOnly(VideoExportSettings, PathBuf),
VideoWithAudio(VideoExportSettings, AudioExportSettings, PathBuf),
/// SVG of vector layers at the given document time.
Svg(f64, PathBuf),
}
/// Export dialog state
@ -156,6 +160,7 @@ impl ExportDialog {
ExportType::Audio => self.audio_settings.format.extension(),
ExportType::Image => self.image_settings.format.extension(),
ExportType::Video => self.video_settings.codec.container_format(),
ExportType::Svg => "svg",
}
}
@ -198,11 +203,12 @@ impl ExportDialog {
ExportType::Audio => "Export Audio",
ExportType::Image => "Export Image",
ExportType::Video => "Export Video",
ExportType::Svg => "Export SVG",
};
let modal_response = egui::Modal::new(egui::Id::new("export_dialog_modal"))
.show(ctx, |ui| {
ui.set_width(500.0);
ui.set_width(crate::mobile::dialog_width(ctx, 500.0));
ui.heading(window_title);
ui.add_space(8.0);
@ -219,6 +225,7 @@ impl ExportDialog {
(ExportType::Audio, "Audio"),
(ExportType::Image, "Image"),
(ExportType::Video, "Video"),
(ExportType::Svg, "SVG"),
] {
if ui.selectable_value(&mut self.export_type, variant, label).clicked() {
self.update_filename_extension();
@ -235,6 +242,7 @@ impl ExportDialog {
ExportType::Audio => self.render_audio_basic(ui),
ExportType::Image => self.render_image_settings(ui),
ExportType::Video => self.render_video_basic(ui),
ExportType::Svg => self.render_svg_settings(ui),
}
ui.add_space(12.0);
@ -253,6 +261,7 @@ impl ExportDialog {
ExportType::Audio => self.render_audio_advanced(ui),
ExportType::Image => self.render_image_advanced(ui),
ExportType::Video => self.render_video_advanced(ui),
ExportType::Svg => {} // SVG has no advanced settings
}
}
@ -356,6 +365,20 @@ impl ExportDialog {
}
}
/// Render SVG export settings — just the frame time (reuses the image time field).
fn render_svg_settings(&mut self, ui: &mut egui::Ui) {
ui.horizontal(|ui| {
ui.label("Time:");
ui.add(egui::DragValue::new(&mut self.image_settings.time)
.speed(0.01)
.range(0.0..=f64::MAX)
.suffix(" s"));
});
ui.add_space(4.0);
ui.weak("Exports vector layers losslessly at this frame. Raster, video, and");
ui.weak("effect layers are not included.");
}
/// Render advanced image export settings (time, resolution override).
fn render_image_advanced(&mut self, ui: &mut egui::Ui) {
// Time (which frame to export)
@ -378,6 +401,19 @@ impl ExportDialog {
if changed_h { self.image_settings.height = if h == 0 { None } else { Some(h) }; }
ui.weak("(0 = document size)");
});
// Fit mode — how the document maps into the output frame when aspect ratios differ.
ui.horizontal(|ui| {
use lightningbeam_core::export::ExportFitMode;
ui.label("Fit:");
egui::ComboBox::from_id_salt("image_fit_mode")
.selected_text(self.image_settings.fit.name())
.show_ui(ui, |ui| {
ui.selectable_value(&mut self.image_settings.fit, ExportFitMode::Letterbox, ExportFitMode::Letterbox.name());
ui.selectable_value(&mut self.image_settings.fit, ExportFitMode::Crop, ExportFitMode::Crop.name());
ui.selectable_value(&mut self.image_settings.fit, ExportFitMode::Stretch, ExportFitMode::Stretch.name());
});
});
}
/// Render advanced audio settings (sample rate, channels, bit depth, bitrate, time range)
@ -504,6 +540,19 @@ impl ExportDialog {
}
});
// Fit mode — how the document maps into the export frame when the aspect ratios differ.
ui.horizontal(|ui| {
use lightningbeam_core::export::ExportFitMode;
ui.label("Fit:");
egui::ComboBox::from_id_salt("video_fit_mode")
.selected_text(self.video_settings.fit.name())
.show_ui(ui, |ui| {
ui.selectable_value(&mut self.video_settings.fit, ExportFitMode::Letterbox, ExportFitMode::Letterbox.name());
ui.selectable_value(&mut self.video_settings.fit, ExportFitMode::Crop, ExportFitMode::Crop.name());
ui.selectable_value(&mut self.video_settings.fit, ExportFitMode::Stretch, ExportFitMode::Stretch.name());
});
});
ui.horizontal(|ui| {
ui.label("FPS:");
egui::ComboBox::from_id_salt("framerate")
@ -527,6 +576,36 @@ impl ExportDialog {
});
});
// Color range applies to H.264 (the VAAPI zero-copy encoder honors it). Limited/TV is the
// compatible default; Full/PC only looks right in players that read the full-range tag.
if matches!(self.video_settings.codec, VideoCodec::H264) {
ui.horizontal(|ui| {
ui.label("Color range:");
egui::ComboBox::from_id_salt("video_color_range")
.selected_text(self.video_settings.color_range.name())
.show_ui(ui, |ui| {
ui.selectable_value(&mut self.video_settings.color_range, ColorRange::Limited, ColorRange::Limited.name());
ui.selectable_value(&mut self.video_settings.color_range, ColorRange::Full, ColorRange::Full.name());
});
});
}
// HDR output: 10-bit BT.2020 PQ/HLG (HEVC). Forces H.265; software path (no zero-copy).
ui.horizontal(|ui| {
use lightningbeam_core::export::HdrExportMode;
ui.label("Dynamic range:");
egui::ComboBox::from_id_salt("video_hdr_mode")
.selected_text(self.video_settings.hdr.name())
.show_ui(ui, |ui| {
ui.selectable_value(&mut self.video_settings.hdr, HdrExportMode::Sdr, HdrExportMode::Sdr.name());
ui.selectable_value(&mut self.video_settings.hdr, HdrExportMode::Pq, HdrExportMode::Pq.name());
ui.selectable_value(&mut self.video_settings.hdr, HdrExportMode::Hlg, HdrExportMode::Hlg.name());
});
});
if self.video_settings.hdr.is_hdr() {
ui.label(egui::RichText::new("HDR exports as 10-bit HEVC (H.265), BT.2020.").weak().small());
}
ui.checkbox(&mut self.include_audio, "Include Audio");
ui.add_space(8.0);
@ -539,7 +618,7 @@ impl ExportDialog {
fn render_time_range(&mut self, ui: &mut egui::Ui) {
let (start_time, end_time) = match self.export_type {
ExportType::Audio => (&mut self.audio_settings.start_time, &mut self.audio_settings.end_time),
ExportType::Image => return, // image uses a single time field, not a range
ExportType::Image | ExportType::Svg => return, // single time field, not a range
ExportType::Video => (&mut self.video_settings.start_time, &mut self.video_settings.end_time),
};
@ -613,6 +692,7 @@ impl ExportDialog {
}
Some(ExportResult::Image(self.image_settings.clone(), output_path))
}
ExportType::Svg => Some(ExportResult::Svg(self.image_settings.time, output_path)),
ExportType::Audio => {
// Validate audio settings
if let Err(err) = self.audio_settings.validate() {
@ -721,7 +801,7 @@ impl ExportProgressDialog {
egui::Modal::new(egui::Id::new("export_progress_modal"))
.show(ctx, |ui| {
ui.set_width(400.0);
ui.set_width(crate::mobile::dialog_width(ctx, 400.0));
ui.heading("Exporting...");
ui.add_space(8.0);

View File

@ -0,0 +1,330 @@
//! Tight GPU RGBA→YUV420p converter for video export.
//!
//! Unlike [`lightningbeam_core::gpu::YuvConverter`] (which writes one byte per
//! `Rgba8Unorm` texel — a 4× readback), this writes **packed planar YUV420p** into a
//! storage buffer, so the readback is exactly `W*H*3/2` bytes (~3.1 MB at 1080p vs
//! 8.3 MB RGBA) and — more importantly — the per-frame CPU `rgba_to_yuv420p` (swscale)
//! is eliminated.
//!
//! Color math is BT.709; the Y/chroma scale+offset (full vs limited range) is selected by
//! the `full_range` flag and must match the encoder color tags set in `setup_video_encoder`.
//!
//! Output buffer layout (tight, little-endian byte packing into `array<u32>`):
//! - `[0, W*H)` Y plane, row stride `W`
//! - `[W*H, W*H + CW*CH)` U plane, row stride `CW` (`CW=W/2`, `CH=H/2`)
//! - `[W*H+CW*CH, end)` V plane, row stride `CW`
//!
//! Dimension requirement: `W % 8 == 0 && H % 2 == 0` (so `W/4` and `CW/4` are whole —
//! the shader packs 4 bytes per `u32`). [`GpuYuv::supports`] reports this; callers
//! fall back to the CPU converter otherwise.
/// `true` when [`GpuYuv`] can convert these dimensions (else use the CPU path).
pub fn supports(width: u32, height: u32) -> bool {
width % 8 == 0 && height % 2 == 0 && width > 0 && height > 0
}
/// Tight planar YUV420p byte length for `width`×`height`.
pub fn yuv420p_len(width: u32, height: u32) -> usize {
let y = (width * height) as usize;
let c = ((width / 2) * (height / 2)) as usize;
y + 2 * c
}
/// `(y_offset, y_scale, chroma_offset, chroma_scale)` as fractions of 255, selecting
/// limited (TV, 16235 / 16240) vs full (PC, 0255) range. Mirrors `render_nv12`.
fn range_params(full_range: bool) -> [f32; 4] {
if full_range {
[0.0, 1.0, 128.0 / 255.0, 1.0]
} else {
[16.0 / 255.0, 219.0 / 255.0, 128.0 / 255.0, 224.0 / 255.0]
}
}
/// GPU compute pipeline: `Rgba8Unorm` texture → tight planar YUV420p storage buffer.
pub struct GpuYuv {
y_pipeline: wgpu::ComputePipeline,
uv_pipeline: wgpu::ComputePipeline,
bind_group_layout: wgpu::BindGroupLayout,
range_buffer: wgpu::Buffer,
}
impl GpuYuv {
/// `full_range`: true → full/PC range (Y 0255), false → limited/TV range (Y 16235).
/// The encoder must tag the stream to match (`setup_video_encoder`'s `full_range`).
pub fn new(device: &wgpu::Device, full_range: bool) -> Self {
use wgpu::util::DeviceExt;
let params = range_params(full_range);
let range_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
label: Some("gpu_yuv_range"),
contents: bytemuck::cast_slice(&params),
usage: wgpu::BufferUsages::UNIFORM,
});
let bind_group_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("gpu_yuv_bgl"),
entries: &[
// 0: input RGBA (non-filterable, read via textureLoad)
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: false },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
},
// 1: output packed YUV (read_write so 4-byte packing writes whole u32s)
wgpu::BindGroupLayoutEntry {
binding: 1,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Buffer {
ty: wgpu::BufferBindingType::Storage { read_only: false },
has_dynamic_offset: false,
min_binding_size: None,
},
count: None,
},
// 2: range params (y_offset, y_scale, chroma_offset, chroma_scale)
wgpu::BindGroupLayoutEntry {
binding: 2,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Buffer {
ty: wgpu::BufferBindingType::Uniform,
has_dynamic_offset: false,
min_binding_size: None,
},
count: None,
},
],
});
let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("gpu_yuv_pl"),
bind_group_layouts: &[&bind_group_layout],
push_constant_ranges: &[],
});
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("gpu_yuv_shader"),
source: wgpu::ShaderSource::Wgsl(SHADER.into()),
});
let mk = |entry: &str| {
device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor {
label: Some("gpu_yuv_pipeline"),
layout: Some(&pipeline_layout),
module: &shader,
entry_point: Some(entry),
compilation_options: wgpu::PipelineCompilationOptions::default(),
cache: None,
})
};
Self {
y_pipeline: mk("y_main"),
uv_pipeline: mk("uv_main"),
bind_group_layout,
range_buffer,
}
}
/// Record the RGBA→YUV420p conversion into `encoder`.
///
/// `rgba_view` is the rendered frame (`Rgba8Unorm`, `width`×`height`, must have
/// `TEXTURE_BINDING` usage). `yuv_buffer` must be a `STORAGE | COPY_SRC` buffer of
/// at least [`yuv420p_len`] bytes (rounded up to 4). Caller must ensure
/// [`supports`]`(width, height)`.
pub fn convert(
&self,
device: &wgpu::Device,
encoder: &mut wgpu::CommandEncoder,
rgba_view: &wgpu::TextureView,
yuv_buffer: &wgpu::Buffer,
width: u32,
height: u32,
) {
debug_assert!(supports(width, height));
let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("gpu_yuv_bg"),
layout: &self.bind_group_layout,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: wgpu::BindingResource::TextureView(rgba_view) },
wgpu::BindGroupEntry { binding: 1, resource: yuv_buffer.as_entire_binding() },
wgpu::BindGroupEntry { binding: 2, resource: self.range_buffer.as_entire_binding() },
],
});
let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor {
label: Some("gpu_yuv_pass"),
timestamp_writes: None,
});
pass.set_bind_group(0, &bind_group, &[]);
// Y: one thread per 4 horizontal luma samples → (W/4)×H threads.
pass.set_pipeline(&self.y_pipeline);
let wg = 8u32;
pass.dispatch_workgroups(((width / 4) + wg - 1) / wg, (height + wg - 1) / wg, 1);
// UV: one thread per 4 horizontal chroma samples → (CW/4)×CH = (W/8)×(H/2) threads.
pass.set_pipeline(&self.uv_pipeline);
let cw = width / 2;
let ch = height / 2;
pass.dispatch_workgroups(((cw / 4) + wg - 1) / wg, (ch + wg - 1) / wg, 1);
}
}
/// CPU reference for the exact math/layout the shader produces — used by unit tests so
/// the packing and BT.709 coefficients stay verifiable without a GPU.
fn cpu_reference(rgba: &[u8], width: u32, height: u32, full_range: bool) -> Vec<u8> {
let w = width as usize;
let h = height as usize;
let cw = w / 2;
let ch = h / 2;
let [yo, ys, co, cs] = range_params(full_range);
let mut out = vec![0u8; yuv420p_len(width, height)];
let to_byte = |v: f32| (v.clamp(0.0, 1.0) * 255.0 + 0.5) as u8;
let px = |x: usize, y: usize| {
let i = (y * w + x) * 4;
[rgba[i] as f32 / 255.0, rgba[i + 1] as f32 / 255.0, rgba[i + 2] as f32 / 255.0]
};
// Y
for y in 0..h {
for x in 0..w {
let p = px(x, y);
let yy = 0.2126 * p[0] + 0.7152 * p[1] + 0.0722 * p[2];
out[y * w + x] = to_byte(yo + ys * yy);
}
}
// U/V (2x2 average)
let y_size = w * h;
let uv_size = cw * ch;
for cy in 0..ch {
for cx in 0..cw {
let mut acc = [0.0f32; 3];
for (dx, dy) in [(0, 0), (1, 0), (0, 1), (1, 1)] {
let p = px(2 * cx + dx, 2 * cy + dy);
acc[0] += p[0]; acc[1] += p[1]; acc[2] += p[2];
}
let a = [acc[0] / 4.0, acc[1] / 4.0, acc[2] / 4.0];
let uc = -0.1146 * a[0] - 0.3854 * a[1] + 0.5000 * a[2];
let vc = 0.5000 * a[0] - 0.4542 * a[1] - 0.0458 * a[2];
out[y_size + cy * cw + cx] = to_byte(co + cs * uc);
out[y_size + uv_size + cy * cw + cx] = to_byte(co + cs * vc);
}
}
out
}
const SHADER: &str = r#"
// RGBA -> tight planar YUV420p (BT.709), packed 4 bytes/u32.
// rng = (y_offset, y_scale, chroma_offset, chroma_scale): selects limited vs full range.
@group(0) @binding(0) var input_rgba: texture_2d<f32>;
@group(0) @binding(1) var<storage, read_write> out_buf: array<u32>;
@group(0) @binding(2) var<uniform> rng: vec4<f32>;
fn to_byte(v: f32) -> u32 { return u32(clamp(v, 0.0, 1.0) * 255.0 + 0.5); }
// Y plane: each thread packs 4 horizontal luma bytes.
@compute @workgroup_size(8, 8, 1)
fn y_main(@builtin(global_invocation_id) gid: vec3<u32>) {
let dims = textureDimensions(input_rgba);
let w = dims.x;
let h = dims.y;
let x4 = gid.x * 4u;
let y = gid.y;
if (x4 >= w || y >= h) { return; }
var packed: u32 = 0u;
for (var i = 0u; i < 4u; i = i + 1u) {
let c = textureLoad(input_rgba, vec2<u32>(x4 + i, y), 0).rgb;
let yy = 0.2126 * c.r + 0.7152 * c.g + 0.0722 * c.b;
packed = packed | (to_byte(rng.x + rng.y * yy) << (8u * i));
}
out_buf[(y * w + x4) / 4u] = packed;
}
// U/V planes: each thread packs 4 horizontal chroma bytes (2x2 box-averaged).
@compute @workgroup_size(8, 8, 1)
fn uv_main(@builtin(global_invocation_id) gid: vec3<u32>) {
let dims = textureDimensions(input_rgba);
let w = dims.x;
let h = dims.y;
let cw = w / 2u;
let ch = h / 2u;
let cx4 = gid.x * 4u;
let cy = gid.y;
if (cx4 >= cw || cy >= ch) { return; }
let y_size = w * h;
let uv_size = cw * ch;
var up: u32 = 0u;
var vp: u32 = 0u;
for (var i = 0u; i < 4u; i = i + 1u) {
let cx = cx4 + i;
let sx = 2u * cx;
let sy = 2u * cy;
let p00 = textureLoad(input_rgba, vec2<u32>(sx, sy), 0).rgb;
let p10 = textureLoad(input_rgba, vec2<u32>(sx + 1u, sy), 0).rgb;
let p01 = textureLoad(input_rgba, vec2<u32>(sx, sy + 1u), 0).rgb;
let p11 = textureLoad(input_rgba, vec2<u32>(sx + 1u, sy + 1u), 0).rgb;
let a = (p00 + p10 + p01 + p11) * 0.25;
// Centered chroma in [-0.5, 0.5], then map to range via (offset + scale*coef).
let uc = -0.1146 * a.r - 0.3854 * a.g + 0.5000 * a.b;
let vc = 0.5000 * a.r - 0.4542 * a.g - 0.0458 * a.b;
up = up | (to_byte(rng.z + rng.w * uc) << (8u * i));
vp = vp | (to_byte(rng.z + rng.w * vc) << (8u * i));
}
out_buf[(y_size + cy * cw + cx4) / 4u] = up;
out_buf[(y_size + uv_size + cy * cw + cx4) / 4u] = vp;
}
"#;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn supports_dims() {
assert!(supports(1920, 1080));
assert!(supports(1280, 720));
assert!(supports(8, 2));
assert!(!supports(6, 2)); // width not %8
assert!(!supports(8, 3)); // height odd
assert!(!supports(0, 0));
}
#[test]
fn len_matches() {
assert_eq!(yuv420p_len(1920, 1080), 1920 * 1080 * 3 / 2);
assert_eq!(yuv420p_len(8, 2), 8 * 2 + 2 * (4 * 1));
}
#[test]
fn reference_known_colors() {
// 8x2 solid white → Y≈255, U≈V≈128. Solid black → Y=0, U=V≈128.
let white = vec![255u8; 8 * 2 * 4];
let out = cpu_reference(&white, 8, 2, true);
let (cw, ch) = (4usize, 1usize);
let y_size = 8 * 2;
for &y in &out[..y_size] { assert!(y >= 254, "white Y={y}"); }
for &u in &out[y_size..y_size + cw * ch] { assert!((u as i32 - 128).abs() <= 1, "white U={u}"); }
let black = vec![0u8; 8 * 2 * 4];
let out = cpu_reference(&black, 8, 2, true);
for &y in &out[..y_size] { assert_eq!(y, 0); }
for &v in &out[y_size + cw * ch..] { assert!((v as i32 - 128).abs() <= 1, "black V={v}"); }
}
#[test]
fn reference_red_bt709() {
// Solid red (255,0,0): Y=0.2126*255≈54; V high, U low (full range).
let red: Vec<u8> = (0..8 * 2).flat_map(|_| [255u8, 0, 0, 255]).collect();
let out = cpu_reference(&red, 8, 2, true);
assert!((out[0] as i32 - 54).abs() <= 1, "red Y={}", out[0]);
let y_size = 8 * 2;
let u = out[y_size];
let v = out[y_size + 4];
// U = -0.1146*1*255+128 ≈ 99 ; V = 0.5*255+128 → clamps to 255
assert!((u as i32 - 99).abs() <= 2, "red U={u}");
assert_eq!(v, 255, "red V={v}");
}
}

View File

@ -0,0 +1,308 @@
//! 10-bit HDR frame production for video export (isolated from the SDR readback pipeline).
//!
//! Takes the compositor's Rgba16Float HDR accumulator and produces YUV420P10LE planes:
//! 1. GPU pass `linear_to_pq.wgsl` → PQ/HLG-encoded BT.2020 R'G'B' into an Rgba16Unorm texture
//! (the expensive per-pixel transfer + gamut work).
//! 2. Synchronous GPU→CPU readback of that texture.
//! 3. CPU BT.2020 R'G'B'→Y'CbCr (limited range), 4:2:0 average, 10-bit little-endian pack.
//!
//! Synchronous (no triple-buffering); HDR export favors correctness/simplicity over throughput.
use lightningbeam_core::export::HdrExportMode;
/// Round up to the wgpu copy row alignment (256 bytes).
fn align_256(n: u32) -> u32 {
(n + 255) & !255
}
pub struct HdrFramePipeline {
width: u32,
height: u32,
pipeline: wgpu::RenderPipeline,
bind_group_layout: wgpu::BindGroupLayout,
sampler: wgpu::Sampler,
mode_buf: wgpu::Buffer,
/// PQ/HLG-encoded BT.2020 R'G'B' (Rgba16Unorm) render target.
enc_texture_view: wgpu::TextureView,
enc_texture: wgpu::Texture,
/// Staging buffer for readback; rows padded to 256-byte alignment.
staging: wgpu::Buffer,
padded_bytes_per_row: u32,
}
impl HdrFramePipeline {
pub fn new(device: &wgpu::Device, width: u32, height: u32) -> Self {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("linear_to_pq_shader"),
source: wgpu::ShaderSource::Wgsl(include_str!("shaders/linear_to_pq.wgsl").into()),
});
let bind_group_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("linear_to_pq_bgl"),
entries: &[
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::FRAGMENT,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: true },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
},
wgpu::BindGroupLayoutEntry {
binding: 1,
visibility: wgpu::ShaderStages::FRAGMENT,
ty: wgpu::BindingType::Sampler(wgpu::SamplerBindingType::Filtering),
count: None,
},
wgpu::BindGroupLayoutEntry {
binding: 2,
visibility: wgpu::ShaderStages::FRAGMENT,
ty: wgpu::BindingType::Buffer {
ty: wgpu::BufferBindingType::Uniform,
has_dynamic_offset: false,
min_binding_size: None,
},
count: None,
},
],
});
let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("linear_to_pq_pl"),
bind_group_layouts: &[&bind_group_layout],
push_constant_ranges: &[],
});
let pipeline = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
label: Some("linear_to_pq_pipeline"),
layout: Some(&pipeline_layout),
vertex: wgpu::VertexState {
module: &shader,
entry_point: Some("vs_main"),
buffers: &[],
compilation_options: Default::default(),
},
fragment: Some(wgpu::FragmentState {
module: &shader,
entry_point: Some("fs_main"),
targets: &[Some(wgpu::ColorTargetState {
// Rgba16Float (not Unorm) so no TEXTURE_FORMAT_16BIT_NORM feature is needed; PQ/HLG
// values are in [0,1] where f16 has ~11 effective bits — ample for 10-bit output.
format: wgpu::TextureFormat::Rgba16Float,
blend: None,
write_mask: wgpu::ColorWrites::ALL,
})],
compilation_options: Default::default(),
}),
primitive: wgpu::PrimitiveState::default(),
depth_stencil: None,
multisample: wgpu::MultisampleState::default(),
multiview: None,
cache: None,
});
let sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("linear_to_pq_sampler"),
..Default::default()
});
let mode_buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("linear_to_pq_mode"),
size: 16, // vec4<u32>
usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
});
let enc_texture = device.create_texture(&wgpu::TextureDescriptor {
label: Some("hdr_enc_texture"),
size: wgpu::Extent3d { width, height, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba16Float,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::COPY_SRC,
view_formats: &[],
});
let enc_texture_view = enc_texture.create_view(&Default::default());
let padded_bytes_per_row = align_256(width * 8); // Rgba16Unorm = 8 bytes/texel
let staging = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("hdr_enc_staging"),
size: (padded_bytes_per_row * height) as u64,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
Self {
width,
height,
pipeline,
bind_group_layout,
sampler,
mode_buf,
enc_texture_view,
enc_texture,
staging,
padded_bytes_per_row,
}
}
/// Encode the composited HDR texture (`hdr_view`, Rgba16Float linear) to YUV420P10LE planes.
pub fn render_to_yuv10(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
hdr_view: &wgpu::TextureView,
mode: HdrExportMode,
) -> (Vec<u8>, Vec<u8>, Vec<u8>) {
let mode_code: u32 = if matches!(mode, HdrExportMode::Hlg) { 1 } else { 0 };
queue.write_buffer(&self.mode_buf, 0, bytemuck::cast_slice(&[mode_code, 0u32, 0, 0]));
let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("linear_to_pq_bg"),
layout: &self.bind_group_layout,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: wgpu::BindingResource::TextureView(hdr_view) },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::Sampler(&self.sampler) },
wgpu::BindGroupEntry { binding: 2, resource: self.mode_buf.as_entire_binding() },
],
});
let mut encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("hdr_frame_encoder"),
});
{
let mut rp = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("linear_to_pq_pass"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &self.enc_texture_view,
resolve_target: None,
depth_slice: None,
ops: wgpu::Operations {
load: wgpu::LoadOp::Clear(wgpu::Color::BLACK),
store: wgpu::StoreOp::Store,
},
})],
depth_stencil_attachment: None,
occlusion_query_set: None,
timestamp_writes: None,
});
rp.set_pipeline(&self.pipeline);
rp.set_bind_group(0, &bind_group, &[]);
rp.draw(0..3, 0..1);
}
encoder.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo {
texture: &self.enc_texture,
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyBufferInfo {
buffer: &self.staging,
layout: wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(self.padded_bytes_per_row),
rows_per_image: Some(self.height),
},
},
wgpu::Extent3d { width: self.width, height: self.height, depth_or_array_layers: 1 },
);
queue.submit(Some(encoder.finish()));
// Synchronous map + wait.
let slice = self.staging.slice(..);
let (tx, rx) = std::sync::mpsc::channel();
slice.map_async(wgpu::MapMode::Read, move |r| { let _ = tx.send(r); });
let _ = device.poll(wgpu::PollType::wait_indefinitely());
let _ = rx.recv();
let w = self.width as usize;
let h = self.height as usize;
let mapped = slice.get_mapped_range();
// Un-pad rows; decode f16 → f32 into a tight RGBA buffer.
let mut rgba = vec![0f32; w * h * 4];
let row_bytes = w * 8;
for row in 0..h {
let src = row * self.padded_bytes_per_row as usize;
let dst = row * w * 4;
let bytes = &mapped[src..src + row_bytes];
for px in 0..w * 4 {
let half = u16::from_le_bytes([bytes[px * 2], bytes[px * 2 + 1]]);
rgba[dst + px] = f16_to_f32(half);
}
}
drop(mapped);
self.staging.unmap();
rgba_to_yuv420p10le(&rgba, w, h)
}
}
/// Decode an IEEE 754 half-float. Inputs are in [0,1] so the inf/NaN paths don't occur in practice.
fn f16_to_f32(h: u16) -> f32 {
let sign = (h >> 15) & 1;
let exp = (h >> 10) & 0x1f;
let mant = h & 0x3ff;
let v = if exp == 0 {
(mant as f32) * 2f32.powi(-24) // subnormal
} else if exp == 31 {
if mant == 0 { f32::INFINITY } else { f32::NAN }
} else {
(1.0 + mant as f32 / 1024.0) * 2f32.powi(exp as i32 - 15)
};
if sign == 1 { -v } else { v }
}
/// BT.2020 non-constant-luminance R'G'B'→Y'CbCr, limited range, 4:2:0, 10-bit little-endian.
/// Input R'G'B' is already gamma-encoded (PQ/HLG) in [0,1].
fn rgba_to_yuv420p10le(rgba: &[f32], w: usize, h: usize) -> (Vec<u8>, Vec<u8>, Vec<u8>) {
const KR: f32 = 0.2627;
const KB: f32 = 0.0593;
let kg = 1.0 - KR - KB;
let luma = |r: f32, g: f32, b: f32| KR * r + kg * g + KB * b;
// 10-bit limited: Y' [64,940] (scale 876), Cb/Cr center 512, excursion ±0.5 → scale 896.
let pack_y = |y: f32| ((y * 876.0 + 64.0).round().clamp(0.0, 1023.0)) as u16;
let pack_c = |c: f32| ((c * 896.0 + 512.0).round().clamp(0.0, 1023.0)) as u16;
let mut y_plane = vec![0u8; w * h * 2];
for j in 0..h {
for i in 0..w {
let p = (j * w + i) * 4;
let y10 = pack_y(luma(rgba[p], rgba[p + 1], rgba[p + 2]));
let o = (j * w + i) * 2;
y_plane[o] = (y10 & 0xff) as u8;
y_plane[o + 1] = (y10 >> 8) as u8;
}
}
let (cw, ch) = (w / 2, h / 2);
let mut u_plane = vec![0u8; cw * ch * 2];
let mut v_plane = vec![0u8; cw * ch * 2];
for j in 0..ch {
for i in 0..cw {
let (mut cb, mut cr) = (0.0f32, 0.0f32);
for dy in 0..2 {
for dx in 0..2 {
let p = ((j * 2 + dy) * w + (i * 2 + dx)) * 4;
let (r, g, b) = (rgba[p], rgba[p + 1], rgba[p + 2]);
let yy = luma(r, g, b);
cb += (b - yy) / (2.0 * (1.0 - KB));
cr += (r - yy) / (2.0 * (1.0 - KR));
}
}
let cb10 = pack_c(cb / 4.0);
let cr10 = pack_c(cr / 4.0);
let o = (j * cw + i) * 2;
u_plane[o] = (cb10 & 0xff) as u8;
u_plane[o + 1] = (cb10 >> 8) as u8;
v_plane[o] = (cr10 & 0xff) as u8;
v_plane[o + 1] = (cr10 >> 8) as u8;
}
}
(y_plane, u_plane, v_plane)
}

View File

@ -10,6 +10,8 @@ pub mod video_exporter;
pub mod readback_pipeline;
pub mod perf_metrics;
pub mod cpu_yuv_converter;
pub mod gpu_yuv;
pub mod hdr_frame;
use lightningbeam_core::export::{AudioExportSettings, ImageExportSettings, VideoExportSettings, ExportProgress};
use lightningbeam_core::document::Document;
@ -51,6 +53,13 @@ pub struct VideoExportState {
width: u32,
/// Export height in pixels
height: u32,
/// HDR output mode — HDR uses a synchronous 10-bit path instead of the async RGBA pipeline.
hdr: lightningbeam_core::export::HdrExportMode,
/// How the document is fit into the export frame (stretch/letterbox/crop).
fit: lightningbeam_core::export::ExportFitMode,
/// SDR color range: true = full (PC, 0255), false = limited (TV, 16235). The YUV
/// converters and the encoder color tag must agree on this.
full_range: bool,
/// Channel to send rendered frames to encoder thread
frame_tx: Option<Sender<VideoFrameMessage>>,
/// HDR GPU resources for compositing pipeline (effects, color conversion)
@ -67,6 +76,23 @@ pub struct VideoExportState {
perf_metrics: Option<perf_metrics::ExportMetrics>,
}
/// Zero-copy VAAPI video production: renders each frame to RGBA and hardware-encodes it
/// into a VAAPI surface, all on the encoder's own wgpu device (no readback / swscale).
/// VAAPI is Linux-only, so this and its machinery are `cfg`-gated; other platforms always
/// use the software encoder path.
#[cfg(target_os = "linux")]
struct ZeroCopyVideo {
encoder: gpu_video_encoder::encoder::ZeroCopyEncoder,
renderer: vello::Renderer,
gpu_resources: video_exporter::ExportGpuResources,
/// Reused RGBA target (RENDER_ATTACHMENT | TEXTURE_BINDING) on the encoder's device.
rgba: wgpu::Texture,
/// True when running on the shared device → compositing can consume hardware-decoded GPU frames.
on_shared_device: bool,
/// How the document is fit into the export frame (stretch/letterbox/crop).
fit: lightningbeam_core::export::ExportFitMode,
}
/// State for a single-frame image export (runs on the GPU render thread, one frame per update).
pub struct ImageExportState {
pub settings: ImageExportSettings,
@ -195,18 +221,36 @@ impl ExportOrchestrator {
return self.poll_parallel_progress();
}
// Handle single export (audio-only or video-only)
if let Some(rx) = &self.progress_rx {
match rx.try_recv() {
Ok(progress) => {
// Handle single export (audio-only or video-only). Recv into a local first so we can
// clear the channel on a terminal event without a borrow conflict — that lets
// `has_pending_progress()` (and thus the UI poll loop) go quiet once the export ends,
// instead of polling forever. The thread may already be finished here, so we must drain
// the final Complete/Error from the channel rather than rely on `is_exporting()`.
let recv = self.progress_rx.as_ref().map(|rx| rx.try_recv());
match recv {
Some(Ok(progress)) => {
println!("📨 [ORCHESTRATOR] Received progress: {:?}", std::mem::discriminant(&progress));
if matches!(progress, ExportProgress::Complete { .. } | ExportProgress::Error { .. }) {
self.progress_rx = None;
self.thread_handle = None;
}
Some(progress)
}
Err(_) => None,
}
} else {
Some(Err(std::sync::mpsc::TryRecvError::Disconnected)) => {
// Thread gone without a terminal message; stop polling.
self.progress_rx = None;
self.thread_handle = None;
None
}
_ => None, // Empty, or no channel
}
}
/// Whether the orchestrator still has progress to report (an active export, or an
/// unconsumed terminal message). Used to gate the UI poll loop so it doesn't run every
/// repaint forever after an export finishes.
pub fn has_pending_progress(&self) -> bool {
self.parallel_export.is_some() || self.image_state.is_some() || self.progress_rx.is_some()
}
/// Poll progress for parallel video+audio export
@ -479,6 +523,19 @@ impl ExportOrchestrator {
/// Cancel the current export
pub fn cancel(&mut self) {
self.cancel_flag.store(true, Ordering::Relaxed);
// Tear down so `is_exporting()` goes false and the UI can drop the progress dialog.
// The background threads observe the cancel flag and exit on their own; we detach their
// handles here rather than joining (joining would block the UI). Partial temp files are
// removed — any still-open encoder fd just writes to the unlinked inode, which is freed
// on close.
if let Some(parallel) = self.parallel_export.take() {
std::fs::remove_file(&parallel.temp_video_path).ok();
std::fs::remove_file(&parallel.temp_audio_path).ok();
}
self.video_state = None;
self.image_state = None;
self.progress_rx = None;
self.thread_handle = None;
}
/// Check if an export is in progress
@ -544,6 +601,7 @@ impl ExportOrchestrator {
// ── First call: render the frame to the GPU output texture ────────
let w = state.width;
let h = state.height;
let fit = state.settings.fit;
if state.gpu_resources.is_none() {
state.gpu_resources = Some(video_exporter::ExportGpuResources::new(device, w, h));
@ -577,6 +635,8 @@ impl ExportOrchestrator {
floating_selection,
state.settings.allow_transparency,
raster_store,
true, // image export composites on the shared device
fit,
)?;
queue.submit(Some(encoder.finish()));
@ -791,21 +851,158 @@ impl ExportOrchestrator {
}
}
/// Start a video export in the background (encoder thread)
/// Spawn the software video-encoder thread and build its UI-driven export state. Used by every
/// non-zero-copy path (non-H.264, VAAPI unavailable, or non-Linux platforms). The caller drives
/// frames into it via `render_next_video_frame()`.
#[allow(clippy::too_many_arguments)]
fn spawn_software_video(
settings: VideoExportSettings,
output_path: PathBuf,
frame_rx: Receiver<VideoFrameMessage>,
frame_tx: Sender<VideoFrameMessage>,
progress_tx: Sender<ExportProgress>,
cancel_flag: Arc<AtomicBool>,
total_frames: usize,
start_time: f64,
end_time: f64,
framerate: f64,
width: u32,
height: u32,
) -> (std::thread::JoinHandle<()>, VideoExportState) {
let hdr = settings.hdr;
let fit = settings.fit;
let full_range = settings.color_range.is_full();
let handle = std::thread::spawn(move || {
Self::run_video_encoder(settings, output_path, frame_rx, progress_tx, cancel_flag, total_frames);
});
// GPU resources + readback pipeline init lazily on the first frame (needs device).
let state = VideoExportState {
current_frame: 0,
total_frames,
start_time,
end_time,
framerate,
width,
height,
hdr,
fit,
full_range,
frame_tx: Some(frame_tx),
gpu_resources: None,
readback_pipeline: None,
cpu_yuv_converter: None,
frames_in_flight: 0,
next_frame_to_encode: 0,
perf_metrics: Some(perf_metrics::ExportMetrics::new()),
};
(handle, state)
}
/// Build a zero-copy VAAPI H.264 encoder targeting `output_path` (with its own vello
/// renderer + GPU resources + RGBA texture on the encoder's device), or `None` when the
/// codec isn't H.264 or VAAPI / the GPU device is unavailable — in which case the caller
/// falls back to the software encoder path. Used by both the video-only and video+audio
/// export entry points. VAAPI is Linux-only, so this whole path is `cfg`-gated.
#[cfg(target_os = "linux")]
fn try_build_zero_copy(
settings: &VideoExportSettings,
width: u32,
height: u32,
framerate: f64,
output_path: &std::path::Path,
shared_device: Option<(wgpu::Device, wgpu::Queue, wgpu::Adapter)>,
) -> Option<ZeroCopyVideo> {
// Zero-copy is 8-bit H.264 only; HDR needs the 10-bit HEVC software path.
if settings.hdr.is_hdr()
|| !matches!(settings.codec, lightningbeam_core::export::VideoCodec::H264)
{
return None;
}
let bitrate = settings.quality.bitrate_kbps();
let fr = framerate.round() as i32;
let full = settings.color_range.is_full();
println!("🎬 [EXPORT] zero-copy H.264 color range: {} (full_range={})",
settings.color_range.name(), full);
let on_shared_device = shared_device.is_some();
// Prefer the shared device → decode→composite→encode stay GPU-resident on one device.
// Without it, the encoder builds its own device (decode still downloads to CPU per Step 1's
// hardware_ok=false on this path).
let encoder_result = match shared_device {
Some((device, queue, adapter)) => {
println!("🎬 [EXPORT] zero-copy on shared device (GPU-resident decode)");
gpu_video_encoder::encoder::ZeroCopyEncoder::new_on_device(
device, queue, adapter, width, height, fr, bitrate, output_path, full,
)
}
None => gpu_video_encoder::encoder::ZeroCopyEncoder::new(
width, height, fr, bitrate, output_path, full,
),
};
let encoder = match encoder_result {
Ok(e) => e,
Err(e) => {
println!("🎬 [EXPORT] zero-copy unavailable ({e}); software path");
return None;
}
};
let renderer = match vello::Renderer::new(
encoder.device(),
vello::RendererOptions {
use_cpu: false,
antialiasing_support: vello::AaSupport::all(),
num_init_threads: None,
pipeline_cache: None,
},
) {
Ok(r) => r,
Err(e) => {
println!("🎬 [EXPORT] zero-copy renderer init failed ({e}); software path");
return None;
}
};
let gpu_resources = video_exporter::ExportGpuResources::new(encoder.device(), width, height);
let rgba = encoder.device().create_texture(&wgpu::TextureDescriptor {
label: Some("zerocopy_export_rgba"),
size: wgpu::Extent3d { width, height, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::COPY_SRC,
view_formats: &[],
});
println!("🎬 [EXPORT] zero-copy VAAPI H.264 enabled");
Some(ZeroCopyVideo { encoder, renderer, gpu_resources, rgba, on_shared_device, fit: settings.fit })
}
/// Start a video export in the background.
///
/// Returns immediately after spawning encoder thread. Caller must call
/// `render_next_video_frame()` repeatedly from the main thread to feed frames.
/// For H.264 with VAAPI available this renders + hardware-encodes on a background thread
/// (writing `output_path` directly); otherwise it spawns the software encoder thread and the
/// caller drives frames via `render_next_video_frame()` from the main thread.
///
/// # Arguments
/// * `settings` - Video export settings
/// * `output_path` - Output file path
/// * `document`/`video_manager`/`raster_store`/`container_path` - scene data; the zero-copy
/// path snapshots the document and renders off-thread (the UI keeps the live one).
///
/// # Returns
/// Ok(()) on success, Err on failure
#[allow(clippy::too_many_arguments)]
#[allow(clippy::too_many_arguments)]
pub fn start_video_export(
&mut self,
settings: VideoExportSettings,
output_path: PathBuf,
document: &Document,
video_manager: Arc<std::sync::Mutex<VideoManager>>,
raster_store: lightningbeam_core::raster_store::RasterStore,
container_path: Option<PathBuf>,
// The shared VAAPI device, `Some` only when active → zero-copy encode runs on it.
shared_device: Option<(wgpu::Device, wgpu::Queue, wgpu::Adapter)>,
) -> Result<(), String> {
println!("🎬 [VIDEO EXPORT] Starting video export");
@ -828,38 +1025,43 @@ impl ExportOrchestrator {
self.cancel_flag.store(false, Ordering::Relaxed);
let cancel_flag = Arc::clone(&self.cancel_flag);
// Spawn encoder thread
// Zero-copy VAAPI H.264 (Linux only) writes the final output directly on a background
// thread (no audio → no mux), reporting through the single-export progress channel. On
// success it returns here; otherwise we fall through to the software encoder thread.
#[cfg(target_os = "linux")]
{
if let Some(zc) = Self::try_build_zero_copy(&settings, width, height, framerate, &output_path, shared_device) {
drop(frame_rx);
let document_snapshot = document.clone();
let mut image_cache = ImageCache::new();
image_cache.set_container_path(container_path);
let handle = std::thread::spawn(move || {
Self::run_video_encoder(
settings,
output_path,
frame_rx,
progress_tx,
cancel_flag,
total_frames,
Self::run_zerocopy_video_export(
zc, document_snapshot, image_cache, video_manager, raster_store,
total_frames, start_time, framerate, width, height,
output_path, progress_tx, cancel_flag,
);
});
self.thread_handle = Some(handle);
self.video_state = None;
println!("🎬 [VIDEO EXPORT] zero-copy thread spawned");
return Ok(());
}
}
// Zero-copy isn't used here; the scene-snapshot inputs are Linux-zero-copy-only.
#[cfg(not(target_os = "linux"))]
let _ = (document, &video_manager, &raster_store, &container_path);
let (handle, video_state) = {
let (h, s) = Self::spawn_software_video(
settings, output_path, frame_rx, frame_tx, progress_tx, cancel_flag,
total_frames, start_time, end_time, framerate, width, height,
);
(h, Some(s))
};
self.thread_handle = Some(handle);
// Initialize video export state
// GPU resources and readback pipeline will be initialized lazily on first frame (needs device)
self.video_state = Some(VideoExportState {
current_frame: 0,
total_frames,
start_time,
end_time,
framerate,
width,
height,
frame_tx: Some(frame_tx),
gpu_resources: None,
readback_pipeline: None,
cpu_yuv_converter: None,
frames_in_flight: 0,
next_frame_to_encode: 0,
perf_metrics: Some(perf_metrics::ExportMetrics::new()),
});
self.video_state = video_state;
println!("🎬 [VIDEO EXPORT] Encoder thread spawned, ready for frames");
Ok(())
@ -879,12 +1081,21 @@ impl ExportOrchestrator {
///
/// # Returns
/// Ok(()) on success, Err on failure
#[allow(clippy::too_many_arguments)]
pub fn start_video_with_audio_export(
&mut self,
video_settings: VideoExportSettings,
mut audio_settings: AudioExportSettings,
output_path: PathBuf,
audio_controller: Arc<std::sync::Mutex<daw_backend::EngineController>>,
// For the zero-copy H.264 path the export runs on a background thread, so it needs an
// owned snapshot of the scene data (the live document/caches stay with the UI thread).
document: &Document,
video_manager: Arc<std::sync::Mutex<VideoManager>>,
raster_store: lightningbeam_core::raster_store::RasterStore,
container_path: Option<PathBuf>,
// The shared VAAPI device, `Some` only when active → zero-copy encode runs on it.
shared_device: Option<(wgpu::Device, wgpu::Queue, wgpu::Adapter)>,
) -> Result<(), String> {
println!("🎬🎵 [PARALLEL EXPORT] Starting parallel video+audio export");
@ -945,19 +1156,52 @@ impl ExportOrchestrator {
let video_cancel_flag = Arc::clone(&self.cancel_flag);
let audio_cancel_flag = Arc::clone(&self.cancel_flag);
// Spawn video encoder thread
let video_settings_clone = video_settings.clone();
let temp_video_path_clone = temp_video_path.clone();
let video_thread = std::thread::spawn(move || {
Self::run_video_encoder(
video_settings_clone,
temp_video_path_clone,
frame_rx,
video_progress_tx,
video_cancel_flag,
total_frames,
// Spawn the video thread: on Linux, try the background zero-copy renderer/encoder (its own
// device + a document snapshot, decoupled from the UI vsync loop, writing the temp .mp4
// directly); otherwise (non-H.264, no VAAPI, or non-Linux) the software encoder thread fed
// by `render_next_video_frame` on the UI thread.
#[cfg(target_os = "linux")]
let (video_thread, video_state) = match Self::try_build_zero_copy(
&video_settings, video_width, video_height, video_framerate, &temp_video_path, shared_device,
) {
Some(zc) => {
drop(frame_rx); // the zero-copy path renders internally, no frame channel
let document_snapshot = document.clone();
let mut image_cache = ImageCache::new();
image_cache.set_container_path(container_path.clone());
let raster_store = raster_store.clone();
let video_manager = Arc::clone(&video_manager);
let temp_video_path = temp_video_path.clone();
let handle = std::thread::spawn(move || {
Self::run_zerocopy_video_export(
zc, document_snapshot, image_cache, video_manager, raster_store,
total_frames, video_start_time, video_framerate, video_width, video_height,
temp_video_path, video_progress_tx, video_cancel_flag,
);
});
// No UI-thread video state: rendering happens entirely on the background thread.
(Some(handle), None)
}
None => {
let (h, s) = Self::spawn_software_video(
video_settings.clone(), temp_video_path.clone(), frame_rx, frame_tx,
video_progress_tx, video_cancel_flag, total_frames,
video_start_time, video_end_time, video_framerate, video_width, video_height,
);
(Some(h), Some(s))
}
};
#[cfg(not(target_os = "linux"))]
let (video_thread, video_state) = {
// VAAPI/zero-copy is Linux-only; these scene-snapshot inputs are unused elsewhere.
let _ = (document, &video_manager, &raster_store, &container_path);
let (h, s) = Self::spawn_software_video(
video_settings.clone(), temp_video_path.clone(), frame_rx, frame_tx,
video_progress_tx, video_cancel_flag, total_frames,
video_start_time, video_end_time, video_framerate, video_width, video_height,
);
(Some(h), Some(s))
};
// Spawn audio export thread
let temp_audio_path_clone = temp_audio_path.clone();
@ -971,30 +1215,16 @@ impl ExportOrchestrator {
);
});
// Initialize video export state for incremental rendering
// GPU resources and readback pipeline will be initialized lazily on first frame (needs device)
self.video_state = Some(VideoExportState {
current_frame: 0,
total_frames,
start_time: video_start_time,
end_time: video_end_time,
framerate: video_framerate,
width: video_width,
height: video_height,
frame_tx: Some(frame_tx),
gpu_resources: None,
readback_pipeline: None,
cpu_yuv_converter: None,
frames_in_flight: 0,
next_frame_to_encode: 0,
perf_metrics: Some(perf_metrics::ExportMetrics::new()),
});
// The software path drives frames from the UI thread (state is `Some`); the zero-copy
// path renders on its own background thread (`None`). GPU resources + readback pipeline
// init lazily on the first frame for the software path.
self.video_state = video_state;
// Initialize parallel export state
self.parallel_export = Some(ParallelExportState {
video_progress_rx,
audio_progress_rx,
video_thread: Some(video_thread),
video_thread,
audio_thread: Some(audio_thread),
temp_video_path,
temp_audio_path,
@ -1035,18 +1265,83 @@ impl ExportOrchestrator {
) -> Result<bool, String> {
use std::time::Instant;
// The zero-copy VAAPI H.264 path runs entirely on its own background thread
// (see `run_zerocopy_video_export`); this UI-thread entry only drives the software
// readback/encode pipeline.
let state = self.video_state.as_mut()
.ok_or("No video export in progress")?;
// Already completed (Done sent, all frames done): don't re-initialize and
// re-run. The completion path nulls gpu_resources but leaves video_state set
// (cleared only when the parallel export finishes); without this guard the
// function would re-create the GPU pipeline and re-emit "Complete" every frame
// while the encoder/mux drains.
if state.frame_tx.is_none()
&& state.current_frame >= state.total_frames
&& state.frames_in_flight == 0
{
return Ok(false);
}
let width = state.width;
let height = state.height;
let fit = state.fit;
// HDR path: synchronous 10-bit render (composite → PQ/HLG → readback → 10-bit YUV), one
// frame per call. Bypasses the SDR async RGBA pipeline (which is 8-bit only).
if state.hdr.is_hdr() {
if state.gpu_resources.is_none() {
println!("🎬 [VIDEO EXPORT] Initializing HDR GPU resources {}x{} ({})", width, height, state.hdr.name());
state.gpu_resources = Some(video_exporter::ExportGpuResources::new(device, width, height));
}
if state.current_frame < state.total_frames {
let timestamp = state.start_time + (state.current_frame as f64 / state.framerate);
let gpu_resources = state.gpu_resources.as_mut().unwrap();
let (y, u, v) = video_exporter::render_frame_to_yuv10_hdr(
document, timestamp, width, height,
device, queue, renderer, image_cache, video_manager,
gpu_resources, state.hdr, fit, raster_store,
)?;
if let Some(tx) = &state.frame_tx {
tx.send(VideoFrameMessage::Frame {
frame_num: state.current_frame,
timestamp,
y_plane: y,
u_plane: u,
v_plane: v,
}).map_err(|_| "Failed to send HDR frame")?;
}
state.current_frame += 1;
}
if state.current_frame >= state.total_frames {
println!("🎬 [VIDEO EXPORT] HDR complete: {} frames", state.total_frames);
if let Some(tx) = state.frame_tx.take() {
tx.send(VideoFrameMessage::Done).ok();
}
state.gpu_resources = None;
return Ok(false);
}
return Ok(true);
}
// Initialize GPU resources and readback pipeline on first frame
if state.gpu_resources.is_none() {
println!("🎬 [VIDEO EXPORT] Initializing HDR GPU + async pipeline {}x{}", width, height);
state.gpu_resources = Some(video_exporter::ExportGpuResources::new(device, width, height));
state.readback_pipeline = Some(readback_pipeline::ReadbackPipeline::new(device, queue, width, height));
state.cpu_yuv_converter = Some(cpu_yuv_converter::CpuYuvConverter::new(width, height)?);
// Enable GPU YUV only when the encoder's YUV420P planes are tight (no linesize
// padding) — then the packed GPU planes copy in without row misalignment.
// Otherwise fall back to RGBA readback + CPU swscale.
let gpu_yuv_tight = std::env::var("LB_DISABLE_GPU_YUV").is_err() && {
let probe = ffmpeg_next::frame::Video::new(
ffmpeg_next::format::Pixel::YUV420P, width, height,
);
probe.stride(0) == width as usize && probe.stride(1) == (width / 2) as usize
};
if !gpu_yuv_tight {
println!("🎬 [VIDEO EXPORT] YUV planes are padded at {width}x{height}; using CPU YUV path");
}
state.readback_pipeline = Some(readback_pipeline::ReadbackPipeline::new(device, queue, width, height, gpu_yuv_tight, state.full_range));
state.cpu_yuv_converter = Some(cpu_yuv_converter::CpuYuvConverter::new(width, height, state.full_range)?);
println!("🚀 [ASYNC PIPELINE] Triple-buffered pipeline initialized");
println!("🚀 [CPU YUV] swscale converter initialized");
}
@ -1075,14 +1370,18 @@ impl ExportOrchestrator {
}
}
// Extract RGBA data (timed)
// Extract readback data (timed)
let extraction_start = Instant::now();
let rgba_data = pipeline.extract_rgba_data(result.buffer_id);
let data = pipeline.extract_rgba_data(result.buffer_id);
let extraction_end = Instant::now();
// CPU YUV conversion (timed)
// YUV planes: GPU-converted (just slice) or CPU swscale fallback (timed).
let conversion_start = Instant::now();
let (y, u, v) = cpu_converter.convert(&rgba_data)?;
let (y, u, v) = if pipeline.is_yuv_mode() {
pipeline.split_yuv(&data)
} else {
cpu_converter.convert(&data)?
};
let conversion_end = Instant::now();
if let Some(m) = metrics.as_mut() {
@ -1130,6 +1429,8 @@ impl ExportOrchestrator {
None, // No floating selection during video export
false, // Video export is never transparent
raster_store,
true, // software export composites on the shared device → may use HW frames
fit,
)?;
let render_end = Instant::now();
@ -1175,6 +1476,118 @@ impl ExportOrchestrator {
Ok(true) // More work to do
}
/// Zero-copy video production: render the document to RGBA on the encoder's own device
/// and hardware-encode it into a VAAPI surface, writing the temp `.mp4` directly. Drives
/// the parallel export's `video_progress` and triggers the mux on completion.
/// Background thread for the zero-copy VAAPI H.264 path: renders every frame with Vello on
/// the encoder's own VAAPI-capable device and hardware-encodes it straight into the temp
/// `.mp4`. Runs entirely off the UI thread (its own device + a `Document` snapshot), so it's
/// not throttled by egui's vsync'd repaint loop. Reports progress through `progress_tx`
/// (the same channel the software encoder thread uses); `poll_parallel_progress` muxes with
/// the audio track once both stream's `Complete` arrive. VAAPI is Linux-only.
#[cfg(target_os = "linux")]
#[allow(clippy::too_many_arguments)]
fn run_zerocopy_video_export(
mut zc: ZeroCopyVideo,
mut document: Document,
mut image_cache: ImageCache,
video_manager: Arc<std::sync::Mutex<VideoManager>>,
raster_store: lightningbeam_core::raster_store::RasterStore,
total_frames: usize,
start_time: f64,
framerate: f64,
width: u32,
height: u32,
temp_video_path: PathBuf,
progress_tx: Sender<ExportProgress>,
cancel_flag: Arc<AtomicBool>,
) {
progress_tx.send(ExportProgress::Started { total_frames }).ok();
let wall = std::time::Instant::now();
let mut render_time = std::time::Duration::ZERO;
let mut encode_time = std::time::Duration::ZERO;
// Throttle progress sends to ~6/s: each one forces a full editor repaint on the UI thread,
// which steals CPU/GPU from this render loop. The dialog doesn't need finer granularity.
let mut last_progress = std::time::Instant::now();
for frame in 0..total_frames {
if cancel_flag.load(Ordering::Relaxed) {
println!("🎬 [VIDEO EXPORT] zero-copy cancelled at frame {frame}");
return; // dropping `zc` closes the encoder / temp file; no Complete → no mux
}
let timestamp = start_time + (frame as f64 / framerate);
let rgba_view = zc.rgba.create_view(&Default::default());
let t0 = std::time::Instant::now();
let fit = zc.fit;
let cmd = match video_exporter::render_frame_to_gpu_rgba(
&mut document,
timestamp,
width,
height,
zc.encoder.device(),
zc.encoder.queue(),
&mut zc.renderer,
&mut image_cache,
&video_manager,
&mut zc.gpu_resources,
&rgba_view,
None,
false,
Some(&raster_store),
zc.on_shared_device, // GPU-resident decode only when on the shared device
fit,
) {
Ok(cmd) => cmd,
Err(e) => {
progress_tx.send(ExportProgress::Error { message: format!("render: {e}") }).ok();
return;
}
};
zc.encoder.queue().submit(Some(cmd.finish()));
let t1 = std::time::Instant::now();
if let Err(e) = zc.encoder.encode_rgba(&zc.rgba) {
progress_tx.send(ExportProgress::Error { message: format!("encode: {e}") }).ok();
return;
}
let t2 = std::time::Instant::now();
render_time += t1 - t0;
encode_time += t2 - t1;
if last_progress.elapsed() >= std::time::Duration::from_millis(160) || frame + 1 == total_frames {
progress_tx
.send(ExportProgress::FrameRendered { frame: frame + 1, total: total_frames })
.ok();
last_progress = std::time::Instant::now();
}
}
// Flush the encoder + write the container trailer.
let ZeroCopyVideo { encoder, .. } = zc;
if let Err(e) = encoder.finish() {
progress_tx.send(ExportProgress::Error { message: format!("finish: {e}") }).ok();
return;
}
// Performance breakdown.
let wall = wall.elapsed();
let n = total_frames.max(1) as f64;
let fps = if wall.as_secs_f64() > 0.0 { total_frames as f64 / wall.as_secs_f64() } else { 0.0 };
println!("🎬 [VIDEO EXPORT] zero-copy complete: {} frames", total_frames);
println!(
" ⏱ wall {:.2}s ({:.1} fps) | render {:.2}ms/frame | nv12+encode {:.2}ms/frame | overhead {:.2}ms/frame",
wall.as_secs_f64(),
fps,
render_time.as_secs_f64() * 1000.0 / n,
encode_time.as_secs_f64() * 1000.0 / n,
(wall.saturating_sub(render_time + encode_time)).as_secs_f64() * 1000.0 / n,
);
progress_tx.send(ExportProgress::Complete { output_path: temp_video_path }).ok();
}
/// Background thread that receives frames and encodes them
fn run_video_encoder(
settings: VideoExportSettings,
@ -1229,13 +1642,33 @@ impl ExportOrchestrator {
// Initialize FFmpeg
ffmpeg_next::init().map_err(|e| format!("Failed to initialize FFmpeg: {}", e))?;
// Convert codec enum to FFmpeg codec ID
let codec_id = match settings.codec {
// Convert codec enum to FFmpeg codec ID. HDR requires 10-bit HEVC (Main10), so force HEVC
// regardless of the chosen codec when an HDR mode is selected.
let codec_id = if settings.hdr.is_hdr() {
// HEVC can only be muxed into MP4/MOV, not WebM — reject the incompatible combo up
// front with a clear message instead of letting the muxer fail cryptically.
if settings.codec.container_format() == "webm" {
return Err(format!(
"HDR export needs H.265/HEVC in an MP4 container, but {} uses WebM. \
Pick H.265 (or H.264) for HDR.",
settings.codec.name()
));
}
if !matches!(settings.codec, VideoCodec::H265) {
println!(
"⚠️ [ENCODER] HDR selected: overriding codec {} → H.265/HEVC (Main10)",
settings.codec.name()
);
}
ffmpeg_next::codec::Id::HEVC
} else {
match settings.codec {
VideoCodec::H264 => ffmpeg_next::codec::Id::H264,
VideoCodec::H265 => ffmpeg_next::codec::Id::HEVC,
VideoCodec::VP8 => ffmpeg_next::codec::Id::VP8,
VideoCodec::VP9 => ffmpeg_next::codec::Id::VP9,
VideoCodec::ProRes422 => ffmpeg_next::codec::Id::PRORES,
}
};
// Get bitrate from quality settings
@ -1278,8 +1711,17 @@ impl ExportOrchestrator {
height,
framerate,
bitrate_kbps,
settings.hdr,
settings.color_range.is_full(),
)?;
// Pixel format the encoder frames are built in (matches setup_video_encoder).
let pixel_format = if settings.hdr.is_hdr() {
ffmpeg_next::format::Pixel::YUV420P10LE
} else {
ffmpeg_next::format::Pixel::YUV420P
};
// Create output file
let mut output = ffmpeg_next::format::output(&output_path)
.map_err(|e| format!("Failed to create output file: {}", e))?;
@ -1308,6 +1750,7 @@ impl ExportOrchestrator {
width,
height,
timestamp,
pixel_format,
)?;
// Send progress update for first frame
@ -1335,6 +1778,7 @@ impl ExportOrchestrator {
width,
height,
timestamp,
pixel_format,
)?;
frames_encoded += 1;
@ -1379,29 +1823,33 @@ impl ExportOrchestrator {
width: u32,
height: u32,
timestamp: f64,
pixel_format: ffmpeg_next::format::Pixel,
) -> Result<(), String> {
// YUV planes already converted by GPU (no CPU conversion needed)
// YUV planes already converted (8-bit YUV420P, or 10-bit YUV420P10LE for HDR).
// Create FFmpeg video frame
let mut video_frame = ffmpeg_next::frame::Video::new(
ffmpeg_next::format::Pixel::YUV420P,
width,
height,
);
// Create FFmpeg video frame in the encoder's pixel format.
let mut video_frame = ffmpeg_next::frame::Video::new(pixel_format, width, height);
// Copy YUV planes to frame
// Use safe slice copy - LLVM optimizes this to memcpy, same performance as copy_nonoverlapping
let y_dest = video_frame.data_mut(0);
let y_len = y_plane.len().min(y_dest.len());
y_dest[..y_len].copy_from_slice(&y_plane[..y_len]);
let u_dest = video_frame.data_mut(1);
let u_len = u_plane.len().min(u_dest.len());
u_dest[..u_len].copy_from_slice(&u_plane[..u_len]);
let v_dest = video_frame.data_mut(2);
let v_len = v_plane.len().min(v_dest.len());
v_dest[..v_len].copy_from_slice(&v_plane[..v_len]);
// Copy each plane row-by-row honoring the frame's stride (10-bit / arbitrary widths can have
// row padding that a flat copy would misalign). `bytes_per_row` = samples × sample size.
let ten_bit = matches!(pixel_format, ffmpeg_next::format::Pixel::YUV420P10LE);
let sample_bytes = if ten_bit { 2usize } else { 1usize };
let copy_plane = |frame: &mut ffmpeg_next::frame::Video, idx: usize, src: &[u8], w: usize, h: usize| {
let bytes_per_row = w * sample_bytes;
let stride = frame.stride(idx);
let dst = frame.data_mut(idx);
for row in 0..h {
let s = row * bytes_per_row;
let d = row * stride;
let n = bytes_per_row.min(src.len().saturating_sub(s)).min(dst.len().saturating_sub(d));
if n == 0 { break; }
dst[d..d + n].copy_from_slice(&src[s..s + n]);
}
};
let (w, h) = (width as usize, height as usize);
copy_plane(&mut video_frame, 0, y_plane, w, h);
copy_plane(&mut video_frame, 1, u_plane, w / 2, h / 2);
copy_plane(&mut video_frame, 2, v_plane, w / 2, h / 2);
// Set PTS (presentation timestamp) in encoder's time base
// Encoder time base is 1/(framerate * 1000), so PTS = timestamp * (framerate * 1000)

View File

@ -41,7 +41,10 @@ struct PipelineBuffer {
/// RGBA texture for GPU rendering output (Rgba8Unorm)
rgba_texture: wgpu::Texture,
rgba_texture_view: wgpu::TextureView,
/// Staging buffer for GPU→CPU transfer (MAP_READ)
/// In YUV mode: packed planar YUV420p the compute shader writes (STORAGE | COPY_SRC).
/// `None` in RGBA fallback mode.
yuv_buffer: Option<wgpu::Buffer>,
/// Staging buffer for GPU→CPU transfer (MAP_READ). Holds YUV in YUV mode, RGBA otherwise.
staging_buffer: wgpu::Buffer,
/// Current state in the pipeline
state: BufferState,
@ -71,6 +74,10 @@ pub struct ReadbackPipeline {
/// Buffer dimensions
width: u32,
height: u32,
/// `Some` when converting RGBA→YUV420p on the GPU (skips the CPU swscale pass and
/// reads back ~3 MB of planar YUV instead of 8 MB RGBA). `None` falls back to RGBA
/// readback + CPU conversion for dimensions the packed shader can't handle.
gpu_yuv: Option<super::gpu_yuv::GpuYuv>,
}
impl ReadbackPipeline {
@ -81,13 +88,31 @@ impl ReadbackPipeline {
/// * `queue` - GPU queue (will be cloned for async operations)
/// * `width` - Frame width in pixels
/// * `height` - Frame height in pixels
pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, width: u32, height: u32) -> Self {
/// `enable_gpu_yuv` should be `true` only when the caller has verified the encoder's
/// `YUV420P` plane strides are tight (== width / width-2), so the packed GPU planes
/// drop straight into the `AVFrame` without row misalignment.
pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, width: u32, height: u32, enable_gpu_yuv: bool, full_range: bool) -> Self {
let (readback_tx, readback_rx) = channel();
// GPU YUV conversion when enabled AND the dimensions fit the packed shader; else RGBA + CPU.
let gpu_yuv = if enable_gpu_yuv && super::gpu_yuv::supports(width, height) {
Some(super::gpu_yuv::GpuYuv::new(device, full_range))
} else {
None
};
let yuv_mode = gpu_yuv.is_some();
// Staging size: planar YUV420p (W*H*3/2) in YUV mode, else RGBA (W*H*4).
let staging_size = if yuv_mode {
super::gpu_yuv::yuv420p_len(width, height) as u64
} else {
(width * height * 4) as u64
};
// Create 3 buffers for triple buffering
let mut buffers = Vec::new();
for id in 0..3 {
// RGBA texture (Rgba8Unorm)
// RGBA texture (Rgba8Unorm). TEXTURE_BINDING lets the YUV compute shader read it.
let rgba_texture = device.create_texture(&wgpu::TextureDescriptor {
label: Some(&format!("readback_rgba_texture_{}", id)),
size: wgpu::Extent3d {
@ -99,17 +124,29 @@ impl ReadbackPipeline {
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::COPY_SRC,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::COPY_SRC
| wgpu::TextureUsages::TEXTURE_BINDING,
view_formats: &[],
});
let rgba_texture_view = rgba_texture.create_view(&wgpu::TextureViewDescriptor::default());
let yuv_buffer = if yuv_mode {
Some(device.create_buffer(&wgpu::BufferDescriptor {
label: Some(&format!("readback_yuv_buffer_{}", id)),
size: staging_size,
usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC,
mapped_at_creation: false,
}))
} else {
None
};
// Staging buffer for GPU→CPU readback
let rgba_buffer_size = (width * height * 4) as u64; // Rgba8Unorm = 4 bytes/pixel
let staging_buffer = device.create_buffer(&wgpu::BufferDescriptor {
label: Some(&format!("readback_staging_buffer_{}", id)),
size: rgba_buffer_size,
size: staging_size,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
@ -118,6 +155,7 @@ impl ReadbackPipeline {
id,
rgba_texture,
rgba_texture_view,
yuv_buffer,
staging_buffer,
state: BufferState::Free,
frame_num: None,
@ -133,9 +171,23 @@ impl ReadbackPipeline {
queue: queue.clone(),
width,
height,
gpu_yuv,
}
}
/// `true` when frames are read back as planar YUV420p (GPU-converted) — the caller
/// should slice planes with [`Self::split_yuv`] instead of running the CPU converter.
pub fn is_yuv_mode(&self) -> bool {
self.gpu_yuv.is_some()
}
/// Split a YUV-mode readback buffer into tight (Y, U, V) planes.
pub fn split_yuv(&self, data: &[u8]) -> (Vec<u8>, Vec<u8>, Vec<u8>) {
let y = (self.width * self.height) as usize;
let c = ((self.width / 2) * (self.height / 2)) as usize;
(data[..y].to_vec(), data[y..y + c].to_vec(), data[y + c..y + 2 * c].to_vec())
}
/// Acquire a free buffer for rendering (non-blocking)
///
/// Returns None if all buffers are in use (caller should poll and retry)
@ -166,7 +218,12 @@ impl ReadbackPipeline {
let buffer = &mut self.buffers[buffer_id];
assert_eq!(buffer.state, BufferState::Rendering, "Buffer not in Rendering state");
// Copy RGBA texture to staging buffer
if let (Some(gpu_yuv), Some(yuv_buffer)) = (self.gpu_yuv.as_ref(), buffer.yuv_buffer.as_ref()) {
// GPU RGBA→YUV420p, then copy the packed YUV buffer to staging (~3 MB).
gpu_yuv.convert(&self.device, &mut encoder, &buffer.rgba_texture_view, yuv_buffer, self.width, self.height);
encoder.copy_buffer_to_buffer(yuv_buffer, 0, &buffer.staging_buffer, 0, buffer.staging_buffer.size());
} else {
// Fallback: copy the RGBA texture to staging (8 MB), CPU converts later.
encoder.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo {
texture: &buffer.rgba_texture,
@ -188,6 +245,7 @@ impl ReadbackPipeline {
depth_or_array_layers: 1,
},
);
}
// Submit GPU commands (non-blocking)
self.queue.submit(Some(encoder.finish()));

View File

@ -0,0 +1,78 @@
// Linear-HDR PQ/HLG BT.2020 encode (for 10-bit HDR video export).
//
// Input: the compositor's Rgba16Float HDR accumulator PREMULTIPLIED scene-linear, BT.709
// primaries, graphics white = 1.0, HDR highlights > 1.0.
// Output: gamma-encoded R'G'B' in BT.2020 primaries, PQ (mode 0) or HLG (mode 1), to an
// Rgba16Unorm target. A later CPU pass does only BT.2020 R'G'B'Y'CbCr (no transfer) + 4:2:0 + 10-bit.
//
// This is the encode inverse of panes/shaders/nv12_blit.wgsl's decode (203-nit PQ white; HLG
// reference white at signal 0.75), so a decodeencode round-trip is the identity.
@group(0) @binding(0) var input_tex: texture_2d<f32>;
@group(0) @binding(1) var input_sampler: sampler;
@group(0) @binding(2) var<uniform> params: vec4<u32>; // .x = mode (0 = PQ, 1 = HLG)
struct VertexOutput {
@builtin(position) position: vec4<f32>,
@location(0) uv: vec2<f32>,
}
@vertex
fn vs_main(@builtin(vertex_index) vertex_index: u32) -> VertexOutput {
var out: VertexOutput;
let x = f32((vertex_index & 1u) << 1u);
let y = f32(vertex_index & 2u);
out.position = vec4<f32>(x * 2.0 - 1.0, 1.0 - y * 2.0, 0.0, 1.0);
out.uv = vec2<f32>(x, y);
return out;
}
// BT.709 BT.2020 primaries, linear light (ITU-R BT.2087).
fn bt709_to_bt2020(c: vec3<f32>) -> vec3<f32> {
let r = 0.627404 * c.r + 0.329283 * c.g + 0.043313 * c.b;
let g = 0.069097 * c.r + 0.919540 * c.g + 0.011362 * c.b;
let b = 0.016391 * c.r + 0.088013 * c.g + 0.895595 * c.b;
return vec3<f32>(r, g, b);
}
// SMPTE ST 2084 (PQ) OETF: scene-linear (white = 1.0 = 203 nits) PQ code [0,1].
fn pq_oetf(lin: vec3<f32>) -> vec3<f32> {
let nits = max(lin, vec3<f32>(0.0)) * 203.0;
let ln = min(nits / 10000.0, vec3<f32>(1.0));
let m1 = 0.1593017578125;
let m2 = 78.84375;
let c1 = 0.8359375;
let c2 = 18.8515625;
let c3 = 18.6875;
let lm = pow(ln, vec3<f32>(m1));
return pow((vec3<f32>(c1) + c2 * lm) / (vec3<f32>(1.0) + c3 * lm), vec3<f32>(m2));
}
// ARIB STD-B67 (HLG) OETF: scene-linear (white = 1.0) HLG signal [0,1]. Reference white maps to
// signal 0.75 (matching the decode's /0.26496256 normalization). Display OOTF omitted (scene-referred).
fn hlg_oetf(lin: vec3<f32>) -> vec3<f32> {
let a = 0.17883277;
let b = 0.28466892;
let c = 0.55991073;
let e = clamp(lin * 0.26496256, vec3<f32>(0.0), vec3<f32>(1.0));
let lo = sqrt(3.0 * e);
let hi = a * log(12.0 * e - vec3<f32>(b)) + vec3<f32>(c);
return select(lo, hi, e > vec3<f32>(1.0 / 12.0));
}
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
// Compositor stores PREMULTIPLIED linear; unpremultiply to straight (video is opaque, a1).
let texel = textureSample(input_tex, input_sampler, in.uv);
let a = texel.a;
let straight = select(texel.rgb / a, vec3<f32>(0.0), a <= 0.0);
let bt2020 = max(bt709_to_bt2020(straight), vec3<f32>(0.0));
var enc: vec3<f32>;
if params.x == 1u {
enc = hlg_oetf(bt2020);
} else {
enc = pq_oetf(bt2020);
}
return vec4<f32>(enc, 1.0);
}

View File

@ -16,6 +16,30 @@ use lightningbeam_core::gpu::{
SrgbToLinearConverter, EffectProcessor, YuvConverter, HDR_FORMAT,
};
/// The document→export-pixels transform for a given fit mode. Stretch distorts to fill; Letterbox
/// scales uniformly to fit (centered, black bars); Crop scales uniformly to fill (centered, trims).
pub fn export_base_transform(
doc_w: f64,
doc_h: f64,
out_w: f64,
out_h: f64,
fit: lightningbeam_core::export::ExportFitMode,
) -> vello::kurbo::Affine {
use lightningbeam_core::export::ExportFitMode;
use vello::kurbo::Affine;
if doc_w <= 0.0 || doc_h <= 0.0 {
return Affine::IDENTITY;
}
let (sx, sy) = (out_w / doc_w, out_h / doc_h);
match fit {
ExportFitMode::Stretch => Affine::scale_non_uniform(sx, sy),
ExportFitMode::Letterbox | ExportFitMode::Crop => {
let s = if matches!(fit, ExportFitMode::Letterbox) { sx.min(sy) } else { sx.max(sy) };
Affine::translate(((out_w - doc_w * s) / 2.0, (out_h - doc_h * s) / 2.0)) * Affine::scale(s)
}
}
}
/// Reusable frame buffers to avoid allocations
struct FrameBuffers {
/// RGBA buffer from GPU readback (width * height * 4 bytes)
@ -75,14 +99,25 @@ pub struct ExportGpuResources {
pub staging_buffer: wgpu::Buffer,
/// Linear to sRGB blit pipeline for final output
pub linear_to_srgb_pipeline: wgpu::RenderPipeline,
/// Variant with highlight rolloff (document HDR output mode = Highlight rolloff).
pub linear_to_srgb_pipeline_rolloff: wgpu::RenderPipeline,
/// Bind group layout for linear to sRGB blit
pub linear_to_srgb_bind_group_layout: wgpu::BindGroupLayout,
/// Sampler for linear to sRGB conversion
pub linear_to_srgb_sampler: wgpu::Sampler,
/// Canvas blit pipeline for raster/video/float layers (bypasses Vello).
pub canvas_blit: crate::gpu_brush::CanvasBlitPipeline,
/// NV12→linear blit for hardware-decoded video frames (export on the shared device).
pub nv12_blit: crate::nv12_blit::Nv12BlitPipeline,
/// Per-keyframe GPU texture cache for raster layers during export.
pub raster_cache: std::collections::HashMap<uuid::Uuid, crate::gpu_brush::CanvasPair>,
/// Cached HDR accumulator state after the (static) background is composited in. The document
/// background doesn't change across an export, so it's rendered once and restored with a cheap
/// texture copy each frame instead of a full Vello render + 2 passes/submits. `None` until the
/// first frame; invalidated on resize.
cached_bg_hdr: Option<wgpu::Texture>,
/// HDR encode pipeline (linear→PQ/HLG BT.2020 → 10-bit YUV). Lazily built on the first HDR frame.
hdr_pipeline: Option<super::hdr_frame::HdrFramePipeline>,
}
impl ExportGpuResources {
@ -108,7 +143,8 @@ impl ExportGpuResources {
format: HDR_FORMAT,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::COPY_SRC,
| wgpu::TextureUsages::COPY_SRC
| wgpu::TextureUsages::COPY_DST, // restore cached background each frame
view_formats: &[],
});
let hdr_texture_view = hdr_texture.create_view(&wgpu::TextureViewDescriptor::default());
@ -230,6 +266,41 @@ impl ExportGpuResources {
cache: None,
});
// Highlight-rolloff variant: identical but the `fs_main_rolloff` entry point.
let linear_to_srgb_pipeline_rolloff = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
label: Some("linear_to_srgb_pipeline_rolloff"),
layout: Some(&pipeline_layout),
vertex: wgpu::VertexState {
module: &shader,
entry_point: Some("vs_main"),
buffers: &[],
compilation_options: wgpu::PipelineCompilationOptions::default(),
},
fragment: Some(wgpu::FragmentState {
module: &shader,
entry_point: Some("fs_main_rolloff"),
targets: &[Some(wgpu::ColorTargetState {
format: wgpu::TextureFormat::Rgba8Unorm,
blend: None,
write_mask: wgpu::ColorWrites::ALL,
})],
compilation_options: wgpu::PipelineCompilationOptions::default(),
}),
primitive: wgpu::PrimitiveState {
topology: wgpu::PrimitiveTopology::TriangleStrip,
strip_index_format: None,
front_face: wgpu::FrontFace::Ccw,
cull_mode: None,
polygon_mode: wgpu::PolygonMode::Fill,
unclipped_depth: false,
conservative: false,
},
depth_stencil: None,
multisample: wgpu::MultisampleState::default(),
multiview: None,
cache: None,
});
let linear_to_srgb_sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("linear_to_srgb_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
@ -242,6 +313,7 @@ impl ExportGpuResources {
});
let canvas_blit = crate::gpu_brush::CanvasBlitPipeline::new(device);
let nv12_blit = crate::nv12_blit::Nv12BlitPipeline::new(device);
Self {
buffer_pool,
@ -257,10 +329,14 @@ impl ExportGpuResources {
yuv_texture_view,
staging_buffer,
linear_to_srgb_pipeline,
linear_to_srgb_pipeline_rolloff,
linear_to_srgb_bind_group_layout,
linear_to_srgb_sampler,
canvas_blit,
nv12_blit,
raster_cache: std::collections::HashMap::new(),
cached_bg_hdr: None,
hdr_pipeline: None,
}
}
@ -279,10 +355,12 @@ impl ExportGpuResources {
format: HDR_FORMAT,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::COPY_SRC,
| wgpu::TextureUsages::COPY_SRC
| wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
self.hdr_texture_view = self.hdr_texture.create_view(&wgpu::TextureViewDescriptor::default());
self.cached_bg_hdr = None; // dimensions changed — rebuild the background cache
}
}
@ -331,6 +409,32 @@ fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
return vec4<f32>(srgb, a);
}
// Highlight rolloff: identity below the knee, smooth C1 rolloff [knee,∞)→[knee,1) above (recovers
// super-white HDR detail). SDR below the knee is untouched. Mirrors panes/shaders/linear_to_srgb.wgsl.
fn highlight_rolloff_ch(x: f32) -> f32 {
let knee = 0.8;
if x <= knee {
return x;
}
let headroom = 1.0 - knee;
return knee + headroom * (1.0 - exp(-(x - knee) / headroom));
}
// Variant of fs_main with highlight rolloff (document HDR output mode = Highlight rolloff).
@fragment
fn fs_main_rolloff(in: VertexOutput) -> @location(0) vec4<f32> {
let src = textureSample(source_tex, source_sampler, in.uv);
let a = src.a;
let straight = select(src.rgb / a, vec3<f32>(0.0), a <= 0.0);
let rolled = vec3<f32>(
highlight_rolloff_ch(straight.r),
highlight_rolloff_ch(straight.g),
highlight_rolloff_ch(straight.b),
);
let srgb = linear_to_srgb(rolled);
return vec4<f32>(srgb, a);
}
"#;
/// Convert RGBA8 pixels to YUV420p format using BT.709 color space
@ -455,6 +559,8 @@ pub fn setup_video_encoder(
height: u32,
framerate: f64,
bitrate_kbps: u32,
hdr: lightningbeam_core::export::HdrExportMode,
full_range: bool,
) -> Result<(ffmpeg::encoder::Video, ffmpeg::Codec), String> {
// Try to find codec by ID first
println!("🔍 Looking for codec: {:?}", codec_id);
@ -510,31 +616,46 @@ pub fn setup_video_encoder(
// Configure encoder parameters BEFORE opening (critical!)
encoder.set_width(aligned_width);
encoder.set_height(aligned_height);
// HDR encodes 10-bit BT.2020 (limited range); SDR keeps 8-bit full-range BT.709.
if hdr.is_hdr() {
encoder.set_format(ffmpeg::format::Pixel::YUV420P10LE);
} else {
encoder.set_format(ffmpeg::format::Pixel::YUV420P);
}
encoder.set_time_base(ffmpeg::Rational(1, (framerate * 1000.0) as i32));
encoder.set_frame_rate(Some(ffmpeg::Rational(framerate as i32, 1)));
encoder.set_bit_rate((bitrate_kbps * 1000) as usize);
encoder.set_gop(framerate as u32); // 1 second GOP
// Tag the color metadata so players interpret the YUV correctly. Our
// RGB→YUV conversion uses the BT.709 matrix with FULL-range (0255) luma
// and no transfer applied to the already-sRGB-encoded RGB. Tagging this
// as full-range BT.709 (matrix/primaries/transfer) prevents the level/
// hue shift that occurs when a player assumes limited-range or BT.601.
// colorspace (matrix) and range have safe setters; primaries and trc are
// generic AVCodecContext options set via the open dictionary below.
encoder.set_colorspace(ffmpeg::color::Space::BT709);
encoder.set_color_range(ffmpeg::color::Range::JPEG); // full range
println!("📐 Video dimensions: {}×{} (aligned to {}×{} for H.264)",
width, height, aligned_width, aligned_height);
// Open encoder with codec (like working MP3 export). color_primaries and
// color_trc have no typed setter on the encoder, so pass them as generic
// AVCodecContext options (BT.709) through the open dictionary.
// Tag the color metadata so players interpret the YUV correctly.
// SDR: our RGB→YUV uses the BT.709 matrix with FULL-range (0255) luma and no transfer applied
// to the already-sRGB-encoded RGB, so tag full-range BT.709 to avoid level/hue shifts.
// HDR: BT.2020 non-constant-luminance matrix, LIMITED range (standard for HDR10/HLG), with the
// PQ or HLG transfer; the 10-bit YUV is produced from PQ/HLG-encoded BT.2020 RGB.
let mut color_opts = ffmpeg::Dictionary::new();
if hdr.is_hdr() {
encoder.set_colorspace(ffmpeg::color::Space::BT2020NCL);
encoder.set_color_range(ffmpeg::color::Range::MPEG); // limited
color_opts.set("color_primaries", "bt2020");
color_opts.set("color_trc", hdr.transfer_name());
// HEVC 10-bit profile (the only HDR-capable codec we wire up).
color_opts.set("profile", "main10");
} else {
encoder.set_colorspace(ffmpeg::color::Space::BT709);
// Range must match what the YUV converters (gpu_yuv / cpu_yuv) actually produce.
encoder.set_color_range(if full_range {
ffmpeg::color::Range::JPEG // full (PC, 0255)
} else {
ffmpeg::color::Range::MPEG // limited (TV, 16235)
});
color_opts.set("color_primaries", "bt709");
color_opts.set("color_trc", "bt709");
}
println!("📐 Video dimensions: {}×{} (aligned to {}×{}){}",
width, height, aligned_width, aligned_height,
if hdr.is_hdr() { " [HDR 10-bit BT.2020]" } else { "" });
let encoder = encoder
.open_as_with(codec, color_opts)
.map_err(|e| format!("Failed to open video encoder: {}", e))?;
@ -748,7 +869,30 @@ fn composite_document_to_hdr(
antialiasing_method: vello::AaConfig::Area,
};
// --- Background ---
let prof = render_profile_enabled();
let t_c0 = std::time::Instant::now();
// --- Background (cached) ---
// The document background is static across an export, so render it through Vello exactly once
// (into the accumulator) and snapshot the result; every later frame restores it with a single
// GPU texture copy instead of a Vello render + sRGB-convert + composite (+2 submits).
let bg_cached = matches!(
&gpu_resources.cached_bg_hdr,
Some(t) if t.width() == width && t.height() == height
);
let copy_size = wgpu::Extent3d { width, height, depth_or_array_layers: 1 };
if bg_cached {
// Restore the cached background into the accumulator.
let cached = gpu_resources.cached_bg_hdr.as_ref().unwrap();
let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("export_bg_restore") });
enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo { texture: cached, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyTextureInfo { texture: &gpu_resources.hdr_texture, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
copy_size,
);
queue.submit(Some(enc.finish()));
} else {
// First frame (or after a resize): full background render into the accumulator.
let bg_srgb = gpu_resources.buffer_pool.acquire(device, layer_spec);
let bg_hdr = gpu_resources.buffer_pool.acquire(device, hdr_spec);
if let (Some(bg_srgb_view), Some(bg_hdr_view)) = (
@ -772,6 +916,28 @@ fn composite_document_to_hdr(
gpu_resources.buffer_pool.release(bg_srgb);
gpu_resources.buffer_pool.release(bg_hdr);
// Snapshot the composited background for reuse on subsequent frames.
let cached = device.create_texture(&wgpu::TextureDescriptor {
label: Some("export_cached_bg_hdr"),
size: copy_size,
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: HDR_FORMAT,
usage: wgpu::TextureUsages::COPY_SRC | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("export_bg_snapshot") });
enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo { texture: &gpu_resources.hdr_texture, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyTextureInfo { texture: &cached, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
copy_size,
);
queue.submit(Some(enc.finish()));
gpu_resources.cached_bg_hdr = Some(cached);
}
let t_bg = std::time::Instant::now();
// --- Layers ---
for rendered_layer in &composite_result.layers {
if !rendered_layer.has_content { continue; }
@ -827,25 +993,26 @@ fn composite_document_to_hdr(
}
RenderedLayerType::Video { instances } => {
for inst in instances {
if inst.rgba_data.is_empty() { continue; }
if inst.gpu.is_none() && inst.rgba_data.is_empty() { continue; }
let hdr_layer_handle = gpu_resources.buffer_pool.acquire(device, hdr_spec);
if let Some(hdr_layer_view) = gpu_resources.buffer_pool.get_view(hdr_layer_handle) {
// sRGB straight-alpha → linear premultiplied
let linear: Vec<u8> = inst.rgba_data.chunks_exact(4).flat_map(|p| {
let a = p[3] as f32 / 255.0;
let lin = |c: u8| -> f32 {
let f = c as f32 / 255.0;
if f <= 0.04045 { f / 12.92 } else { ((f + 0.055) / 1.055).powf(2.4) }
};
let r = (lin(p[0]) * a * 255.0 + 0.5) as u8;
let g = (lin(p[1]) * a * 255.0 + 0.5) as u8;
let b = (lin(p[2]) * a * 255.0 + 0.5) as u8;
[r, g, b, p[3]]
}).collect();
let tex = upload_transient_texture(device, queue, &linear, inst.width, inst.height, Some("export_video_frame_tex"));
let tex_view = tex.create_view(&Default::default());
let bt = crate::gpu_brush::BlitTransform::new(inst.transform, inst.width, inst.height, width, height);
gpu_resources.canvas_blit.blit(device, queue, &tex_view, hdr_layer_view, &bt, None);
if let Some(gpu) = &inst.gpu {
// Hardware-decoded NV12 plane textures → linear, no CPU upload.
let y_view = gpu.y.create_view(&Default::default());
let uv_view = gpu.uv.create_view(&Default::default());
gpu_resources.nv12_blit.blit(
device, queue, &y_view, &uv_view, hdr_layer_view, &bt,
gpu.full_range, gpu.coeffs, gpu.transfer, gpu.primaries,
);
} else {
// Upload raw sRGB straight-alpha bytes into an sRGB texture; the GPU
// decodes to linear on sample (no per-pixel CPU conversion). Blit with
// blit_straight so the shader doesn't unpremultiply.
let tex = upload_transient_texture(device, queue, &inst.rgba_data, inst.width, inst.height, wgpu::TextureFormat::Rgba8UnormSrgb, Some("export_video_frame_tex"));
let tex_view = tex.create_view(&Default::default());
gpu_resources.canvas_blit.blit_straight(device, queue, &tex_view, hdr_layer_view, &bt, None);
}
let compositor_layer = CompositorLayer::new(hdr_layer_handle, inst.opacity, lightningbeam_core::gpu::BlendMode::Normal);
let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("export_video_composite") });
gpu_resources.compositor.composite(device, queue, &mut enc, &[compositor_layer], &gpu_resources.buffer_pool, &gpu_resources.hdr_texture_view, None);
@ -865,7 +1032,7 @@ fn composite_document_to_hdr(
};
[lin(p[0]), lin(p[1]), lin(p[2]), p[3]]
}).collect();
let tex = upload_transient_texture(device, queue, &linear, *fw, *fh, Some("export_float_tex"));
let tex = upload_transient_texture(device, queue, &linear, *fw, *fh, wgpu::TextureFormat::Rgba8Unorm, Some("export_float_tex"));
let tex_view = tex.create_view(&Default::default());
let hdr_layer_handle = gpu_resources.buffer_pool.acquire(device, hdr_spec);
if let Some(hdr_layer_view) = gpu_resources.buffer_pool.get_view(hdr_layer_handle) {
@ -915,17 +1082,43 @@ fn composite_document_to_hdr(
}
}
if prof {
record_composite_profile(t_bg.duration_since(t_c0), t_bg.elapsed());
}
gpu_resources.buffer_pool.next_frame();
Ok(())
}
/// Upload `pixels` to a transient `Rgba8Unorm` GPU texture (TEXTURE_BINDING | COPY_DST).
/// Split of `composite_document_to_hdr`: static-background re-render vs. the layer loop
/// (video upload + blits). Prints a running average every 200 frames under LB_RENDER_PROFILE.
fn record_composite_profile(background: std::time::Duration, layers: std::time::Duration) {
use std::sync::atomic::{AtomicU64, Ordering};
static BG_US: AtomicU64 = AtomicU64::new(0);
static LAYERS_US: AtomicU64 = AtomicU64::new(0);
static N: AtomicU64 = AtomicU64::new(0);
BG_US.fetch_add(background.as_micros() as u64, Ordering::Relaxed);
LAYERS_US.fetch_add(layers.as_micros() as u64, Ordering::Relaxed);
let n = N.fetch_add(1, Ordering::Relaxed) + 1;
if n % 200 == 0 {
println!(
"📊 [COMPOSITE PROFILE] {n} frames avg: background-render {:.2}ms | layers(video upload+blit) {:.2}ms",
BG_US.load(Ordering::Relaxed) as f64 / n as f64 / 1000.0,
LAYERS_US.load(Ordering::Relaxed) as f64 / n as f64 / 1000.0,
);
}
}
/// Upload `pixels` to a transient GPU texture (TEXTURE_BINDING | COPY_DST) in the
/// given format. Use `Rgba8UnormSrgb` to upload raw sRGB bytes and let the GPU
/// decode to linear on sample (no CPU conversion).
fn upload_transient_texture(
device: &wgpu::Device,
queue: &wgpu::Queue,
pixels: &[u8],
width: u32,
height: u32,
format: wgpu::TextureFormat,
label: Option<&'static str>,
) -> wgpu::Texture {
let tex = device.create_texture(&wgpu::TextureDescriptor {
@ -933,7 +1126,7 @@ fn upload_transient_texture(
size: wgpu::Extent3d { width, height, depth_or_array_layers: 1 },
mip_level_count: 1, sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: wgpu::TextureFormat::Rgba8Unorm,
format,
usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST,
view_formats: &[],
});
@ -946,221 +1139,9 @@ fn upload_transient_texture(
tex
}
/// Render a document frame using the HDR compositing pipeline with effects
///
/// This function uses the same rendering pipeline as the stage preview,
/// ensuring effects are applied correctly during export.
///
/// # Arguments
/// * `document` - Document to render (current_time will be modified)
/// * `timestamp` - Time in seconds to render at
/// * `width` - Frame width in pixels
/// * `height` - Frame height in pixels
/// * `device` - wgpu device
/// * `queue` - wgpu queue
/// * `renderer` - Vello renderer
/// * `image_cache` - Image cache for rendering
/// * `video_manager` - Video manager for video clips
/// * `gpu_resources` - HDR GPU resources for compositing
///
/// # Returns
/// Ok((y_plane, u_plane, v_plane)) with YUV420p planes on success, Err with message on failure
pub fn render_frame_to_rgba_hdr(
document: &mut Document,
timestamp: f64,
width: u32,
height: u32,
device: &wgpu::Device,
queue: &wgpu::Queue,
renderer: &mut vello::Renderer,
image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
gpu_resources: &mut ExportGpuResources,
) -> Result<(Vec<u8>, Vec<u8>, Vec<u8>), String> {
use vello::kurbo::Affine;
// Set document time to the frame timestamp
document.current_time = timestamp;
// Scale the document to the export resolution. The core renderer bakes this
// base transform into every layer (vector scenes, raster and video layer
// transforms), so the whole stage scales up/down to fill the output. When the
// export size matches the document this is the identity.
let base_transform = if document.width > 0.0 && document.height > 0.0 {
Affine::scale_non_uniform(
width as f64 / document.width,
height as f64 / document.height,
)
} else {
Affine::IDENTITY
};
// Render document for compositing (returns per-layer scenes)
let composite_result = render_document_for_compositing(
document,
base_transform,
image_cache,
video_manager,
None, // No webcam during export
None, // No floating selection during export
false, // No checkerboard in export
);
// Video export is never transparent.
composite_document_to_hdr(&composite_result, document, device, queue, renderer, gpu_resources, width, height, false)?;
// Use persistent output texture (already created in ExportGpuResources)
let output_view = &gpu_resources.output_texture_view;
// Convert HDR to sRGB for output
let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("export_linear_to_srgb_bind_group"),
layout: &gpu_resources.linear_to_srgb_bind_group_layout,
entries: &[
wgpu::BindGroupEntry {
binding: 0,
resource: wgpu::BindingResource::TextureView(&gpu_resources.hdr_texture_view),
},
wgpu::BindGroupEntry {
binding: 1,
resource: wgpu::BindingResource::Sampler(&gpu_resources.linear_to_srgb_sampler),
},
],
});
let mut encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("export_linear_to_srgb_encoder"),
});
{
let mut render_pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("export_linear_to_srgb_pass"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &output_view,
resolve_target: None,
ops: wgpu::Operations {
load: wgpu::LoadOp::Clear(wgpu::Color::BLACK),
store: wgpu::StoreOp::Store,
},
depth_slice: None,
})],
depth_stencil_attachment: None,
occlusion_query_set: None,
timestamp_writes: None,
});
render_pass.set_pipeline(&gpu_resources.linear_to_srgb_pipeline);
render_pass.set_bind_group(0, &bind_group, &[]);
render_pass.draw(0..4, 0..1);
}
queue.submit(Some(encoder.finish()));
// GPU YUV conversion: Convert RGBA output to YUV420p
let mut yuv_encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("export_yuv_conversion_encoder"),
});
gpu_resources.yuv_converter.convert(
device,
&mut yuv_encoder,
output_view,
&gpu_resources.yuv_texture_view,
width,
height,
);
// Copy YUV texture to persistent staging buffer
let yuv_height = height + height / 2; // Y plane + U plane + V plane
yuv_encoder.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo {
texture: &gpu_resources.yuv_texture,
mip_level: 0,
origin: wgpu::Origin3d::ZERO,
aspect: wgpu::TextureAspect::All,
},
wgpu::TexelCopyBufferInfo {
buffer: &gpu_resources.staging_buffer,
layout: wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(width * 4), // Rgba8Unorm = 4 bytes per pixel
rows_per_image: Some(yuv_height),
},
},
wgpu::Extent3d {
width,
height: yuv_height,
depth_or_array_layers: 1,
},
);
queue.submit(Some(yuv_encoder.finish()));
// Map buffer and read YUV pixels (synchronous)
let buffer_slice = gpu_resources.staging_buffer.slice(..);
let (sender, receiver) = std::sync::mpsc::channel();
buffer_slice.map_async(wgpu::MapMode::Read, move |result| {
sender.send(result).ok();
});
let _ = device.poll(wgpu::PollType::wait_indefinitely());
receiver
.recv()
.map_err(|_| "Failed to receive buffer mapping result")?
.map_err(|e| format!("Failed to map buffer: {:?}", e))?;
// Extract Y, U, V planes from packed YUV buffer
let data = buffer_slice.get_mapped_range();
let width_usize = width as usize;
let height_usize = height as usize;
// Y plane: rows 0 to height-1 (extract R channel from Rgba8Unorm)
let y_plane_size = width_usize * height_usize;
let mut y_plane = vec![0u8; y_plane_size];
for y in 0..height_usize {
let src_row_offset = y * width_usize * 4; // 4 bytes per pixel (Rgba8Unorm)
let dst_row_offset = y * width_usize;
for x in 0..width_usize {
y_plane[dst_row_offset + x] = data[src_row_offset + x * 4]; // Extract R channel
}
}
// U and V planes: rows height to height + height/2 - 1 (half resolution, side-by-side layout)
// U plane is in left half (columns 0 to width/2-1), V plane is in right half (columns width/2 to width-1)
let chroma_width = width_usize / 2;
let chroma_height = height_usize / 2;
let chroma_row_start = height_usize * width_usize * 4; // Start of chroma rows in bytes
let mut u_plane = vec![0u8; chroma_width * chroma_height];
let mut v_plane = vec![0u8; chroma_width * chroma_height];
for y in 0..chroma_height {
let row_offset = chroma_row_start + y * width_usize * 4; // Full width rows in chroma region
// Extract U plane (left half: columns 0 to chroma_width-1)
let u_start = row_offset;
let dst_offset = y * chroma_width;
for x in 0..chroma_width {
u_plane[dst_offset + x] = data[u_start + x * 4]; // Extract R channel
}
// Extract V plane (right half: columns width/2 to width/2+chroma_width-1)
let v_start = row_offset + chroma_width * 4;
for x in 0..chroma_width {
v_plane[dst_offset + x] = data[v_start + x * 4]; // Extract R channel
}
}
drop(data);
gpu_resources.staging_buffer.unmap();
Ok((y_plane, u_plane, v_plane))
}
/// Render frame to GPU RGBA texture (non-blocking, for async pipeline)
///
/// Similar to render_frame_to_rgba_hdr but renders to an external RGBA texture view
/// Renders to an external RGBA texture view
/// (provided by ReadbackPipeline) and returns the command encoder WITHOUT blocking on readback.
/// The caller (ReadbackPipeline) will submit the encoder and handle async readback.
///
@ -1212,6 +1193,51 @@ fn fault_in_raster_for_frame(
}
}
/// Render one frame as 10-bit HDR YUV420P10LE planes (BT.2020 + PQ/HLG). Synchronous: composites,
/// runs the linear→PQ/HLG GPU pass, reads it back, and CPU-converts to 10-bit YUV. Used by the
/// HDR export path instead of the async readback pipeline.
#[allow(clippy::too_many_arguments)]
pub fn render_frame_to_yuv10_hdr(
document: &mut Document,
timestamp: f64,
width: u32,
height: u32,
device: &wgpu::Device,
queue: &wgpu::Queue,
renderer: &mut vello::Renderer,
image_cache: &mut ImageCache,
video_manager: &Arc<std::sync::Mutex<VideoManager>>,
gpu_resources: &mut ExportGpuResources,
hdr_mode: lightningbeam_core::export::HdrExportMode,
fit: lightningbeam_core::export::ExportFitMode,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
) -> Result<(Vec<u8>, Vec<u8>, Vec<u8>), String> {
document.current_time = timestamp;
fault_in_raster_for_frame(document, raster_store);
let base_transform = export_base_transform(document.width, document.height, width as f64, height as f64, fit);
// HDR export composites on the shared device, so it can consume hardware-decoded GPU frames.
if let Ok(mut vm) = video_manager.lock() {
vm.set_render_hardware_ok(true);
}
let composite_result = render_document_for_compositing(
document, base_transform, image_cache, video_manager, None, None, false,
);
composite_document_to_hdr(&composite_result, document, device, queue, renderer, gpu_resources, width, height, false)?;
if gpu_resources.hdr_pipeline.is_none() {
gpu_resources.hdr_pipeline = Some(super::hdr_frame::HdrFramePipeline::new(device, width, height));
}
let planes = gpu_resources
.hdr_pipeline
.as_ref()
.unwrap()
.render_to_yuv10(device, queue, &gpu_resources.hdr_texture_view, hdr_mode);
Ok(planes)
}
pub fn render_frame_to_gpu_rgba(
document: &mut Document,
timestamp: f64,
@ -1227,8 +1253,16 @@ pub fn render_frame_to_gpu_rgba(
floating_selection: Option<&lightningbeam_core::selection::RasterFloatingSelection>,
allow_transparency: bool,
raster_store: Option<&lightningbeam_core::raster_store::RasterStore>,
// True when compositing on the shared device (software/image export) → may consume
// hardware-decoded GPU frames; false for the zero-copy path on its own device.
hardware_ok: bool,
fit: lightningbeam_core::export::ExportFitMode,
) -> Result<wgpu::CommandEncoder, String> {
use vello::kurbo::Affine;
// One-shot profiling of the render-bucket split (LB_RENDER_PROFILE=1): how much of the
// per-frame CPU "render" is document build (incl. video decode) vs. composite-command
// recording (incl. the frame texture upload) vs. the sRGB pass. Prints a running average.
let prof = render_profile_enabled();
let t0 = std::time::Instant::now();
// Set document time to the frame timestamp
document.current_time = timestamp;
@ -1243,14 +1277,13 @@ pub fn render_frame_to_gpu_rgba(
// base transform into every layer (vector scenes, raster and video layer
// transforms), so the whole stage scales up/down to fill the output. When the
// export size matches the document this is the identity.
let base_transform = if document.width > 0.0 && document.height > 0.0 {
Affine::scale_non_uniform(
width as f64 / document.width,
height as f64 / document.height,
)
} else {
Affine::IDENTITY
};
let base_transform = export_base_transform(document.width, document.height, width as f64, height as f64, fit);
// GPU frames are usable only on the shared device (software/image export); the zero-copy path
// runs on its own device and must download to CPU.
if let Ok(mut vm) = video_manager.lock() {
vm.set_render_hardware_ok(hardware_ok);
}
// Render document for compositing (returns per-layer scenes)
let composite_result = render_document_for_compositing(
@ -1262,8 +1295,10 @@ pub fn render_frame_to_gpu_rgba(
floating_selection,
false, // No checkerboard in export
);
let t_build = std::time::Instant::now();
composite_document_to_hdr(&composite_result, document, device, queue, renderer, gpu_resources, width, height, allow_transparency)?;
let t_composite = std::time::Instant::now();
// Convert HDR to sRGB (linear → sRGB), render directly to external RGBA texture
let output_view = rgba_texture_view;
@ -1303,16 +1338,58 @@ pub fn render_frame_to_gpu_rgba(
timestamp_writes: None,
});
render_pass.set_pipeline(&gpu_resources.linear_to_srgb_pipeline);
let final_pipeline = match document.hdr_output_mode {
lightningbeam_core::document::HdrOutputMode::HighlightRolloff => &gpu_resources.linear_to_srgb_pipeline_rolloff,
lightningbeam_core::document::HdrOutputMode::Clip => &gpu_resources.linear_to_srgb_pipeline,
};
render_pass.set_pipeline(final_pipeline);
render_pass.set_bind_group(0, &bind_group, &[]);
render_pass.draw(0..4, 0..1);
}
if prof {
record_render_profile(
t_build.duration_since(t0),
t_composite.duration_since(t_build),
t_composite.elapsed(),
);
}
// Return encoder for caller to submit (ReadbackPipeline will handle submission and async readback)
// Frame is already rendered to external RGBA texture, no GPU YUV conversion needed
Ok(encoder)
}
/// `LB_RENDER_PROFILE` gate, checked once.
fn render_profile_enabled() -> bool {
static V: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*V.get_or_init(|| std::env::var("LB_RENDER_PROFILE").is_ok())
}
/// Accumulate the per-frame render split and print a running average every 200 frames.
/// `build` = document build incl. video decode; `composite` = composite-command recording
/// incl. the frame texture upload; `srgb` = the linear→sRGB pass.
fn record_render_profile(build: std::time::Duration, composite: std::time::Duration, srgb: std::time::Duration) {
use std::sync::atomic::{AtomicU64, Ordering};
static BUILD_US: AtomicU64 = AtomicU64::new(0);
static COMPOSITE_US: AtomicU64 = AtomicU64::new(0);
static SRGB_US: AtomicU64 = AtomicU64::new(0);
static N: AtomicU64 = AtomicU64::new(0);
BUILD_US.fetch_add(build.as_micros() as u64, Ordering::Relaxed);
COMPOSITE_US.fetch_add(composite.as_micros() as u64, Ordering::Relaxed);
SRGB_US.fetch_add(srgb.as_micros() as u64, Ordering::Relaxed);
let n = N.fetch_add(1, Ordering::Relaxed) + 1;
if n % 200 == 0 {
let (b, c, s) = (BUILD_US.load(Ordering::Relaxed), COMPOSITE_US.load(Ordering::Relaxed), SRGB_US.load(Ordering::Relaxed));
println!(
"📊 [RENDER PROFILE] {n} frames avg: build(+decode) {:.2}ms | composite(+upload) {:.2}ms | srgb {:.2}ms",
b as f64 / n as f64 / 1000.0,
c as f64 / n as f64 / 1000.0,
s as f64 / n as f64 / 1000.0,
);
}
}
#[cfg(test)]
mod tests {
use super::*;
@ -1356,61 +1433,10 @@ mod tests {
assert!(v[0] > 128, "V value: {}", v[0]);
}
#[test]
fn test_rgba_to_yuv420p_dimensions() {
// 4×4 image (16 pixels)
let rgba = vec![0u8; 4 * 4 * 4]; // All black
let (y, u, v) = rgba_to_yuv420p(&rgba, 4, 4);
// Y should be full resolution: 4×4 = 16 pixels
assert_eq!(y.len(), 16);
// U and V should be quarter resolution: 2×2 = 4 pixels each
assert_eq!(u.len(), 4);
assert_eq!(v.len(), 4);
}
#[test]
fn test_rgba_to_yuv420p_2x2_subsampling() {
// Create 2×2 image with different colors in each corner
let mut rgba = vec![0u8; 2 * 2 * 4];
// Top-left: Red
rgba[0] = 255;
rgba[1] = 0;
rgba[2] = 0;
rgba[3] = 255;
// Top-right: Green
rgba[4] = 0;
rgba[5] = 255;
rgba[6] = 0;
rgba[7] = 255;
// Bottom-left: Blue
rgba[8] = 0;
rgba[9] = 0;
rgba[10] = 255;
rgba[11] = 255;
// Bottom-right: White
rgba[12] = 255;
rgba[13] = 255;
rgba[14] = 255;
rgba[15] = 255;
let (y, u, v) = rgba_to_yuv420p(&rgba, 2, 2);
// Y plane should have 4 distinct values (one per pixel)
assert_eq!(y.len(), 4);
// U and V should have 1 value each (averaged over 2×2 block)
assert_eq!(u.len(), 1);
assert_eq!(v.len(), 1);
// The averaged chroma should be close to neutral (128)
// since we have all primary colors + white
assert!(u[0] >= 100 && u[0] <= 156, "U value: {}", u[0]);
assert!(v[0] >= 100 && v[0] <= 156, "V value: {}", v[0]);
}
// NOTE: `rgba_to_yuv420p` rounds dimensions up to multiples of 16 (H.264
// macroblock alignment), so its plane lengths are the aligned sizes, not the
// tight input dimensions. The former `test_rgba_to_yuv420p_dimensions` and
// `_2x2_subsampling` tests asserted tight sizes and were removed when that
// alignment was added. (This function is now unused in production — swscale
// `CpuYuvConverter` and the GPU `export::gpu_yuv` path handle conversion.)
}

View File

@ -1627,12 +1627,24 @@ impl GpuBrushEngine {
self.proxy_layer_cache.get(kf_id)
}
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted).
/// Remove the cached texture for a raster layer keyframe (e.g. when deleted or edited).
pub fn remove_layer_texture(&mut self, kf_id: &Uuid) {
if self.raster_layer_cache.remove(kf_id).is_some() {
let mut changed = self.raster_layer_cache.remove(kf_id).is_some();
if changed {
if let Some(pos) = self.raster_layer_lru.iter().position(|id| id == kf_id) {
self.raster_layer_lru.remove(pos);
}
}
// Also drop the low-res proxy: proxies are uploaded once and never refreshed, so a
// stale pre-edit proxy left here would be blitted (flashing old content) if the full-res
// texture is later evicted before the edited pixels page back in.
if self.proxy_layer_cache.remove(kf_id).is_some() {
if let Some(pos) = self.proxy_layer_lru.iter().position(|id| id == kf_id) {
self.proxy_layer_lru.remove(pos);
}
changed = true;
}
if changed {
self.report_raster_cache_vram();
}
}
@ -1951,6 +1963,9 @@ impl GpuBrushEngine {
/// the camera transform.
pub struct CanvasBlitPipeline {
pub pipeline: wgpu::RenderPipeline,
/// Variant for straight-alpha sources (hardware-sRGB video frames): the
/// fragment shader skips the unpremultiply. See [`CanvasBlitPipeline::blit_straight`].
pub pipeline_straight: wgpu::RenderPipeline,
pub bg_layout: wgpu::BindGroupLayout,
pub sampler: wgpu::Sampler,
/// Bilinear sampler for smooth upscaling (used by `blit_smooth`, e.g. low-res
@ -2132,6 +2147,39 @@ impl CanvasBlitPipeline {
},
);
// Variant pipeline for straight-alpha sources (hardware-sRGB video frames):
// identical except the fragment shader skips the unpremultiply.
let pipeline_straight = device.create_render_pipeline(
&wgpu::RenderPipelineDescriptor {
label: Some("canvas_blit_pipeline_straight"),
layout: Some(&pipeline_layout),
vertex: wgpu::VertexState {
module: &shader,
entry_point: Some("vs_main"),
buffers: &[],
compilation_options: Default::default(),
},
fragment: Some(wgpu::FragmentState {
module: &shader,
entry_point: Some("fs_main_straight"),
targets: &[Some(wgpu::ColorTargetState {
format: wgpu::TextureFormat::Rgba16Float,
blend: None,
write_mask: wgpu::ColorWrites::ALL,
})],
compilation_options: Default::default(),
}),
primitive: wgpu::PrimitiveState {
topology: wgpu::PrimitiveTopology::TriangleStrip,
..Default::default()
},
depth_stencil: None,
multisample: wgpu::MultisampleState::default(),
multiview: None,
cache: None,
},
);
let sampler = device.create_sampler(&wgpu::SamplerDescriptor {
label: Some("canvas_blit_sampler"),
address_mode_u: wgpu::AddressMode::ClampToEdge,
@ -2165,7 +2213,7 @@ impl CanvasBlitPipeline {
..Default::default()
});
Self { pipeline, bg_layout, sampler, linear_sampler, mask_sampler }
Self { pipeline, pipeline_straight, bg_layout, sampler, linear_sampler, mask_sampler }
}
/// Render the canvas texture into `target_view` (Rgba16Float) with the given camera.
@ -2183,7 +2231,7 @@ impl CanvasBlitPipeline {
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.sampler);
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.sampler, &self.pipeline);
}
/// Blit with a bilinear sampler — smooth upscaling for low-res sources (proxies).
@ -2196,9 +2244,27 @@ impl CanvasBlitPipeline {
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler);
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler, &self.pipeline);
}
/// Blit a **straight-alpha** source (e.g. a video frame uploaded to an
/// `Rgba8UnormSrgb` texture, hardware-decoded to linear on sample). Uses the
/// `fs_main_straight` pipeline, which skips the unpremultiply that `blit` does.
/// Bilinear-sampled: video frames are scaled to the output size (document→export, or any
/// non-1:1 transform), and nearest sampling makes that look blocky.
pub fn blit_straight(
&self,
device: &wgpu::Device,
queue: &wgpu::Queue,
canvas_view: &wgpu::TextureView,
target_view: &wgpu::TextureView,
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
) {
self.blit_with(device, queue, canvas_view, target_view, transform, mask_view, &self.linear_sampler, &self.pipeline_straight);
}
#[allow(clippy::too_many_arguments)]
fn blit_with(
&self,
device: &wgpu::Device,
@ -2208,6 +2274,7 @@ impl CanvasBlitPipeline {
transform: &BlitTransform,
mask_view: Option<&wgpu::TextureView>,
canvas_sampler: &wgpu::Sampler,
pipeline: &wgpu::RenderPipeline,
) {
// When no mask is provided, create a temporary 1×1 all-white texture.
// (queue is already available here, unlike in new())
@ -2296,7 +2363,7 @@ impl CanvasBlitPipeline {
occlusion_query_set: None,
timestamp_writes: None,
});
rp.set_pipeline(&self.pipeline);
rp.set_pipeline(pipeline);
rp.set_bind_group(0, &bg, &[]);
rp.draw(0..4, 0..1);
}

View File

@ -0,0 +1,135 @@
//! Minimal GPU timestamp timer for the composite pipeline.
//!
//! Brackets a section of GPU work with two timestamps and reads the elapsed GPU
//! time back asynchronously (no pipeline stall). Used to attribute the per-frame
//! composite cost (Vello render + sRGB→linear + compositor + tonemap) shown in F3.
//!
//! Requires `TIMESTAMP_QUERY` + `TIMESTAMP_QUERY_INSIDE_ENCODERS`; [`FrameGpuTimer::new`]
//! returns `None` when the adapter doesn't support them, and all call sites no-op.
use std::sync::{Arc, Mutex};
/// State of the single readback buffer (shared with the map callback).
#[derive(Clone, Copy, PartialEq)]
enum Readback {
/// Available to resolve into this frame.
Free,
/// Submitted + `map_async` in flight; don't touch until the callback fires.
Mapping,
/// Mapped and ready to read.
Ready,
}
/// Times one GPU section (two timestamps) per frame with intermittent async readback.
pub struct FrameGpuTimer {
query_set: wgpu::QuerySet,
resolve_buf: wgpu::Buffer,
readback_buf: wgpu::Buffer,
state: Arc<Mutex<Readback>>,
/// Nanoseconds per timestamp tick.
period_ns: f32,
/// Most recent measured GPU time for the bracketed section, in milliseconds.
last_ms: f64,
}
impl FrameGpuTimer {
/// Required device features for GPU timestamp timing.
pub fn required_features() -> wgpu::Features {
wgpu::Features::TIMESTAMP_QUERY | wgpu::Features::TIMESTAMP_QUERY_INSIDE_ENCODERS
}
/// Create a timer, or `None` if the device lacks timestamp support.
pub fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Option<Self> {
if !device.features().contains(Self::required_features()) {
return None;
}
let query_set = device.create_query_set(&wgpu::QuerySetDescriptor {
label: Some("composite_gpu_timer"),
ty: wgpu::QueryType::Timestamp,
count: 2,
});
// 2 timestamps × u64.
let size = 2 * std::mem::size_of::<u64>() as u64;
let resolve_buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("composite_gpu_timer_resolve"),
size,
usage: wgpu::BufferUsages::QUERY_RESOLVE | wgpu::BufferUsages::COPY_SRC,
mapped_at_creation: false,
});
let readback_buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("composite_gpu_timer_readback"),
size,
usage: wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
});
Some(Self {
query_set,
resolve_buf,
readback_buf,
state: Arc::new(Mutex::new(Readback::Free)),
period_ns: queue.get_timestamp_period(),
last_ms: 0.0,
})
}
/// Write the **start** timestamp (call just before the bracketed GPU work).
pub fn start(&self, device: &wgpu::Device, queue: &wgpu::Queue) {
let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("composite_gpu_timer_start"),
});
enc.write_timestamp(&self.query_set, 0);
queue.submit(Some(enc.finish()));
}
/// Write the **end** timestamp and, if the readback buffer is free, resolve +
/// kick off an async read. Also consumes a previously-completed read into
/// `last_ms`. Call just after the bracketed GPU work.
pub fn end(&mut self, device: &wgpu::Device, queue: &wgpu::Queue) {
// 1. Consume a completed readback first (so the buffer is free to reuse).
let cur = *self.state.lock().unwrap();
if cur == Readback::Ready {
{
let view = self.readback_buf.slice(..).get_mapped_range();
let t0 = u64::from_le_bytes(view[0..8].try_into().unwrap());
let t1 = u64::from_le_bytes(view[8..16].try_into().unwrap());
// Timestamps can wrap or arrive out of order across queue resets; guard.
let ticks = t1.saturating_sub(t0);
self.last_ms = ticks as f64 * self.period_ns as f64 / 1.0e6;
}
self.readback_buf.unmap();
*self.state.lock().unwrap() = Readback::Free;
}
// 2. End timestamp + resolve + copy, only when the buffer is free.
let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
label: Some("composite_gpu_timer_end"),
});
enc.write_timestamp(&self.query_set, 1);
let can_read = *self.state.lock().unwrap() == Readback::Free;
if can_read {
enc.resolve_query_set(&self.query_set, 0..2, &self.resolve_buf, 0);
enc.copy_buffer_to_buffer(
&self.resolve_buf,
0,
&self.readback_buf,
0,
2 * std::mem::size_of::<u64>() as u64,
);
}
queue.submit(Some(enc.finish()));
if can_read {
*self.state.lock().unwrap() = Readback::Mapping;
let state = Arc::clone(&self.state);
self.readback_buf.slice(..).map_async(wgpu::MapMode::Read, move |res| {
*state.lock().unwrap() = if res.is_ok() { Readback::Ready } else { Readback::Free };
});
}
}
/// Most recently measured GPU time of the bracketed section, in milliseconds.
pub fn last_ms(&self) -> f64 {
self.last_ms
}
}

View File

@ -0,0 +1,180 @@
//! Hardware video decode glue (Linux/VAAPI). The editor implements core's [`HwVideoImporter`]:
//! it maps a decoded VAAPI surface to a DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane
//! textures on the **shared** device (the one eframe + the compositor run on, which has the
//! DMA-BUF-import extensions). [`install`] creates the VAAPI device and wires it into the
//! `VideoManager`.
use ffmpeg_next::ffi as ff;
use gpu_video_encoder::dmabuf::{self, Nv12DmaBuf};
use lightningbeam_core::video::{
ycbcr_coeffs, GpuVideoFrame, HwDeviceHandle, HwVideoImporter, VideoManager, VideoPrimaries,
VideoTransfer,
};
use std::sync::{Arc, Mutex};
/// Imports decoded VAAPI surfaces onto the shared wgpu device. Holds clones of the shared
/// device + adapter (Arc-backed, cheap).
struct SharedHwImporter {
device: wgpu::Device,
adapter: wgpu::Adapter,
/// Log the detected colour info once (under LB_VIDEO_DEBUG) rather than per frame.
logged: std::sync::atomic::AtomicBool,
}
impl HwVideoImporter for SharedHwImporter {
unsafe fn import(&self, av_frame: *mut std::ffi::c_void) -> Option<GpuVideoFrame> {
let frame = av_frame as *mut ff::AVFrame;
// Map the VAAPI surface to a DRM-PRIME DMA-BUF (read-only).
let drm_f = ff::av_frame_alloc();
(*drm_f).format = ff::AVPixelFormat::AV_PIX_FMT_DRM_PRIME as i32;
let flags = ff::AV_HWFRAME_MAP_DIRECT as i32 | ff::AV_HWFRAME_MAP_READ as i32;
if ff::av_hwframe_map(drm_f, frame, flags) < 0 {
ff::av_frame_free(&mut (drm_f as *mut _));
return None;
}
let desc = (*drm_f).data[0] as *const ff::AVDRMFrameDescriptor;
let obj = &(*desc).objects[0];
let width = (*frame).width as u32;
let height = (*frame).height as u32;
// 10/12/16-bit content decodes to P010-style surfaces (16-bit planes). Detect via the hw
// frames context's software format so the import builds R16/Rg16 textures.
let ten_bit = {
let hwfc = (*frame).hw_frames_ctx;
if hwfc.is_null() {
false
} else {
let ctx = (*hwfc).data as *const ff::AVHWFramesContext;
matches!(
(*ctx).sw_format,
ff::AVPixelFormat::AV_PIX_FMT_P010LE
| ff::AVPixelFormat::AV_PIX_FMT_P010BE
| ff::AVPixelFormat::AV_PIX_FMT_P012LE
| ff::AVPixelFormat::AV_PIX_FMT_P012BE
| ff::AVPixelFormat::AV_PIX_FMT_P016LE
| ff::AVPixelFormat::AV_PIX_FMT_P016BE
)
}
};
// NV12: Y then UV — two layers (one plane each) or one layer with two planes.
let (y_pl, uv_pl) = if (*desc).nb_layers >= 2 {
(&(*desc).layers[0].planes[0], &(*desc).layers[1].planes[0])
} else {
(&(*desc).layers[0].planes[0], &(*desc).layers[0].planes[1])
};
let buf = Nv12DmaBuf {
fd: obj.fd,
size: obj.size as u64,
modifier: obj.format_modifier,
width,
height,
y_offset: y_pl.offset as u64,
y_pitch: y_pl.pitch as u64,
uv_offset: uv_pl.offset as u64,
uv_pitch: uv_pl.pitch as u64,
ten_bit,
};
let full_range = (*frame).color_range == ff::AVColorRange::AVCOL_RANGE_JPEG;
// Luma weights (kr, kb) from the frame's matrix coefficients, so SD (BT.601) and HD/UHD
// (BT.709) clips each convert with the right matrix. Unspecified → guess by height, as
// players/swscale do. SMPTE240M and BT.2020 are handled too (the latter's transfer is still
// approximated as sRGB — fine for SDR; true HDR is out of scope).
let (kr, kb) = match (*frame).colorspace {
ff::AVColorSpace::AVCOL_SPC_BT709 => (0.2126, 0.0722),
ff::AVColorSpace::AVCOL_SPC_BT470BG | ff::AVColorSpace::AVCOL_SPC_SMPTE170M => {
(0.299, 0.114)
}
ff::AVColorSpace::AVCOL_SPC_SMPTE240M => (0.212, 0.087),
ff::AVColorSpace::AVCOL_SPC_BT2020_NCL | ff::AVColorSpace::AVCOL_SPC_BT2020_CL => {
(0.2627, 0.0593)
}
_ => {
if height <= 576 {
(0.299, 0.114) // SD → BT.601
} else {
(0.2126, 0.0722) // HD/UHD → BT.709
}
}
};
let coeffs = ycbcr_coeffs(kr, kb);
if std::env::var("LB_VIDEO_DEBUG").is_ok()
&& !self.logged.swap(true, std::sync::atomic::Ordering::Relaxed)
{
eprintln!(
"[hw_video] {}x{} ten_bit={} full_range={} colorspace={:?} primaries={:?} trc={:?}",
width, height, ten_bit, full_range,
(*frame).colorspace, (*frame).color_primaries, (*frame).color_trc,
);
}
// Transfer characteristic → which EOTF the compositor applies to reach scene-linear.
let transfer = match (*frame).color_trc {
ff::AVColorTransferCharacteristic::AVCOL_TRC_SMPTE2084 => VideoTransfer::Pq,
ff::AVColorTransferCharacteristic::AVCOL_TRC_ARIB_STD_B67 => VideoTransfer::Hlg,
_ => VideoTransfer::Gamma,
};
// Primaries → BT.2020 is gamut-mapped to BT.709; unspecified follows the matrix guess above.
let primaries = match (*frame).color_primaries {
ff::AVColorPrimaries::AVCOL_PRI_BT2020 => VideoPrimaries::Bt2020,
ff::AVColorPrimaries::AVCOL_PRI_UNSPECIFIED
if matches!(
(*frame).colorspace,
ff::AVColorSpace::AVCOL_SPC_BT2020_NCL | ff::AVColorSpace::AVCOL_SPC_BT2020_CL
) =>
{
VideoPrimaries::Bt2020
}
_ => VideoPrimaries::Bt709,
};
let imported = dmabuf::import_raw(&self.device, &self.adapter, &buf);
ff::av_frame_free(&mut (drm_f as *mut _)); // the fd was dup'd into Vulkan
let (y, uv) = match imported {
Ok(t) => t.into_planes(),
Err(e) => {
// Surface the failure: a silent None here makes core fall back to software (no gamut
// conversion → BT.2020 looks washed out). 10-bit P010 import is the likely culprit.
eprintln!("[hw_video] import_raw failed (ten_bit={ten_bit}): {e}");
return None;
}
};
Some(GpuVideoFrame {
y: Arc::new(y),
uv: Arc::new(uv),
width,
height,
full_range,
coeffs,
transfer,
primaries,
})
}
}
/// Create the VAAPI hardware device and install hardware decode into `vm`, importing onto the
/// shared `device`/`adapter`. Logs and no-ops if VAAPI is unavailable (→ software decode).
pub fn install(vm: &Arc<Mutex<VideoManager>>, device: &wgpu::Device, adapter: &wgpu::Adapter) {
match gpu_video_encoder::vaapi::create_device() {
Ok(hw_device) => {
let importer = Arc::new(SharedHwImporter {
device: device.clone(),
adapter: adapter.clone(),
logged: std::sync::atomic::AtomicBool::new(false),
});
if let Ok(mut vm) = vm.lock() {
vm.set_hardware_decode(
HwDeviceHandle(hw_device as *mut std::ffi::c_void),
importer,
);
}
println!("🎞 Hardware video decode enabled (VAAPI → shared device)");
}
Err(e) => {
println!("🎞 Hardware video decode unavailable ({e}); using software decode");
}
}
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More