- Mobile double-tap is now a single tool-independent priority chain: object →
enter (movie clip or group); empty inside a clip → exit one level; empty at
root → zoom-to-fit. Desktop keeps its own select-tool double-click (gated).
- Inspector "tap outside to dismiss" now only HIDES the sheet (with a dismissed
flag reset on selection change) instead of clearing the selection — the earlier
clear clobbered the selection before menu/omnibutton actions ran, breaking
Group/Convert/Cut/Copy. Delete also clears focus so it dismisses the inspector.
- Breadcrumb shows the full editing path (Scene 1 › … › current) via a new
SharedPaneState.editing_clip_path / EditingContext::clip_path(); each ancestor
crumb jumps to that level (pending_exit_to_depth → EditingContext::exit_to_depth).
- Remove the "Vello Stage" debug overlay; move the breadcrumb up to the top-left.
Gestures (Stage + Timeline):
- Pinch-zoom via zoom_delta() (touch pinch + Ctrl+wheel), unified with the raw-wheel
path so Ctrl+wheel zooms exactly once; plain wheel zoom + trackpad pan preserved.
- Double-tap empty → zoom-to-fit (Stage: artboard via zoom_to_fit; Timeline: new
fit_to_project over the full duration).
- Double-tap-drag on empty → transient marquee regardless of the active tool.
Long-press context menu: a shared SharedPaneState.mobile_context_menu that panes
populate with (label, MenuAction) items on secondary_clicked(); the mobile shell
renders one persistent popup (styled like the timeline menu) and dispatches via
pending_menu_actions. Stage offers Cut/Copy/Duplicate/Delete for clip instances and
Cut/Copy/Convert-to-movie-clip/Delete for geometry.
Fixes surfaced along the way:
- Geometry delete now frees selected fills (free_fill) and GCs isolated vertices
(new VectorGraph::gc_isolated_vertices) — no more orphaned fills / phantom snap
points; applies to the Delete key too.
- clipboard_delete_selection clears focus so deleting dismisses the mobile inspector.
- Mobile inspector: appears on pointer release (not press) so drags aren't
interrupted; reflows only when the tapped point is actually behind the sheet; taps
outside the sheet dismiss it; rounded-corner border no longer detaches.
Add mobile::apply_touch_style(ctx) — applied every frame when mobile is active —
enlarging egui's global spacing/sizing (interact_size, button padding, slider
width/rail, combo width/height, icon sizes, scrollbar, body/button text) so the
standard widgets in panes and dialogs are finger-sized. Desktop is untouched.
Rework the mobile inspector header into a single row using real egui widgets:
title (left, truncating) + Timeline/Nodes jump buttons + ✕ close, laid out with
egui layouts so they inherit the touch sizing and theme visuals instead of being
hand-painted.
Add Theme::var(name, ctx) (CSS custom-property getter) and Theme::apply_to_egui(ctx)
which maps the palette onto egui's global Visuals so standard widgets share the
theme colors. Add a mobile::palette::Palette built from the theme each frame and
replace the duplicated per-file C_* color constants across all mobile modules
(omni, topbar, stack, inspector, transport, intent, shell) with it. Add --scrim
and --accent-* category vars to styles.css. Apply the theme visuals every frame in
update(). Mobile and the main UI now share one CSS-variable-driven palette that
responds to light/dark/user themes.
Add mobile::dialog_width(ctx, desired) — a min()-clamp that's a no-op on wide
desktop screens. Clamp the Export and Export Progress modals and the Sample
Import window to it. Render Preferences as an egui::Modal (dim backdrop, centered,
no title bar) with a screen-fit scroll height when mobile_active(); desktop keeps
its window. Extracted the preferences body into render_body() shared by both paths.
Reserve a slim top bar in the mobile shell showing the project filename (or
"Lightningbeam" when unsaved) on the left, and ⌕/⋯ on the right. ⋯ opens a
curated overflow sheet (Save/Save As/Open/New/Import/Export/Undo/Redo/
Preferences); ⌕ opens a command palette that searches the whole menu tree
(flattened from MenuItemDef::menu_structure()). Both dispatch MenuActions.
Completes the P4 omnibutton + global menu.
Omnibutton (mobile/omni.rs): a context FAB whose radial holds the active layer's
drawing tools laid out in a staggered inner/outer arc, plus a "+New" petal
(create Vector/Audio/MIDI/Raster/Video layer, Group, Import, and Split/Duplicate
when a clip is selected — dispatched as MenuActions) and a "more" petal that
opens a full-tools grid when the layer has extra tools. Contextual: the tool ring
only appears when the Stage is in the visible window; off-Stage the FAB is a
"+New" create button. Tools/creates use Lucide icons. Pane-internal adds
(nodes/notes/instruments) deferred to P6.
Playback fix: document.current_time (read by the renderer for animation eval and
video-frame decode) was only synced from the audio playback position inside
TimelinePane::render_content. In the mobile shell, panes render on demand, so with
the timeline off-screen current_time went stale and the Stage froze during
playback. Move the sync into the per-frame block in update() so it runs regardless
of which panes are visible.
- Add `is_mobile` to SharedPaneState (set in build_frame). On mobile the timeline
layer-header column collapses to a minimal type-colored swatch (1:2 w:h, ~30px),
brighter/outlined when active/selected; tapping it selects the layer (handled by
the existing header-column input path, which now uses the narrower width).
Desktop headers unchanged.
- Inspector reflow: the sheet overlaps the stack by default, but when the selected
pane would be covered by the sheet, the stack reflows (renders above the sheet)
so it stays visible. Restore on dismiss is automatic since reflow only changes
the render rect; manual divider moves while the sheet is up persist. Adds
stack::pane_bottom_in and inspector::target_slot.
- New mobile/inspector.rs: when something is selected/focused, a bottom sheet
overlays the lower stack (no reflow) above the transport, showing the object's
properties by reusing InfopanelPane full-bleed. Header shows a title + ✕ to
deselect; jump-to chips (Timeline, Nodes) reframe the stack to that surface
with the selection kept. Grab handle resizes the sheet.
- Make the Infopanel Layer section editable (benefits desktop too): Name (text,
commit on Enter/blur), Opacity and Volume sliders (live preview, single-undo
on release), and Mute/Solo/Lock toggles. Backs the plan to reduce mobile layer
headers to a minimal swatch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When LB_MOBILE_UI is set, the desktop start screen is replaced by a phone
intent picker: a 2x3 grid of icon cards (Draw / Animate / Compose / Record /
Edit video / Blank) taking the top ~2/3, and a scrollable Recent-projects list
in the bottom ~1/3.
Each intent reuses EditorApp::create_new_project_with_focus and then sets the
initial stack window (e.g. Draw -> fullscreen Stage, Compose -> Timeline +
Piano Roll + Keys). Recent rows load the project and enter the editor. Cards are
icon-only (Lucide glyph + label) for legibility on a phone. Adds the intent
icons to the Lucide set.
Rework 3-pane divider behavior and animate every layout change:
- All transitions now ease over ~100ms via a LayoutAnim between two full stack
configs (resize, slide, collapse, fullscreen) instead of jumping; edge
transitions continue from the drag's progress.
- 3-pane dividers are linked along a single path parameter through the three
presets ([.5,.25,.25] / even / [.25,.25,.5]) so both dividers move together
and reach their snaps simultaneously. Over-grow past the extreme preset is
anchored to the path endpoint so the non-dragged divider stays continuous.
- Growing a pane past ~66% collapses the window to 2 panes (dropping the
opposite-end pane); the surviving divider snaps to the nearest 2-pane preset
(1/4, 1/2, 3/4) based on the drop position. Dragging to the slide extreme
still slides the window.
- 2-pane snaps to 1/4, 1/2, 3/4.
After releasing a divider, the boundary now eases (cubic ease-out, ~100ms) from
where the finger lifted to the snapped fraction instead of jumping. Starting a
new drag cancels an in-flight ease; membership changes still apply instantly.
Dragging a divider now resizes its two adjacent panes live and, on release,
snaps the boundary to the nearest of 1/4, 1/3, 1/2, 2/3, 3/4 — so one pane can
be made dominant within a 2- or 3-pane window. Dragging far enough to squeeze a
pane past the collapse threshold still triggers the membership transition
(slide/shrink per R1-R6), after which sizing resets to even.
MobileState gains a `weights: [f32;3]` (normalized on use); config_rects is
weight-aware; edge drags keep animating toward the even-split target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bundle Lucide (ISC license) at assets/fonts/lucide.ttf and register it as the
egui font family "lucide" via mobile/icons.rs::install (called in EditorApp::new).
Icons are drawn as text with icons::font(size) + PUA codepoint constants.
Switch the stack headers (grip, maximize/minimize, node⇄instrument), the footer
chevron, and the transport play/pause to Lucide glyphs instead of relying on
egui's limited built-in emoji-icon-font.
- Double header height (52px) so the drag targets are easy to grab; footer
stays compact (28px).
- Add a fullscreen button on the right of each header that collapses the
window to that single pane; dragging the sole pane's header down or the
footer up splits it with the adjacent pane (count can now be 1). The button
toggles back to a 2-pane split.
- Use egui's bundled emoji-icon-font glyphs (⛶ fullscreen / ▣ restore) instead
of hand-drawn primitives.
- Cap the drag-to-commit distance (TRIGGER_MAX) so transitions don't require
dragging half the screen.
- Fix panic "attempt to subtract with overflow" when sliding to the end:
the R-ops used then_some(), which eagerly evaluates top-1 even when the
guard is false. Switch to lazy .then(|| ...).
- Replace the thin divider/edge grab bars with full-width header bars: each
band has a labeled header (top of band) that is its drag handle, plus a
reserved bottom footer for the BottomEdge "pull up for more" handle. Band 0's
header = TopEdge, band i's header = the divider above it, footer = BottomEdge.
The Node/Instrument ⇄ toggle lives in that band's header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per design feedback, drop the top-tabs + single-ribbon model. The mobile UI
is now one vertical stack of all panes in a fixed order (Outliner, Asset
Library, Stage, Timeline, Piano Roll, Virtual Piano, Node/Instrument, Script
editor) with a window of 2-3 consecutive panes visible at a time.
Dragging the dividers between panes and the top/bottom screen edges grows,
shrinks, or slides the window via a small state machine (R1-R6 in stack.rs),
with continuous interpolation during the drag and snap-on-release. The
Node/Instrument band toggles between the node editor and the preset browser.
The transport stays pinned as the bottom floor.
Each visible band reuses PaneInstance::render_content full-bleed (surface.rs).
Removes topbar.rs and ribbon.rs; rewrites mod.rs state (window_top,
window_count, show_instruments, drag); adds stack.rs.
New src/mobile/ module renders a phone-style shell: top surface tabs
(Stage/Time/Nodes/Mixer/Tree), a single full-bleed hero pane reusing the
existing PaneInstance renderers, a resizable timeline ribbon (peek/half/full
snap tiers riding above the floor), and a fixed transport bar wired to the
audio controller (play/pause, timecode, project scrub).
Gated entirely on the LB_MOBILE_UI env var: when set, main() opens a
phone-aspect window and update() routes the central panel through
render_mobile_shell instead of render_layout_node, sharing the same
build_frame context and post-render drain. Desktop is unchanged when unset.
Tabs mirror the pane list; per the wireframe the Toolbar and Infopanel do
not become tabs (they become the omnibutton and inspector in later phases),
and the Nodes surface is a placeholder pending the focus/patch rework.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the ~80-field SharedPaneState/RenderContext literal from update()
into EditorApp::build_frame, returning a FrameBundle that also carries the
layout-editing borrows (current_layout, drag_state, etc.). Per-frame locals
move into a FrameScratch struct so the bundle can borrow both self and the
scratch under one lifetime.
This is a behavior-preserving refactor that gives the upcoming mobile UI
path a single shared construction site for SharedPaneState instead of
duplicating the literal.
Introduce editable text layers: a resizable text box with editable text,
font size, color, font family, and alignment.
Core:
- New TextLayer/TextContent (text_layer.rs), wired into AnyLayer/LayerType
and all the exhaustive match sites; structured so content can be keyframed
later via content_at().
- fonts.rs: thread-local parley FontContext with three bundled fonts
(Liberation Sans/Serif/Mono, SIL OFL), system-font enumeration consolidated
to base families, document-embedded fonts, glyph/caret/selection geometry,
and a background preloader for the picker fonts.
- Rendering via parley layout + Scene::draw_glyphs (renderer.rs); text
composites through the vector path.
- Actions: CreateTextClipAction (vector-layer branch, undoable),
SetTextContentAction, ResizeTextBoxAction.
- .beam font embedding: MediaKind::Font rows (content-hash dedupe) written on
save and registered on load, with bundled-default fallback.
- VectorClip content bounds include text boxes so text-only clips are
selectable/draggable.
Editor:
- Text tool: click empty/raster/video to create a top-level text layer, or a
vector layer to create+enter a clip containing the text; click an existing
box to edit it.
- Hybrid in-place editing: a hidden egui TextEdit drives input/IME/caret while
the text and caret/selection render in Vello; empty just-created layers are
removed on commit.
- Selection outline + 8 resize handles (re-wrap text) with hover cursors;
factored the corner/edge resize-cursor mapping shared with the Transform tool.
- Info panel: edit text, size, color, alignment, box size, and a font-family
picker that previews each entry in its own font (fonts preloaded in the
background to avoid hitches).
Deps: add parley (git, pinned to match vello's peniko); bundle Liberation
fonts under lightningbeam-core/assets/fonts. Gitignore the local
.cargo/config.toml used to select a machine's ffmpeg.
- Wire MenuAction::ToggleLayerVisibility (was a stub) to flip the active
layer's `visible` flag via SetLayerPropertiesAction (undoable). This is
what the SVG-export visibility fix depends on, and the "Hide/Show Layer"
menu item now works.
- Log the resolved color range in the zero-copy H.264 path to diagnose
whether `full_range` reaches the encoder (the VAAPI driver may still omit
the SPS VUI full_range flag).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Export correctness:
- Honor the user's color-range (Limited/Full) on the software encode path:
thread full_range through gpu_yuv (new shader range uniform), the CPU
swscale fallback (sws_setColorspaceDetails → BT.709 + range, fixing the
BT.601 hue/level shift on odd-width exports), and the encoder color tags.
- Reject HDR + WebM up front with a clear message and log the forced HEVC
override instead of producing an unplayable file.
- Delete dead render_frame_to_rgba_hdr (hardcoded Stretch; live HDR path
already honors the fit mode).
Decode/playback (video.rs):
- Drain the decoder at EOF (send_eof + flush) so the final B-frame-delayed
frames render instead of erroring; per-frame logic extracted to a helper.
- Missing-PTS frames continue monotonically rather than snapping to ts=0.
- Force exact thumbnail width so sub-128px sources aren't shown stretched.
Resource leaks (gpu-video-encoder):
- dmabuf import_raw: RAII guard frees the duped fd + partial VkImages/memory
on every error path.
- vaapi alloc: free device/frames-ctx/AVFrames on the unexpected-DRM path.
Data model / robustness:
- collapse_boundary_spikes requires a full curve reversal (all control
points) so it no longer deletes a real lens/sliver and drops the fill.
- Export audio spin-wait ignores a stale `finished` flag when a forward
seek is pending (was rendering silence over real audio).
- RasterDiff apply_before/after take current dims and skip on a post-resize
mismatch.
- beam_archive read_media_full caps the preallocation from untrusted total_len.
UI/visual:
- SVG export skips hidden layers/empty groups; import folds fill-opacity into
gradients and surfaces failures as a notification.
- Active raster-layer border uses playback_time + overlay_transform.
- gpu_brush remove_layer_texture also evicts the stale low-res proxy.
- ensure_raster_resident_for_undo registers faulted frames in the LRU so
resident RAM stays bounded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Export (lightningbeam-core/svg_export.rs): document_to_svg walks vector
layers/groups at a given frame and emits <path> per fill (solid or
gradient via <defs>) and per stroked edge. Wired into the export dialog
as an "SVG" tab; written synchronously. Raster/video/effect layers are
skipped (vector-only, lossless), structured for a later rasterize pass.
Import (lightningbeam-editor/svg_import.rs): import_svg parses via usvg,
bakes each path's absolute transform into geometry, converts segments to
cubic edges, and maps solid/linear/radial paint to ShapeColor/
ShapeGradient. .svg is detected in the Ctrl+I Import handler and added as
a new vector layer (keyframe at the playhead). file_types gains
FileType::Vector + VECTOR_EXTENSIONS.
Tests: svg_export::export_tests (core) and svg_import::tests (editor).
Resizing the document leaves raster layers as-is (canvas keeps its old pixel size,
anchored top-left). To reconcile, the info panel now shows the active raster
layer's canvas size — driven by the *active* layer, not selection focus, since
painting doesn't focus the layer — and, when it differs from the document, a
Scale / Expand-Crop toggle + a "Layer to document size" button.
- RasterKeyframe::resize_to(w, h, mode): always applies the new declared size;
resamples (Lanczos3) when Scale, top-left pad/trim when Canvas, and only touches
the buffer when pixels are resident (a blank canvas just takes the new size).
Sets texture_dirty so the stage's dirty-scan refreshes the GPU texture.
- ResizeRasterLayerAction resizes every keyframe with undo, holding a (read-only)
RasterStore so paged-out keyframes are loaded one at a time rather than bulk.
Resized keyframes stay resident + dirty and persist on the next full save (no
incremental store write to page them back out). Scale is lossy/compounds;
Expand-Crop is lossless.
Outline the active raster layer's canvas bounds (document space) so its size is
visible, especially when it differs from the document after a resize. Two strokes
sharing one dash pattern, the yellow offset by a dash to fill the black's gaps -
interlocking marching-ants. Sizes divided by zoom so the dashes stay ~constant on
screen.
ImageExportSettings gains `fit: ExportFitMode` (default Letterbox) + a "Fit"
dropdown in advanced image settings; the image render path uses it instead of the
hardcoded Letterbox. Image export supports a resolution override, so the mode
matters when the override aspect differs from the document.
A) Imported video had a different aspect ratio depending on how it was placed:
direct import used a uniform scale + center (preserve aspect), while the
asset-library timeline drag used an independent scale_x/scale_y (stretch to fill).
Add Transform::fit_centered (uniform scale, centered, aspect-preserving) and route
both paths through it, so a clip looks identical however it's added.
B) Exported video was stretched when the export resolution's aspect differed from
the document's (base_transform was always scale_non_uniform). Add
ExportFitMode {Stretch, Letterbox (default), Crop} on VideoExportSettings + a "Fit"
dropdown in advanced export settings, and a shared export_base_transform() helper:
Letterbox = uniform fit centered (black bars), Crop = uniform fill centered (trim),
Stretch = the old distort-to-fill. Threaded through the software, HDR, and
zero-copy export render paths (image export is doc-sized → identity).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two bugs made 4K (and any) video playback wrong:
1. Sped-up playback. The VideoManager frame cache was keyed by milliseconds, but
GPU (hardware-decoded) frames bypass the decoder's internal cache. A UI that
refreshes finer than the video frame rate (a 60Hz canvas on a 30fps clip)
therefore missed the cache on every sub-frame request and decoded the NEXT
frame each time — advancing the video ~2x faster than the clock, and racing the
decoder toward EOF. Key the cache on the video frame index (round(ts·fps))
instead, so all requests within one frame share an entry and each frame decodes
exactly once → correct speed.
2. Periodic seek/jerk. The seek decision compared the rounded request frame_ts to
the decoded frame's exact PTS, which sit on slightly different grids — so
frame_ts landed ~1 frame "behind" the just-decoded PTS every ~10 frames and
falsely read as a backward seek (40ms seek + GOP re-decode). Track the previous
request (last_requested_ts), which is strictly monotonic during forward
playback, and detect backward from that instead. Real scrubs still seek.
Per-frame HW decode is ~3-5ms; both bugs were spurious decode/seek work on top.
Keeps a gated LB_VIDEO_DEBUG [Video Seek?] diagnostic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The asset-library thumbnail called VideoManager::get_frame, which honors the
render-pass hardware flag — left ON by the preview, so the thumbnail got a GPU
frame with empty rgba_data → an all-black thumbnail. The thumbnail is also the
only consumer that requests a fixed low timestamp (1.0s) on the shared per-clip
decoder, so when it re-decodes during playback it yanks the decoder back to ~1s;
the next playback frame (6.x) is then ">2s forward" and re-seeks to the keyframe
+ re-decodes the whole GOP (the jerk: per-frame decode is ~5ms, but these seeks
cost 40ms + N-frame catch-up).
Add VideoManager::get_frame_cpu (forces want_gpu=false regardless of the render
flag); the thumbnail uses it. Now it produces a real RGBA thumbnail that the
editor texture-caches once, instead of re-decoding black frames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes the common Linux H.264 export fully GPU-resident: decode (HW NV12) →
composite → VAAPI encode all on one device, no CPU round-trips.
- gpu-video-encoder: ZeroCopyEncoder now holds wgpu (device,queue,adapter) handles
instead of owning a DrmDevice. New `new_on_device(device,queue,adapter,...)` runs
the RGBA→NV12 render + DMA-BUF import on a passed device; `new` keeps building its
own. The encoder only *imports* VAAPI surfaces (not export), so the shared
import-capable device works directly.
- editor: stash the shared device handles on EditorApp (set in the creation closure
from the eframe render_state when the shared device is active) and thread them
through start_video_export / start_video_with_audio_export → try_build_zero_copy,
which uses new_on_device when available. ZeroCopyVideo carries on_shared_device →
the export composite uses hardware_ok=true (consuming HW-decoded GPU frames from
pt 1) only then.
Falls back to the own-device encoder (decode downloads to CPU) when no shared
device. Tradeoff: the export now shares the GPU with the UI thread (was a separate
device); acceptable for the GPU-resident-decode win. Compiles; crate tests build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The software and HDR export paths already composite on the shared device (the
eframe device, via render_next_video_frame), so they can consume the same
hardware-decoded NV12 GPU frames the preview uses — no CPU download/upload:
- ExportGpuResources gains the Nv12BlitPipeline; the export compositor's Video arm
branches on inst.gpu (NV12 blit, mirroring stage.rs) vs the RGBA upload path.
- render_frame_to_gpu_rgba takes a hardware_ok flag: true for the software video +
image export (shared device), false for the zero-copy path (own device, must
download). render_frame_to_yuv10_hdr sets it true.
Drops the 4K software/HDR-export decode wall (HW decode + GPU composite, no
per-frame RGBA upload). The common Linux H.264 zero-copy path is unchanged
(separate device) — that's pt 2. Falls back to software when no shared device /
importer (flag is harmless then: get_frame returns CPU anyway). Compiles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire HDR frame production end-to-end (encoder side was pt 1):
- linear_to_pq.wgsl: linear scene HDR (BT.709, white=1.0) → BT.2020 primaries →
PQ (203-nit) or HLG OETF → gamma-encoded R'G'B'. Inverse of the nv12 decode, so
a decode→encode round-trip is the identity.
- hdr_frame.rs (isolated module): runs that pass into an Rgba16Float target, does a
synchronous readback (f16-decoded on CPU), then BT.2020 R'G'B'→Y'CbCr limited
4:2:0 10-bit pack → YUV420P10LE planes. Rgba16Float avoids the 16BIT_NORM device
feature dependency.
- video_exporter::render_frame_to_yuv10_hdr: composite + HDR encode, returning
10-bit planes; lazily builds the pipeline.
- orchestrator: HDR uses a synchronous 1-frame-per-call path (the async RGBA
pipeline is 8-bit only); zero-copy is skipped for HDR.
- dialog: "Dynamic range" dropdown (SDR / HDR10 PQ / HLG).
Software HEVC Main10 path; favors correctness over throughput. Compiles; runtime
verification needs a GPU + HDR-capable player.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settings + encoder side of HDR video export (frame data still SDR until pt 2):
- core: HdrExportMode {Sdr (default), Pq, Hlg} on VideoExportSettings (serde
default), with transfer-name helpers.
- setup_video_encoder takes the mode: HDR → YUV420P10LE, BT.2020 NCL matrix,
limited range, color_primaries=bt2020, color_trc=smpte2084/arib-std-b67,
profile=main10. SDR path unchanged (8-bit full-range BT.709).
- run_video_encoder forces HEVC when HDR is selected (the only 10-bit codec wired
up). encode_frame is parameterized by pixel format and now copies planes
row-by-row honoring stride (10-bit / non-aligned widths can have row padding).
Dormant: no UI exposes the mode yet and the frame data is still 8-bit SDR, so
selecting HDR is not yet wired. The render→PQ/HLG 10-bit frame path is pt 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A failed DMA-BUF import returned None silently → core falls back to software,
whose RGBA path does no BT.2020→709 gamut conversion, so HDR clips render washed
out. The detection log fires before import_raw, so it didn't reveal this. Log the
actual import error (with ten_bit) instead of swallowing it, to confirm whether
10-bit P010 import is failing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Symptom: properly-converted BT.2020 PQ/HLG clips rendered progressively
desaturated vs the SDR reference — the signature of the transfer/primaries tags
not reaching the NV12→RGB shader, so HDR code values were decoded as plain
sRGB/BT.709 (no gamut expansion, wrong EOTF).
VAAPI hardware-decoded frames frequently leave color_trc/color_primaries/
colorspace/color_range unspecified — the authoritative values live on the codec
context (parsed from the bitstream), which the importer never sees. Before
importing, copy any unspecified colour field from the codec context onto the
frame, so the importer detects PQ/HLG + BT.2020 correctly.
Also add a one-time importer log (LB_VIDEO_DEBUG) of the detected
ten_bit/full_range/colorspace/primaries/trc to confirm what's picked up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
R16Unorm/Rg16Unorm aren't renderable, but the import gave every plane texture
RENDER_ATTACHMENT/COLOR_ATTACHMENT usage (needed only for the encoder's RGBA→NV12
write path) — so create_view panicked on 10-bit decode ("R16Unorm is not
renderable"). Gate the render-target usage (vk COLOR_ATTACHMENT, hal COLOR_TARGET,
wgpu RENDER_ATTACHMENT) on 8-bit; 16-bit P010 planes are sample-only
(TEXTURE_BINDING/RESOURCE + COPY_SRC), which is all the decode path needs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
10/12-bit HDR decodes to P010-style VAAPI surfaces (16-bit planes), which the
8-bit NV12 import couldn't handle — real HDR clips fell back to software (losing
the HDR). Import them as R16/Rg16 plane textures:
- dmabuf::Nv12DmaBuf gains `ten_bit`; import_raw picks R16_UNORM/R16G16_UNORM (vk)
+ R16Unorm/Rg16Unorm (wgpu) when set, else the existing R8/Rg8.
- hw_video.rs detects 10-bit from the hw frames context sw_format (P010/P012/P016)
and sets it. The NV12→RGB shader is unchanged: it samples normalized floats, and
the limited/full de-quant lands at the same values regardless of bit depth.
- vk_device requests TEXTURE_FORMAT_16BIT_NORM when the adapter supports it (R16/
Rg16 need it); absent → 10-bit falls back to software, 8-bit unaffected.
Pairs with the PQ/HLG/BT.2020 colour math so HDR10/HLG clips now reach the GPU
path end-to-end. NV12 callers (encoder, decode primitive) pass ten_bit=false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a per-document HDR→SDR mapping applied at the final linear→sRGB encode, so
super-white (HDR) video highlights can be recovered instead of hard-clipped:
- core: HdrOutputMode {Clip (default), HighlightRolloff} on Document (serde
default), with a SetDocumentPropertiesAction variant for undoable edits.
- shaders: a fs_main_rolloff entry point (preview linear_to_srgb.wgsl + the export
inline shader) applying a C1 highlight knee — identity below 0.8, smooth rolloff
[0.8,∞)→[0.8,1). SDR below the knee is untouched; Clip stays the historical path.
- preview (stage.rs) and both export encodes (video_exporter.rs) pick the pipeline
variant from document.hdr_output_mode — one value per frame, so no per-pixel
uniform; mirrors the existing fs_main_straight pattern.
- UI: an "HDR output" dropdown in the Document section of the info panel.
Default (Clip) is bit-identical to previous behaviour. Completes Stage A:
HDR-correct input (pt 1) + SDR-safe output mapping. HDR export (10-bit P010/PQ)
and HDR display remain Stages B/C.
Decode HDR video into the linear compositor correctly instead of approximating
everything as sRGB/BT.709:
- Read the frame's color_trc and color_primaries in the importer → VideoTransfer
{Gamma,Pq,Hlg} + VideoPrimaries {Bt709,Bt2020} on GpuVideoFrame.
- nv12_blit.wgsl: branch the EOTF — sRGB gamma (SDR), SMPTE2084 PQ (normalized so
203-nit graphics white = 1.0; highlights exceed 1.0), or HLG inverse-OETF
(reference white ≈ 1.0). Then BT.2020→BT.709 primaries in linear light when
wide-gamut, clamping out-of-709 colours.
Establishes the white=1.0 scene-linear convention: SDR content is unchanged
(stays in [0,1]); HDR video carries super-white highlights through compositing.
SDR-output mapping (clip default vs highlight rolloff) is Part 2. HLG's display
OOTF is omitted (scene-referred) — approximate but reasonable for SDR-out.
The NV12→RGB pass hardcoded BT.709, so SD (BT.601) clips had slightly wrong hues.
Read each frame's AVColorSpace in the importer and derive the Y'CbCr→R'G'B'
matrix (BT.709/601/240M/2020; Unspecified guessed by height like swscale/players),
carry the four coefficients on GpuVideoFrame, and apply them in the shader.
- core: GpuVideoFrame.coeffs + ycbcr_coeffs(kr, kb) helper.
- hw_video.rs: map AVColorSpace → (kr, kb) → coeffs.
- nv12_blit{.rs,.wgsl}: uniform grows to 80 bytes (adds a coeffs vec4); the matrix
multiply uses params.coeffs instead of literals.
BT.2020's transfer is still approximated as sRGB. The DRM-modifier-without-SAMPLED case stays a graceful software fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The WGSL struct trailed `full_range: u32` with `_pad: vec3<u32>`, but vec3 has
16-byte alignment so the struct rounded up to 80 bytes while the Rust-side
Nv12Params (BlitTransform + u32 + [u32;3]) is 64 — wgpu rejected the draw
("Buffer is bound with size 64 where the shader expects 80"). Pack the flag as a
single `flags: vec4<u32>` (.x = full_range) so both sides are 64 bytes.
Make the dormant core HW-decode engine live for the preview path:
- hw_video.rs: editor's HwVideoImporter — maps a decoded VAAPI surface to a
DRM-PRIME DMA-BUF and imports it as wgpu NV12 plane textures on the *shared*
device (the only one with the import extensions). install() creates the VAAPI
device and injects it + the importer into the VideoManager.
- main.rs: track whether the shared device is actually in use; only then (Linux,
not LB_NO_SHARED_DEVICE) install hardware decode, using the CreationContext's
shared device + adapter.
- nv12_blit.rs + nv12_blit.wgsl: NV12 plane textures → BT.709 → sRGB-encoded →
linear, written straight into the Rgba16Float HDR layer (no CPU upload). Colour
math mirrors the software path so HW/SW video match; honours full_range.
- stage.rs: the preview Video arm branches on inst.gpu (NV12 blit) vs rgba_data
(existing upload+blit_straight); sets render_hardware_ok = !cpu_renderer so the
CPU fallback still gets software frames.
- video_exporter.rs: sets render_hardware_ok(false) before both compositing
passes — export composites on the encoder's separate device, so a hardware
decoder downloads to CPU instead (export stays software, correct).
- dmabuf.rs: imported plane textures now also carry SAMPLED/TEXTURE_BINDING so
they can be sampled by the NV12 blit (they were render-target-only); into_planes
hands the textures to the longer-lived GpuVideoFrame.
- video.rs: cache-key the GPU/CPU representation on want_gpu (HW-configured AND
render_hardware_ok) so software-only decode keeps a single cache entry.
Preview only this pass; export GPU-residency is the 3c-export follow-up. Untested
at runtime here (no GPU/display in container) — both crates compile.
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):
- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
downloads the surface via av_hwframe_transfer_data then swscales — so export
stays software/correct and only the preview goes GPU-resident. HW init or
import failure falls back to software per clip.
Dormant until the editor injects an importer (next): no importer => software,
unchanged.
import_raw now takes (&wgpu::Device, &wgpu::Adapter) and extracts the raw Vulkan
handles via as_hal, instead of taking the crate's own DrmDevice. This lets a
VAAPI surface be imported onto ANY DMA-BUF-import-capable wgpu device — the
encoder/decoder's own device today, and the editor's shared device (Stage 3a)
next, so hardware-decoded frames are usable by the preview compositor.
Encoder/decoder pass their DrmDevice's device+adapter; round-trip decode test
still passes.
Foundation for hardware video decode in BOTH preview and export: wgpu textures
can't cross devices, and a hardware-decoded frame is a DMA-BUF-imported texture
that needs the import extensions (only addable via wgpu-hal device_from_raw). So
eframe + the compositor + decode + encode must share ONE custom device.
- vk_device::create_windowed(): the existing import-capable DrmDevice plus
VK_KHR_swapchain (the WSI surface instance extensions are already enabled by
Instance::init), for use as the editor's main device.
- main.rs: on Linux, build the shared device and inject it into eframe via
WgpuSetup::Existing (the egui fork supports it); fall back to wgpu's normal
device + software decode on any failure, on other platforms, or via
LB_NO_SHARED_DEVICE. The DrmDevice's wgpu handles are cloned into eframe
(Arc-backed), so the VkDevice persists with them.
Runtime-verified: the editor launches and renders identically (canvas/vello,
video, panels) on the shared device, and the env-var fallback works.
Headless decode primitive, the mirror of the encoder: VaapiDecoder opens a file,
hardware-decodes H.264 into VAAPI NV12 surfaces (hw_device_ctx + a get_format
callback selecting AV_PIX_FMT_VAAPI), maps each surface to a DRM-PRIME DMA-BUF,
and imports it as two wgpu plane textures via the existing dmabuf::import_raw —
the same path the encoder uses, in the read direction. Frames stay GPU-resident
(no CPU copy).
Validated by a round-trip test: encode solid gray with ZeroCopyEncoder, decode
it back, read the Y plane (mean ≈ 128). All 9 crate tests pass on the container's
Intel GPU.
Next (Stage 3): wire this into VideoManager/get_frame so the compositor consumes
a GPU texture directly (no write_texture upload), with software decode fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The zero-copy VAAPI encoder emitted full-range BT.709 NV12 but wrote no color
tags, so players assumed limited range and stretched it — the H.264 output
looked dark and oversaturated (preview and VP9/software were fine).
- Rgba2Nv12 takes `full_range` and applies the matching Y/chroma scale+offset
(limited 16-235 / full 0-255) via a uniform; the encoder sets color_range +
bt709 colorspace/primaries/transfer tags to match. ffprobe-confirmed.
- New ColorRange { Limited, Full } on VideoExportSettings (default Limited, the
universally-correct choice; serde(default) so saved settings still load),
surfaced as a "Color range" dropdown in advanced export settings for H.264.
The swscale software fallback still emits Limited regardless of the toggle
(Full only affects the VAAPI zero-copy path).
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.
Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
Skyler Lehmkuhl