Three bugs in a row came from the same root: a time value crossing an API
boundary as a bare f64, with the caller and the callee disagreeing about
whether it meant seconds or beats. Recording landed at the wrong time,
MIDI clips grew too fast, and a 1-second split played back as half a
second. Each was "obviously" one domain at the call site and read as the
other on the far side. This makes the mismatch a compile error.
Backend API — every time-carrying f64 is gone:
- Commands: Seek/SetOffset/SetTrimStart/SetTrimEnd -> Seconds; MoveClip/
ExtendClip/CreateMidiClip/AddMidiNote/AddLoadedMidiClip/
UpdateMidiClipNotes/AddMidiClipSync and all four automation commands ->
Beats; TrimClip -> TrimRange.
- Events/queries: PlaybackPosition, WaveformChunksReady's time range,
AudioFileReady::duration, PoolFileInfo, get_playhead_seconds -> Seconds.
- Serialized: MidiClipData::duration and AutomationKeyframeData::time ->
Beats. Both newtypes are #[serde(transparent)], so the .beam on-disk
format is unchanged.
- Several controller methods ALREADY took Beats and unwrapped it to shove
into the command — the newtype was being discarded at the very boundary
it existed to protect.
TrimRange, for the domain-polymorphic case: a clip's content time is
SECONDS for sampled audio but BEATS for MIDI, so a single newtype can't
express it (there was even a comment in engine.rs saying so, and that
rationalization is what let the bug through). A domain-tagged enum can.
The engine rejects a range whose domain doesn't match the track, and the
range is built from the clip (clip.trim_range()) so callers can't pick
the wrong variant.
ContentTime, for the trim fields: ClipInstance::trim_start/trim_end are
content times, and were the last untyped f64 — the actual root of the
split bug. ContentTime is deliberately a DEAD END: no .to_seconds(), no
.to_beats(), no arithmetic with Seconds or Beats. Content times combine
freely with each other (same clip, same domain — safe), so the ~100
passthrough sites cost nothing; the only exit is resolving against the
clip that knows the domain (AudioClip::resolve_content_time /
Document::resolve_content_time / ClipDuration::same_domain). Mixing
domains no longer compiles.
Two more live bugs the types surfaced:
- ClipInstance::effective_duration_beats took a SECONDS clip duration and
subtracted trim_start from it. For a TRIMMED MIDI clip that subtracted
a beats offset from a seconds duration, so the clip's timeline length
was wrong at any tempo but 60 BPM. Untrimmed clips happened to work,
which is why it hid. It now takes a ClipDuration and resolves in the
clip's own domain: beats content carries over directly (tempo-
invariant), wall-clock content converts at the clip's position.
Regression test asserts a clip trimmed to beats 2..6 is 4 beats long at
60/90/120 BPM.
- Trim validation clamped a content-domain trim against a wall-clock gap.
gap_to_content/content_to_secs now convert at the clip's position.
Also folds two more copies of the backend add-logic into
BackendContext::add_clip_instance (split and remove_clip_instances both
re-add clips), so the trim/duration conversions live in exactly one place
instead of four.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Type ClipInstance.timeline_start/timeline_duration/loop_before as Beats and
thread Beats/Seconds through core so the compiler catches the seconds-vs-beats
mismatches behind the audio-clip placement/drag/trim bugs.
Fixes latent mixups surfaced by the types:
- add/remove/split clip instance: the audio "effective duration" fallback was
the content-seconds span treated as beats (clips stopped early off 60 BPM);
now converted via the tempo map at the clip's start.
- trim validation: extend-left/right clamped a content-seconds delta against a
timeline-beats gap (and moved trim_start + timeline_start by the same raw
amount, assuming 1:1). Now the gap is converted to content seconds and the
timeline moves by the beats-equivalent.
- split content-split point mixed beats into a seconds trim value.
- set_keyframe: visibility-start fallback returned beats where seconds expected.
- hit_test: timeline_time (s) compared against beats without conversion.
Typed backend boundaries that were passing beats/seconds as bare f64:
add_audio_clip (start/dur Beats, offset Seconds), move_clip/extend_clip (Beats),
set_offset (Seconds). Command enum transport stays f64.
Serialization is unchanged (Beats/Seconds are #[serde(transparent)]).
lightningbeam-core builds; 299 core tests pass. The editor call sites
(recording/drop/paste/drag placement) are updated in stage 2.
Introduce editable text layers: a resizable text box with editable text,
font size, color, font family, and alignment.
Core:
- New TextLayer/TextContent (text_layer.rs), wired into AnyLayer/LayerType
and all the exhaustive match sites; structured so content can be keyframed
later via content_at().
- fonts.rs: thread-local parley FontContext with three bundled fonts
(Liberation Sans/Serif/Mono, SIL OFL), system-font enumeration consolidated
to base families, document-embedded fonts, glyph/caret/selection geometry,
and a background preloader for the picker fonts.
- Rendering via parley layout + Scene::draw_glyphs (renderer.rs); text
composites through the vector path.
- Actions: CreateTextClipAction (vector-layer branch, undoable),
SetTextContentAction, ResizeTextBoxAction.
- .beam font embedding: MediaKind::Font rows (content-hash dedupe) written on
save and registered on load, with bundled-default fallback.
- VectorClip content bounds include text boxes so text-only clips are
selectable/draggable.
Editor:
- Text tool: click empty/raster/video to create a top-level text layer, or a
vector layer to create+enter a clip containing the text; click an existing
box to edit it.
- Hybrid in-place editing: a hidden egui TextEdit drives input/IME/caret while
the text and caret/selection render in Vello; empty just-created layers are
removed on commit.
- Selection outline + 8 resize handles (re-wrap text) with hover cursors;
factored the corner/edge resize-cursor mapping shared with the Transform tool.
- Info panel: edit text, size, color, alignment, box size, and a font-family
picker that previews each entry in its own font (fonts preloaded in the
background to avoid hitches).
Deps: add parley (git, pinned to match vello's peniko); bundle Liberation
fonts under lightningbeam-core/assets/fonts. Gitignore the local
.cargo/config.toml used to select a machine's ffmpeg.
Export (lightningbeam-core/svg_export.rs): document_to_svg walks vector
layers/groups at a given frame and emits <path> per fill (solid or
gradient via <defs>) and per stroked edge. Wired into the export dialog
as an "SVG" tab; written synchronously. Raster/video/effect layers are
skipped (vector-only, lossless), structured for a later rasterize pass.
Import (lightningbeam-editor/svg_import.rs): import_svg parses via usvg,
bakes each path's absolute transform into geometry, converts segments to
cubic edges, and maps solid/linear/radial paint to ShapeColor/
ShapeGradient. .svg is detected in the Ctrl+I Import handler and added as
a new vector layer (keyframe at the playhead). file_types gains
FileType::Vector + VECTOR_EXTENSIONS.
Tests: svg_export::export_tests (core) and svg_import::tests (editor).
Extend the existing VideoDecoder with an optional hardware path, reusing its
demux/seek/keyframe/blob engine (no duplication):
- GpuVideoFrame (NV12 plane wgpu textures) + HwVideoImporter trait (editor
implements the DMA-BUF import; the AVFrame crosses as an opaque pointer so
core needn't reference the GPU crate) + HwDeviceHandle (opaque AVBufferRef).
- VideoManager::set_hardware_decode injects the VAAPI device + importer; each
decoder attaches hw_device_ctx + a get_format(VAAPI) callback before opening,
decodes into VAAPI surfaces, and imports them (no CPU copy).
- get_frame returns DecodedFrame::{Cpu,Gpu}; VideoFrame/VideoRenderInstance gain
an optional `gpu`. The frame cache budgets GPU frames as ~w*h*3/2 and keys on
whether the consumer wanted GPU.
- A hardware decoder serving a CPU consumer (export, render_hardware_ok=false)
downloads the surface via av_hwframe_transfer_data then swscales — so export
stays software/correct and only the preview goes GPU-resident. HW init or
import failure falls back to software per clip.
Dormant until the editor injects an importer (next): no importer => software,
unchanged.
The decoder's output size was frozen to the document size at import, and export
reused that decoder — so exporting above document res upscaled the video (real
source detail discarded) and a document resize never re-targeted the decode.
Decode size is now chosen per get_frame call: VideoDecoder::get_frame and
VideoManager::get_frame take a target (w, h), capped to native (never upscale),
with the swscale context and frame caches keyed on the output size so preview
(preview res) and an in-progress export (export res) don't collide. The renderer
derives the target from the document->output base_transform, so export decodes
at export res (full detail) and the canvas at preview res. Thumbnails/asset
library pass small targets.
Imported video is a Group[Video, Audio] that rendered as a Vello-baked
Vector layer, re-uploading the full frame to Vello's image atlas every
frame (~17ms/frame at 1080p, hitting playback and export alike). Extract
video frames out of the Group/clip scene recursion into
VideoRenderInstances so they composite via the GPU Video path; mixed
video+vector containers fall back to Vello (correct, unaccelerated).
Also route video through hardware sRGB decode: upload raw sRGB bytes to an
Rgba8UnormSrgb texture and blit with a non-unpremultiplying shader variant
(blit_straight), removing the per-frame per-pixel CPU sRGB->linear pass.
Add an F3 GPU-timestamp timer and a per-frame video texture cache.
Drops the live composite of a 1080p video from ~17ms to ~2-3ms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tween_after == Shape` was stored on keyframes but never read. Now the
render path morphs geometry across a shape-tween span:
- VectorGraph::interpolated(other, t): same-topology lerp of vertex
positions, edge curves, stroke widths and stroke/fill colours. Returns
None when topology differs (counts, deleted flags, edge endpoints, fill
boundaries), so the caller holds the source keyframe.
- VectorLayer::tweened_graph_at(time): returns an owned morphed graph for
a shape-tween span whose two keyframes share topology, else borrows the
held keyframe. Editing still uses graph_at_time (the held keyframe).
- Renderer (Vello + CPU paths) renders via tweened_graph_at.
- SetTweenAction + wired the previously-stubbed "Add Shape Tween" menu.
The typical workflow — keyframe, duplicate it (same topology), move
vertices, Add Shape Tween — now morphs between the two. Non-matching
topology falls back to a hold.
Within a vector layer, groups and movie clips (clip instances) were drawn
before the layer's own VectorGraph, so they appeared underneath the loose
shapes. Draw the loose shapes first and the clip instances on top, in both
the Vello and CPU render paths.
`assets_needed_at(document, time)` (core) enumerates the image asset ids referenced by
the visible vector layers' active keyframes at a time (top-level + group children).
During playback the stage decodes the images needed ~0.5s ahead into the bounded
ImageCache, so a keyframe that swaps image fills doesn't hitch when the playhead
reaches it. Gated on is_playing; nested clip-instance recursion + background decode
are refinements.
Completes Phase 4 (image asset paging: Tier 2 decoded-cache LRU, Tier 1 lazy bytes,
playback prefetch).
Project load no longer eager-reads all image bytes — `ImageAsset.data` stays empty
and the renderer's ImageCache pages compressed bytes from the `.beam` on a decode
miss (read_packed_media_readonly by asset id), decoding into the byte-bounded Tier-2
cache. Result: instant load, and compressed bytes don't accumulate on the heap.
- ImageCache: `container_path` + `resolve_bytes` (asset.data if resident — fresh
import or old base64 project — else page from the container); decoders take `&[u8]`
and use the decoded dimensions.
- Container path threaded App.current_file_path → SharedPaneState → VelloRenderContext,
set on the cache each prepare.
- load_beam_sqlite drops the 3.5b eager read.
(Refinement: a persistent read connection instead of open-per-miss.)
The decoded-image cache (peniko ImageBrush + tiny-skia Pixmap, ~w·h·4 each) was
unbounded — the main asset-memory cost. Now capped at 256 MB with usage-LRU eviction:
every get_or_decode bumps the asset's recency, and inserts past the budget evict the
least-recently-used (a miss re-decodes from the resident asset.data). Images actually
rendered each frame stay resident; unused ones age out under pressure. invalidate/clear
keep the lru + byte accounting in sync.
Next (4b): lazy-load asset.data from the container instead of eager on project open.
The renderer painted the image brush at its native origin (0,0) with no brush
transform, so an image-filled rect drawn anywhere but the world origin only showed
the overlapping corner of the image. Both render paths now map the image's native
pixel space onto the fill's bounding box (Vello brush_transform; tiny-skia Pattern
transform) — 1:1 for an image-sized rectangle, stretch-to-bbox for arbitrary shapes.
Surround → stereo downmix:
- render_from_file folds multichannel sources (5.1/7.1/…) down to stereo with
proper coefficients (full level for the matching front channel, 1/√2 for centre
+ each surround, LFE dropped), normalized per row to avoid clipping (matching
ffmpeg's default). Applied uniformly to both the direct-copy and sinc-resample
paths and to every storage type (PCM, compressed, video audio), only when
dst==2 && src>2; unknown layouts fall back to front L/R. Previously it just took
FL/FR, dropping centre dialog + surrounds.
Proper video-audio reload:
- A video's audio track is now stored as a path reference to the video (never
packed/embedded as audio media) and re-probed via FFmpeg on load into a
streaming VideoAudio entry, so multichannel audio survives reload (the old
Symphonia reconstitution collapsed it, breaking the downmix). Driven by a new
AudioPoolEntry.is_video_audio flag across serialize / save_beam / load. Also
removes the decode-whole-video-to-RAM + temp-file path on load.
Fix video scaling:
- Any video with dimensions larger than the stage was being scaled down into the corner incorrectly; we now bake the frame-clip scale into the instance transform.
Skyler Lehmkuhl