One entryMerged PR· crunch.stumason.dev· PHP
PR #36 merged: feat: consume roll's new pack inputs (keyframes, sysaudio, typed text)
additions
+214
deletions
-13
files changed
13
roll's capture now emits richer inputs (still tagged schema 0.0.14) that /pack was silently dropping. This teaches the endpoint to consume them.
What changed in the pack contract
| New in the take | Before | Now |
|---|---|---|
keyframes/<t_ms>.png (pre-rendered lossless screen frames) | ignored; pipeline ffmpeg-extracts everything | OCR the keyframes directly at their t_ms (lossless PNG), and keep baseline extraction from screen.mp4 for coverage |
sysaudio + sysAudioSyncOffsetMs (system-audio track) | dropped | parsed into PackManifest (captured, not yet processed) |
text events (the typed string, e.g. "shop") | left in raw, lost from the join | promoted onto the event and emitted as a scored type_text moment (source: telemetry) |
Decisions (agreed with @StuMason)
- Keyframes = supplement (not replace): reuse roll's PNGs where they align, still sample the rest so persistent on-screen text between interactions is captured.
- sysaudio = parse-only for now (stop the data loss; transcription is a later call).
- text = promote + moment (typed input is a strong edit landmark).
Implementation notes
PackReaderdiscoverskeyframes/<t_ms>.png(non-numeric names ignored).ProcessPack::timesNeedingExtraction()is pure (unit-tested) — splits keyframe-covered vs still-extract within a merge window; keyframes are pack input and never unlinked.- All new manifest/event fields are appended with defaults — no positional-constructor breakage.
Validation
- Parsed a real 0.0.14 take end-to-end (10 keyframes,
"shop"text event, sysaudio all parsed; 6 keyframes reused, 215 baseline frames still extracted). - 10 new unit tests; full suite green (131 passed). Pint + PHPStan clean.
Note: version string is still 0.0.14 — roll added these fields without bumping it.
🤖 Generated with Claude Code