fix(stt): chunked transcription with progress, retry, and no 300s cliff - #232
Merged
Conversation
A 30-minute recording was one `/inference` call: ~10 minutes with no progress, no recovery from a transient failure, and — the reported bug — killed outright before it ever finished. Node's global fetch (undici) applies a 300s `headersTimeout`, and whisper sends no response header until the whole upload is transcribed. Measured on the reporter's file: whisper needed 574s, undici cut the connection at 300s, and the renderer surfaced an unactionable "Transcription failed / fetch failed". The helper kept burning CPU on a transcription nobody was listening for. Split the audio into ~90s chunks and run them one at a time: - `chunking.ts` nudges each boundary to the quietest 20ms frame within ±3s of the ideal cut, so a chunk break lands in a pause instead of mid-word. Energy minimum, not a VAD: whisper.cpp's Silero VAD runs per REQUEST, so it cannot tell us where to cut before we upload. - `SttManager.transcribe` shifts each chunk's timestamps to absolute time, emits `completedSec`/`totalSec` per chunk, retries a chunk up to 3 times, and re-runs `server.start()` between attempts (idempotent when the helper is alive, a respawn when it isn't) — the usual cause of a mid-run failure is a dead helper, not a bad chunk. - The language detected on the first chunk is pinned for the rest. Left to auto-detect, whisper can flip mid-recording on a chunk that opens with a proper noun and "transcribe" the remainder as another language. - `whisperServer` bounds each request at 280s and names the failure with the helper's stderr, instead of letting undici's 300s ceiling surface as a bare "fetch failed". Chunks run SEQUENTIALLY, measured rather than assumed: whisper-stt-server holds a single model context, and two 120s chunks took 76.9s one after the other vs 144.1s fired together (0.53x — concurrency is ~1.9x SLOWER). A client-side worker pool would be a pessimisation. The progress reaches the UI: the renderer's status callback forwarded only a phase string and never subscribed to the main process's events, so the toast showed one static "transcribing" for the whole run. It now carries the chunk progress and renders a real bar. Verified end to end on a 32-minute source: 22 chunks, timestamps monotonic across every seam, last segment at 1950.7s of 1951.0s, slowest chunk 76.8s against the 280s ceiling.
…nner The chunked pipeline now reports how much audio it has transcribed, so surface it. A 30-minute recording spends minutes in "Transcribing…", and a spinner that never changes is indistinguishable from a hang. Progress travels the same road the phase already did — `status.ts` owns the vocabulary, the store owns the queue, `TranscriptionStatus.tsx` owns how a job reads on screen: - `TranscriptionProgress` + `progressFraction` join `TranscriptionPhase` in status.ts, and `deriveAssetStatus` carries them onto the view. - The store's `onStatus` no longer casts its argument to `TranscriptionPhase`: the renderer's `TranscribeStatus` and `TranscriptionPhase` are now genuinely the same vocabulary, including the `"loading-model"` phase the cast used to paper over. Failure paths clear `progress` with `phase`, so a failed job cannot leave a stale bar. - `TranscriptionStatusDot`'s sibling `TranscriptionProgressBar` renders a determinate bar, and the label gains a percentage. Both render nothing until the run reports measurable progress. Queued, extracting audio and downloading the model have no fraction to report, and a bar pinned at 0% reads as "stuck" where the spinner reads as "working".
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The mock listener in transcribe.test.ts declared its own event shape and still had the pre-progress one, so `tsc -p tsconfig.test.json` rejected the `completedSec`/`totalSec` the test itself asserts on. Runtime was fine, which is why vitest passed and only the CI typecheck caught it.
Five defects from the review of this PR, all in the new chunk loop. `"auto"` defeated the cross-chunk language pin. The request contract spells it as the explicit way to ask for detection, and it is truthy, so `if (!language)` never fired and every chunk detected independently — the exact flip the pin exists to prevent. Normalized to `undefined`. Verified end-to-end on 352s of real speech: the chunks now go `[undefined, "en", "en", "en"]` where they were all-detect before. Nothing could stop a run. The chunk boundary is the natural interruption point and went unused: cancelling in the renderer left main transcribing every remaining chunk while the replacement request queued behind it, which is what made "regenerate in another language" look dead. `cancel()` bumps an epoch the loop checks between chunks, reachable over a new `stt:cancel` channel, and surfaces as an `AbortError` so the store drops the job quietly. Measured on a 375s clip: cancelling after chunk 1 returns in 19s, not 143s. The IPC status sink was a single slot each request saved and restored, so two overlapping transcriptions ended with the first one to finish silencing the other for the rest of its run — no progress, which reads as a hang. It is a Set now, and each request detaches only its own. Every fetch rejection claimed "after 280s". A helper that died a moment ago rejects in a millisecond, and telling the reader it spent 280s on an over-long chunk sends them somewhere else entirely. The timeout wording is reserved for an actual timeout, the cause is attached, and the body read is named too. A chunk that exhausts its retries now says where it died rather than failing a 30-minute recording with no position at all. Energy ties broke toward the earliest frame, so digital silence — a muted track, a gap between takes — pulled every cut back by the whole search window: 90s chunks became 87s, and the silent test case 10s became 5s. Ties now break toward the target. The pin's comment also records what a stale staged helper does to it, since that cost a full debugging detour: electron/native/bin is gitignored, a binary from before cc78180 echoes the request instead of resolving it, and the failure is completely silent.
…nload The bar was wrapped in a div carrying `margin: -8px 0 16px`, rendered unconditionally. `.mediaDetail` is a flex column, so those margins do not collapse: every media card that was not transcribing paid 8px of dead gap, and the failure hint below it sat 8px lower than intended. A wrapper cannot render itself away with its child — the margin belongs on the bar. `"loading-model"` was added to `TranscriptionPhase` in this PR, plumbed through `TranscribeStatus` and the store, and then read by nothing: every label switches on `status`, so the 253MB first-run download was labelled "Transcribing" like everything else. That download is the wait most often mistaken for a hang, which is what this PR is about, so it gets its own words rather than the plumbing getting deleted.
`--connect-timeout` only covers the handshake. An origin that accepts the
connection and then trickles never errors, so nothing retried and nothing
fell through to the Debian mirror — the job would sit until the runner's
six-hour limit, which is the failure the mirror was added to survive.
`--speed-limit 1024 --speed-time 30` bounds the part `--connect-timeout`
cannot reach, and cannot fire on a merely slow link for a ~10MB tarball.
`spawnSync` reports a missing curl as `{ status: null, error: ENOENT }`, so
"curl exited with null" twice over sent the reader hunting a network problem
instead of a missing binary. The error is surfaced when present.
The comment claimed a 404 mirror is "simply skipped"; with
`--retry-all-errors` it is retried three times first. The flags stay — that
retry is the whole point on a connection reset — and the comment now says
what actually happens.
nix/package.nix restated the version by hand and had already drifted two
minors behind the app it names (1.6.0, then 1.8.0 against a 1.8.0-rc.6
tree). It reads package.json now. `npmDepsHash` still needs updating by
hand, but that one fails loudly.
The language whisper resolved on chunk 1 — the one every later chunk is pinned to — reached the document and was rendered nowhere. It had a pill, in `SourceTranscriptModal`. That modal is mounted by `LeftPanel`'s `MediaPane`, and `LeftPanel` is `active === "chat" ? <ChatStripPanel /> : <MediaPane />` with exactly one mount site passing the literal `active="chat"`. So `MediaPane`, `MediaList` and the pill are unreachable, and "did the language pin work?" had no answer in the UI at all — only in the saved .openscreen. Put it on the live surface, next to the status badge in the v4 MediaStage detail panel. It also belongs beside "Regenerate as": that selector is the control you set BECAUSE of what was detected, and it currently reads "Auto" next to a transcript that resolved "en".
Scrubbing a 30-minute recording (4501 words) stutters. The transcript pane
subscribes to the playhead and re-renders per frame, which is by design —
`TranscriptClipBlock` is memoised on `cueWordId` precisely so that the frames
in between cost nothing.
That memo's premise is playback: "cueWordId changes at word boundaries, a few
times per second, not sixty". A scrub breaks it. Dragging the playhead crosses
many words per frame, so `cueWordId` changes on EVERY frame, the block
re-renders every frame, and it renders one component per transcript word.
`TranscriptWord` had no memo, so all 4501 re-rendered to move one underline.
Measured over a 40-frame scrub in jsdom, median per frame:
words before after
100 19.6 ms 11.5 ms
500 24.0 ms 13.9 ms
1500 83.9 ms 24.7 ms
4501 132.6 ms 57.4 ms
The same benchmark with the cue word held inside one word — so the block's
memo bails out — costs 0.1 ms/frame at 4501 words. That isolates it: the
linear `findCueWordId` scan and the playhead subscription are both free, and
100% of what remains is the block re-render itself.
So this halves it but does not fix the shape: the cost is still proportional
to transcript length, because the block still builds 4501 React elements per
frame for the memo to then discard. The O(1) fix is to stop routing the cue
through render at all — subscribe outside React and toggle a class on the two
nodes that changed. That is a real change to a contentEditable with caret and
selection handling, so it is not smuggled in here.
The Media tab's timeline exists to add, remove and reorder clips. It carried the Edit tab's furniture anyway: a transport (play, prev/next, timecode, scrub bar), the Shift/Ctrl+Scroll hints, the zoom/pan window, and a playhead — none of which has anything to act on when there is no playback and no per-clip editing on that surface. All four are now gated on the `variant === "edit"` flag the component already had, and the "Arrange clips" caption centres, being alone in the header. Two behaviours go with them rather than being left inert, which is the part that would have bitten: - `startScrub` returns early. Seeking a playhead that is not rendered still wrote `currentTimeSec`, so a click on the Media timeline silently moved the Edit tab's preview from a screen displaying no time at all. - the Ctrl/Shift+wheel listener is not attached. Zooming with the zoom window gone leaves no control to undo it and no ruler reading to explain it.
Measured head-to-head in Chromium on a 32-minute recording (68 MB on disk),
which corrects what an earlier version of this message claimed:
STREAMING total 12296 ms (peaks 12259 ms, ~192 kB peak memory)
IN-MEMORY total 12198 ms (decodeAudioData 12003 ms → 714 MB,
channel slice copy 160 ms)
So the two pipelines take the SAME time, and the allocation is not the cost:
the slice copy I suspected is 160 ms, 1.3% of the total. Browser audio
decoding is the cost, in both paths, at roughly 160x realtime.
What this commit therefore does and does not do:
- Routing now estimates DECODED bytes from duration instead of comparing the
file's size, which says nothing about decoded size on compressed video (68 MB
→ 714 MB here). That is a MEMORY fix — 714 MB down to ~192 kB — and buys no
speed. The existing 256 MB threshold was already meant to protect memory; it
just measured the wrong quantity, so the streaming path never ran for
recordings that clearly needed it.
- The cache moves from a `useRef` (one per mounted component) to module scope,
with an in-flight map so N clips of one asset share a single decode. Every
clip used to decode independently, and every unmount threw the result away,
so a Media↔Edit switch re-decoded the whole recording. That is a real fix,
but only for REPEAT mounts — the first waveform of a session still waits the
full 12 s.
Making the first one fast needs a different pipeline, not a different route:
the bundled ffmpeg produces waveform-grade PCM for this same file in 2038 ms
(`-vn -ac 1 -ar 1000 -f s16le`, 3.9 MB out), 6x faster than the browser and in
the main process. With peaks cached on disk it would be paid once per
recording, ever. Not attempted here.
Both behaviours are covered, and both tests fail against the previous code.
The waveform took ~12s to appear on a 32-minute recording because both renderer pipelines decode the whole audio track in Chromium. Measured on the same 68 MB file: decodeAudioData (whole track) 12003 ms 714 MB resident WebCodecs chunk-by-chunk 12259 ms ~192 kB resident ffmpeg in the main process 3382 ms nothing resident 3.6x, off the UI process, and the result is cached on disk keyed by path+size+mtime — so it is paid once per recording rather than once per session. The renderer keeps both old pipelines and falls back to them when no native binary resolves, so nothing loses its waveform. WHICH ffmpeg, because this is where it would have broken silently: electron-builder deliberately excludes the static `ffmpeg.exe` (109 MB, "nothing in the app spawns" it). Spawning that one would have worked in dev and failed in every installed build. The SHARED build is 1 MB and links the same av*.dll set the compositor already ships, so fetch-ffmpeg.mjs now stages it as `ffmpeg-shared.exe` — named apart so the packager's `!win32-*/ffmpeg.exe` rule keeps dropping the static build while this one ships under the existing `win32-*/*` include. No packaging rule changes. Peaks are folded incrementally out of ffmpeg's stdout (mono s16 at 16 kHz), so the 62 MB of PCM never exists at once, and the block maths matches audioPeaksWorker.ts and streamingAudioPeaks.ts exactly — a clip must not change shape depending on which pipeline drew it. macOS and Linux have no binary staged in electron/native/bin, so `resolveFfmpeg` returns null there and they keep today's behaviour unchanged. ponytail: the CLI, not libav bindings in the compositor addon. Skipping the spawn saves ~20 ms against a ~2000 ms decode, and would cost a new Rust surface, an N-API entry point and a build story on three platforms. Parallelism was measured and rejected too: 4 processes over segments ran 1735 ms against 1893 ms for one, and 8 ran 2100 ms — it is demux- and spawn-bound, not CPU-bound. The disk cache is the real win.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Transcribing a 30-minute import fails. Reproduced on the reporter's file (
podcastproject, 32 min):whisperServer.runMultipartInferuses Node's globalfetch, i.e. undici, whoseheadersTimeoutdefaults to 300 s. whisper.cpp sends no response header until it has transcribed the whole upload. Measured on this file: whisper needed 574 s, undici cut the connection at 300 s, and the renderer surfaced an unactionable "Transcription failed / fetch failed". The helper process kept burning CPU on a transcription nobody was listening for — the reporter's machine still had one alive a day later.Audio extraction was never the problem: replayed as-is in Chromium it completes in 75 s and yields 31 215 421 samples @16 kHz with real signal (peak 0.85).
The fix
Split the audio into ~90 s chunks and run them one at a time.
electron/stt/chunking.tsnudges each boundary to the quietest 20 ms frame within ±3 s of the ideal cut, so a break lands in a pause instead of mid-word. Energy minimum, not a VAD: whisper.cpp's Silero VAD runs per request, so it cannot tell us where to cut before we upload.SttManager.transcribeshifts each chunk's timestamps to absolute time, emitscompletedSec/totalSecper chunk, and retries a chunk up to 3 times — re-runningserver.start()between attempts (idempotent when the helper is alive, a respawn when it isn't), because the usual cause of a mid-run failure is a dead helper, not a bad chunk.whisperServerbounds each request at 280 s and names the failure with the helper's stderr, instead of letting undici's 300 s ceiling surface as a bare "fetch failed".Progress reaches the UI
The renderer's status callback forwarded only a phase string and never subscribed to the main process's events, so the toast showed one static "transcribing" for the entire run. Progress now travels the road the phase already had —
status.tsowns the vocabulary, the store owns the queue,TranscriptionStatus.tsxowns how a job reads on screen. The store'sonStatusalso loses itsas TranscriptionPhasecast: the two vocabularies are now genuinely the same type,"loading-model"included.Both the bar and the percentage render nothing until the run reports measurable progress — queued, extracting audio and downloading the model have no fraction, and a bar pinned at 0 % reads as "stuck" where a spinner reads as "working".
Why chunks run sequentially
Measured, not assumed. whisper-stt-server holds a single model context:
0.53× — concurrency is ~1.9× slower. A client-side worker pool would be a pessimisation. Real parallelism needs several server processes, each with its own copy of the model resident on the GPU; worth revisiting only if a much smaller model ever becomes the default. This is recorded in the code so nobody "optimises" it later.
Verification
End to end on the 32-minute source, real
planChunksdriving the real server:Boundaries land at 87–93 s rather than on a 90 s grid — the pause search is doing its job.
vitest run electron/stt src/lib/captioning src/lib/ai-edition/transcription— 85/85 pass (10 new: 5 on chunk planning, 5 on orchestration — absolute offsets across seams, monotonic progress ending at 100 %, language pinning, retry, terminal failure; plus 2 on progress propagation).tsc --noEmitclean.Known gaps
src/lib/ai-edition/store/transcriptionStore.test.tswas not executed locally. It fails to load becausei18nextis missing from this machine'snode_modulesand no package manager is available here to install it — a pre-existing environment gap that also breaks 7 untouched component test files on this branch. The store change is covered bytscand by the purestatus.tstests; CI should be the judge.undicidependency andnew Agent({ headersTimeout: 0, bodyTimeout: 0 })as the fetch dispatcher (verified working). Deliberately not done here to avoid adding a dependency for a hypothetical machine — flagged in aponytail:comment.decodeAudioData44 s +mixToMono26 s, both synchronous). Out of scope here, worth its own pass.