Skip to content

fix(stt): chunked transcription with progress, retry, and no 300s cliff - #232

Merged
EtienneLescot merged 12 commits into
release/v1.8.0from
fix/stt-chunked-transcription
Aug 3, 2026
Merged

fix(stt): chunked transcription with progress, retry, and no 300s cliff#232
EtienneLescot merged 12 commits into
release/v1.8.0from
fix/stt-chunked-transcription

Conversation

@EtienneLescot

@EtienneLescot EtienneLescot commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

The bug

Transcribing a 30-minute import fails. Reproduced on the reporter's file (podcast project, 32 min):

wav 62430920 bytes
FETCH THREW after 311s: TypeError fetch failed — Headers Timeout Error

whisperServer.runMultipartInfer uses Node's global fetch, i.e. undici, whose headersTimeout defaults to 300 s. whisper.cpp sends no response header until it has transcribed the whole upload. Measured on this file: whisper needed 574 s, undici cut the connection at 300 s, and the renderer surfaced an unactionable "Transcription failed / fetch failed". The helper process kept burning CPU on a transcription nobody was listening for — the reporter's machine still had one alive a day later.

Audio extraction was never the problem: replayed as-is in Chromium it completes in 75 s and yields 31 215 421 samples @16 kHz with real signal (peak 0.85).

The fix

Split the audio into ~90 s chunks and run them one at a time.

  • electron/stt/chunking.ts nudges each boundary to the quietest 20 ms frame within ±3 s of the ideal cut, so a break lands in a pause instead of mid-word. Energy minimum, not a VAD: whisper.cpp's Silero VAD runs per request, so it cannot tell us where to cut before we upload.
  • SttManager.transcribe shifts each chunk's timestamps to absolute time, emits completedSec/totalSec per chunk, and retries a chunk up to 3 times — re-running server.start() between attempts (idempotent when the helper is alive, a respawn when it isn't), because the usual cause of a mid-run failure is a dead helper, not a bad chunk.
  • Language is pinned to whatever the first chunk detected. Left to auto-detect, whisper can flip mid-recording on a chunk that opens with a proper noun and "transcribe" the remainder as another language.
  • whisperServer bounds each request at 280 s and names the failure with the helper's stderr, instead of letting undici's 300 s ceiling surface as a bare "fetch failed".

Progress reaches the UI

The renderer's status callback forwarded only a phase string and never subscribed to the main process's events, so the toast showed one static "transcribing" for the entire run. Progress now travels the road the phase already had — status.ts owns the vocabulary, the store owns the queue, TranscriptionStatus.tsx owns how a job reads on screen. The store's onStatus also loses its as TranscriptionPhase cast: the two vocabularies are now genuinely the same type, "loading-model" included.

Both the bar and the percentage render nothing until the run reports measurable progress — queued, extracting audio and downloading the model have no fraction, and a bar pinned at 0 % reads as "stuck" where a spinner reads as "working".

Why chunks run sequentially

Measured, not assumed. whisper-stt-server holds a single model context:

wall time
two 120 s chunks, one after the other 76.9 s
the same two, fired together 144.1 s

0.53× — concurrency is ~1.9× slower. A client-side worker pool would be a pessimisation. Real parallelism needs several server processes, each with its own copy of the model resident on the GPU; worth revisiting only if a much smaller model ever becomes the default. This is recorded in the code so nobody "optimises" it later.

Verification

End to end on the 32-minute source, real planChunks driving the real server:

audio 1951.0s → 22 chunks (91,88,92,89,91,92,93,89,88,90,89,90,92,91,93,87,87,89,88,88,92,61s)
segments=485
coverage: last segment ends at 1950.7s / 1951.0s
slowest chunk 76.8s vs the 280s per-request ceiling
timestamps monotonic across chunk seams: true

Boundaries land at 87–93 s rather than on a 90 s grid — the pause search is doing its job.

  • vitest run electron/stt src/lib/captioning src/lib/ai-edition/transcription — 85/85 pass (10 new: 5 on chunk planning, 5 on orchestration — absolute offsets across seams, monotonic progress ending at 100 %, language pinning, retry, terminal failure; plus 2 on progress propagation).
  • tsc --noEmit clean.

Known gaps

  • src/lib/ai-edition/store/transcriptionStore.test.ts was not executed locally. It fails to load because i18next is missing from this machine's node_modules and no package manager is available here to install it — a pre-existing environment gap that also breaks 7 untouched component test files on this branch. The store change is covered by tsc and by the pure status.ts tests; CI should be the judge.
  • The 280 s per-request bound leaves one real ceiling: a machine so slow that a 90 s chunk needs more than 280 s (~0.3× realtime). Removing it for good means a direct undici dependency and new Agent({ headersTimeout: 0, bodyTimeout: 0 }) as the fetch dispatcher (verified working). Deliberately not done here to avoid adding a dependency for a hypothetical machine — flagged in a ponytail: comment.
  • Audio extraction still blocks the renderer for ~70 s on a 32-minute file (decodeAudioData 44 s + mixToMono 26 s, both synchronous). Out of scope here, worth its own pass.

A 30-minute recording was one `/inference` call: ~10 minutes with no
progress, no recovery from a transient failure, and — the reported bug —
killed outright before it ever finished. Node's global fetch (undici)
applies a 300s `headersTimeout`, and whisper sends no response header
until the whole upload is transcribed. Measured on the reporter's file:
whisper needed 574s, undici cut the connection at 300s, and the renderer
surfaced an unactionable "Transcription failed / fetch failed". The
helper kept burning CPU on a transcription nobody was listening for.

Split the audio into ~90s chunks and run them one at a time:

- `chunking.ts` nudges each boundary to the quietest 20ms frame within
  ±3s of the ideal cut, so a chunk break lands in a pause instead of
  mid-word. Energy minimum, not a VAD: whisper.cpp's Silero VAD runs per
  REQUEST, so it cannot tell us where to cut before we upload.
- `SttManager.transcribe` shifts each chunk's timestamps to absolute
  time, emits `completedSec`/`totalSec` per chunk, retries a chunk up to
  3 times, and re-runs `server.start()` between attempts (idempotent when
  the helper is alive, a respawn when it isn't) — the usual cause of a
  mid-run failure is a dead helper, not a bad chunk.
- The language detected on the first chunk is pinned for the rest. Left
  to auto-detect, whisper can flip mid-recording on a chunk that opens
  with a proper noun and "transcribe" the remainder as another language.
- `whisperServer` bounds each request at 280s and names the failure with
  the helper's stderr, instead of letting undici's 300s ceiling surface
  as a bare "fetch failed".

Chunks run SEQUENTIALLY, measured rather than assumed: whisper-stt-server
holds a single model context, and two 120s chunks took 76.9s one after
the other vs 144.1s fired together (0.53x — concurrency is ~1.9x SLOWER).
A client-side worker pool would be a pessimisation.

The progress reaches the UI: the renderer's status callback forwarded
only a phase string and never subscribed to the main process's events, so
the toast showed one static "transcribing" for the whole run. It now
carries the chunk progress and renders a real bar.

Verified end to end on a 32-minute source: 22 chunks, timestamps
monotonic across every seam, last segment at 1950.7s of 1951.0s, slowest
chunk 76.8s against the 280s ceiling.
…nner

The chunked pipeline now reports how much audio it has transcribed, so
surface it. A 30-minute recording spends minutes in "Transcribing…", and
a spinner that never changes is indistinguishable from a hang.

Progress travels the same road the phase already did — `status.ts` owns
the vocabulary, the store owns the queue, `TranscriptionStatus.tsx` owns
how a job reads on screen:

- `TranscriptionProgress` + `progressFraction` join `TranscriptionPhase`
  in status.ts, and `deriveAssetStatus` carries them onto the view.
- The store's `onStatus` no longer casts its argument to
  `TranscriptionPhase`: the renderer's `TranscribeStatus` and
  `TranscriptionPhase` are now genuinely the same vocabulary, including
  the `"loading-model"` phase the cast used to paper over. Failure paths
  clear `progress` with `phase`, so a failed job cannot leave a stale bar.
- `TranscriptionStatusDot`'s sibling `TranscriptionProgressBar` renders
  a determinate bar, and the label gains a percentage.

Both render nothing until the run reports measurable progress. Queued,
extracting audio and downloading the model have no fraction to report,
and a bar pinned at 0% reads as "stuck" where the spinner reads as
"working".
@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bd2a4e45-df37-4eed-9da6-97bccd687218

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The mock listener in transcribe.test.ts declared its own event shape and
still had the pre-progress one, so `tsc -p tsconfig.test.json` rejected the
`completedSec`/`totalSec` the test itself asserts on. Runtime was fine,
which is why vitest passed and only the CI typecheck caught it.
Five defects from the review of this PR, all in the new chunk loop.

`"auto"` defeated the cross-chunk language pin. The request contract spells
it as the explicit way to ask for detection, and it is truthy, so
`if (!language)` never fired and every chunk detected independently — the
exact flip the pin exists to prevent. Normalized to `undefined`. Verified
end-to-end on 352s of real speech: the chunks now go
`[undefined, "en", "en", "en"]` where they were all-detect before.

Nothing could stop a run. The chunk boundary is the natural interruption
point and went unused: cancelling in the renderer left main transcribing
every remaining chunk while the replacement request queued behind it, which
is what made "regenerate in another language" look dead. `cancel()` bumps an
epoch the loop checks between chunks, reachable over a new `stt:cancel`
channel, and surfaces as an `AbortError` so the store drops the job quietly.
Measured on a 375s clip: cancelling after chunk 1 returns in 19s, not 143s.

The IPC status sink was a single slot each request saved and restored, so
two overlapping transcriptions ended with the first one to finish silencing
the other for the rest of its run — no progress, which reads as a hang. It
is a Set now, and each request detaches only its own.

Every fetch rejection claimed "after 280s". A helper that died a moment ago
rejects in a millisecond, and telling the reader it spent 280s on an
over-long chunk sends them somewhere else entirely. The timeout wording is
reserved for an actual timeout, the cause is attached, and the body read is
named too. A chunk that exhausts its retries now says where it died rather
than failing a 30-minute recording with no position at all.

Energy ties broke toward the earliest frame, so digital silence — a muted
track, a gap between takes — pulled every cut back by the whole search
window: 90s chunks became 87s, and the silent test case 10s became 5s. Ties
now break toward the target.

The pin's comment also records what a stale staged helper does to it, since
that cost a full debugging detour: electron/native/bin is gitignored, a
binary from before cc78180 echoes the request instead of resolving it, and
the failure is completely silent.
…nload

The bar was wrapped in a div carrying `margin: -8px 0 16px`, rendered
unconditionally. `.mediaDetail` is a flex column, so those margins do not
collapse: every media card that was not transcribing paid 8px of dead gap,
and the failure hint below it sat 8px lower than intended. A wrapper cannot
render itself away with its child — the margin belongs on the bar.

`"loading-model"` was added to `TranscriptionPhase` in this PR, plumbed
through `TranscribeStatus` and the store, and then read by nothing: every
label switches on `status`, so the 253MB first-run download was labelled
"Transcribing" like everything else. That download is the wait most often
mistaken for a hang, which is what this PR is about, so it gets its own
words rather than the plumbing getting deleted.
`--connect-timeout` only covers the handshake. An origin that accepts the
connection and then trickles never errors, so nothing retried and nothing
fell through to the Debian mirror — the job would sit until the runner's
six-hour limit, which is the failure the mirror was added to survive.
`--speed-limit 1024 --speed-time 30` bounds the part `--connect-timeout`
cannot reach, and cannot fire on a merely slow link for a ~10MB tarball.

`spawnSync` reports a missing curl as `{ status: null, error: ENOENT }`, so
"curl exited with null" twice over sent the reader hunting a network problem
instead of a missing binary. The error is surfaced when present.

The comment claimed a 404 mirror is "simply skipped"; with
`--retry-all-errors` it is retried three times first. The flags stay — that
retry is the whole point on a connection reset — and the comment now says
what actually happens.

nix/package.nix restated the version by hand and had already drifted two
minors behind the app it names (1.6.0, then 1.8.0 against a 1.8.0-rc.6
tree). It reads package.json now. `npmDepsHash` still needs updating by
hand, but that one fails loudly.
The language whisper resolved on chunk 1 — the one every later chunk is
pinned to — reached the document and was rendered nowhere.

It had a pill, in `SourceTranscriptModal`. That modal is mounted by
`LeftPanel`'s `MediaPane`, and `LeftPanel` is `active === "chat" ?
<ChatStripPanel /> : <MediaPane />` with exactly one mount site passing the
literal `active="chat"`. So `MediaPane`, `MediaList` and the pill are
unreachable, and "did the language pin work?" had no answer in the UI at
all — only in the saved .openscreen.

Put it on the live surface, next to the status badge in the v4 MediaStage
detail panel. It also belongs beside "Regenerate as": that selector is the
control you set BECAUSE of what was detected, and it currently reads "Auto"
next to a transcript that resolved "en".
Scrubbing a 30-minute recording (4501 words) stutters. The transcript pane
subscribes to the playhead and re-renders per frame, which is by design —
`TranscriptClipBlock` is memoised on `cueWordId` precisely so that the frames
in between cost nothing.

That memo's premise is playback: "cueWordId changes at word boundaries, a few
times per second, not sixty". A scrub breaks it. Dragging the playhead crosses
many words per frame, so `cueWordId` changes on EVERY frame, the block
re-renders every frame, and it renders one component per transcript word.
`TranscriptWord` had no memo, so all 4501 re-rendered to move one underline.

Measured over a 40-frame scrub in jsdom, median per frame:

     words    before    after
       100   19.6 ms   11.5 ms
       500   24.0 ms   13.9 ms
      1500   83.9 ms   24.7 ms
      4501  132.6 ms   57.4 ms

The same benchmark with the cue word held inside one word — so the block's
memo bails out — costs 0.1 ms/frame at 4501 words. That isolates it: the
linear `findCueWordId` scan and the playhead subscription are both free, and
100% of what remains is the block re-render itself.

So this halves it but does not fix the shape: the cost is still proportional
to transcript length, because the block still builds 4501 React elements per
frame for the memo to then discard. The O(1) fix is to stop routing the cue
through render at all — subscribe outside React and toggle a class on the two
nodes that changed. That is a real change to a contentEditable with caret and
selection handling, so it is not smuggled in here.
The Media tab's timeline exists to add, remove and reorder clips. It carried
the Edit tab's furniture anyway: a transport (play, prev/next, timecode,
scrub bar), the Shift/Ctrl+Scroll hints, the zoom/pan window, and a playhead
— none of which has anything to act on when there is no playback and no
per-clip editing on that surface.

All four are now gated on the `variant === "edit"` flag the component already
had, and the "Arrange clips" caption centres, being alone in the header.

Two behaviours go with them rather than being left inert, which is the part
that would have bitten:

- `startScrub` returns early. Seeking a playhead that is not rendered still
  wrote `currentTimeSec`, so a click on the Media timeline silently moved the
  Edit tab's preview from a screen displaying no time at all.
- the Ctrl/Shift+wheel listener is not attached. Zooming with the zoom window
  gone leaves no control to undo it and no ruler reading to explain it.
Measured head-to-head in Chromium on a 32-minute recording (68 MB on disk),
which corrects what an earlier version of this message claimed:

  STREAMING   total 12296 ms   (peaks 12259 ms, ~192 kB peak memory)
  IN-MEMORY   total 12198 ms   (decodeAudioData 12003 ms → 714 MB,
                                channel slice copy 160 ms)

So the two pipelines take the SAME time, and the allocation is not the cost:
the slice copy I suspected is 160 ms, 1.3% of the total. Browser audio
decoding is the cost, in both paths, at roughly 160x realtime.

What this commit therefore does and does not do:

- Routing now estimates DECODED bytes from duration instead of comparing the
  file's size, which says nothing about decoded size on compressed video (68 MB
  → 714 MB here). That is a MEMORY fix — 714 MB down to ~192 kB — and buys no
  speed. The existing 256 MB threshold was already meant to protect memory; it
  just measured the wrong quantity, so the streaming path never ran for
  recordings that clearly needed it.

- The cache moves from a `useRef` (one per mounted component) to module scope,
  with an in-flight map so N clips of one asset share a single decode. Every
  clip used to decode independently, and every unmount threw the result away,
  so a Media↔Edit switch re-decoded the whole recording. That is a real fix,
  but only for REPEAT mounts — the first waveform of a session still waits the
  full 12 s.

Making the first one fast needs a different pipeline, not a different route:
the bundled ffmpeg produces waveform-grade PCM for this same file in 2038 ms
(`-vn -ac 1 -ar 1000 -f s16le`, 3.9 MB out), 6x faster than the browser and in
the main process. With peaks cached on disk it would be paid once per
recording, ever. Not attempted here.

Both behaviours are covered, and both tests fail against the previous code.
The waveform took ~12s to appear on a 32-minute recording because both
renderer pipelines decode the whole audio track in Chromium. Measured on the
same 68 MB file:

  decodeAudioData (whole track)      12003 ms   714 MB resident
  WebCodecs chunk-by-chunk           12259 ms   ~192 kB resident
  ffmpeg in the main process          3382 ms   nothing resident

3.6x, off the UI process, and the result is cached on disk keyed by
path+size+mtime — so it is paid once per recording rather than once per
session. The renderer keeps both old pipelines and falls back to them when
no native binary resolves, so nothing loses its waveform.

WHICH ffmpeg, because this is where it would have broken silently:
electron-builder deliberately excludes the static `ffmpeg.exe` (109 MB,
"nothing in the app spawns" it). Spawning that one would have worked in dev
and failed in every installed build. The SHARED build is 1 MB and links the
same av*.dll set the compositor already ships, so fetch-ffmpeg.mjs now stages
it as `ffmpeg-shared.exe` — named apart so the packager's
`!win32-*/ffmpeg.exe` rule keeps dropping the static build while this one
ships under the existing `win32-*/*` include. No packaging rule changes.

Peaks are folded incrementally out of ffmpeg's stdout (mono s16 at 16 kHz),
so the 62 MB of PCM never exists at once, and the block maths matches
audioPeaksWorker.ts and streamingAudioPeaks.ts exactly — a clip must not
change shape depending on which pipeline drew it.

macOS and Linux have no binary staged in electron/native/bin, so `resolveFfmpeg`
returns null there and they keep today's behaviour unchanged.

ponytail: the CLI, not libav bindings in the compositor addon. Skipping the
spawn saves ~20 ms against a ~2000 ms decode, and would cost a new Rust
surface, an N-API entry point and a build story on three platforms.
Parallelism was measured and rejected too: 4 processes over segments ran
1735 ms against 1893 ms for one, and 8 ran 2100 ms — it is demux- and
spawn-bound, not CPU-bound. The disk cache is the real win.
@EtienneLescot
EtienneLescot merged commit 37a48df into release/v1.8.0 Aug 3, 2026
12 checks passed
@EtienneLescot
EtienneLescot deleted the fix/stt-chunked-transcription branch August 3, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant