fix(voice): don't count discarded room audio as played - #2256
fix(voice): don't count discarded room audio as played#2256rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 1768f45 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| pause(): void { | ||
| if (this.playbackEnabledFuture.done) { | ||
| this.playbackEnabledFuture = new Future(); | ||
| } | ||
| // Drop already-buffered audio so playback stops promptly instead of draining the prebuffer. | ||
| this.audioSource.clearQueue(); | ||
| super.pause(); | ||
| } |
There was a problem hiding this comment.
🔴 Pausing the agent no longer stops audio that is already buffered, so the agent keeps talking
Already-buffered agent audio is no longer thrown away when playback is paused (the audioSource.clearQueue() call that used to run in pause() at agents/src/voice/room_io/_output.ts:423-428 was removed and only re-added inside the frame-forwarding path at agents/src/voice/room_io/_output.ts:480-496), so a pause that arrives after the last frame was handed over leaves the buffered audio playing to the end.
Impact: When a user starts speaking near the end of an agent utterance, the agent keeps talking for up to the source queue length (~1s) instead of going quiet immediately.
Why the deferred clear does not cover the tail of an utterance
Before this PR, pause() dropped the prebuffer directly ("Drop already-buffered audio so playback stops promptly instead of draining the prebuffer", introduced in #1579). Now the queue is only cleared when the next captureFrame reaches the pause gate at agents/src/voice/room_io/_output.ts:480. During continuous TTS streaming that happens roughly one frame later, but once all frames of the segment have been submitted (the common end-of-utterance case, or a frame currently blocked inside audioSource.captureFrame which never re-checks the gate) no further capture arrives, so nothing clears the queue.
ParticipantAudioOutput.pause() is used by the false-interruption flow in agents/src/voice/agent_activity.ts:1474 and agents/src/voice/agent_activity.ts:1588, whose whole purpose is to silence the agent promptly when the user starts speaking. The new test at agents/src/voice/room_io/_output.test.ts:384-396 locks in the new behavior (pause after flush still reports the full playback position).
A fix would perform the same discarded-duration accounting that captureFrame does and then clear the queue inside pause().
Prompt for agents
In agents/src/voice/room_io/_output.ts, ParticipantAudioOutput.pause() used to call this.audioSource.clearQueue() so that pausing stopped already-buffered playback immediately. That call was removed and the clearing (plus the new discarded-duration accounting) now only happens inside captureFrame when a subsequent frame reaches the pause gate. If no further frame is captured — the normal situation once all frames of an utterance have been submitted, or when a frame is blocked inside audioSource.captureFrame — the queued audio keeps playing, so pause() no longer stops the agent promptly. pause() is used by the false-interruption path in agent_activity.ts (around lines 1474 and 1588) to silence the agent when the user starts speaking. Consider factoring the discard accounting in captureFrame (attributing audioSource.queuedDuration to the most recent captureDurations entries and bumping sourceDiscardedDuration) into a helper and invoking it from pause() before clearing the queue, so pause stops playback promptly while still not counting the discarded audio as played. Note that the new test 'finishes playout when paused after forwarding drains' in _output.test.ts encodes the current behavior and may need revisiting.
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.flushTask.cancel(); | ||
| } |
There was a problem hiding this comment.
🟡 A flush that arrives after an interruption can start a second playback-completion task and emit extra playback-finished events
A flush arriving right after an interruption is no longer treated as a no-op (the early-return guard at agents/src/voice/room_io/_output.ts:634 now also requires no pending segments), so it cancels the in-flight completion task and starts a duplicate one that reports the same audio as finished twice.
Impact: A spurious error is logged and a later piece of agent speech can be reported as finished before it actually played, ending a turn early.
Sequence that produces the duplicate task
clearBuffer() internally calls flush() (agents/src/voice/room_io/_output.ts:676), which records flushPushedDuration = pushedDuration (a non-zero value) and starts a waitForPlayoutTask. Immediately afterwards clearBuffer() zeroes this.pushedDuration (agents/src/voice/room_io/_output.ts:700).
If a producer then calls flush() while that task is still pending (e.g. forwardAudio's finally in agents/src/voice/generation.ts:1047 running after cancelAndWait times out in agents/src/voice/agent_activity.ts:2713-2717; the comment at agents/src/voice/avatar/queue_io.ts:100-105 also notes producers typically flush after clearBuffer):
- the new guard passes because
pendingPlayoutSegments > 0, flushPushedDuration(old non-zero value) !==pushedDuration(0), sologger.error('flush called while playback is in progress')fires andflushTask.cancel()is called.
Task.cancel() only aborts the controller; waitForPlayoutTask has already passed its abort race, so its body runs to completion anyway while a second task is created. Both bodies then emit onPlaybackFinished for the same interruption snapshot. The base class drops surplus finishes with a warning, but if a new segment has been captured in the meantime the extra finish is consumed by that segment and its waitForPlayout() resolves prematurely with a stale interrupted event.
Before this PR, clearBuffer() did not reset pushedDuration, so the second flush() matched flushPushedDuration and returned early.
(Refers to lines 634-645)
Prompt for agents
In agents/src/voice/room_io/_output.ts, flush() now proceeds when pushedDuration is 0 as long as pendingPlayoutSegments > 0. After clearBuffer() has zeroed pushedDuration (line ~700) while its own flush task is still running, a subsequent producer-side flush() reaches the 'flush called while playback is in progress' branch (flushPushedDuration !== pushedDuration), logs an error, cancels the running task (which keeps running because waitForPlayoutTask does not observe the abort signal after its race) and creates a second task. Both tasks then emit onPlaybackFinished for the same interruption snapshot, which can prematurely finish a later segment. Consider making flush() a no-op when an interruption-driven flush task is already in flight (e.g. track that the current flushTask was started by clearBuffer / an interruption snapshot exists for the current interruptedFuture), or keep flushPushedDuration consistent with the zeroed pushedDuration so the existing equality guard still short-circuits.
Was this helpful? React with 👍 or 👎 to provide feedback.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
1 participant
Ports livekit/agents#6728.
Summary
@livekit/agentsSource diff coverage
Source diff coverage
livekit-agents/livekit/agents/voice/recorder_io/recorder_io.pytoagents/src/voice/recorder_io/recorder_io.ts: clipped pause intervals to each recorder segment's actual first-capture timestamp, using the target per-segment recorder bookkeeping.livekit-agents/livekit/agents/voice/room_io/_output.pytoagents/src/voice/room_io/_output.ts: ported source-submitted/discarded duration accounting, active-submission draining, interruption generation tracking, and paused-drain completion into the target promise-gated capture architecture. Capture-sequence bookkeeping preserves the target support for overlapping segments.tests/test_recording.pytoagents/src/voice/recorder_io/recorder_io.test.ts: ported all 3 source pause-alignment scenarios to Vitest.tests/test_room_io.pytoagents/src/voice/room_io/_output.test.ts: ported all 5 source room-audio scenarios with target-side fake audio sources and promise controls.No source diff files were omitted, and no required infrastructure gap remains.
Validation
pnpm exec vitest run agents(1,596 passed, 5 skipped)pnpm --filter @livekit/agents buildpnpm --filter @livekit/agents lintpnpm format:checkcue-clivoice interruption run: observedoverlapping_speech(is_interruption=true), an interrupted spoken-prefix commit, and a fresh non-interrupted assistant commitPorted from livekit/agents#6728
Original PR description
Problem: Pausing RoomIO cleared queued agent audio, but interruption accounting later treated it as played. RecorderIO then padded the user track to match audio that never played.
Behavior: Interrupted playback now derives position from audio sent to the source, excluding queued and discarded durations. Buffered or pause-held frames never count as playout.
Fixes AGT-3231