fix(realtime): support externally transcribed user turns

[TonyG-FWE]

Fixes #5408

Summary

This fixes two turn-semantics problems in realtime sessions that use external turn detection:

1. Externally bounded Google audio turns could receive a second activity end plus the legacy "." placeholder when generate_reply() ran, conflicting with the activity-backed turn and eventually timing out.
2. AgentActivity discarded finalized external STT messages whenever the LLM was a RealtimeModel, preventing an external-STT text-input path and dropping edits made by on_user_turn_completed.

The change adds the provider-neutral turn_handling.realtime_input_mode option so applications explicitly choose whether a realtime model receives native audio or finalized external-STT text.

Input modes

• "audio" (default) preserves existing realtime behavior. With client-side turn detection, audio is sent through one provider activity and Google waits for the natural generation_created acknowledgement without adding a synthetic user turn.
• "text" (opt-in) routes microphone audio only to external VAD/STT. The finalized ChatMessage, including on_user_turn_completed edits, is synchronized to the realtime provider exactly once before one generation is requested. Raw audio is not also sent to the model.

Text mode validates its required boundaries: a realtime model, external STT, mutable realtime chat context, and provider-side turn detection disabled or disableable. It is never inferred merely from the presence of STT. When text mode has streaming external STT, no VAD, and no explicitly selected detector, the activity automatically uses STT end-of-speech detection; explicit None, "manual", "stt", and other valid explicit selections retain their existing meaning.

The built-in Agent.stt_node now rejects a non-streaming STT with no VAD synchronously instead of starting a session whose STT pump will fail. Runtime STT/VAD updates validate the prospective pair before mutation. A pre-wrapped stt.StreamAdapter remains valid without a session VAD. Custom STT nodes remain supported: when model capabilities cannot establish their turn boundary, they require a compatible explicit detector at startup, while an already-resolved "stt" boundary remains valid across runtime model swaps.

Google's placeholder behavior remains available for legitimate application-initiated generations that have no activity-backed or text-backed user input.

Lifecycle and provider ownership

• Realtime audio input has a private logical-input identity spanning buffering, activity boundaries, deferred replay, generation setup, cancellation, and cleanup. A finishing reply can clear only the input it owns, never a next-turn input.
• End-of-turn skip_reply and reply_already_triggered disposition is owned by one logical turn and one bounce generation. Speech onset, turn-detector mode changes, explicit clear, failed commit, and cancellation abandon only the affected disposition.
• Google manual-input events carry provider-local sequence ownership. Discard invalidates queued and in-flight events from the exact input sequence, removes only the exact pending text item, quarantines post-discard audio, and restarts the provider epoch only when provider-visible abandoned input requires it.
• A required Google restart is deferred while unrelated output or tool continuation still owns the provider turn; a new explicit user activity can force the fresh epoch and replay only post-discard audio.
• Native audio, server-side turn detection, local/provider chat context, conversation items, metrics, cancellation, skip_reply, StopResponse, empty transcripts, reconnects, close, and terminal failures retain clean per-turn settlement.

Regression coverage

Hermetic coverage uses fake realtime sessions and captured provider events. It includes:

• the exact old-reply/new-turn cleanup interleaving, pre-VAD deferred audio, consecutive commits, setup/replay failure, cancellation, close, and pause/resume;
• turn-disposition abandonment through VAD, STT, provider onset, task replacement, mode transition, failed manual commit, and clear;
• Google local-only versus provider-visible discard, queued/dequeued/in-flight sender races, exact pending-text removal, active output and tool continuation, reconnect, quarantine bounds, and next-sequence ownership;
• automatic STT fallback with one combined finalized turn at END_OF_SPEECH, runtime option reset, and preservation of explicit detector choices;
• fail-fast validation for the default non-streaming-STT/no-VAD path, atomic runtime STT/VAD rejection, simultaneous valid updates, pre-wrapped adapters, custom nodes, and already-resolved custom STT boundaries;
• the original [google] RealtimeModel + external VAD: generate_reply() conflicts with activity-based audio flow, STT transcript discarded #5408 audio and text paths, default compatibility, edits from on_user_turn_completed, empty turns, skip_reply, StopResponse, and the legitimate placeholder path.

Fail-before evidence was captured against published head 5c4c1ac285ee04fae8a62ec00b9136ff21026fd5 for the four original review defects. The no-VAD streaming-STT configuration resolved to no detector and finalized both "hello" and "hello world"; the corrected path resolves to "stt" and finalizes one "hello world" turn at STT end-of-speech.

The later non-streaming-STT review finding was reproduced against aec93d628b9a423c414e75ad8b94d404592cdb7c: construction succeeded with no usable detector, and runtime STT swap/VAD removal replaced a working pipeline before its pump raised the existing VAD/StreamAdapter error. The focused tests failed 3/6 before the initial fix; an additional custom-node/implicit-detector test and resolved-boundary runtime-swap test each failed before their corresponding lifecycle correction.

Validation

At head 90f92f31db0be3fae9217ce7822e9db98d00f198:

• GitHub full test workflow: 2,074 passed, 5 skipped in the unit job; blockguard passed on Ubuntu, macOS, and Windows; aggregate dumps passed.
• GitHub CI workflow: Ruff and Python 3.10/3.13 type checks passed.
• Release gate: passed.
• CLA Assistant: passed; every branch commit is associated with TonyG-FWE.
• Focused non-streaming/default-node/custom-node/runtime regressions: 10 passed.
• \.venv\Scripts\python.exe -m pytest -q tests/test_realtime_external_input.py tests/test_agent_update_options.py tests/test_plugin_google_realtime.py tests/test_realtime_adaptive_interruption.py tests/test_false_interruption_resume.py tests/test_audio_recognition_handoff.py tests/test_speech_start_time_persistence.py tests/test_agent_session.py tests/test_audio_recognition_push_audio.py tests/test_audio_recognition_aclose.py: 259 passed.
• \.venv\Scripts\python.exe -m pytest -q tests/test_audio_recognition_turn_detection.py: 35 passed, 3 failed; all three failures are unrelated and reproduce on exact base 3568970323b04a2f71a134a8cc43e7b30f3e00e2 (two tests call nonexistent public methods; one expects stale warning text).
• \.venv\Scripts\ruff.exe format --check .: 932 files already formatted.
• \.venv\Scripts\ruff.exe check .: passed.
• \.venv\Scripts\mypy.exe --platform linux -p livekit.agents: no issues in 206 source files; the Python 3.13 language target also passed. The local Python 3.10 target was blocked by the installed NumPy stub using Python 3.12 syntax; the authoritative GitHub 3.10 job passed.
• git diff --check: passed.
• \.venv\Scripts\python.exe -m pytest --unit -q: the Windows/Python 3.14 environment reached 1,649 passed, 6 skipped before the same two exact-base-reproduced failures, nine LiveKit test-server startup errors, and the existing concurrency-plugin closed-loop error. The clean GitHub unit workflow above passed in full.

Local make check was unavailable because this Windows environment has no make; its Ruff and type-check components ran directly. The repository type wrapper could not launch its bare uv subprocess because uv.exe is not exposed on this shell's PATH; direct package typing and the authoritative GitHub matrix passed.

No Google, Gemini, Vertex, or LiveKit credential environment variables were present, so no live-provider integration test was run. Required coverage is hermetic.

Related work and scope

Agents JS #779 reports the same architectural distinction between provider-consumed audio and externally transcribed text; this PR changes only the Python repository.

PR #6537 remains open and changes adjacent realtime preemptive-generation scheduling. It does not add the input-mode distinction or solve the duplicate Google activity/placeholder trigger, so this work does not use or copy its branch.

PR #6481 is adjacent Google session-restart/tool-result replay work and also touches legitimate placeholder generation. It targets #6479, remains open and currently conflicts with main, and does not implement external-input modes or input-sequence ownership. This PR preserves its own legitimate application-initiated placeholder path and does not use that branch.

This does not address Gemini's upstream audio-context growth, latency escalation, or context-exhaustion behavior.

[CLAassistant]

All committers have signed the CLA.

[TonyG-FWE]

@codex review

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 90f92f31db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[devin-ai-integration]

Devin Review found 2 new potential issues.

View 7 additional findings in Devin Review.

[devin-ai-integration]

🔴 Tool results can be silently dropped while a Gemini realtime connection is being re-established

Tool results are thrown away (early return at livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py:935-942) whenever the chat context is updated while the connection is being re-established, so the model waits forever for an answer it never receives.

Impact: A tool call made just before a reconnect can leave the conversation permanently stalled with no reply.

Mechanism: early return skips the tool-response send path and reconnect never replays function outputs

RealtimeSession.update_chat_ctx gained a new early-return branch:

if not self._active_session or self._session_should_close.is_set():
    if turns and self._session_resumption_handle is not None:
        self._mark_restart_needed()
    self._chat_ctx = chat_ctx
    ...
    return

This returns before the if append_ctx.items: block that builds and sends tool_results via get_tool_results_for_realtime(...) / _send_client_event(...) (realtime_api.py:960-977).

Previously the guard was only if not self._active_session: — while a restart was pending (_session_should_close set) the session object still existed, so the tool response was enqueued on the freshly-created _msg_ch and delivered by the next session's send task.

The dropped tool response is never recovered: the reconnect replay in _main_task copies the chat context with exclude_function_call=True (realtime_api.py:1390), so function_call_output items are never re-sent.

The _session_should_close window is not short: it stays set through asyncio.wait teardown, _close_active_session() (network close), and, on the error path, through await asyncio.sleep(retry_interval) in the retry loop.

The same omission exists in the second new early return at realtime_api.py:947-958 (turns and self._input_state in (AUDIO_ACTIVE, INTERRUPT_ONLY)), which also returns before the tool-result send when an update carries both a new user turn and a tool output.

Prompt for agents

In livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py, RealtimeSession.update_chat_ctx now has two new early-return branches (one when there is no active session or a restart is pending, one when new turns arrive while a realtime audio/interruption activity is open). Both return before the block that computes `tool_results = get_tool_results_for_realtime(append_ctx, ...)` and sends them with `_send_client_event`. Because the reconnect replay in `_main_task` copies the chat context with `exclude_function_call=True`, a function_call_output that hits either early return is never delivered to Gemini, and the model can wait indefinitely for the tool response. Rework these branches so that pending tool responses are still enqueued (or otherwise persisted and replayed after the new session is established), while keeping the new client-content/turn-state handling intact.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Transcription-timeout notifications start firing for realtime sessions that never enabled them

A transcription-timeout notification is now raised (timeout defaulted from max_delay at livekit-agents/livekit/agents/voice/audio_recognition.py:2022-2027) even when the application left the timeout feature turned off, so apps suddenly receive an event they explicitly opted out of.

Impact: Existing realtime voice apps begin receiving unexpected timeout notifications on any turn where speech-to-text returns nothing, which can trigger unwanted "please repeat yourself" prompts.

Mechanism: unconditional hook call after the newly synthesized timeout

transcription_timeout defaults to None and is documented as "Disabled by default". _arm_transcription_timeout used to return immediately in that case. It now substitutes self._endpointing.max_delay whenever finalize_empty_transcript_on_timeout is set.

That flag is enabled for the default realtime configuration — AgentActivity passes isinstance(self.llm, llm.RealtimeModel) and not self._rt_turn_detection_enabled and self._turn_detection != "manual" (livekit-agents/livekit/agents/voice/agent_activity.py:1252-1254).

When the synthesized timer fires, _on_transcription_timeout calls the hook unconditionally before the new finalization logic:

self._hooks.on_transcription_timeout(
    speech_duration=self._turn_speech_duration, turn_start=self._user_turn_start
)
if self._finalize_empty_transcript_on_timeout:
    self._run_eou_detection(...)

AgentActivity.on_transcription_timeout (livekit-agents/livekit/agents/voice/agent_activity.py:2504-2511) emits the public user_transcription_timeout session event. So a realtime + external-STT app that never set transcription_timeout now gets that event on every turn where STT produced no text, whereas the intent of the change is only to bound and finalize the turn internally.

Prompt for agents

In livekit-agents/livekit/agents/voice/audio_recognition.py, `_arm_transcription_timeout` now synthesizes a timeout from `self._endpointing.max_delay` when `transcription_timeout` is unset but `_finalize_empty_transcript_on_timeout` is enabled (which is the default for realtime models with client-side turn detection). `_on_transcription_timeout` then calls `self._hooks.on_transcription_timeout(...)` unconditionally, which emits the public `user_transcription_timeout` session event. Applications that left `transcription_timeout` at its default of None explicitly disabled that event, so they now receive notifications they never opted into. Track whether the timer was armed from a user-configured timeout or from the internal realtime finalization bound, and only emit the public event in the former case while still running the empty-transcript end-of-turn finalization in the latter.

---

Was this helpful? React with 👍 or 👎 to provide feedback.