feat(voice): add speech_id to assistant message metrics - #6787
Open
hassannaftabb wants to merge 1 commit into
Open
feat(voice): add speech_id to assistant message metrics#6787hassannaftabb wants to merge 1 commit into
hassannaftabb wants to merge 1 commit into
Conversation
TTSMetrics and the llm_node/tts_node spans both carry a speech_id, but the assistant ChatMessage they describe did not, so a stored message could not be joined to its trace or to the metrics_collected events of the same turn. Populate speech_id on the assistant metrics in all three paths that store an assistant message: session.say, the reply pipeline, and realtime generation. Follows the provider_request_ids precedent: an optional key on the existing MetricsReport TypedDict, so no existing field changes. Closes livekit#4387
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4387.
Problem
TTSMetricscarries aspeech_id, and thellm_node/tts_nodespans setlk.speech_id, but the assistantChatMessagethose describe carries no such id.metrics_collectedevents are not part ofSessionReport, so joining a stored message to its trace or to the per-component metrics of the same turn takes external bookkeeping. One commenter on #4387 keeps anitem.id -> speech_idmap by hand.Change
MetricsReportgains an optionalspeech_id, populated in the three paths that store an assistant message:session.say(_tts_task_impl)_pipeline_reply_task_impl)_realtime_generation_task_impl)This follows the
provider_request_idsprecedent: a new key on the existingTypedDict, so no existing field changes._early_assistant_metricsis left alone on purpose — it only feeds console rendering and never reaches a stored message.One correction to the issue's framing
#4387 asks for this so
tts_node_ttfb - TTSMetrics.ttfbgives sentence-tokenizer overhead. That subtraction no longer measures anything.tts_node_ttfbnow anchors on the text handed to the TTS provider, so it reads the same asTTSMetrics.ttfb: both came out at 0.19998 for the same turn on the fake harness. The LLM-to-TTS buffering the reporter wanted is already exposed asllm_node_ttfs.The id still looks worth having for the other reason, joining a persisted message to its
lk.speech_idspan, so I've written it up that way. Close this if you'd rather not add the field.Tests
All three failed before the change:
test_assistant_metrics_share_speech_id_with_tts_metrics— pipeline path; asserts the message's id matches the one on thetts_metricsevent from the same runtest_say_assistant_metrics_carry_speech_id—session.saypathtest_realtime_response_id_is_available_on_assistant_message— realtime pathpytest --unitgives 1520 passed with no new failures against main.ruff check,ruff format --check, andmypy --platform linux -p livekit.agentsare clean.