feat(expressive): expose expressive mode and add expressive agent example - #2245
feat(expressive): expose expressive mode and add expressive agent example#2245rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: ec5b051 The changes in this PR will be included in the next version bump. This PR includes changesets to release 36 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| for (const mood of MOOD_PRIORITY) { | ||
| const score = Object.entries(MOOD_KEYWORDS[mood]).reduce( | ||
| (total, [keyword, weight]) => total + (matchesWord(text, keyword) ? weight : 0), | ||
| 0, | ||
| ); | ||
| if (score > bestScore) { | ||
| best = mood; | ||
| bestScore = score; | ||
| } | ||
| } |
There was a problem hiding this comment.
🟡 Delivery labels that describe curiosity are reported as empathy instead
A label whose descriptive words tie in weight is resolved by a fixed priority list (matchMood at agents/src/tts/_mood.ts:54-63) that ranks empathy above curiosity, so "gently curious, welcoming" is published as empathetic and the PR's own expectation for that label does not hold.
Impact: Frontends reading the published mood get the wrong feeling for common mixed labels, and the new mood test in this PR fails.
Score computation and tie-break mechanism
Scoring is a sum over keyword prefix matches. For the lowercased label gently curious, welcoming:
empatheticmatches BOTHgentle(weight 1,agents/src/tts/_mood_data.ts:189) andgently(weight 1,agents/src/tts/_mood_data.ts:190) becausematchesWordonly requires the match to start at a word start and does not require a word end — the single word "gently" is therefore counted twice → score 2.curiousmatchescurious(weight 2) → score 2.
MOOD_PRIORITY lists empathetic (index 5) before curious (index 7) and the loop updates only on score > bestScore, so the first mood with the maximum score wins → empathetic.
agents/src/tts/mood.test.ts:17 asserts matchMood('gently curious, welcoming') === 'curious', so this case is a failing test as well as a questionable classification. Either the overlapping gentle/gently stems must be de-duplicated (keep only the gentl stem) or the priority/tie-break must match the upstream Python ordering.
Prompt for agents
matchMood in agents/src/tts/_mood.ts sums keyword weights per mood and breaks ties by the order of MOOD_PRIORITY (first mood with the strictly-highest score wins). Two problems interact: (1) MOOD_KEYWORDS.empathetic in agents/src/tts/_mood_data.ts contains both the stem 'gentle' and the longer form 'gently', and matchesWord only anchors at a word start (no word-end check), so a single occurrence of the word 'gently' contributes 2 points instead of 1; (2) MOOD_PRIORITY ranks 'empathetic' ahead of 'curious'. As a result matchMood('gently curious, welcoming') returns 'empathetic', while agents/src/tts/mood.test.ts:17 expects 'curious'. Verify against the upstream Python _mood.py/_mood_data.py whether the keyword table should contain only one stem ('gentl') and/or whether MOOD_PRIORITY ordering differs, and align the port so the shipped test passes and overlapping stems cannot double-count.
Was this helpful? React with 👍 or 👎 to provide feedback.
| export type AgentMood = | ||
| | 'excited' | ||
| | 'happy' | ||
| | 'playful' | ||
| | 'curious' | ||
| | 'surprised' | ||
| | 'hopeful' | ||
| | 'empathetic' | ||
| | 'sad' | ||
| | 'angry' | ||
| | 'anxious' | ||
| | 'calm'; | ||
|
|
||
| export const MOOD_PRIORITY: AgentMood[] = [ | ||
| 'angry', | ||
| 'sad', | ||
| 'anxious', | ||
| 'surprised', | ||
| 'playful', | ||
| 'empathetic', | ||
| 'excited', | ||
| 'curious', | ||
| 'hopeful', | ||
| 'happy', | ||
| 'calm', | ||
| ]; | ||
| export const DEFAULT_MOOD: AgentMood = 'calm'; | ||
|
|
||
| function matchesWord(text: string, keyword: string): boolean { | ||
| let start = 0; | ||
| while (true) { | ||
| const at = text.indexOf(keyword, start); | ||
| if (at === -1) return false; | ||
| if (at === 0 || !/\p{L}/u.test(text[at - 1]!)) return true; | ||
| start = at + 1; | ||
| } | ||
| } | ||
|
|
||
| export function matchMood(label: string): AgentMood; | ||
| export function matchMood(label: string, fallback: AgentMood): AgentMood; | ||
| export function matchMood(label: string, fallback: null): AgentMood | null; | ||
| export function matchMood( | ||
| label: string, | ||
| fallback: AgentMood | null = DEFAULT_MOOD, | ||
| ): AgentMood | null { |
There was a problem hiding this comment.
🟡 New public helpers in the mood module ship without documentation comments
The newly added exported type, constants and function (AgentMood, MOOD_PRIORITY, DEFAULT_MOOD, matchMood at agents/src/tts/_mood.ts:6-50) carry no TSDoc, which the repository's contribution rules require for every new interface/method addition.
Impact: Generated API docs omit the mood normalization surface, and reviewers/consumers have no stated contract for the fallback behaviour.
Rule reference
CONTRIBUTING.md: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."
The same omission applies to SessionConfig and parseSessionConfig in examples/src/expressive_agent/protocol.ts:54-60.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
ExpressiveOptionsfrom the public voice APIlk.expressionPorts livekit/agents#6698 from exact aggregate commit
404a3b70.Validation
pnpm buildpassed for all 37 workspace packagespnpm lintpassed for all 37 workspace packages with existing warningspnpm format:checkpassedpnpm --filter @livekit/agents typecheckpassednode ./examples/dist/expressive_agent.js --helpCEREBRAS_API_KEY, unavailable/corrupt Silero assets, and a Hugging Face network timeoutexport * asdeclarations; the repository-wide command also reaches an existing plugin with noapi-extractor.jsoncue-cliis installed, but runtime validation could not dispatch because no LiveKit credentials or.envare available in this workspaceSource diff coverage
.github/workflows/deploy-examples.yml- not applicable: target has no Jukebox or per-example Python deployment workflow..github/workflows/evals.yml- not applicable: source only follows the Pythondrive-thrutodrive_thrurename; target has no counterpart.examples/README.md- not applicable: target has no examples catalog counterpart.examples/drive_thru/.dockerignore- not applicable: rename-only and target has no per-example Docker context.examples/drive_thru/Dockerfile- not applicable: rename-only and target has no per-example Dockerfile convention.examples/drive_thru/README.md- adapted toexamples/src/drive-thru/README.md, retaining target naming and TypeScript setup.examples/drive_thru/agent.py- adapted toexamples/src/drive-thru/drivethru_agent.ts; the target retains its idiomatic directory and provider APIs.examples/drive_thru/bg_noise.mp3- not applicable: rename-only asset not used by the target implementation.examples/drive_thru/database.py- adapted toexamples/src/drive-thru/database.ts.examples/drive_thru/order.py- adapted toexamples/src/drive-thru/order.ts.examples/drive_thru/pyproject.toml- not applicable: target examples shareexamples/package.json.examples/drive_thru/test_agent.py- adapted toexamples/src/drive-thru/test_agent.test.ts.examples/expressive_agent/.dockerignore- not applicable: target examples are not independent deployment units.examples/expressive_agent/Dockerfile- not applicable: Python and uv deployment infrastructure has no target counterpart.examples/expressive_agent/README.md- adapted toexamples/src/expressive_agent/README.mdwith JS commands, dispatch name, and pipeline terminology.examples/expressive_agent/agent.py- adapted toexamples/src/expressive_agent.ts, translating the friend pipeline, AssemblyAI STT, Gemma LLM, adaptive interruption, runtime voices, and attributes to JS APIs while retaining explicit cue-cli dispatch.examples/expressive_agent/prompt.md- adapted toexamples/src/expressive_agent/prompt.md, preserving persona, output, venting, and guardrail rules.examples/expressive_agent/protocol.py- adapted toexamples/src/expressive_agent/protocol.tswith equivalent metadata parsing, malformed fallback, four-voice table, expressive toggle, and participant attributes.examples/expressive_agent/pyproject.toml- adapted through the existing sharedexamples/package.json; no dependency change was required.examples/playground.yaml- not applicable: Python Jukebox deployment metadata and hosted agent IDs have no target infrastructure counterpart.livekit-agents/livekit/agents/__init__.py- adapted toagents/src/index.tsandagents/src/voice/index.ts; root wildcard export plus the added voice export exposeExpressiveOptions, while the other option types were already public.livekit-agents/livekit/agents/tts/_mood.py- adapted toagents/src/tts/_mood.tswith equivalent mood type, priority, word-start weighting, tie-breaking, and calm fallback.livekit-agents/livekit/agents/tts/_mood_data.py- adapted toagents/src/tts/_mood_data.tswith the complete weighted source table.livekit-agents/livekit/agents/tts/_provider_format.py- adapted toagents/src/tts/_provider_format.ts, preserving target document-order abstractions while porting mood payloads, Fish vocabulary and tones, provider guidance, aliases, xAI chunking, and bracket-prose preservation.livekit-agents/livekit/agents/voice/__init__.py- adapted toagents/src/voice/index.tsby exportingExpressiveOptions.livekit-agents/livekit/agents/voice/agent_activity.py- ported inagents/src/voice/agent_activity.ts; the target base already resolves and gates public expressive settings.livekit-agents/livekit/agents/voice/agent_session.py- ported inagents/src/voice/agent_session.ts; the target base already exposes expressive settings and defaults.livekit-agents/livekit/agents/voice/room_io/_output.py- adapted toagents/src/voice/room_io/_output.ts; normalizedlk.expressionis attached to the opening delta header because rtc-nodeclose()cannot carry attributes.pyproject.toml- not applicable: Python workspace membership only; target uses pnpm.tests/test_expressive_toggle.py- adapted toagents/src/voice/agent_session.test.tswith equivalent default, boolean, and object assertions; broader expressive tests already existed on the target base.tests/test_mood.py- adapted toagents/src/tts/mood.test.tswith equivalent table completeness, priority, weighted match, word-start, fallback, and payload tests.tests/test_tokenizer_xml_markup.py- adapted acrossagents/src/tokenize/xml_markup.test.ts,agents/src/tts/expr_markup.test.ts, andagents/src/tts/markup_utils.test.tsfor Fish expansion, tones, aliases, normalized payloads, provider prompts, and bracket prose.uv.lock- not applicable: Python lock and workspace registration only; no target dependency changed.Ported from livekit/agents#6698
Original PR description
No description.