Provider-agnostic AI layer with AssemblyAI LLM Gateway support

[richiemcilroy]

What

All four LLM features (AI titles/summaries/chapters, transcript translation, docs Ask AI, and the support messenger) now go through one provider-agnostic layer built on the Vercel AI SDK (apps/web/lib/ai/), replacing per-feature hand-rolled Groq/OpenAI/Anthropic clients. groq-sdk is removed.

Self-hosters can now pick any provider via env:

• AI_PROVIDER — assemblyai | openai | anthropic | groq | openai-compatible
• AI_MODEL / AI_CHAT_MODEL / AI_STREAM_MODEL — per-role model overrides
• AI_BASE_URL / AI_API_KEY — any OpenAI-compatible endpoint (Ollama, OpenRouter, LM Studio…)

assemblyai routes through the AssemblyAI LLM Gateway using the existing ASSEMBLY_API_KEY, so one key can cover transcription and AI features. It is never auto-detected — transcription-only instances don't start incurring LLM charges unless they opt in.

Behavior at merge: unchanged

With AI_PROVIDER unset (the default everywhere), providers are auto-detected per role to reproduce the current behavior exactly: generation prefers Groq → OpenAI → Anthropic (same gpt-oss-120b / gpt-4o-mini), chat features prefer Anthropic → OpenAI → Groq (same claude-sonnet-5). Later providers in each chain are automatic fallbacks — previously only summaries had any fallback at all.

Design notes

• generateText + the existing JSON extraction (not generateObject): the AssemblyAI gateway rejects response_format on gpt-oss models. The two map-reduce parse sites upgraded from cleanJsonResponse to the more tolerant extractJsonObject, and the gateway path opts into server-side json-repair.
• Capability flags per selection (supportsStreaming, supportsTemperature, token budgets) encode live-verified gateway rules: no streaming for non-OpenAI-family models, no temperature for Claude models, max_tokens must always be explicit.
• Messenger keeps its exact one-support-email-per-turn guarantee via three layers (stepCountIs(2), per-provider parallel-tool-call disable, an in-execute latch), and never retries another provider after an email was sent.
• Docs Ask AI keeps its byte-identical text/plain streaming contract; an empty completion (e.g. a thinking model consuming the whole token budget) now falls through to the next provider instead of streaming an empty answer.
• Model instances are always created inside workflow steps — never across "use step" boundaries.

Verification

• Typecheck clean; full unit suite green (only the pre-existing slack-manifest failure remains); new unit suites for provider resolution and the fallback chain.
• New env-gated live e2e (__tests__/e2e/ai-provider-live-e2e.test.ts, same pattern as live-transcribe-local-e2e): real providers, real local MySQL/MinIO/media-server, no AI mocks — fresh recording → real AssemblyAI transcription → real title/summary; short + 24k-char map-reduce generation; docs-ask incremental streaming; live messenger reply. 6/6 under auto-detect and 6/6 under AI_PROVIDER=assemblyai, including a live rate-limit event absorbed by the fallback chain.

Docs

self-hosting.mdx and docker-compose.coolify.env.example document all providers, the per-role auto-detect order, and the EU-gateway caveat.

Greptile Summary

The PR centralizes AI generation, translation, docs Ask AI, and messenger behavior behind a provider-agnostic Vercel AI SDK layer with AssemblyAI gateway and OpenAI-compatible endpoint support.

• Adds role-aware provider selection, capability flags, model overrides, and fallback chains.
• Moves output validation into provider attempts so unusable completions can fall through.
• Documents the new self-hosting configuration and adds focused unit and live E2E coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/web/lib/ai/provider.ts	Adds provider discovery, role-specific model resolution, capability filtering, and lazy model construction; the prior role-awareness issue is fixed.
apps/web/workflows/generate-ai.ts	Migrates title, summary, chapter, and map-reduce generation to the shared provider fallback layer with in-loop output validation.
apps/web/app/api/docs/ask/route.ts	Migrates Ask AI to role-aware provider streaming while retaining its plain-text response contract.
apps/web/lib/messenger/agent.ts	Moves messenger generation to the shared provider chain while retaining tool-call safeguards.
packages/env/server.ts	Adds validated environment settings for provider selection, model overrides, credentials, and compatible base URLs.
apps/web/tests/e2e/ai-provider-live-e2e.test.ts	Adds an environment-gated live integration suite spanning transcription, generation, streaming, and messenger behavior.

_{Reviews (4): Last reviewed commit: "fix(web): sanitize map-reduce keyPoints ..." | Re-trigger Greptile}

Context used (3)

• Context used - AGENTS.md (source)
• Knowledge Base — Web App (apps/web)
• Knowledge Base — Infra, Storage and Config

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Obfuscated code: npm json-schema is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: pnpm-lock.yaml → npm/json-schema@0.4.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/json-schema@0.4.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

[richiemcilroy]

hey @greptileai, please re-review the PR

[richiemcilroy]

hey @greptileai, please re-review the PR

[richiemcilroy]

hey @greptileai, please re-review the PR