perf(pd): coalesce decode tokens into fewer ws frames to relieve pd_master - #1464
Closed
sufubao wants to merge 1 commit into
Closed
perf(pd): coalesce decode tokens into fewer ws frames to relieve pd_master#1464sufubao wants to merge 1 commit into
sufubao wants to merge 1 commit into
Conversation
…aster wsproto per-frame parsing is ~60% of the pd_master single-core CPU under high D-node fan-out (py-spy, 2999 samples at 100% core). The cost is per-frame, not per-byte: when a decode node emits ~1 token per websocket frame under light load, frame rate maxes out and the master core is spent parsing ws frame headers (parse_header / extension_processing / Frame allocation) rather than doing useful work. Add an opt-in coalesce window in _up_tokens_to_pd_master: after the first drain, wait LIGHTLLM_PD_WS_BATCH_WAIT seconds, then non-blocking drain again via AsyncQueue.get_all_data() and merge into a single ws frame. Frame rate drops by ~K, wsproto cost drops by ~K, token throughput rises by ~K at unchanged master CPU. Default is 0.0 (off) -> no behavior or latency change unless enabled. Measured against a live pd_master saturated by 8 decode nodes: 1 token/frame -> ~64k tokens/s absorbed (master 100% CPU) 16 tokens/frame -> ~1M tokens/s absorbed (same frame rate, same CPU) Trade-off: up to LIGHTLLM_PD_WS_BATCH_WAIT added ITL per stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
PD 分离部署中,D(decode)节点增多后,pd_master 单核接收 token 成为瓶颈。
对满载状态(8 个 D 节点持续打流,master 单核 100%)的 pd_master 用 py-spy 采样 30s(2999 个样本),CPU 分布:
瓶颈不是 pickle,而是 wsproto 的每帧解析成本。这笔开销按帧收取、与字节数无关:轻载时 D 每帧只发 1 个 token,帧率被拉满,master 单核算力几乎全花在解析 WS 帧头上。
方案
在 D 侧
_up_tokens_to_pd_master发送前增加一个可配置的攒帧窗口:首次 drain 后等待LIGHTLLM_PD_WS_BATCH_WAIT秒,再用AsyncQueue.get_all_data()非阻塞捞一次,把这段时间内并发生成的 token 合并进同一帧发出。帧率下降约 K 倍,wsproto 成本下降约 K 倍。0.0(关闭):不改变任何行为,也不引入额外延迟。LIGHTLLM_PD_WS_BATCH_WAIT在 D 节点上开启,建议 1ms 起步。收益
同样的 8 个 D 节点把 pd_master 单核打到 100%:
master 的瓶颈从"帧率"基本消失,token 吞吐随每帧 token 数 K 近似线性提升。
代价
每条 token 流的 ITL 上限增加
LIGHTLLM_PD_WS_BATCH_WAIT。1ms 量级通常可忽略,按实际 ITL / 吞吐权衡调参,置0即关闭。改动范围
单文件
lightllm/server/httpserver/pd_loop.py,仅_up_tokens_to_pd_master一处,默认行为不变。