#

continuous-batching

Here are 69 public repositories matching this topic...

waybarrios / vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

Updated Aug 13, 2026
Python

cubist38 / mlx-openai-server

A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

flux queue speech-recognition image-generation whisper vision-api mlx fastapi multi-models apple-silicon continuous-batching tool-calling structured-outputs mlx-lm mlx-vlm openai-compatible

Updated Aug 10, 2026
Python

lumia431 / photon_infer

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

modern-cpp inference-engine ai-infra vllm llm-inference paged-attention continuous-batching

Updated Jan 2, 2026
C++

gty111 / gLLM

An Efficient and Versatile Inference Engine for Distributed LLM Serving

pipeline-parallelism tensor-parallelism llm-serving llm-inference pagedattention continuous-batching qwen3 token-throttling chunked-prefill

Updated Aug 10, 2026
Python

200lz / llm-inference-optimization-lab

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

kv-cache llm-serving llama-cpp llm-inference continuous-batching prefix-caching

Updated Jul 19, 2026
C++

CarolBaggins2023 / baby-vllm

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

llm-serving llm-inference continuous-batching chunked-prefill paged-kv-cache

Updated Jun 20, 2026
Python

iFurySt / nanoLLMServe

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

Updated May 28, 2026
Python

caimari / vtts

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

text-to-speech pytorch tts speech-synthesis voice-synthesis voice-cloning voice-agent gpu-inference vllm continuous-batching real-time-tts qwen3-tts

Updated Mar 15, 2026
Python

pjdurden / nanoserve

An AI inference engine from scratch. Like nanoGPT, but for serving.

machine-learning cuda inference pytorch triton llama from-scratch kv-cache llm vllm llm-inference paged-attention continuous-batching

Updated Aug 11, 2026
Python

JohnScheuer / continuous-batching-profiler

Real PyTorch inference server comparing FCFS, EagerContBatch, and ChunkedPrefill on RTX 2070. Key finding: ChunkedPrefill is not a throughput optimizer but a fairness mechanism — it reduces worst-case decode stalls from 58.6ms to 34.1ms while EagerContBatch maximizes mean TTFT and throughput.

python gpu latency scheduling inference pytorch throughput serving llm continuous-batching chunked-prefill rtx-2070

Updated Jul 12, 2026
Python

Mihawii / mini-vllm

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.

inference pytorch fastapi kv-cache llm continuous-batching

Updated Jul 4, 2026
Python

likhith-v1 / inferd

Local-first LLM stack on a single RTX 5090: QLoRA fine-tuning, exact speculative decoding, paged KV-cache, and continuous batching — served via FastAPI with a live React dashboard.

react cuda pytorch triton lora inference-engine fine-tuning fastapi kv-cache llm local-llm llm-inference qlora qwen speculative-decoding paged-attention continuous-batching rtx-5090

Updated Aug 6, 2026
Python

nandanadileep / vllm-like-llm-serving

vLLM-inspired LLM inference server for Apple Silicon using MLX, with continuous batching, prefix caching, chunked prefill, streaming, and an OpenAI-compatible API.

benchmarking llama mlx inference-engine fastapi kv-cache apple-silicon openai-api llm-serving vllm local-llm paged-attention continuous-batching mlx-lm

Updated Aug 11, 2026
Python

JohnScheuer / llm-serving-sim

End-to-end LLM serving simulator integrating scheduling, prefix caching, tensor allocation, and KV-cache management. 168-run sweep (72 baseline + 96 pressure). Key finding: ChunkedPrefill + LFU cache achieves 41% lower TTFT p95 and 94% prefix hit rate, but hits OOM first under memory pressure.

simulation gpu scheduler inference memory-management systems-programming serving cpp20 kv-cache llm continuous-batching prefix-cache

Updated Jul 10, 2026
C++

stmay4 / llama.csharp

.NET wrapper library for llama.cpp that provides batch processing (Continuous batching) and context sequence cache management

dotnet llama sequence gpt parallel-processing embeding wrapper-lib kv-cache rwkv llamacpp llama-cpp continuous-batching

Updated Aug 11, 2026
C#

achi9629 / llm-inference-engine

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

nlp deep-learning transformers autoregressive inference-engine model-serving fastapi gpt2 kv-cache llm llm-serving llm-inference paged-attention mistral-7b continuous-batching paged-kv-cache

Updated May 8, 2026
Python

Ray-RP / mini-sglang-ascend

Mini-SGLang port for Ascend NPUs with FIA, paged KV cache, ragged continuous batching, and request lifecycle safety.

inference-engine ascend kv-cache llm llm-serving llm-inference ascend-npu qwen paged-attention sglang continuous-batching torch-npu mini-sglang

Updated Jul 12, 2026
Python

swaynehales / vllm-mlx

Fork of OpenAI and Anthropic compatible server for Apple Silicon. Native MLX backend, 500+ tok/s. Run LLMs and vision-language models with continuous batching, MCP tool calling, and multimodal support.

inference-server mlx multimodal apple-silicon llm vllm local-ai continuous-batching tool-calling openai-compatible

Updated May 20, 2026
Python

Dev228-afk / LLM-Sim-Bench

High-performance discrete-event simulator (C++20/Python) for modeling agentic LLM traffic, KV cache dynamics, Prefill-Decode Disaggregation (PDD), and scheduling policies. Features roofline model analysis, K-Means request clustering, and a real-time web dashboard.

machine-learning data-visualization pybind11 scheduling-algorithms queuing-theory discrete-event-simulation cpp20 performance-modeling roofline-model kv-cache llm-inference ai-infrastructure continuous-batching datacenter-traffic

Updated Jun 29, 2026
C++

JohnScheuer / adaptive-batching-policy-bench

Event-driven benchmark of adaptive batch composition policies for LLM serving, measuring how prefill and decode interference affects TTFT, TPOT, and throughput under different memory pressure regimes.

python performance-engineering benchmark simulation latency scheduler inference batching systems decode gpu-memory throughput serving prefill event-driven-simulation kv-cache llm continuous-batching

Updated Jul 24, 2026
Python

Improve this page

Add a description, image, and links to the continuous-batching topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the continuous-batching topic, visit your repo's landing page and select "manage topics."