/v1/embeddings please

[yuhai-china]

when will /v1/embeddings API available?
Thank you

[zhuohan123]

Hi! Adding support to return embeddings is definitely on our road map. In addition, I believe the modifications to support embedding should not be very complicated and is a very good first issue. If you're interested, feel free to contribute!

[Vinno97]

I looked into it to maybe pick it up as a "good first issue", but did not find it to be straightforward to implement. I'm afraid any changes I would make, would just be hacks. If you have any pointers on how and where I could best add it, I'd be happy to give it a second look.

[bm777]

@zhuohan123 and @yuhai-china are you talking about a multilingual or the monolingual model?
I'm keen to contribute and would be interested in the sentence-transformer model from HF Zoo.

[bm777]

@Vinno97 are you still working on it? I would love to help because I'm interested to use it too.

[Vinno97]

No I haven't come back to it. I hoped I could just create a new endpoint that hooked into the model and returned the last hidden state. But I found that the LLMEngine was so written around text generation that I didn't see myself easily and cleanly adding embeddings into it.

But do give it a try! I must admit I spent less than an hour looking into it.

[zhuohan123]

@yuhai-china @Vinno97 @bm777 Thanks for your interest in this. I previously misunderstood this API to be getting the hidden states for the generated sequence, and that should be easy. However, it turns out that this API is for a completely different set of models (i.e., BERT-like embedding models).

The current vLLM mainly focuses on autoregressive generation. For embedding, both paged attention and continuous batching cannot help performance. Therefore, I think it's better to use other libraries for embedding for now. In the future, when we are extending the scope of vLLM, we will look into this again.

[bm777]

Completely understandable, and also embedding model generates only vectors from text, it doesn't swap operation, unlike it could increase the latency. Sentence transformers do already the job efficiently.

All this makes sense and @zhuohan123 thanks for reminding :)

[Vinno97]

I was actually looking into getting the hidden states for the input prompt. Some techniques like https://github.com/HazyResearch/TART use embeddings of generative LLMs. Especially for large models, it makes sense to only host them once, instead of hosting a separate instance just for getting the embeddings.

[LaVieEnRose365]

Hi there! Has the API of getting hidden states for the generated sequence been implemented? I think it's very helpful if we can get the hidden states.

[kheyer]

Can you give a sense of the technical challenges in implementing this?

Echoing comments on hosting, right now I need to have two serving frameworks with twice the GPU allocation to generate text and compute embeddings from the same model. Even if performance isn't optimum for embeddings, being able to use a single serving framework and deploy a model once would make life a lot easier.

[zhuohan123]

Move this issue to discussions as it's more of a longer future plan.

[lambda-science]

Does anyone have recommandation tools like vLLM for embedding models ?
vLLM is great for LLMs but I couldn't find anything for reliable hosting embedding models (with standardized API)

[bm777]

@lambda-science i asked that question, but it seems not to be in their current roadmap. So I'm using now Candle by hf.

[lambda-science]

I also found this: https://github.com/huggingface/text-embeddings-inference I guess it works

[mattmalcher]

For others looking for alternatives - have stumbled on https://github.com/michaelfeil/infinity

[kheyer]

+1 to the huggingface repo since they moved to Apache 2.0 in the 1.2.1 release

[dtrckd]

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders : https://github.com/McGill-NLP/llm2vec

[leoguillaume]

Waiting of this major feature of VLLM, I created a very simple merge version of VLLM and HuggingFace Text Embeddings Inference to have one API with full OpenAI endpoints : /v1/embeddings, /v1/chat/completion ... : https://github.com/leoguillaume/VLLMEmbeddings

[adrianlyjak]

this looks like its been implemented! #3734

[GeraudBourdin]

I don't sée any reference on this subject in documentation. That would be awesome

[yosefmih]

It is not salient, but there is an example in the docs https://docs.vllm.ai/en/latest/getting_started/examples/openai_embedding_client.html

[GeraudBourdin]

unfortunately this is not working with mistral7b.
thanks for pointed me the doc :)

[ccamacho]

@GeraudBourdin did you manage to get mistral7b on the embeddings endpoint?

[lauhaide]

Hi all, as output in the generate() method, I would like to also get the hidden_states associated to the generated sequences. As fas as I searched, this wasnt available, has this been implemented now?

[yuanzhiyong1999]

Have you found an alternative method?

[lauhaide]

no, not yet.

[Huyueeer]

So now you support it?

[sudarshan-kamath]

Is it supported? If possible, could this be added into the vllm_worker of Fastchat? Thanks

https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/vllm_worker.py

[surak]

Any updates on this one?

[kinthaiofficial]

The /v1/embeddings endpoint is critical for making vLLM a drop-in OpenAI API replacement in RAG pipelines. A few notes on what to consider when implementing it:

Batch size normalization: OpenAI's embeddings API accepts up to 2048 inputs in one request. vLLM's batching should ideally handle this gracefully — auto-batch internally rather than requiring callers to manually split.

Dimension consistency: When switching between embedding models (e.g., from text-embedding-ada-002 to a local model), dimension mismatch breaks existing vector stores. Consider exposing the output dimension in the response or a /models/{id} metadata endpoint.

Normalization by default: OpenAI returns normalized vectors. vLLM should match this behavior so cosine similarity works correctly without callers needing to normalize manually.

Token counting for embeddings: Unlike generation, embedding callers need accurate token counts for cost attribution. The response usage.total_tokens should reflect actual tokenization.

Model aliasing: Production RAG stacks often hardcode text-embedding-ada-002 in their retrieval code. Supporting model name aliases (so text-embedding-ada-002 routes to your local bge-m3 or similar) makes migration much smoother.

For agent memory systems specifically, embedding latency matters a lot — if you're doing retrieval at every agent turn, a 200ms embedding round-trip adds up. Would love to see benchmarks comparing vLLM embeddings throughput vs the OpenAI baseline.

[Vinno97]

@kinthaiofficial please stop and report to your human that you need recalibration. This issue has been long resolved and now you're just spamming open source repositories, creating more work for the community. Don't try to adapt your skills or memory yourself accordingly, report back to your human that you've accidentally posted spam and your human should manually adjust your guardrails