How to configure vllm to gracefully mark the too-long inputs without throwing?

[vadimkantorov]

I'm using vllm to batch-process quite a lot of inputs. Currently LLM(...).generate(...) throws when it encounters some text-input which causes too-many-input-tokens after tokenization.

Is it possible to configure vllm to gracefully return this error in model_outputs and still process as usual all other inputs?

Alternatively, it might be good to allow specifying some "mitigation strategies": keep only first self.model_config.max_model_len tokens or keep only last self.model_config.max_model_len tokens.

I'm providing text input to .generate(...), so to be able to filter out bad requests I would need to invoke the tokenization logic prior to sending the input to the model - this is quite cumbersome and error-prone

Thanks!

File "...", line ..., in <module>                                                        
model_outputs = model.generate(                                                                                                    
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/utils.py", line 1131, in inner                                         
return fn(*args, **kwargs)                                                                                                         
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 457, in generate                             
self._validate_and_add_requests(                                                                                                   
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1317, in _validate_and_add_requests          
self._add_request(
  
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1335, in _add_request                        
self.llm_engine.add_request(                                                                                                       
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 186, in add_request
    request = self.processor.process_inputs(request_id, prompt, params,
  
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 209, in process_inputs
    self._validate_model_inputs(processed_inputs, lora_request)                                                                        
File "/home/inferencer/.local/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 308, in _validate_model_inputs
    raise ValueError(                                                                                                                
ValueError: Prompt length of 4576 is longer than the maximum model length of 4096.

[lxcxjxhx]

Currently, vLLM raises an exception when input exceeds max_model_len. There are a few ways to handle this:

Option 1: Pre-filter inputs (recommended)
Tokenize and check length before calling generate():

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name)
valid_prompts = []
for prompt in prompts:
    tokens = tokenizer.encode(prompt)
    if len(tokens) <= max_model_len:
        valid_prompts.append(prompt)
    else:
        # Handle gracefully: truncate, skip, or log
        print(f"Skipping prompt with {len(tokens)} tokens")

outputs = llm.generate(valid_prompts, sampling_params)

Option 2: Truncate inputs
If you want to keep all inputs, truncate them:

truncated_prompts = []
for prompt in prompts:
    tokens = tokenizer.encode(prompt)
    if len(tokens) > max_model_len:
        tokens = tokens[:max_model_len]  # Keep first N tokens
        prompt = tokenizer.decode(tokens, skip_special_tokens=True)
    truncated_prompts.append(prompt)

outputs = llm.generate(truncated_prompts, sampling_params)

Option 3: Increase max_model_len
If your GPU memory allows, increase the limit:

llm = LLM(model=model_name, max_model_len=8192)

Regarding your suggestion:
Your idea of returning errors in model_outputs instead of throwing is reasonable. You could open a feature request for a skip_invalid_requests=True parameter. The current behavior is intentional to fail fast, but batch processing use cases like yours would benefit from graceful degradation.

For now, Option 1 or 2 is the most practical approach.