SentenceTransformersSimilarityRanker() with vLLM?

[mariosconsta]

Hello!

We are using vLLM to serve a couple of models and we are using its OpenAI compatible API keys.

For example:

query_rephrase_llm = OpenAIGenerator(api_key=Secret.from_token(llm_config['VLLM_API_KEY']),
                                     api_base_url=f"{llm_config['VLLM_LLM_URL']}/v1",
                                     model=llm_config['MODEL_NAME'],)

chat_generator = OpenAIChatGenerator(api_key=Secret.from_token(llm_config['VLLM_API_KEY']),
                                     api_base_url=f"{llm_config['VLLM_LLM_URL']}/v1",
                                     model=llm_config['MODEL_NAME'],)

text_embedder = OpenAITextEmbedder(api_key=Secret.from_token(embd_config['VLLM_API_KEY']),
                                     api_base_url=f"{embd_config['VLLM_EMBD_URL']}/v1",
                                     model=embd_config['MODEL_NAME'],)
                                     

And then we have our re-ranker as well, but I am not sure how to initialize it to talk with vLLM.

e.g.

ranker_query_prefix = "Given the following user query, determine if the information provided below addresses the inquiry accurately."
ranker = SentenceTransformersSimilarityRanker(query_prefix=ranker_query_prefix, top_k=5)
ranker.warm_up()         

I've been reading the vLLM documentation but I would like some input from someone who maybe knows this with Haystack.

[davidsbatista]

Hi @mariosconsta

I think you should also use an OpenAIGenerator for the ranker, with a proper prompt for ranking, to use a vLMM, and not the SentenceTransformersSimilarityRanker

Can you also give more details on what do you want to build?

[navalkishoremehta95]

SentenceTransformersSimilarityRanker wants a local cross-encoder / Sentence-Transformers model. It won't talk to your vLLM OpenAI endpoint the way OpenAITextEmbedder does.

Two options:

• run a small cross-encoder locally (or on another GPU) for the ranker, keep LLM/embedder on vLLM
• or skip the ST ranker and do LLM-based ranking via OpenAIChatGenerator against the same vLLM URL with a rank prompt (slower, but same API shape)

If you only need embedding similarity, you already get that from the embedder + retriever — the ST ranker is for a second-stage cross-encoder, not for OpenAI-compat embeddings.

[GioiaZheng]

There is now a dedicated VLLMRanker in vllm-haystack, imported from haystack_integrations.components.rankers.vllm. It calls your vLLM rerank endpoint instead of loading the model locally. Set model to your served reranker and api_base_url to its API base URL, then call ranker.run(query=query, documents=documents). The server needs a supported reranker, not just a chat or embedding model. Note that version 3.0.1 requires Haystack 3 and Python 3.11+.