Beyond Semantic Similarity: Introducing NVIDIA NeMo Retriever’s Generalizable Agentic Retrieval Pipeline
In the rapidly evolving landscape of AI retrieval, many solutions are highly specialized, engineered to perform exceptionally well on specific, narrow tasks. However, real-world enterprise applications rarely have the luxury of perfectly curated, single-domain data. They require systems that can seamlessly adapt to a wide variety of challenges—from parsing complex visual layouts to executing deep logical reasoning.
The Motivation: Why Semantic Similarity Isn’t Enough
For years, dense retrieval based on semantic similarity has been the standard for finding information. However, as the applications of retrieval expand, finding relevant documents goes beyond semantic similarity alone. Complex document search requires reasoning skills, an understanding of real-world systems, and iterative exploration.
There is a fundamental gap: LLMs are great at thinking and reasoning but cannot process millions of documents at once. Conversely, retrievers can easily sift through millions of documents but possess limited reasoning skills. Agentic retrieval bridges this gap by creating an active, iterative loop between the LLM and the retriever.
How It Works: The Agentic Loop
Our agentic retrieval pipeline relies on a ReACT architecture. Instead of a single ‘one-and-done’ query, the agent iteratively searches, evaluates, and refines its approach.
The agent utilizes built-in tools like think to plan its approach and final_results to output the exact documents needed for a given query, alongside a retrieve (query, top_k) tool to explore the corpus. Through this loop, we observed successful search patterns emerge naturally:
- Generating better queries: The agent dynamically adjusts its search queries based on newly discovered information.
- Persistent rephrasing: It continually rephrases queries until useful information is found.
- Breaking down complexity: It translates complex, multi-part queries into multiple simpler queries with clear goals.
Finally, to synthesize the iterative discoveries, the agent calls a final_results tool to output the most relevant documents, ranked by their relevance to a given query. As a safety net—for example, when the agent hits the maximum number of steps or the context length limit—the pipeline falls back to Reciprocal Rank Fusion (RRF), which scores documents based on their ranks across all retrieval attempts in the agent trajectory.
Engineering for Speed and Scale
Agentic workflows are notoriously slow and resource-intensive. To make this pipeline viable for leaderboard-scale evaluation, we had to rethink how the LLM agent and the retriever communicate.
Initially, the retriever was exposed to the agent via a Model Context Protocol (MCP) server—a natural choice, since MCP is designed precisely for giving LLMs access to external tools. But in practice, this architecture imposed a compounding tax on experiment velocity. Every run required spinning up a separate MCP server, loading the right dataset corpus into GPU memory, and orchestrating the lifecycle of both client and server. Each of these steps was an opportunity for silent misconfiguration or a server freeze under high-volume requests. The network round-trips added latency to every retrieval call, and the overall cognitive burden of managing the two-process setup made it significantly harder for other teams to adopt and iterate on the pipeline.
To resolve this, we replaced the MCP server with a thread-safe singleton retriever that lives in-process. The singleton loads the model and corpus embeddings once, protects all access with a reentrant lock, and exposes the same retrieve() interface to arbitrarily many concurrent agent tasks—achieving the key benefit of an MCP server (safe, shared access to a GPU-resident retriever from multiple threads) without incurring network serialization overhead or requiring a separate server process. This single architectural change eliminated an entire class of deployment errors and dramatically improved both GPU utilization and experiment throughput.
Generalization vs. Specialization Across Benchmarks
A common observation in modern retrieval evaluation is that solutions highly optimized for one specific type of task often experience a performance gap when applied to a completely different domain.
We placed #2 on the reasoning-intensive BRIGHT leaderboard with an NDCG@10 of 50.90. The #1 solution on that leaderboard, INF-X-Retriever, achieves an impressive 63.40. However, to test cross-domain adaptability, we benchmarked the INF-X pipeline (coupled with the same nemotron-colembed-vl-8b-v2 embedding model used in our agentic pipeline) on ViDoRe v3, a dataset focusing on visually rich and diverse enterprise documents. On this different task, its performance landed at an NDCG@10 of 62.31, lower than the dense retrieval score 64.36. In other words, INF-Query-Aligner does not improve over the dense retrieval baseline on ViDoRe v3.
In contrast, our same agentic pipeline (pairing Opus 4.5 with nemotron-colembed-vl-8b-v2) achieved the #1 spot on ViDoRe v3 with a score of 69.22.
This highlights a core strength of our approach: generalizability. Rather than relying on dataset-specific heuristics or a query-rewriter/aligner, our agentic loop naturally adapts its strategy to the dataset at hand, whether it requires multi-step logical reasoning or parsing complex visual layouts.
Ablation Studies: Open vs. Closed Models
We conducted ablation studies to evaluate the impact of using open vs. closed models in our agentic pipeline.
ViDoRe v3
Agent Embedding Model NDCG @10 Average sec/query Total input tokens (M) Total output tokens (M) Average retrieval calls
Opus 4.5 nemotron-colembed-vl-8b-v2 69.22 136.3 1837 15 9.2
gpt-oss-120b nem 66.51 123.4 1574 12 8.5
gpt-oss-120b nemotron-colembed-vl-8b-v2 65.23 118.1 1521 11 8.1
Our results show that using a closed model (Opus 4.5) with a pre-trained embedding model (nemotron-colembed-vl-8b-v2) achieves the best performance on ViDoRe v3, with an NDCG@10 of 69.22.
