Introduction: Beyond Traditional Retrieval Metrics
For years, retrieval in information systems revolved around simple questions: Did we find relevant content? Did recall improve? Did the ranking model perform better? While these metrics remain useful, they’re incomplete for modern agentic systems. The Bits over Random (BoR) framework, introduced in the ICLR 2026 blogpost, reveals a critical blind spot: retrieval success isn’t just about finding relevant material—it’s about avoiding irrelevant noise that harms downstream performance.
What is Bits over Random (BoR)?
BoR measures retrieval selectivity relative to random chance. Imagine two librarians: one delivers a concise, relevant bundle of books, while the other floods the room with 100 random volumes. The first librarian’s work is selective; the second’s is context pollution. BoR quantifies this selectivity. When BoR scores drop, it signals that retrieval is becoming less discriminative—often correlating with worse model performance due to attention dilution and prompt overload.
Why BoR Matters for Agent Systems
Retrieval dashboards often mislead agent teams. High Success@K or recall metrics can mask a critical issue: the model is forced to process irrelevant or overlapping content. Unlike humans, LLMs don’t skim or filter—they process every retrieved item. This makes retrieval a “reasoning-budget allocation problem.” The key question becomes: How much noise did we force the model to process to achieve relevance?
Context Engineering: A New Discipline
Building agentic systems requires context engineering—a discipline focused on:
- What enters the prompt: Prioritize high-signal, low-noise content.
- When it enters: Stage retrieval to narrow options incrementally.
- What’s excluded: Actively remove irrelevant tools or chunks.
The context window is contested cognitive real estate. Every irrelevant passage competes with the model’s working memory. BoR formalizes this intuition, urging teams to evaluate retrieval not just by success, but by how much better it is than random chance.
Tool Overload and Agent Performance
Tool selection in agents often leads to overload. A model choosing among 50 tools with vague descriptions is akin to a student given 50 textbooks without guidance. The solution lies in:
- Staged retrieval: Narrow options in steps, not all at once.
- Domain routing: First identify the task’s domain (e.g., finance, coding), then select tools.
- Compressed summaries: Replace verbose tool specs with concise, high-signal descriptions.
- Explicit exclusion: Remove tools irrelevant to the current task.
BoR helps identify when tool overload turns into noise. A low BoR score here signals that the model is no longer making meaningful choices.
Practical Strategies for Better Retrieval
Optimize retrieval with these actionable steps:
- Monitor BoR alongside traditional metrics to detect context pollution.
- Use domain routing to reduce tool overload before final selection.
- Compress tool descriptions to highlight unique use cases.
- Stage retrieval to prioritize relevance over volume.
These strategies align with BoR’s core insight: retrieval success depends on both finding relevant content and minimizing irrelevant noise.
Conclusion: Rethinking Retrieval for LLM Agents
The Bits over Random framework challenges us to rethink retrieval as a balance between relevance and selectivity. Traditional metrics like recall are necessary but insufficient for agentic systems. By adopting BoR, teams can build cleaner, more efficient retrieval pipelines that align with how LLMs actually process information. Start measuring selectivity, not just success—and watch your agent’s performance improve.
