Reliable Retrieval for Production AI Systems
At QCon London 2026, Lan Chu, AI Tech Lead at Rabobank, shared critical insights into building production-ready AI search systems. Her work highlights that most failures in Retrieval-Augmented Generation (RAG) systems stem from indexing and retrieval, not the language model itself. This article breaks down her strategies for overcoming these challenges and ensuring reliable performance.
Why Document Parsing Matters
Enterprise documents often contain complex layouts like tables, infographics, and nested data. Traditional text extraction methods can strip away critical structure, leading to misread numbers or misinterpreted visuals. Chu’s solution? A hybrid pipeline combining text extraction with visual-language models to preserve document context. This approach ensures retrieval systems can accurately interpret even the most intricate layouts.
Key Takeaways for Document Parsing
- Use visual-language models to retain layout structure
- Test parsing accuracy with real-world datasets
- Validate outputs against known document patterns
Optimizing Chunking Strategies
Chunking—breaking documents into manageable pieces—is essential for cost efficiency and model performance. Chu tested multiple strategies and found that section-based chunking worked best for her dataset. However, she emphasized that the optimal approach depends on your specific data. For example, legal documents might require paragraph-level chunks, while technical reports may benefit from section-based splits.
Proven Chunking Tips
- Align chunk size with model context limits
- Test different chunking methods on real data
- Balance accuracy with cost constraints
Enhancing Retrieval Beyond Vector Similarity
Standard vector similarity can miss critical context like document timing. Chu’s system adds temporal scoring to prioritize newer documents and includes a routing layer to decide between retrieval and API calls. When models struggle with tool parameters, the system asks users to confirm inputs—ensuring accuracy without overcomplicating the process.
Advanced Retrieval Techniques
- Temporal scoring for time-sensitive documents
- Routing layers for API vs. retrieval decisions
- User confirmation for ambiguous inputs
Building Evaluation Frameworks
Evaluation is often overlooked but critical for long-term success. Chu recommends creating datasets from real user queries and tracking failure modes like routing errors or temporal mismatches. Statistical methods help verify improvements, and real-world queries often provide more value than synthetic datasets.
Effective Evaluation Practices
- Build datasets from actual user interactions
- Track specific failure patterns
- Use statistical validation for iterative improvements
Key Lessons for Production AI Systems
Chu’s work underscores that reliable AI search requires attention to parsing, chunking, retrieval, and evaluation. By combining visual-language models with temporal scoring and robust evaluation, teams can build systems that scale effectively. Agentic architectures offer additional power but require careful balancing of complexity and performance.
Takeaway: Start with real-world data, test strategies iteratively, and prioritize structured evaluation. These steps ensure your AI systems deliver consistent, accurate results in production environments.
