Production Ready RAG | RAISE Summit 2024 | Paris

David Leconte, Solution Engineer at DataStax, addresses the critical gap between experimental RAG implementations in labs and reliable deployment in production environments.

DataStax leverages over a decade of enterprise experience with high-latency, high-availability applications, citing users like Apple, Netflix, Spotify, and Starbucks, to pivot into the Generative AI space.

Three primary challenges hinder RAG production adoption:

• Production Stability: Inability to quickly identify, prevent, or diagnose failures when deployed applications break.
• Regulatory & Support Gaps: Regulatory sectors (e.g., banking) struggle with open-source frameworks lacking formal support, validation, and compliance seals required for production.
• Library Complexity: RAG systems rely on hundreds of asynchronous open-source libraries that are not designed to work together, leading to API incompatibilities and unsupported integration risks.

DataStax introduces "RackStack," a curated, supported layer of fundamental RAG components designed to de-risk production:

• Bundles popular frameworks like LangChain and LlamaIndex into verified, compatible packages.
• Includes automated daily testing and security scans (e.g., for SQL injection) to ensure library stability.
• Provides a single pip install command to deploy the entire stack, eliminating the need for developers to hunt for compatible library versions.
• Offers 24/7 follow-the-sun support and compliance with security standards, contrasting with unsupported open-source reliance on community forums.
• Harrison Chase (creator of LangChain) reportedly uses RackStack for his own production testing.
• Prevents the need for risky or resource-intensive alternatives like forking open-source frameworks or rebuilding solutions from scratch.

The underlying database architecture, AstraDB, addresses performance bottlenecks in vector storage:

• Built on 14 years of Apache Cassandra expertise, offering a serverless, multi-cloud, and multi-region engine.
• Achieves 10-millisecond latency for LLM queries, ensuring ingestion, embedding generation, and retrieval do not create system bottlenecks.
• Claims a 74% performance advantage over dedicated vector databases in retrieval speed based on internal studies.
• Maintains superior accuracy (F1 score) and relevance compared to competitors like Pinecone, while offering a lower Total Cost of Ownership (TCO).
• Integrates with NVIDIA NEMO microservices to accelerate embedding mechanisms at reduced costs.

DataStax recently acquired Longflow to streamline the RAG development lifecycle:

• Functions as both a user interface for designing RAG systems and a workflow engine.
• Automates code generation to ensure "pure, well-written" code without manual coding.
• Enables deployment through API calls with a single click.
• Shifts focus from debugging code correctness to managing system logic and deployment.

The proposed three-layer solution (RackStack libraries + AstraDB + Longflow) allows organizations to:

• Scale with high throughput and low latency.
• Maintain real-time updates and ingestion capabilities.
• Operate within regulated markets by ensuring support, compliance, and security.
• Avoid the "race" against competitors by reducing the complexity of following rapid innovation cycles in open-source libraries.