Interview
Giving New Life to Unstructured Data with LLMs and Agents
- Core Hypothesis: Robotic Process Automation (RPA), which records human desktop clicks to automate tasks, is predicted to be "fully eaten" by AI-driven automation capable of handling unstructured data.
- Unstructured Data Definition: Defined as any data that cannot be stored in relational database tables or queried via SQL, including PDFs, images, and documents lacking a fixed schema.
- Historical Context: Early research at MIT (circa 2015) focused on the "Data Hub" project to answer questions across heterogeneous, unstructured data sources (file systems, databases, applications).
- Early Technique Failure: Previous methods for extracting data from unstructured documents were deemed "brutal" and unreliable, specifically:
- Template-based: Hard-coded coordinate logic (e.g., "10 pixels from the right") that broke with minor formatting changes.
- Rule-based: Keyword searches that failed on context variations.
- Feature-engineered ML: Required manual feature writing for specific document types, proving difficult to scale.
- Program Synthesis: Attempted to generate regular expressions on the fly but remained brittle when input structures shifted.
- Technical Breakthrough (InstaLM): In 2017, the speaker's team developed "InstaLM," a model encoding both token position and X/Y coordinates to understand 2D document layout, a technique that became a standard for document understanding.
- Generative AI Impact: The release of ChatGPT in November 2022 validated that LLMs could process unstructured data, causing the speaker's team to triple revenue between 2021 and 2022.
- Reliability Gap: LLMs alone suffer from "surprising errors" (e.g., missing random cells in a table) and lack completeness guarantees, making them unsuitable for high-stakes decisions (lending, immigration, insurance claims) without system-level safeguards.
- Proposed Architecture: Enterprises require a compound AI system that includes:
- Specialized algorithms for table detection, signature recognition, and checkbox classification rather than passing raw text to an LLM.
- Validation and cross-validation steps (e.g., verifying W2 forms against pay stubs).
- Auditable workflows that ensure 100% data completeness for critical decisions.
- Enterprise Adoption Barriers: Primary hurdles for enterprise adoption are:
- Data Security: Guaranteeing that proprietary data remains safe and secure.
- Auditability: The ability to explain the specific steps an AI took to reach a decision to satisfy compliance and legal committees.
- Acceptance Criteria Shift: Enterprises do not demand 100% AI accuracy; instead, they require predictability, where the system explicitly identifies which results (e.g., the bottom 20%) need human review.
- Agent Paradigm: The speaker argues against autonomous runtime agents for enterprise use; instead, agents should function at compile/build time:
- Build Time: Agents generate initial workflow drafts and logic.
- Runtime: The final workflow must be deterministic, auditable, and strictly controlled to ensure consistency.
- Future Execution Model: The vision shifts toward decentralized, federated execution where thousands of autonomous agents dynamically discover and communicate to solve tasks, replacing the need for a single central controller.
- RPA Replacement Strategy: AI automation can replace RPA by utilizing protocols like Model Context Protocol (MCP) and "identity pass-through" to interact with legacy systems, allowing agents to operate with capped permissions (acting as a "good intern") rather than full user access.
- Customer Experience Evolution: AI enables new interactive interfaces, such as conversational lending over WhatsApp where users upload piecemeal documents and receive real-time feedback, moving away from static, non-interactive submission forms.
- Comparative Advantage: AI reduces the time to make complex decisions (e.g., loan approval) from weeks to under five seconds while maintaining reliability through system-level validation.