Interview, Fireside Chat, Conference Presentation
Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin
- Models are projected to learn evolving context with the same depth as static facts within the short-to-medium term, challenging the separation of fact and skill learning.
- Collective token generation is anticipated to reach tens of millions per day soon, necessitating new solutions to address resulting storage and search costs.
- A significant trade-off involves burning compute upfront to internalize context, expected to yield a 100x (two orders of magnitude) reduction in inference consumption compared to current models.
- A consistent three-to-six-month gap is predicted where frontier models lack proficiency in bespoke tasks, creating a window for models capable of autonomous learning.
- Technology adoption will likely shift from generic models to personalized models for individuals and organizations, moving away from a "one big model" paradigm.
- Future infrastructure is expected to support training small models for mass adoption, with team-level data serving as the initial entry point before expanding to individual devices.
- The industry expects a convergence where specific company facts and stories are mixed into model weights, rather than keeping generic models strictly separate from specific contexts.
- A 80 GB HBM requirement for a single Wikipedia article via KV caches is cited as inefficient, with compression techniques potentially reducing this state by a factor of 1,000.
- Language is expected to dominate over vision in the short term, though both modalities are predicted to eventually combine into a unified system.
- Within the next five to ten years, hundreds of millions of personalized neural memories are expected to function as a neural interface to personal and corporate data planes.
- A "memory wallet" allowing individuals to carry a sanitized set of skills across different jobs and companies is expected to become feasible.
- Frontier Labs face P0 pressure to deliver AGI and generic models, while alternative views suggest exclusive funding is needed for breakthroughs in memory and continual learning.
- Research and product integration is predicted to evolve so that user inputs are intricately tied to training signals rather than treated as a separate process.
- A proof of concept demonstrating an "intern model" taught over time to achieve noticeable improvement remains a prerequisite milestone not yet achieved by current context engineering.
- Open source models are identified as the easiest target for white-box access, though partnerships with closed-source providers remain viable for the proposed approach.
- Future AI systems may incorporate "dreaming" capabilities to experiment with affordances and handle tail extreme events by retreating from active interaction.