Interview, Fireside Chat
Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil
- Core Mission: Core Automation is founded by Jerry Liu and Rohan Ramachandran to build a new class of AI systems capable of continual, test-time learning, moving beyond the static nature of current models.
- Architectural Thesis: Jerry Liu asserts that the Transformer architecture has reached a "ceiling" where further scaling yields diminishing returns; the primary bottleneck is no longer data or compute, but the fundamental architecture itself.
- Inference Bottleneck: Rohan Ramachandran identifies that Transformers are inefficient at spending computation during inference, relying on slow, autoregressive token-by-token generation rather than optimizing for computational depth.
- The "Depth" Problem: Current models are limited to shallow depths (approx. 100 layers); increasing effective depth via Chain of Thought reasoning requires generating many tokens, which is computationally expensive and inefficient compared to biological learning.
- Test-Time Learning Gap: Standard Transformers lack the ability to adapt continuously to new distributions; they require lab retraining for new data, whereas the founders seek models that learn from user interactions and real-world environments autonomously.
- Reinforcement Learning (RL) Limitations: Jerry Liu argues that while scaling RL is vital, it is not the "end of learning from experience" and does not replicate how humans learn distinct concepts (e.g., mathematics vs. motor skills) efficiently.
- Optimization vs. Architecture: Rohan Ramachandran notes that optimization methods (like Shampoo) and architecture design are symbiotic; better optimizers allow for deeper, more complex architectures that were previously untrainable.
- Hardware Constraints: Kernels for advanced linear algebra operations (e.g., QR decomposition) remain a major bottleneck; current coding agents can achieve ~7x speedups, whereas human experts achieve 60x, leaving a massive efficiency gap.
- AGI Definition: Liu defines AGI as a system that can improve itself without human intervention, contrasting this with current LLMs which still require significant human oversight and cannot operate effectively "human-in-the-loop" for self-improvement tasks.
- Strategic Differentiation: Core Automation aims to fill a niche ignored by major labs focused on short-term release cycles and Transformer scaling, by building a fully automated research lab with a small, agent-enabled team.
- 6-Month Roadmap: The lab's immediate focus is on automating kernel generation and building a stack where models can execute experiments and optimize code to run at maximum TFLOPs on hardware.
- Success Metric: The team plans to validate their new architecture by measuring whether the lab's systems improve autonomously over time, humorously noting they will "extend their vacation" as long as the systems produce better results without human input.
- Biological Comparison: Rohan Ramachandran states that current silicon-based neural networks are unlikely to surpass biological learning efficiency without a shift to analog or neuromorphic hardware that mimics how brains build learning algorithms during growth.