Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI

The global economic impact of AI is projected to reach quadrillions of dollars, driven by artificial engineers that could increase global GDP by orders of magnitude.

• Currently, 25% of code characters checked into Google's codebase are generated by AI coding models.
• Future scaling may require millions of automated researchers to accelerate invention cycles.
• Noam Shazeer anticipates that the compute allocated to AI per person will reach astronomical levels.

Jeff Dean and Noam Shazeer, both Google veterans of roughly 25 years and co-leads of Gemini, discuss the evolution of Google from a small team where everyone knew everything to a massive organization requiring deep networks of specialized contacts.

• Shazeer joined Google in 2000 after initially assuming the company was too large to join in 1999, originally intending to work on AI full-time only after making money elsewhere.
• Dean joined in 2000, noting that early Google had a "crayon chart" of daily search queries, signaling exponential growth and a commitment to AI ambitions.

The scaling of AI hardware has shifted from relying on Moore's Law for general-purpose CPUs to requiring specialized ML accelerators like TPUs and GPUs.

• Modern arithmetic is extremely cheap, making data movement the primary bottleneck, which has driven the success of deep learning architectures based on matrix multiplications.
• Google's TPU evolution has moved from 8-bit integer quantization for inference to supporting int4, FP4, and even 1-bit or 2-bit quantization for both training and inference.
• Chip design cycles, traditionally 18–24 months, are being targeted for reduction to a few months using automated search processes to allow hardware to keep pace with algorithmic changes.

Historical milestones in Google's AI development include Jeff Dean's 1990 senior thesis on parallelizing backpropagation and a 2007 team winning a DARPA translation contest with a 2-trillion-token n-gram model.

• The 2007 translation system was optimized from a 12-hour processing time per sentence to ~100 milliseconds by moving 2 trillion n-gram statistics into memory on 200 machines.
• The 2012 Google Brain project demonstrated unsupervised learning on 10 million YouTube frames, identifying a neuron that activated specifically for images of cats without explicit labeling.

Google's mission is expanding from "organizing information" to creating and synthesizing new information across multimodal contexts (text, video, code, genomics).

• The company envisions AI as a tool to make any content available to any person regardless of language, including translating video and text across thousands of languages.
• Current challenges include hallucinations and factuality issues, which the team aims to solve by merging the "sharp" context of search with the generative capabilities of LLMs.

Future AI systems are expected to utilize "inference time compute" scales of 10x to 1,000x to improve reasoning and problem-solving reliability.

• Techniques to extend context windows include algorithmic approximations to move beyond the quadratic cost of naive attention, aiming to attend to trillions of tokens (e.g., entire codebases or personal data).
• "Drafting" models are used to predict multiple tokens in parallel, with a larger model acting as a verifier to speed up inference without sacrificing accuracy.

A feedback loop is anticipated where AI systems design better hardware and algorithms, potentially accelerating the timeline for superintelligence.

• Algorithmic improvements and data mix changes are driving efficiency gains (improvements per flop) that may eventually outweigh raw hardware scaling.
• The team projects that as models can break down complex tasks into hundreds of steps with higher reliability, the "intelligence explosion" could occur within a few generations (2–3 years).

Safety and alignment are prioritized as capabilities increase, with a focus on "shaping" AI deployment rather than a laissez-faire approach.

• Google's strategy involves human oversight in the loop for automated research, where AI explores ideas but humans make final decisions on incorporating them into production models.
• Safeguards include using AI to analyze and filter its own output, leveraging the fact that language analysis is easier than generation to detect and mitigate harmful behaviors.

Future model architectures may shift toward "organic," modular structures (continual learning) where different experts can be updated independently without retraining the entire model.

• This "mixture of experts" vision allows for parallel development of specialized modules (e.g., math, coding, specific languages) that can be glued together or swapped out.
• Distillation is viewed as a key mechanism to transfer knowledge from large, complex "blob" models into efficient, specialized versions for specific use cases.

Data efficiency remains a critical area of improvement, with potential gains from using dropout techniques more extensively in text training and moving from passive observation to active, action-based learning.

• Shazeer suggests changing training objectives beyond next-token prediction to include tasks like answering questions at the end of a chapter or reasoning through thought experiments.
• The team believes the world has not yet exhausted its textual data and that better learning techniques could make models as sample-efficient as humans.

Google's historical approach to open-sourcing (e.g., the Transformer paper) was driven by a belief in an expanding economic pie rather than a fixed one, though recent strategies involve more selective publication to protect product roadmaps.

• Dean and Shazeer attribute their long-term success to intellectual humility, the willingness to pivot away from failed ideas, and the practice of collaborating with experts from diverse domains (e.g., chip design, healthcare).
• The internal culture is evolving to balance top-down resource allocation for large projects (like Gemini) with bottom-up incentives to encourage innovation and rapid iteration on smaller scales.