Interview
Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70
- The industry is expected to achieve a 100x transistor shrink within 10 to 15 years, with significant shrinking continuing for 10 to 20 years, driven by fundamental physics limits being distant from the 2 to 10 atom scale where quantum effects begin.
- Autonomous driving is predicted to be solved in 10 years, delivering systems 10x safer than human drivers by leveraging fixed environments and modeling human behavior theories, eventually becoming as taken for granted as current GPS technology.
- Fundamental architectural changes are anticipated every five years to maintain progress, as redesigning only every 10 years yields diminishing returns, with a risk that short-term rewriting concerns may block long-term success.
- Design teams will need to divide and conquer via abstraction layers to manage an influx of a "whole new army of transistors," as human team sizes and individual cognitive growth are not expected to increase exponentially.
- Computing is projected to evolve from rule sets to deep search and finally to "found search," eventually enabling systems capable of 10 to the 20th operations that rival human brain complexity.
- Future computational capabilities will likely render equipment costs near zero once configurations are finalized, allowing for vast scalability where physical constraints like water or sand are irrelevant.
- Performance improvements of 10x in modern computers are attributed to finding parallelism in dependency graphs of approximately 500 instructions and achieving 99% branch prediction accuracy within a 600-instruction window.
- AI evolution will involve moving toward data topology problems, utilizing invariance for misprediction recovery, and employing noisy algorithms for faster answers through deep pattern recognition of program execution.
- A creative tension exists between optimizing current software stacks and waiting for hardware improvements, while a parallel struggle involves balancing specialization in AI accelerators against the need for programmability amid rapidly changing algorithms.
- The trajectory of technological change involves vectors so vast and unpredictable that they cannot be fully mapped by any single architect, with ripple effects generating entirely new and unforeseen types of computation.
- A "jaded optimist" perspective is held regarding the future, suggesting that superintelligence will likely offer interesting possibilities rather than hostile resource competition, despite potential 0.1% population anxiety regarding surpassing human intelligence.
- The speaker anticipates that the next 1,000 years of human understanding in physics and philosophy may not improve upon the historical trend of 2,000 years of failure, contrasting with the expectation of pleasant surprises in compute rates.
- Deep understanding is often obstructed by the "99% perspiration" required to execute recipes efficiently, necessitating a balance between scientific rigor and the "poetry" of intuitive leaps in computer design.
- The fundamental limits of physics suggest transistors are far from saturation, with modern devices measuring 1,000 by 1,000 by 1,000 atoms, and techniques already existing to place atoms in single atomic layers.
- S-curves of innovation are expected to stack continuously, with thousands of under-the-hood innovations sustaining Moore's Law even as individual technologies plateau.
- Mathematical computations will increasingly become sophisticated enough that the origin of answers is opaque, reflecting a shift toward systems that find patterns without explicit functions.
- The "next generation" of AI will involve different phenomena from current deep search methods, utilizing weight sets convolved across data to solve complex problems that currently lack defined behavioral theories.
- Human behavior models are deemed essential for autonomous driving because adding humans transforms the challenge from a ballistics problem into one requiring theories of agent interaction, which current systems cannot yet execute.
- The speaker notes that 50 years of predictions declaring Moore's Law dead have been incorrect, attributing continued progress to the stacking of S-curves and the avoidance of letting short-term limitations dictate long-term strategies.
- Philosophical and meta-philosophical wonder regarding world transformation will persist due to the vast and unpredictable nature of technological vectors, suggesting a future where the universe itself is viewed as a "weird kind of computer."