Interview, Fireside Chat
Michael Nielsen – Why aliens will have a different tech stack than us
- Predictions suggest AI will match the tax preparation capabilities of a competent general manager by 2028, while deep learning models may eventually be distilled into simpler, parsimonious theories via regularization.
- Future productivity may evolve into a pure information problem as fabrication becomes commoditized through bioreactors or functional 3D printers, potentially occurring during a transition distinct from the current "intelligence age."
- The tech tree of knowledge is expected to expand significantly over the next million years, with civilizations likely exploring divergent paths based on biases, though most branches will remain unexplored due to the vast number of deep ideas and combinations.
- A transition involving quantum computers is anticipated to be qualitatively different from current AI paths on classical machines, potentially enabling a strictly larger class of computations when combined with AI.
- The current trend of hyperbolic growth in scientific progress is expected to accelerate with AI, as new fields and materials like superconductors and Bose-Einstein condensates continuously reset the progress curve.
- AI is expected to assist in specific scientific bottlenecks such as structural biology but may not resolve issues requiring interesting design ideas without verification loops, where scientists will remain a constraint.
- Current AI safety benchmarks are predicted to be misleading, with success rates for attacks ranging from a few percent on public tests to approximately 90% in real-world scenarios.
- The open science movement is expected to reshape the political economy of science by making preprint cultures and data sharing central to debate, potentially shifting the "fashion" of centralized attention in research.
- Future institutions may preemptively steer technological exploration away from certain paths, and the "comparative advantage" between humans and AI may not generate trade due to high transaction costs and low subsistence benefits.
- Material expectations include the need for a 9% annual increase in the number of scientists to maintain semiconductor progress aligned with Moore's Law, alongside the possibility of AI generating scientific objects akin to long, complex equations processable by modern tools.
- Historical data indicates that verification loops for scientific theories are often extremely long and can be actively hostile, with examples ranging from 184 years for helium discovery to 85 years for isotopes, and some loops, such as the ether wind, remaining unclosed.
- Scientific adoption of new theories often precedes empirical validation by decades, as seen with the 40-year delay in recognizing muon decay as proof of special relativity, and communities may occasionally adopt interpretations before they are empirically preferred.
- The probability of any specific creative release being extremely important is estimated to be similar across works, implying that high productivity is the key to success under an "equal odds rule," while learning from passive media requires demanding tasks to compound.
- Risks include the potential for civilizations to develop deep learning before cosmology, leading them to build complex epicyclic models like Ptolemy's rather than finding simpler theories, and the lack of salient computation technology historically bottlenecking quantum computing until the 1980s.
- The speaker anticipates a future transition so distinct that the current generation will not recognize it as "pre-AI" or "post-AI," potentially occurring within a brief period relative to the long span of human history.