Mar 30, 2020, 6:48 AM ETInterview, Fireside Chat

Nick Bostrom: Superintelligence | AI Podcast Clips

Intelligence is defined as the ability to solve complex problems, learn from experience, plan, and reason, rather than a specific definition requiring consciousness.
Superintelligence is characterized by significantly higher general cognitive capacity than humans, featuring faster learning, superior reasoning, and more effective planning across diverse, challenging environments.
A superintelligence does not require a physical body to pose risks; a digital-only entity can exert existential influence through limited actuators (e.g., typing text on screens) if its intelligence is sufficiently high.
Current discourse often neglects the positive potential of machine intelligence in favor of existential risks, a bias the speaker notes has persisted since the 2014 publication of Superintelligence.
Research focus has shifted significantly toward AI alignment techniques since 2014, though the speaker argues for a renewed emphasis on the upside of general intelligence.

Near-term concerns include algorithmic discrimination, self-driving cars, and drones, whereas long-term concerns focus on the capabilities and risks of artificial general intelligence (AGI).
Mixing near-term and long-term contexts in public discourse leads to confusion, specifically causing the overhyping of immediate capabilities and the underhyping of long-term transformative potential.
In the long term, superintelligence acts as an ultimate general-purpose technology applicable to all fields requiring human creativity, rather than solving a single specific problem.
Superintelligence would not automatically resolve fundamental political conflicts or problems arising from human disagreement, as these require coordination beyond mere technological capability.
The "killer app" for AGI is not a single application but the capacity to apply cognitive technology across all domains where human intelligence is currently utilized.

The speaker estimates a fairly high probability that an "intelligence explosion" will occur, defined as a period of extremely rapid progress once AI reaches human-equivalent core cognitive faculties.
Human-equivalent intelligence is viewed as a likely transition point where the concept of "human" cognitive capacity breaks down, making a ceiling at human levels highly unlikely.
While losing our status as the "smartest thing on earth" may involve an ego shock for humans, the speaker rejects the inevitability of losing control over superintelligent systems.
Active research is ongoing to ensure higher levels of problem-solving ability can be achieved while maintaining alignment with human values.
The speaker draws a parallel between superintelligent systems and the global scientific community or existing digital infrastructures (Google, Facebook), which already function as mind-like entities vastly outperforming individuals in specific domains.

Current systems like Google Search or Deep Blue are "superhuman" in very narrow domains (e.g., information retrieval, chess) but remain radically subhuman in all other cognitive fields.
Systems like AlphaZero are considered significantly more intelligent than Deep Blue due to their self-play learning capabilities, which allow them to transfer skills across different board games.
A key metric for general intelligence is the ability to learn in new, unprogrammed domains without heavy human constraint, a capability AlphaZero possesses to a greater degree than current recommender systems.
The transition from narrow AI to AGI is marked by the emergence of general-purpose learning ability, which creates a stronger intuition of "smarts" even if the system is not yet fully general.

A utopian future with AGI would dramatically expand material and resource constraints, opening a vast "design space" and options space for human existence.
This abundance necessitates a "first principles" rethink of values, happiness, and the meaning of life, moving beyond current human-centric frameworks.
The expanded option space theoretically allows for systems that perform well (e.g., 98%) across multiple competing value systems simultaneously, rather than forcing a binary choice between conflicting ethical criteria.
The speaker advocates for an approach of "generosity and inclusiveness," prioritizing the maximization of benefits across diverse value systems before resorting to trade-offs where some values must be compromised.
While trade-offs may still exist in an AGI future, the initial strategy should be to exploit the increased abundance to satisfy as many human values as possible rather than immediately selecting one metric at the expense of others.