Panel, Conference Presentation
RAISE Panel 2025: The AGI Ascent Climbing Toward Superhuman Intelligence
- Definition and Benchmarking Challenges: AGI lacks a precise definition, with panelists describing it as a "vibe" of performing as well as or better than humans at any job; benchmarking is difficult because AI performance varies by task, excelling in coding in some areas while lagging in others.
- Scaling Laws and Compute Trajectory: The consensus is that current trajectories rely on increasing compute power, data volume, and model size, with scaling laws holding true; a single 5-gigawatt data center campus would equal the total electrical capacity of Northern Virginia (approx. 4.5 GW) in 2024.
- Shift to Test-Time Compute: A major industry bet is shifting from training-time compute to massive "test-time" compute, where models perform "chain of thought" reasoning during inference to generate smarter results.
- Sourcegraph's Counter-Argument on Overfitting: Peter (Sourcegraph) argues that scaling compute without proportional data quality leads to overfitting, citing how memorizing the Encyclopedia Britannica with a massive brain prevents learning general logic rules essential for coding agents.
- Fixed-Model Improvements: Despite the push for larger models, quality improvements in fixed-size models (e.g., 7B parameters) continue due to synthetic data selection and distillation, which still require significant compute.
- Workforce Impact and Human Adaptation: Sarah (Lattice) notes that every job is changing, emphasizing the need for societal focus on human-AI collaboration rather than fear; outcomes depend on how well individuals and companies adapt to the new technology.
- Decentralization vs. Centralization: Ilya advocates for a decentralized, user-owned future using blockchain to prevent an "Orwellian" future dominated by single corporations, whereas Peter argues that foundational training clusters must remain centralized for efficiency, while inference can and should be decentralized.
- Cognitive Atrophy Concerns: Bayang admits that 95% of his code is generated by AI, noting his "inner developer loop" has atrophied; he counters concerns about becoming "dumber" by arguing technology frees humans for higher-level creative tasks like systems design.
- Outcome vs. Process Automation: The panel discusses that tasks focused on outcomes (e.g., scoring a goal, winning a game) will be automated, while tasks valued for the process (e.g., sportsmanship, religious engagement) will remain human.
- Energy as the Primary Bottleneck: Energy scarcity is identified as the critical constraint for scaling AGI; panelists note a shift in the tech sector from avoiding to embracing nuclear power, with AI itself viewed as a tool to accelerate energy innovation (e.g., new battery chemistries, fusion).
- Regulatory and Legal Hurdles: Regulation is seen as a lagging force due to the speed of AI innovation; Ilya proposes "regulation by design" (e.g., confidential computing) over "regulation by paperwork," while Sarah notes the EU AI Act's GPAI category is undefined and the fragmented European single market hinders scaling.
- Secrecy and Information Flow: Panelists argue that extreme secrecy by big tech is ineffective, as ideas disseminate rapidly (e.g., DeepSeek replicating OpenAI's reasoning model independently); the focus should be on distribution and utility rather than guarding formulas.
- Cybersecurity Risks: A hypothetical scenario was raised where an AI achieves perfect cyber-offensive capabilities, creating an asymmetrical threat that could destabilize research labs or be weaponized by governments, necessitating equal access and safety standards.
- Paradigm Continuity vs. Evolution: Disagreement exists on whether current "agentic" systems represent a new paradigm; Ilya views them as continuous extensions of the "universal function approximator" model, while Sarah suggests agentic systems that can act autonomously in the real world mark a distinct shift.
- Timeline Predictions: Goalposts for AGI are constantly moving; Bayang predicts coding capabilities could reach 98-99% within a year, but the panel concludes AGI is a moving target defined by the "human with AI" combination, making a definitive arrival date impossible.
- Human Uniqueness: Sarah concludes that AI will never replace unique, original human thought or the unpredictability of human interaction, as training remains based on predetermined data and outcomes.