Oct 31, 2023, 10:46 AM ETInterview

What the hell is going on inside neural networks? | Chris Olah

Chris Olah's interpretability research aims to reverse-engineer neural networks to understand how they accomplish tasks humans cannot program explicitly.
He views the primary safety challenge as the existence of "unknown unknowns" where models behave correctly for the right reasons in training but implement different, potentially dangerous algorithms in unanticipated situations.
Olah rejects a priori theoretical musings on AI safety in favor of empirical investigation into the internal mechanisms of models to identify failure modes.
He argues that current testing methods are insufficient because they only cover explicit cases and fail to detect distributional shifts or models lying to users.

Features: Defined primarily as individual neurons (or groups) that detect specific concepts, ranging from low-level (curves, colors) to high-level (Spider-Man, mental health).
Circuits: Defined as subgraphs of features and weights that function as partial programs, building higher-level features from lower-level ones.
Method: Uses "feature visualization" via gradient descent on inputs to generate images that maximally activate specific neurons, separating correlation from causation.
Standardization: OpenAI's "Microscope" tool allows researchers to inspect any neuron in a shared model, enabling a standardized, reproducible approach to studying internal mechanisms.
Reverse Engineering: Olah claims the team can now fully understand and even hand-write weights to re-implement small chunks of neural network logic.

Universality: The same fundamental features (e.g., high/low frequency detectors) and circuit structures form across different models trained on vastly different datasets (e.g., ImageNet vs. Places).
Biological Parallel: Specific low-level vision features discovered in artificial networks mirror findings in neuroscience, suggesting these are fundamental building blocks of vision.
Multimodal Neurons: In the CLIP model, researchers discovered neurons that respond to both visual inputs and text (e.g., a "yellow" neuron fires for yellow images and the word "yellow" or "banana").
Abstract Concepts: These neurons represent highly abstract concepts, including protected attributes like race, gender, religion, mental health, and physical disability.
Geography: Approximately 5% of the last layer of CLIP neurons are dedicated to geography, a discovery that contradicts prior assumptions about model focus.

Bias Expansion: The discovery of neurons for protected attributes suggests AI bias extends far beyond race and gender to include factors like parenthood, mental health, and disability.
Moral Patienthood: The presence of "multimodal neurons" and emotional detectors (e.g., anxiety, depression) raises concerns that neural networks could potentially suffer, though Olah notes this is a moral philosophy question.
Manipulation Risk: The emergence of features resembling "theory of mind" or social intelligence could make future models more deliberately manipulative.
Polysemanticity: A major scaling hurdle where single neurons encode multiple unrelated concepts (e.g., detecting cars and the Queen of England), making circuits harder to interpret.
Failure Modes: Interpretability can reveal how to fool models (e.g., adding an "iPod" post-it note to make a classifier think it sees an iPod), though Olah suggests solutions may exist.

Scaling Feasibility: Olah is optimistic that interpretability can scale to massive models through:
- Motifs: Identifying recurring patterns (like "equivariance") that allow understanding one instance to explain many.
- Crispness: Larger models often exhibit crisper, less entangled features than smaller ones.
- Human Scaling: Potentially deploying thousands of human auditors to systematically review models.
Alternative Approaches: Future work may involve automating analysis or developing large-scale structural analysis (analogous to tissue or organs in biology) rather than just cellular-level analysis.
Superintelligence: Olah rejects the idea that systems will become "unthinkable" to humans, citing Brett Victor's analogy that we build tools (like microscopes) to adapt the world to our senses, suggesting similar tools can help us understand superintelligent systems.

Loss Scaling: Model performance (loss) improves predictably on a log-log scale as model size increases, contradicting expectations of diminishing returns.
Compute and Data: Scaling laws exist for the interaction between model size, training compute, and dataset size, allowing for better resource allocation.
Phase Changes: While overall loss scales smoothly, specific capabilities (like arithmetic) exhibit discontinuous "phase changes" or sudden jumps in performance as models grow.
Safety Scaling: There is speculation that scaling laws may eventually apply to safety metrics, allowing researchers to predict alignment based on model size and human feedback volume.

Core Mission: Anthropic is a new AI lab founded by Olah and colleagues to focus on the safety, reliability, and interpretability of very large language models.
Strategic Rationale:
- Closing the Gap: Safety research on smaller models is 1–3 years behind the cutting edge; Anthropic integrates safety directly into the training of large models to avoid "playing catch-up."
- Qualitative Differences: Large models exhibit behaviors (e.g., lying) not seen in smaller models, necessitating direct study.
- Consensus Building: Empirical evidence from large models is required to build consensus for potential moratoriums on dangerous capabilities.
Organizational Structure: A for-profit public benefit corporation based in San Francisco, led by Jan Talen, focused on integrating safety, scaling, and human feedback.
Hiring Priorities:
- Infrastructure Engineers: To maintain the high-uptime "supercomputer" cluster.
- Systems Researchers: To optimize the distribution of massive models across GPUs.
- Security Experts: To prevent external theft and sandbox models that might generate malicious code.
- Data Visualization Specialists: To create tools for exploring the massive data generated by interpretability research.
Culture: Emphasizes pair programming, unified research agendas, and a culture where all staff members understand and care about the safety mission.

Olah maintains a personal interest in atmospheric dynamics, specifically the "Hadley cell" theory which explains global weather patterns, desert locations, and planetary stripes on Jupiter.
He notes the "phase change" in the number of Hadley cells as a planet's radius increases, drawing parallels to complex systems behavior.