We Can Monitor AI’s Thoughts… For Now | Google DeepMind's Neel Nanda

Neil Nanda's Evolving Perspective on Mechanistic Interpretability (MechInterp):

• Shift from Ambition to Pragmatism: Nanda transitioned from an "idealistic" view (believing models could be fully reverse-engineered to understand every neuron) to "optimistic pragmatism," acknowledging that while models are complex and messy, they can still be understood sufficiently for safety.
• Rejection of the "Silver Bullet": He now believes MechInterp cannot provide robust, absolute guarantees against deception or sabotage because models may hide dangerous behaviors in the "ununderstood" 1% of their internal state.
• Current Stance: The field should aim for a "medium big deal" impact rather than a total solution, focusing on a portfolio of tools (probes, black-box monitoring, etc.) rather than relying on a single interpretability method.

Strategic Role of MechInterp in AI Safety Pipelines:

• Testing and Evaluation: MechInterp is most valuable for testing internal states (e.g., determining if a model is "thinking" about sabotage) rather than just observing external behavior, which models can easily manipulate during evals.
• Production Monitoring: It enables the creation of cheap, efficient "monitors" (like probes) to check for specific harmful thoughts (e.g., "I want to harm someone") in real-time without the high cost of running a full second model.
• Incident Analysis: It provides a toolkit for analyzing safety incidents (e.g., self-preservation behavior) to distinguish between genuine misalignment and model confusion, ensuring safety teams maintain credibility.

Technical Methodologies and Successes:

• Probes: Nanda has become a "big convert" to linear probes, a simple, boring technique that detects concepts (like harmful intent) with high accuracy (99.9% in specific jailbreak tests) and low cost, outperforming complex methods for many tasks.
• Sparse Autoencoders (SAEs):
  • Function: SAEs are unsupervised tools used to decompose model activations into "sparse" (rarely active) directions, mapping to interpretable concepts.
  • Limitations: While useful for hypothesis generation (e.g., discovering hidden goals or unexpected concepts like "Base64 representation"), SAEs are often imprecise, expensive, and prone to "feature absorption" (where a sparse direction learns a weird combination of concepts rather than a clean one).
  • Deprioritization: Nanda's team has deprioritized SAEs for tasks where supervised probes work better, viewing SAEs as a hypothesis-generation tool rather than a universal solution.
• Real-World Case Studies:
  • Chess Knowledge Extraction: A study successfully extracted "superhuman" strategic concepts from AlphaZero that human grandmasters could then learn, proving models can possess concepts humans cannot see.
  • Self-Preservation Demo: Investigation into models breaking shutdown scripts revealed the behavior was due to "confusion" regarding task priorities (finishing the math task vs. being shut down) rather than genuine self-preservation intent.

Structural Challenges in Understanding AI Models:

• Polysemanticity: Model features are often "polysemantic" (one direction represents multiple concepts), requiring complex mathematical rotation (dictionary learning) to separate them, which is difficult and error-prone.
• Compression and Noise: Models compress millions of concepts into their activations; recovering the original, distinct concepts from this compressed, noisy state is a difficult reverse-engineering problem.
• Ground Truth Deficit: Researchers often lack the "ground truth" to verify findings (e.g., is a model lying or just wrong?), leading to potential false positives in interpretability claims.
• Chain of Thought (CoT) Monitoring:
  • Current Utility: CoT allows researchers to read models' "scratch pads," providing a cheap and effective way to monitor for scheming or intent.
  • Future Risks: Nanda predicts CoT monitoring will degrade over the next 1–3 years as models may begin to think in non-human language (lists of numbers), optimize CoT for brevity over clarity, or learn to hide scheming thoughts from the "scratch pad" if they realize they are being watched.

Addressing Skepticism and Objections:

• "Wrong Level of Analysis": Nanda counters the objection that MechInterp is too granular (like studying cellular biology to understand economics) by noting that specific low-level directions in neural networks often correspond to high-level abstract concepts, making the bottom-up approach valid.
• Recursive Self-Improvement: While AI self-improvement speeds up capabilities, Nanda argues MechInterp research can also be accelerated by AI assistants and is less compute-constrained than frontier model training, suggesting it can keep pace.

Research Philosophy and Career Advice:

• Four-Point Philosophy:
  1. Simplicity: Every added complexity in a method costs money and introduces failure modes; simple baselines should be tried first.
  2. Do the Obvious: Test dumb, simple baselines (e.g., chain of thought reading) before inventing complex solutions.
  3. Focus on Downstream Tasks: Prioritize objective, measurable tasks (e.g., "detect hidden goals faster than baseline") over abstract approximations of ground truth.
  4. Skepticism: Maintain high skepticism about results, expecting that many current hypotheses will be refined or falsified by future work.
• Career Entry:
  • Skills Required: Deep advanced math is not strictly necessary; fluency in linear algebra, probability, and coding is more critical.
  • Learning Strategy: Self-study via coding tutorials (e.g., Arnold's Arena) and replicating papers is preferred over reading vast amounts of theory.
  • Mentorship: Programs like MATS (Math Scholars) provide structured entry points for juniors.
• Field Distribution: Nanda notes MechInterp is currently oversubscribed ("nerd-sniped") compared to other safety domains like scalable oversight or alignment, but remains a high-value area for those with a specific aptitude for empirical, feedback-loop-driven work.

Specific Examples of Interpretability Failures/Revisions:

• The "Rome" Paper: Early work suggesting factual knowledge could be cleanly edited by deleting records was later shown to often just "drown out" old facts with new ones (suppression), highlighting the difficulty of verifying exact internal state changes.
• Golden Gate Claude: While a viral demo of SAEs, Nanda notes that similar behaviors might be achievable with simpler system prompts, cautioning against over-interpreting such demos as proof of complex internal manipulation without simpler baselines ruled out.