Interview
Who's trying to steal AI models? And what could they do with them? | Sella Nevo
- Risk Probability Calculation: Granting full read access to the model weights to 50 highly trusted employees results in a cumulative leak probability of approximately 63% (calculated as $1 - 0.98^{50}$), demonstrating that even with near-perfect individual trust, the aggregate risk of internal leakage is substantial.
- Economic Value of Weights: Model weights represent a unique culmination of costly prerequisites, including an estimated $78 million for GPT-4 and nearly $200 million for Gemini Ultra; possessing the weights allows bypassing these hundreds of millions in compute, talent, and infrastructure costs, reducing inference costs to negligible levels (under half a cent per 1,000 tokens).
- Strategic Vulnerability: Once model weights are stolen, standard defenses such as API monitoring and trained guardrails become ineffective because the attacker can run the model independently, fine-tune it to remove safety constraints, and deploy it without detection.
- Biosecurity Threat: Model weights pose an acute risk for bioweapons development due to AI's growing capability in driving state-of-the-art biology research, with experts citing this as a plausible pathway for global-scale harm in the near future.
- SolarWinds Supply Chain Attack: A 2019 Russian espionage operation (Cozy Bear) infiltrated 18,000 organizations by subverting SolarWinds' build system, allowing attackers to replace legitimate software updates with malware containing backdoors that remained undetected for 14 months.
- Attacker Operational Capacity (OC) Categories: Sela Nevo's RAND report defines five tiers of attackers based on resources:
- OC1 (Amateur): Single individuals investing days and up to $1,000, often motivated by financial gain or "lulz."
- OC2 (Professional Opportunistic): Capable individuals or groups investing weeks and up to $10,000 for untargeted attacks.
- OC3 (Crime Syndicates/Insiders): Organizations like Anonymous or insiders investing up to $1 million; these actors often utilize zero-day vulnerabilities and Advanced Persistent Threat (APT) techniques.
- OC4 (Major Cyber Capable Institutions): Nation-states (e.g., Russia, China) with hundreds of staff, $10 million budgets, and capabilities including human intelligence, physical ops, and legal cover.
- OC5 (Top Priority Operations): The most capable state actors with thousands of personnel, billion-dollar budgets, and decades of head starts; equivalent to military special operations units.
- Zero-Day Exploitation: Malicious actors, particularly nation-states, can systematically undermine defense-in-depth strategies by acquiring hundreds of zero-day vulnerabilities (unpatched flaws) through state-mandated reporting laws (e.g., China's regulations) or by intercepting responsible disclosure channels.
- Human Intelligence (HUMINT) Vectors: Attackers often bypass security by aligning with an insider's existing ideology (e.g., democratization of AI or transparency) rather than offering direct bribes, or by using "grooming" tactics to gradually erode boundaries before resorting to extortion involving family safety or exposure of minor prior violations.
- Side-Channel Attacks: Attackers can infer sensitive data by monitoring physical outputs of computation, including:
- Electromagnetic Radiation: Historical "TEMPEST" attacks collected emissions from CRT monitors and computer hardware to reconstruct data.
- Power Consumption: Analyzing electricity usage fluctuations can reveal encryption keys or model weights (e.g., distinguishing multiplication from squaring operations).
- Acoustic Analysis: Malware or sensors can identify passwords typed on physical keyboards by analyzing distinct acoustic signatures.
- Air-Gapped Breach via USB: Air-gapped networks are vulnerable to malware transfer via USB storage devices, a method exploited by malware like Stuxnet, Agent Btz, and PlugX to infiltrate Iranian nuclear facilities and other high-security US government networks.
- Hardware Supply Chain Threats: Malicious USB cables containing hidden wireless communication modules can be purchased for ~$180, allowing attackers to control systems from afar once plugged in; this hardware supply chain risk is a critical vulnerability even for organizations that avoid physical USB sticks.
- Model Extraction Feasibility: While full extraction of trillion-bit models is theoretically difficult, attackers can potentially infer usable portions of weights via:
- Distillation: Training smaller, cheaper models to mimic the behavior of frontier models, bypassing the need for exact weight reconstruction.
- Quantization: Utilizing attacks where only a fraction of bits (e.g., 4 out of 16) is needed for functional replication.
- Mixture of Experts: Exploiting models where only specific experts are active per query, reducing the amount of data required to reconstruct a functional subset.
- Defense Recommendation 1: Access Control: Labs must centralize weights, strictly limit full read access (avoiding hundreds of users with full permissions), and enforce monitoring to prevent trivial copying (e.g., Ctrl+C/Ctrl+V).
- Defense Recommendation 2: Hardened Interfaces: Internal access should be restricted to pre-approved code paths (inference only) or enforced with output rate limits (e.g., 100 bits per second) to prevent data exfiltration while maintaining necessary flexibility for research.
- Defense Recommendation 3: Confidential Computing: Deploying Trusted Execution Environments (TEEs) where weights are decrypted only inside a secure chip (e.g., NVIDIA H100) to prevent extraction during use; however, this does not protect against side-channel attacks, supply chain compromises, or distillation via API access.
- Defense Recommendation 4: Red Teaming: Organizations must conduct external, incentivized red teaming exercises where attackers are given privileges (e.g., zero-day simulation) to test for undetected breaches, rather than internal teams conducting superficial checks.
- Policy and Talent Gap: There is a critical global shortage of personnel capable of designing security architectures for future AI systems and developing informed policies, with experts identifying this as the primary bottleneck in AI safety.
- Non-AI Work Impact: Sela Nevo also led a Google Research project using LSTM neural networks and physics-hybrid models to predict floods with 50-meter spatial accuracy across 80 countries, significantly reducing flood-related injuries and costs in Africa and Asia.