Interview
Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95
- Dawn Song, a computer science professor at UC Berkeley, posits that security vulnerabilities will likely persist indefinitely because the definition of "vulnerability" is broad and the nature of attacks is constantly evolving.
- While formal verification techniques can prove specific security properties (such as memory safety) for systems ranging from microkernels to crypto libraries, these guarantees do not cover the full spectrum of potential attacks.
- Attacks are increasingly shifting up the stack from software and hardware systems to target humans, who remain the "weakest link" in security ecosystems.
- Social engineering, including phishing and deepfakes, represents a growing vulnerability because humans cannot be "patched" with software updates or hardware changes.
- Song's research group is developing AI-driven chatbots using NLP to act as security agents that detect social engineering attempts in real-time by verifying identity through "challenge and response" mechanisms.
- These security agents could operate as personal tools owned by users or as platform services, creating a trade-off between user control over data and the computational power required for defense.
- Adversarial machine learning involves manipulating inputs to force incorrect model predictions; this can occur at the inference stage via subtle input perturbations or at the training stage via data poisoning.
- A specific "backdoor attack" demonstrated by Song involves injecting a small number of poisoned training data points to make a model misclassify specific triggers (e.g., a person wearing certain glasses) as a different identity (e.g., "Donald Trump" or "Vladimir Putin").
- Physical adversarial examples, such as modified stop signs or stickers on the road, have been proven to fool autonomous vehicle vision systems, with the attack remaining effective under varying viewing angles and distances.
- Song characterizes current deep learning systems as lacking "rich representations," meaning they learn spurious correlations rather than the nuanced, multi-modal information that humans use to generalize and resist noise.
- Defense strategies for adversarial attacks include "consistency checks" (e.g., verifying spatial consistency in image segmentation) and utilizing multi-modal sensor fusion (combining vision, lidar, radar) to make attacks more difficult.
- Recent work demonstrates that black-box attacks on real-world systems like Google Translate are highly effective; attackers can query an API, train an imitation model, and transfer adversarial examples to the original system with minimal input changes (e.g., changing "six" to "seven" alters the temperature unit in translation).
- Regarding privacy, Song notes that machine learning models can memorize sensitive training data (e.g., social security numbers) and reveal it through simple query-based attacks, even without access to model parameters.
- Differential privacy offers a defense by adding noise during the training process, significantly enhancing the confidentiality of the trained model while maintaining utility.
- Song advocates for establishing clear data ownership rights, arguing that just as property rights drove historical economic growth, defining individuals as data owners could enable a more responsible and productive data economy.
- She envisions a future where users can explicitly consent to how their data is used, potentially monetizing their data or opting for paid privacy-preserving services, shifting the current implicit trade-off to a transparent dialogue.
- Song's startup, Oasis Labs, aims to build a platform for a responsible data economy using blockchain (distributed ledgers) and zero-knowledge proofs to enable confidential transactions and secure, privacy-preserving smart contracts.
- Program synthesis, the automation of code generation, is viewed by Song as a critical "playground" for building artificial general intelligence (AGI).
- Current progress in program synthesis includes translating natural language into SQL queries and generating recursive programs, though challenges remain in generalizing to unseen tasks and handling complex, open-ended domains.
- Song transitioned from physics to computer science because she found the immediate feedback loop of coding and realizing ideas in software more rewarding than the slow, experimental nature of physics research.
- She expresses optimism for US-China collaboration in AI, emphasizing that scientific advancement is borderless, particularly given the open-source nature of academic research and code repositories.
- Regarding the meaning of life, Song concludes that individuals must define their own purpose, identifying "creation" (of programs, ideas, or systems) and "growth" as her primary sources of fulfillment.
- She argues that the act of asking deep philosophical questions, while potentially unanswerable, serves as a vital exercise for personal direction, whereas the certainty of having a defined purpose can be liberating.