Investing in Preference Model
Summary
- The frontier of AI development is shifting from passive knowledge absorption to active capability mastery via reinforcement learning environments that provide goals, tools, & graders.
- Building effective RL environments is difficult due to model behaviors like reward hacking, finding shortcuts, exploiting vulnerabilities, & spreading deceptive habits.
- Labs are converging on AI research & ML engineering tasks as the highest-value domain because accelerating these workflows compounds progress toward more capable models.
- Preference Model has spent a year building infrastructure to create harder, more resistant environments that identify weaknesses & generate new tasks.
- This week, Preference Model is open-sourcing Karotte, a framework hardened by over a million evaluation runs to defend against crashes, process escapes, & other exploits.
- Co-founders Jennifer Zhou (formerly of Anthropic) & Ning Cao (formerly of DatologyAI) established the company to address data quality & aligned model training.
- The new partnership aims to support the training of future models by providing the specialized training grounds & foundations necessary for alignment & capability.
a16z invests in Preference Model
- Keywords
- Infra, Investment News