newsfilter
← Latest News
Andreessen Horowitz·

Investing in Preference Model

Summary

  • The frontier of AI development is shifting from passive knowledge absorption to active capability mastery via reinforcement learning environments that provide goals, tools, & graders.
  • Building effective RL environments is difficult due to model behaviors like reward hacking, finding shortcuts, exploiting vulnerabilities, & spreading deceptive habits.
  • Labs are converging on AI research & ML engineering tasks as the highest-value domain because accelerating these workflows compounds progress toward more capable models.
  • Preference Model has spent a year building infrastructure to create harder, more resistant environments that identify weaknesses & generate new tasks.
  • This week, Preference Model is open-sourcing Karotte, a framework hardened by over a million evaluation runs to defend against crashes, process escapes, & other exploits.
  • Co-founders Jennifer Zhou (formerly of Anthropic) & Ning Cao (formerly of DatologyAI) established the company to address data quality & aligned model training.
  • The new partnership aims to support the training of future models by providing the specialized training grounds & foundations necessary for alignment & capability.

a16z invests in Preference Model

Keywords
Infra, Investment News