newsfilter.io

The Problem With Testing AI Architectures at Small Scale | Jerry Tworek, Core Automation

Sequoia CapitalJerry Tworek
Watch on YouTube →
  • Core Automation Mission & Belief: The organization challenges the prevailing industry practice of validating architectures on small datasets and limited compute before attempting to scale.
  • Critique of Current Scaling Practices:
    • The prevailing approach prioritizes testing on small-scale environments first, deferring evaluation of scalability until a model proves itself in that regime.
    • This methodology is deemed insufficiently scaled for effective architectural research.
  • Specific Challenge in Reinforcement Learning (RL):
    • Meaningful results and observable capabilities in RL models require a specific baseline of compute; lower compute regimes fail to yield interesting outcomes.
    • RL systems possess a minimum threshold of ability necessary to function effectively, which small-scale compute cannot provide.
  • Forward-Looking Hypothesis:
    • The author posits that many promising architectural designs may require a foundational level of compute to activate and demonstrate their potential utility.