Conference Presentation, Panel
Going In Deep On Data | YC Paper Club
- Market Validation of Data: The VC community has shifted its view on data from a terminal value of $0 to a sector creating over $100 billion in market cap in the last decade, contradicting early skepticism regarding data businesses like Scale.ai.
- Architectural vs. Data Bottleneck: While early research (e.g., Andrej Karpathy's PhD) allocated 95% of time to architecture, the industry standard in production has flipped to 97% of effort focused on data, particularly with the advent of expressive architectures like Transformers.
- Production vs. Training Distribution: Models trained on clean datasets (e.g., ImageNet) frequently fail in production when encountering out-of-distribution data, such as fogged refrigerators or obstructed views, which architecture changes alone cannot resolve.
- Human Limitations: Current models struggle with tasks where human experts also fail; for instance, distinguishing between similar dog breeds constitutes a ~30-40% error rate in human identification, a signal that data noise cannot be fully corrected by algorithmic improvements.
- Data as Products: Data and RL environments are no longer simple zip files but complex products requiring significant craftsmanship; for example, Focal Systems spent extensive effort distinguishing between "out of stock" and "product pushers" to prevent model confusion.
- Dynamic Data Requirements: High-quality initial datasets are insufficient because external factors, such as UI updates in Salesforce, render captured traces obsolete, necessitating continuous data acquisition to maintain model performance.
- Financial Model Failure: Despite high compute and advanced architectures, no AI model currently outperforms random chance in predicting next 7-day S&P returns, indicating a lack of high-quality, expert-verified financial data.
- App Ecosystem Analogy: The speaker proposes that data companies function like mobile apps; just as there is no "Instacart for Apple" and "Instacart for Android," domain-specific capabilities (e.g., AI doctor, AI lawyer) should be provided by specialized data companies rather than general-purpose model providers.
- Scaling Expertise (Data 2.0): Snorkel's Frontier Lab thesis focuses on "scaling expert supervision," moving from manual labeling to encoding expert judgment into software to handle complex, high-stakes domains like medicine and law.
- Data Programming Evolution: Snorkel's methodology evolved from manual labeling (Data 1.0) to "data programming," where expert rationales are encoded as labeling functions and denoised via weak supervision models to generate scalable, reproducible datasets.
- Senior SWE-bench Introduction: Snorkel introduced "Senior SWE-bench," a benchmark requiring data creation that mirrors the work of senior engineers, including open-ended user stories and nuanced reward systems beyond simple code correctness.
- Validation Agents: To scale expertise in coding evaluation, Snorkel developed "validation agents" that synthesize deterministic tests based on high-level user stories, balancing the reliability of unit tests with the flexibility of LLM judges.
- Benchmark Results: On the Senior SWE-bench, Fable, Opus, and Sol models are tied for first place, introducing a "tasteful pass" metric that evaluates code based on bloat minimization and alignment with senior engineer practices rather than mere mergeability.
- Diffusion Language Models: Inception Labs' Mercury 2 model utilizes diffusion architecture to generate tokens in parallel, achieving inference speeds exceeding 1,000 tokens per second, significantly outpacing sequential token-by-token models.
- Latency-Quality Frontier: Mercury 2 establishes a new Pareto frontier for voice agents, simultaneously improving quality and reducing latency compared to cascaded architectures (STT-LLM-TTS) even on specialized hardware like Cerebras chips.
- TauForge Synthetic Environment: Inception's TauForge system synthesizes realistic RL environments and tasks based on real-world logs and business knowledge graphs, overcoming the limitations of static benchmarks like TauBench which lack real-world complexity.
- Performance Gains from Synthetic Data: Training on tasks synthesized via TauForge improved a model's accuracy on banking and travel domains by over 23% compared to pre-training without domain-specific data, without overfitting to the specific test set.
- Multilingual Synergy Research: MIT/Google research reveals that scaling laws are 99% English-centric, yet performance on low-resource languages (e.g., Thai) can be improved by mixing in "synergistic" high-quality data from unrelated languages.
- Cross-Lingual Transfer Matrix: Empirical analysis generated a transfer matrix showing that language synergy is not symmetric (e.g., Indonesian helps Spanish more than Spanish helps Indonesian) and that script similarity often outweighs language family lineage.
- Model Size and Interference: Small models exhibit high negative transfer (interference) when mixing languages, whereas larger models can accommodate diverse linguistic patterns, turning negative transfer matrices positive through increased capacity.
- Optimized Scaling Laws: Researchers developed a modified Chinchilla scaling law that incorporates a transfer weight factor ($\tau$) derived from the cross-lingual matrix to predict optimal data mixes and model sizes for specific target languages.
- Curse of Multilinguality: The research quantifies the data requirements needed to maintain performance when expanding a model from a few languages to many, providing practitioners with formulas to calculate necessary data scaling factors.
- Speaker Introductions: The session featured Vincent Chen (Snorkel, Frontier Lab), Volo (Inception Labs), and Shane (MIT PhD/Anthropic), with May (originally scheduled) unable to attend.
- Data Provenance Initiative: Shane's recent work focuses on data provenance, emphasizing the need for audit trails regarding how specific data sources contribute to model capabilities and risks.