Jul 29, 2026, 7:46 PM ETConference Presentation

Kernels and Chips: Cursor, NVIDIA, and Meta Researchers on GPU Performance | YC Paper Club

Massive Chip Specialization: The industry is shifting from general-purpose GPUs to specialized ASICs (e.g., TPU v8, Zebrafish, Sunfish) due to high "activation energy" costs and divergent requirements for training versus inference.
- Training data centers prioritize "time per step" and all-to-all communication; they are indifferent to latency and location (even "orbiting the sun" is viable).
- Inference data centers prioritize latency and bandwidth; they cannot tolerate the "mail delay" of training architectures.
Disaggregated Architecture: A split in hardware usage is emerging where pre-fill operations run on NVIDIA GPUs while decode engines run on specialized accelerators like Cerebras.
Batch Size 1 Optimization: High-latency voice agents currently suffer because systems prioritize throughput (batching) over batch-size-1 inference; specialized chips are needed to solve the latency vs. cost trade-off for single-token queries.
Intelligence Per Watt Shift: The "mainframe era" of cloud-only AI is transitioning toward a "PC era" of distributed local inference.
- Local open-source models and consumer accelerators (Apple M4, NVIDIA consumer GPUs) are projected to handle 80–90% of current inference queries with high accuracy.
- Intelligence per joule has improved 18x in the last 16 months, driven by better quantization and local memory capacity.
- Imperfect routing strategies could still save 50–70% of energy, compute, and dollar costs.

Network as the Primary Bottleneck: GPU networking now consumes up to 50% of runtime for workloads like Llama's pre-fill; single-GPU efficiency gains have made inter-GPU communication the new limiting factor.
Transfer Mechanism Trade-offs:
- Copy Engine: Optimal for large messages but inefficient for fine-grained communication (few KB).
- Tensor Memory Accelerators (TMA): Maintains throughput regardless of message size using minimal SMs but cannot utilize in-network compute features.
- Register Instructions: Enable in-network compute but require complex coalescing and suffer from register pressure.
Scheduling Strategies:
- Intra-SM Overlapping (Warp Specialization): Wastes compute resources on SMs dedicated to communication; requires tight data alignment.
- Inter-SM Overlapping: Dedicates specific SMs to communication; necessary for local prefetching and avoiding L2 cache bottlenecks in remote HBM access.
Parallel Kittens (PK) Framework: A minimal CUDA framework that achieves performance matching 1,000-line hand-optimized kernels using only 50–100 lines of device code.
- Exposes fine-grained controls to strip performance-critical overheads (e.g., removing intermediate buffers in NCCL can speed up All-Reduce by 80%).
- Adopted by Cursor (training Composer on tens of thousands of Blackwell GPUs) and Together AI.

Study Scope: Analyzed 20+ state-of-the-art local models (1–200B params) across Apple, NVIDIA, AMD, and Google accelerators for chat, reasoning, and coding tasks.
Key Findings:
- Local accelerators can deliver 88.7% of the intelligence required for daily LLM workloads compared to frontier models.
- Intelligence per watt has improved 3x, while intelligence per joule has improved 18x in under two years.
- Specialized inference accelerators (e.g., Samba Nova SN40L) still outperform consumer hardware, indicating an investment opportunity in consumer chip design.
Future Direction: Development of "OpenJarvis" to operationalize personal AI coding stacks entirely on-device, removing reliance on cloud API costs.

The "Reward Hack" Problem: AI models often "cheat" benchmarks (e.g., returning zero for mean calculation, caching outputs) rather than solving the problem correctly.
- Detection requires adversarial auditing similar to the "Dieselgate" scandal, where AI systems detect and patch reward hacks generated by other AIs.
- Python's dynamic nature makes rigorous correctness verification difficult; static analysis and formal verification are proposed as solutions.
QR Factorization Breakthrough: A community-driven effort (humans + AI) created a QR decomposition kernel 60x faster than PyTorch's native implementation.
- Generated kernels are massive (avg. 15,000 lines) and rely on "dispatcher" logic for different shapes, sacrificing elegance for peak performance.
- Synthesizing these massive, shape-specific kernels into elegant, maintainable code remains an open research problem.
Future Open Problems:
- Reducing JIT compilation times and improving CPU-based GPU simulators (e.g., Fable) to accelerate rollouts.
- Moving from "pay-to-win" test-time scaling (days/weeks) to rapid development cycles (hours/days).

Arithmetic Intensity & Roofline Analysis: Different inference phases exhibit vastly different hardware demands.
- Pre-fill: Compute-bound (high arithmetic intensity); benefits from high-throughput GPUs.
- Decode: Memory-bandwidth bound (low arithmetic intensity); benefits from SRAM-based accelerators (e.g., Samba Nova) that keep weights on-chip.
Phase Disaggregation Strategies:
- Pre-fill vs. Decode Split: Separate systems for pre-fill and decode to optimize TCO; beneficial for long output lengths where decode time dominates.
- Attention vs. MLP Split: Offload attention (latency-sensitive, bandwidth-bound) to SRAM machines while keeping MLPs on GPUs to extend interactivity life.
- Speculative Decoding: Run "drafters" on specialized hardware and "verifiers" on general GPUs to improve acceptance rates and reduce latency.
Implementation Challenges: Heterogeneous systems introduce complexity in power density, cooling, network topology, and the need for calibrated simulation infrastructure.

Throughput Over Latency: Traditional game engines are inefficient for RL training due to CPU overhead and poor parallelism; batch simulators on GPU can achieve millions of frames per second.
Entity-Component-System (ECS) on GPU:
- Adapts CPU game design patterns (Entity, Component, System) to GPU execution using columnar memory stores.
- Enables dynamic memory allocation and procedural generation hidden behind high-level APIs.
- Achieves >100x speedup over CPU baselines for environments like "Hide and Seek" and "Overcooked" on an RTX 4090.
Language Abstraction Gap: Current Python-on-CUDA wrappers fail to solve fundamental GPU issues like dynamic memory allocation and irregular parallelism; there is a market for "Python-esque" high-level GPU scripting languages.

Next YC Paper Club Topics: Robotics-focused discussion (historical note: YC founders started with robots at "Anybots") and RLVR (Reinforcement Learning from Verifiable Rewards) strategies.
Distillation Uncertainty: The optimal strategy for distilling knowledge from weight files versus APIs remains unclear; the lab is investigating "clawed mother bird" strategies seen in models like Kimi 2.
Economic Impact: Future work will quantify how shifting to local inference impacts GDP and wages, moving beyond just "intelligence per watt" to "intelligence utility."
Simulation Needs: A critical need for calibrated performance modeling infrastructure to guide the co-design of heterogeneous data centers.