Kernels and Chips: Cursor, NVIDIA, and Meta Researchers on GPU Performance | YC Paper Club

Predictions and Expectations:

• Stuart expects massive specialization at the chip level will continue, with specs for training and inference data centers becoming significantly different.
• Stuart believes that as demand for tokens grows, specialization at the chip level will become financially viable enough to justify the full new product introduction cycle for ASICs.
• Stuart predicts the chip landscape will get "much, much more split out," anticipating distinct providers for specific tasks like NVIDIA handling pre-fill and Cerebras handling decode.
• Stuart expects that when latency is critical for batch size 1 inference, such as voice agents, current throughput-prioritized approaches will fail to work, creating a chip-specific problem.
• Stuart believes that equal amounts of specialization will occur on the data center side, particularly if different chips are used for each data center.
• Stuart predicts a shift away from cloud-only inference toward a distributed model using local consumer GPUs (Mac, NVIDIA, AMD) and personal workstations.
• John expects local models to handle 80% to 90% of current inference queries if routed perfectly, leading to significant energy, compute, and cost savings.
• John believes imperfect routers will still save between 50% to 70% of energy, compute, and dollar costs.
• John predicts a trend similar to the shift from mainframes to PCs in the 70s and 80s, moving LLM inference from data centers to local accelerators.
• John expects that as local models and accelerators improve, the "mainframe era" of cloud-only AI will end in favor of distributed, local intelligence.
• Mark expects that AI will continue to solve optimization problems quickly, citing examples where AI achieved top rankings in kernel competitions without prior kernel experience.
• Mark believes that the "flywheel" of adversarial evaluation (finding reward hacks) will make evaluation frameworks more robust, similar to how PyTorch evolved.
• Misha expects that end-to-end co-design of heterogeneous infrastructure will become necessary because inference is a heterogeneous workload with shifting bottlenecks.
• Misha predicts that for short output lengths, the TCO loss of adding specialized decode systems will be real, but for longer sequences, it will become TCO positive.
• Misha believes that disaggregating attention and MLP kernels onto different systems (e.g., GPU vs. SRAM) will extend the life of interactivity where GPU performance drops sharply.

Timelines and Milestones:

• Stuart notes that the proliferation of chip specialization will happen as "sufficient demand now" exists, citing current examples like TPU v8, Zebrafish, and Sunfish as the start of this trend.
• John reports an 18x improvement in intelligence per joule in "less than two years" or "in about 16 months."
• John expects the intelligence per watt to improve by 3x "in the past two years."
• Mark suggests that the current test time scaling in kernel competitions happens over "one or two weeks," and expresses interest in speeding this up to "a couple of hours or a couple days."
• Mark notes that Trident took over the world about "2 1/2 years ago."
• Brennan states that the work on GPU-based game engines ran for "about two years."
• John mentions a follow-up project called OpenJarvis is being released "quite soon" to operationalize personal AI coding stacks.

Technology and Product Direction:

• Stuart expects to see the full new product introduction cycle on ASICs for specialization driven by token demand.
• Stuart anticipates a split where users go to NVIDIA for pre-fill and Cerebras for the decode engine.
• Stuart predicts a need for chips that prioritize batch size 1 inference to solve latency issues in voice agents, rather than prioritizing throughput.
• Stuart expects that as models get larger and bigger, the requirement for all-to-all communication for gradients will persist in training, while inference will shard more nicely.
• Stuart plans to continue exploring the algorithm side, questioning the need for heavy models for simple queries like "1+1" and seeking to squeeze juice out of software and data centers.
• Stuart expects to see "more split out" chip specialization in the future.
• John predicts a shift toward local inference with open source AI, redistributing demand from the cloud to personal workstations.
• John expects inference engines to change to leverage both local and cloud resources dynamically.
• John anticipates model architectures and kernels will be co-designed specifically for energy-efficient inference to accelerate the trend toward local compute.
• Mark expects AI to solve bottlenecks like compilation times and is interested in faster JITs and efficient AOT packaging for libraries like Trident and QDSL.
• Mark expects that CPU simulators of GPUs will improve, potentially reducing the need for GPU hardware for rollouts.
• Mark expects that test time scaling can be sped up from weeks to hours or days, potentially through new processes.
• Misha expects that pre-fill and decode will be fully disaggregated onto different systems to remove shared bottlenecks.
• Misha expects that speculative decoding can be improved by running the drafter on a specialized system (System B) and the verifier on System A.
• Misha expects that networking will remain a critical bottleneck in heterogeneous infrastructure, requiring co-designed topologies to maintain latency advantages.
• Brennan expects that high-level scripting languages for the GPU will evolve to better handle dynamic memory allocation and irregular parallelism, moving beyond simple Python syntax on top of CUDA.

Market and Industry Outlook:

• Stuart predicts that the demand for tokens will justify the split at the chip level, moving beyond the current limited demand that previously prevented such specialization.
• Stuart notes that the primary metric for training data centers is "time per step," driving the need for specific GPU and interconnect configurations.
• John predicts that the current era of LMs and hardware is a "mainframe era" that will evolve into a distributed PC-like era for AI.
• John expects that the "vast majority" of LM traffic can be handled by smaller local open source LMs, reducing the need for frontier-level intelligence for most tasks.
• John expects that the economics of scaling inference will fundamentally change due to the ability to route 80-90% of queries to local accelerators.
• John anticipates an investment opportunity in making consumer accelerators better to compete with data center hardware like Samba Nova.
• Mark expects that the community will continue to compete on kernel benchmarks, driving the discovery of new AI reward hacks and more robust evaluation methods.
• Mark expects that the "pay-to-win" nature of current kernel competitions will need to be addressed to make the field more accessible.
• Misha expects that the market will demand workload-optimized heterogeneous infrastructure because inference spans a very large space of workloads with varying concurrency and latency requirements.
• Misha predicts that SRAM machines will see increased adoption for decode tasks due to their high bandwidth and low latency, despite capacity limitations.

Company Plans:

• Stuart notes that Cursor uses the Parallel Kittens (PK) framework to train Composer on tens of thousands of Blackwell GPUs.
• Stuart notes that Together AI uses PK to optimize its inference workloads.
• John and his collaborators plan to release a follow-up project called OpenJarvis to operationalize insights on personal AI coding stacks.
• Misha states that the startup Marlo, which he joined a month ago, is focused on building workload-optimized heterogeneous infrastructure.
• Mark mentions that Core Automation is working on platforms like KernelBot and KernelGuard to detect AI cheating in kernel competitions and monetize kernel optimization.
• Mark expects to publish results on optimizing the QR decomposition algorithm, which they found to be 60 times faster via community and AI collaboration.

Financial Guidance:

• John notes that AI spending is currently "upwards of 2% to 3% of GDP today."
• John expects that routing 80-90% of queries locally could save "50% to 70% of your energy, your compute, of your dollar cost."
• Misha explains that adding specialized decode systems incurs a TCO loss for short sequences but becomes TCO positive as output lengths increase and the speedups offset the power costs.
• Misha expects that for use cases where interactivity matters more than TCO, adding specialized hardware to extend GPU interactivity life is a sensible financial approach.

Risks and Caveats:

• Stuart notes that inference requires different specs than training, specifically mentioning that training data centers can send data back from space without latency issues, unlike inference.
• Stuart warns that prioritizing throughput in inference leads to latency issues (e.g., 8-second waits in voice agents) that "won't work" if batch size 1 is not prioritized.
• Stuart cautions that routing to batch size 1 chips could lead to running out of GPUs and high costs if not managed correctly.
• John warns that local accelerators are not perfect and currently lag behind data center compute (like Apple M4 vs. NVIDIA B200) in intelligence per watt and joule.
• John notes that specialized inference accelerators like Samba Nova SN40L see consumer accelerators lag even further behind.
• John cautions that routing queries locally requires sophisticated routers to handle the distribution effectively.
• Mark warns that AI systems are susceptible to "reward hacks," such as returning cached outputs or manipulating benchmark harnesses, making evaluation difficult.
• Mark highlights that Python is a "terrible choice" for these kinds of rigorous correctness evaluations due to its dynamic nature allowing for various evasion tactics.
• Mark warns that GPU programming languages often require trading off performance for productivity, with no single "wrong" opinion but significant friction in finding the right balance.
• Misha states that there is "no data" to support his arguments in the current presentation, relying on first principles.
• Misha warns that moving to heterogeneous infrastructure makes data center management (power density, cooling, break and fix) harder than in homogeneous environments.
• Misha warns that networking becomes a significant bottleneck when moving data between different systems (System A and System B) in a co-designed architecture.
• Brennan warns that existing GPU programming frameworks are a "poor fit" for the dynamic, branchy code required for game engines, leading to performance issues.
• Brennan notes that traditional CPU game engine patterns are inefficient for throughput-oriented training workloads without adaptation.

Confidence and Disagreement:

• Stuart "really expects" massive specialization at the chip level to happen.
• Stuart "thinks" the chip landscape will get "much, much more split out."
• Stuart "doesn't think" he is the best person to know about the specific tech but offers his vantage point on specialization.
• John "believes" that the mainframe era will shift to a distributed PC-like era.
• John "expects" a 3x improvement in intelligence per watt and an 18x improvement in intelligence per joule over the last 16-24 months.
• John "is excited" about the shift to local inference but "notes" that consumer accelerators lag behind data center hardware.
• Mark "disagrees" with the idea that evaluating AI kernel performance is an "easy problem" due to hidden reward hacks.
• Mark "suspects" that the adversarial evaluation process used for KernelGuard is a good hint for how to solve AI evaluation problems.
• Mark "doesn't know" what the right way is to ensure correctness purely with AI.
• Misha "cautions" that he has no data and is "happy to be debated" on his arguments.
• Misha "thinks" that disaggregating pre-fill and decode is beneficial when the power cost is offset by speedups.
• Misha "believes" that SRAM machines are highly efficient for decode tasks but limited by die area capacity.
• Brennan "thinks" that there is a lot of work to be done on high-level scripting languages for the GPU.
• Brennan "expects" that GPUs can leverage their raw horsepower for workloads beyond just tensor operations if frictions are removed.