Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Company Growth and Organizational Culture

• Semi-Analysis has reportedly generated revenue nearing $100 million with a team of approximately 90 employees.
• The firm's internal culture blends technologists and engineers with former hedge fund professionals, creating an environment where technical feasibility and economic viability are constantly debated.
• The company is rumored to be exploring the launch of a venture fund, leveraging its established trusted brand within the semiconductor ecosystem.
• Revenue accuracy is described as variable depending on the source, though the organization's growth trajectory is confirmed as rapid.

Founder Background and Origin Story

• Dylan Patel grew up in a family business operating a motel and gas station, where he developed early problem-solving skills by profiling customers to optimize cigarette placement.
• His entry into hardware engineering began at age 11 or 12 after repairing an Xbox 360 affected by the "Red Ring of Death" hardware defect.
• By age 12, Patel was moderating technology forums for Android, Apple, Google, and hardware, tracking the shift from simple smartphones to devices surpassing PCs in architectural complexity.
• He transitioned from a background in quantitative risk trading, where he was allegedly denied a bonus despite generating millions in risk-free revenue, to founding Semi-Analysis.
• The company was formally established around Patel's 24th birthday following a personal period of homelessness (mid-2020 to present) involving travel across U.S. national parks while writing blogs.
• The founding motivation was catalyzed by a combination of personal setbacks, including the loss of his grandmother to dementia, a professional grievance at a quant firm, and the COVID-19 lockdowns.

Data Collection Methodology and Conference Strategy

• Patel attends over 40 conferences annually across the entire semiconductor supply chain, ranging from major AI events like NeurIPS to niche technical gatherings in Japan.
• He prioritizes deep technical conferences (e.g., SPI Advanced Lithography) where understanding requires years of immersion, noting that many sessions in these fields involve languages like Japanese with few English speakers.
• Key insights often come from informal conversations at these events, such as historical data regarding a 1980s chemical factory fire in Japan that caused memory prices to double.
• The firm's methodology involves building relationships with supply chain experts to uncover non-public data on costs, shortages, and vendor relationships.

InferenceX and Benchmarking Philosophy

• Semi-Analysis launched InferenceX to address the obsolescence of point-in-time benchmarks caused by rapid updates in software (PyTorch, vLLM, SGLang) and AI models.
• The platform provides "living" benchmarks that run daily on the latest hardware and models, offering a Pareto optimal curve for throughput versus latency.
• InferenceX has secured over $50 million in donated compute from partners including CoreWeave, Oracle, Microsoft, Amazon, Google, OpenAI, and NVIDIA, with potential to exceed $100 million once TPU training capabilities are added.
• The system benchmarks a wide array of models daily, including Chinese labs (Moonshot, Alibaba), open-source models (GPT-OSS, Nemotron), and various private optimization layers.
• Benchmarks are open-sourced, providing optimal container configurations so users can achieve peak performance without managing complex optimization parameters.
• Patel argues that the "throughput vs. latency" curve is the most critical metric, as different workloads require different trade-offs between speed and cost.

Market Forecasts and Strategic Outlook

• Space Compute: Patel predicts that while space-based data centers will comprise less than 1% of compute in 2030, they could account for over 50% of incremental compute growth by 2040 due to the difficulty of scaling terrestrial power grids.
• Power Demand: By 2030, major labs (OpenAI, Anthropic) alone may require over 100 gigawatts of combined power; by 2040, global demand could reach terawatt levels.
• Efficiency Gains: Intelligence per watt has improved approximately 40x annually, though hardware cost per benchmark has dropped 60x.
• Co-Design Revolution: Patel disputes that hardware or kernel optimization alone drives the most value, asserting that the largest gains (e.g., 100x improvements) come from "software-hardware co-design" where model architecture and silicon are optimized together.
• Model Architecture Divergence: Major labs are diverging in architecture; for instance, OpenAI's sparse models are optimized for NVIDIA, while Anthropic and Google's dense models may be better suited for TPUs or Tranium.
• China vs. West: Patel contends that the West does not publish the level of co-optimization achieved by Chinese labs like DeepSeek, rather than the West lacking capability.
• CUDA Moat: The CUDA software moat is becoming less relevant as model companies use AI to write custom kernels for alternative hardware, shifting competition toward hardware-specific model optimization.

Supply Chain and Infrastructure Dynamics

• Memory Bottlenecks: DRAM and NAND cell physics have seen no major breakthroughs in decades; future gains rely on stacking memory directly on the chip rather than traditional HBM stacking.
• Power Density: Silicon power density is breaking the historic 1W/mm² limit, enabling chips to exceed 1,000–2,000 watts and approaching 4,000 watts (e.g., NVIDIA Rubin Ultra).
• Energy Solutions: Patel suggests converting millions of diesel truck engines into stationary power generators for data centers as a rapid, low-tech solution to energy shortages.
• Data Center Quality: There is a significant price and utility disparity in data center construction; Google achieves 1.5x hardware density per gigawatt through advanced power management, while others struggle with grid reliability.
• Compute Crunch: A sustained crunch is expected as model capabilities (TAM) expand faster than the doubling of compute capacity.
• Neo-Cloud Viability: Neo-clouds (e.g., CoreWeave, Crusoe) exist because hyperscaler architectures (e.g., AWS Nitro) prioritize security over AI performance, and hyperscalers often lack the agility to scale compute hardware quickly.

Competitive Landscape and Geopolitics

• NVIDIA vs. TPUs: The competition is defined by hardware-software co-optimization rather than raw chip superiority; TPUs excel in specific Google-optimized architectures, while GPUs remain general-purpose leaders.
• NVIDIA Strategy: Jensen Huang actively funds "Neo-labs" and non-hyperscaler compute to maintain a multipolar world, preventing the consolidation of all AI power within hyperscalers who might stop buying NVIDIA chips.
• Cerebras Risks: Patel identifies a risk for Cerebras (specifically their fast inference mode) if models exceed their memory capacity, as large context windows are difficult to run on SRAM-based chips.
• Google's Divergence: Google is running three distinct TPU architectural designs (including partnerships with Broadcom and MediaTek), acknowledging that different AI workloads require different hardware approaches.
• Future State: The ecosystem will likely bifurcate into specialized ASICs for specific workloads and general-purpose GPUs/TPUs for research and unknown future architectures.

Personal Anecdotes and Philosophy

• Patel describes his journey as a "wrestling with a pig" scenario where he enjoys the chaos of informal technical debate.
• He maintains a disciplined financial tracking system within his company, monitoring daily token spend and ROI for internal tasks.
• He expresses frustration with the narrative that "AI has no ROI," citing the massive expansion of the Total Addressable Market (TAM) for useful AI tasks.
• Patel views the industry as a "local minima" problem where companies race to optimize specific architectures, only to find they need to leap to a different global minimum as model breakthroughs occur.