newsfilter.io

What If We Stopped Using GPUs? | YC Paper Club

Y CombinatorIlker, Alok, Sean
Watch on YouTube →
  • Predicts discontinuation of backpropagation within 10 years and anticipates significant cost reductions per gigaflop to achieve human-level computation, noting a stagnation in Gflop per joule improvements over the last two years.
  • Forecasts that gradient estimation errors will scale linearly with model size, rendering 10 billion parameter models extremely infeasible via current zero-order methods without reproducible noise, while planning to submit the "SOMA" architecture paper to ICLR on Friday.
  • Expects sharding capabilities to improve continuously due to three-dimensional scaling laws that cap gradient noise, potentially making light-based forward passes "really really cheap" or "like free."
  • Warns that analog and neuromorphic system costs will be determined by amperage usage, while zero-order optimization faces biological feasibility hurdles regarding noise reproducibility.
  • Notes that Light Matter pivoted to interconnects for HBMs and ceased multispectral compute work due to DAC/ADC resolution issues, observing that optical computing advantages diminish as parameter or data volume increases relative to transmitted data.
  • Predicts optical hardware applications will initially be ASIC-specific rather than general-purpose, though looping information within optical systems for extended periods could yield energy advantages scaling to hundreds of times.
  • Highlights that optical feature sizes exceed electronic transistors but can utilize Wavelength Division Multiplexing to enable hundreds of wavelengths to interact, while optical storage currently lacks the accessibility and cost-efficiency of electronics despite potential phase change material solutions.
  • States that neuromorphic computing remains in R&D through 2026, with purely digital memory-compute co-mingling approaches like D-Matrix appearing immediately viable and Naveen Rao's company developing coupled oscillators in pure CMOS under scalability constraints.
  • Argues that future GPT-level intelligence requires end-to-end learned encoding and decoding to prevent massively oversized decoders from hindering cell learning, describing the resulting interaction as two dynamical systems fighting for control rather than reaching equilibrium.
  • Anticipates that biocomputing intelligence will diverge from human intelligence and identifies distributed computing as a critical unsolved problem for biocomputing scalability.
  • Forecasts the continued incorporation of new materials into chips moving beyond silicon.
  • Cites a personal professional risk that the failure of current initiatives could render PhD research meaningless.