Y Combinator1h 21m
What If We Stopped Using GPUs? | YC Paper Club
- Predicts discontinuation of backpropagation within 10 years and anticipates significant cost reductions per gigaflop to achieve human-level computation, noting a stagnation in Gflop per joule improvements over the last two years.
- Forecasts that gradient estimation errors will scale linearly with model size, rendering 10 billion parameter models extremely infeasible via current zero-order methods without reproducible noise, while planning to submit the "SOMA" architecture paper to ICLR on Friday.
- Expects sharding capabilities to improve continuously due to three-dimensional scaling laws that cap gradient noise, potentially making light-based forward passes "really really cheap" or "like free."
- Warns that analog and neuromorphic system costs will be determined by amperage usage, while zero-order optimization faces biological feasibility hurdles regarding noise reproducibility.
- Notes that Light Matter pivoted to interconnects for HBMs and ceased multispectral compute work due to DAC/ADC resolution issues, observing that optical computing advantages diminish as parameter or data volume increases relative to transmitted data.
- Predicts optical hardware applications will initially be ASIC-specific rather than general-purpose, though looping information within optical systems for extended periods could yield energy advantages scaling to hundreds of times.
- Highlights that optical feature sizes exceed electronic transistors but can utilize Wavelength Division Multiplexing to enable hundreds of wavelengths to interact, while optical storage currently lacks the accessibility and cost-efficiency of electronics despite potential phase change material solutions.
- States that neuromorphic computing remains in R&D through 2026, with purely digital memory-compute co-mingling approaches like D-Matrix appearing immediately viable and Naveen Rao's company developing coupled oscillators in pure CMOS under scalability constraints.
- Argues that future GPT-level intelligence requires end-to-end learned encoding and decoding to prevent massively oversized decoders from hindering cell learning, describing the resulting interaction as two dynamical systems fighting for control rather than reaching equilibrium.
- Anticipates that biocomputing intelligence will diverge from human intelligence and identifies distributed computing as a critical unsolved problem for biocomputing scalability.
- Forecasts the continued incorporation of new materials into chips moving beyond silicon.
- Cites a personal professional risk that the failure of current initiatives could render PhD research meaningless.