How To Build Generative AI Models Like OpenAI's Sora

OpenAI Sora Capabilities and Limitations

• Sora successfully integrates Transformer and Diffusion models with a temporal component using "space-time patches" (3D matrices of pixels across space and time).
• The model demonstrates accurate long-term visual consistency, maintaining architectural and environmental continuity throughout minute-long videos.
• Text rendering in Sora is a significant breakthrough, with the model capable of spelling words correctly, a known failure point for previous image models like Stable Diffusion.
• Physical simulations are highly advanced but imperfect; specific glitches include a "shuffling" robot walk, a "floating" dog-like object, and disjointed bridge columns.
• Geographical and traffic logic errors persist, such as the Golden Gate Bridge clip showing cars driving on the wrong side of the road and inaccurate terrain topography.
• Fluid dynamics remain partially "wonky," with some water waves appearing static rather than fluid, though other clips show superior motion capture.
• Computational requirements are estimated to be an order of magnitude higher than GPT-4, potentially involving 10 trillion parameters and 10x more GPUs.
• Training data likely includes synthetic video generated by physics engines like Unreal Engine or Unity to enable multi-camera views and simulate real-world physics.

YC Companies Achieving "Magical" Results on Minimal Resources

• Infinity AI: Built an AI replica of Elon Musk and generated video of him speaking by training on only one hour of YouTube footage.
• Sync Lab: Created a real-time lip-syncing API for non-Hindi speakers to appear to speak Hindi using a single A100 GPU by compressing data to low-resolution video.
• Sonato: Developed a text-to-song model capable of mimicking specific artists (e.g., Taylor Swift) in two months by 21-year-old founders with no prior ML expertise.
• Metalware: Built a hardware design copilot using GPT-2.5 (1 billion parameters) by leveraging high-quality, scanned textbook data rather than massive datasets.
• Find: Created a software copilot that outperforms Stack Overflow by generating high-quality synthetic data for programming competitions.
• GuyLab: Developing an explainable foundation model to address the "black box" nature of deep learning by allowing users to understand output logic.
• Playground: Suhail Doshi pivoted to image generation, releasing Playground 2.5 that rivals Stable Diffusion despite a non-AI background and limited funding.

Strategic Hacks for Building Foundation Models

• Data Efficiency: Startups can succeed by using smaller datasets with higher quality, such as synthesizing data or scanning specific technical texts, rather than relying on massive, low-quality public datasets.
• Compute Optimization: Companies can utilize YC's dedicated GPU clusters (over $500k in credits) for instant access, bypassing the need to purchase hardware or wait for cloud availability.
• Model Size Reduction: Specific tasks do not require trillion-parameter models; using smaller architectures like GPT-2.5 can be viable if the data is domain-specific and high-fidelity.
• Synthetic Data Validity: Contrary to earlier skepticism about "circular" training (models training on their own output), synthetic data is now proven effective, particularly for robotics and simulation where real-world data is scarce.
• Expertise Accessibility: Technical expertise is not a prerequisite for success; founders can achieve cutting-edge results by reading academic papers (e.g., the 2020 "Visual Transformer" or 2018 "World Model" papers) and self-studying.

Future Implications of Physics-Simulating AI

• Weather Prediction: The startup Atmo built a foundation model using machine learning to simulate weather physics, achieving accuracy superior to NOAA's billion-dollar supercomputer model at a fraction of the cost.
• Drug Discovery: Diffuse Bio utilizes AI to simulate protein functions, accelerating the creation of new molecules and gene therapies.
• Neurology: Pyramidal is applying space-time chunking techniques to EEG data to predict stroke and eventually read brain signals.
• Robotics and CAD: Kscale Labs is applying physics simulation to consumer humanoid robots, while Draft8 is using AI to shortcut the complex polynomial calculations in CAD design (forces, shear, tolerances).
• Broader Utility: The technology shifts AI from mere content generation to functional simulation of real-world systems, enabling applications in biology, engineering, and scientific discovery.

Narrative on Startup Trajectories and OpenAI

• OpenAI's path was not linear; early efforts included failed reinforcement learning robotics projects before discovering the transformer architecture.
• The current AI landscape allows new entrants to compete with giants like OpenAI or Midjourney by focusing on vertical specialization and leveraging YC resources.
• Founders are encouraged to learn AI fundamentals rapidly, noting that the field's newness allows individuals to reach the cutting edge within months of dedicated study.