Interview, Other
The True Cost of Compute
- Training large language models (LLMs) involves computational scales described as "crazy," with estimates including 175 billion parameters and 350 billion floating-point operations for a single inference pass.
- The total computational effort required to train a model like GPT-3 is approximately $3 \times 10^{23}$ floating-point operations, a magnitude rarely seen in other human undertakings.
- Practical industry costs for training an LLM range in the tens of millions of dollars, significantly exceeding naive estimates of hundreds of thousands or even one million dollars.
- Many AI startups currently spend over 80% of their total raised capital on compute resources, particularly those aiming to train proprietary models.
- The cost of compute is driven by the necessity of reserving capacity for extended periods (e.g., two years), which inflates operational expenses by a factor of 12 compared to short-term rental rates.
- Training a model requires roughly six times the computational effort of inference (measured in floating-point operations per parameter), while inference typically costs a fraction of a cent per token.
- Inference costs can be mitigated for smaller models (e.g., image generation) by utilizing consumer-grade graphics cards, though training requires sophisticated, specialized hardware due to data distribution overheads.
- Founders expect the absolute cost of training to rise as the AI boom continues, while the percentage of total capital allocated to compute may normalize as companies shift toward product development and hiring.
- A data-limited constraint is emerging, suggesting that training costs may top out or decline as models approach the limit of available human-generated knowledge (estimated at only ~10% of internet data currently utilized).
- The capital barrier for training large models acts as a "speed bump" rather than an insurmountable moat, potentially allowing well-funded startups to compete given current capital availability.
- The relationship between model size and data quantity is critical; oversizing a model without sufficient data or vice versa yields diminishing returns.
- GPT-4 training data is estimated to be equivalent to the text content of approximately 100 million books, though newer models like Llama 2 are trained on significantly larger datasets (2 trillion tokens).
- Efficiency gains in training are often limited by hardware utilization, with naive implementations achieving less than 10% utilization, while optimized setups reach the tens of percent.
- Inference provisioning costs are influenced by peak capacity requirements (e.g., Monday morning traffic), forcing companies to pay for idle capacity during low-usage periods.
- The discussion concludes with the view that while hardware is becoming a determining factor for success, the scarcity of high-quality training data may be a more significant constraint than compute itself in the near future.