Apr 18, 2024, 12:00 PM ETInterview, Podcast

Mark Zuckerberg — Llama 3, $10B models, Caesar Augustus, & 1 GW datacenters

Meta is rolling out Llama 3 as open source to the developer community and as the foundation for the new Meta AI assistant.
The Llama 3 lineup includes an 8 billion parameter model and a 70 billion parameter model, both released today; a 405 billion parameter dense model is currently in training and not yet released.
The 70B model achieves 82 MMLU, leading in math and reasoning benchmarks, while the 8B model is nearly as powerful as the largest Llama 2 model.
The 405B model is currently performing at approximately 85 MMLU during training, with expectations to lead on future benchmarks.
Meta AI is being integrated prominently across WhatsApp, Instagram, Facebook, and Messenger with a unified search box for questions.
New Meta AI features include real-time image generation that updates as the user types, and image animation capabilities.
Meta AI integrates Google and Bing for real-time knowledge retrieval to supplement its internal model data.
Future Llama 3 updates will focus on multi-modality, multi-linguality, and larger context windows.

Meta began acquiring H100 GPUs around 2022 to scale capacity for Reels and recommendation algorithms, a move driven by the need to compete with TikTok's content discovery capabilities.
The strategy involved doubling GPU procurement beyond immediate needs to prepare for unforeseen AI infrastructure requirements.
Meta expects to reach 350,000 GPUs by the end of the year, with clusters ranging from 22,000 to 24,000 nodes.
Meta's inference compute requirements significantly exceed training requirements due to the massive scale of its user base.
Current data centers typically operate at 50–150 megawatts; 300–500 megawatt or gigawatt-scale clusters have not yet been built.
Mark Zuckerberg identifies energy acquisition and permitting as the next major bottleneck, noting that building gigawatt-scale power infrastructure takes years due to regulatory hurdles.
Meta is developing custom silicon to eventually handle both inference and training, having already moved ranking and recommendation tasks to their own chips to reserve H100s for training.

Llama 3 was explicitly trained on extensive coding data to improve general reasoning and rigor, even if users primarily use the model for non-coding tasks.
The company is shifting focus from simple chatbots to multi-step agent behaviors, where AI performs complex, iterative tasks rather than just answering questions.
Meta views emotional understanding as a distinct modality requiring specialized training beyond standard text or image processing.
Future AI development will likely utilize synthetic data generation (inference-based) to augment training datasets, potentially shifting the balance between training and inference workloads.
Scaling laws are expected to continue, but physical constraints (energy, hardware) will eventually limit the pace of exponential growth.
The company acknowledges that open source models like Llama 2 can be further improved by other organizations, but major performance leaps still require new architectures and larger training runs.

Meta advocates for open source distribution of models to prevent the concentration of super-intelligent AI in a few closed entities, which they view as a greater security risk than widespread access.
Zuckerberg argues that a "super-strong" AI in the hands of a single adversarial actor or hostile nation-state poses a threat comparable to asymmetric warfare.
Meta's license allows permissive use but restricts the largest companies from reselling the model without negotiation, ensuring Meta shares in the upside of commercial deployment.
Primary safety focus remains on mitigating tangible harms (misinformation, fraud, violence) rather than abstract existential risks, though the company retains the right to restrict open sourcing if qualitative risks emerge.
Meta believes that widespread open source deployment allows the ecosystem to harden security faster through collective scrutiny, similar to the Open Compute Project.
Mark Zuckerberg expresses concern about API-controlled ecosystems (like Apple or Google) dictating what developers can build, preferring open standards where possible.

Zuckerberg attributes Meta's willingness to invest heavily in unproven ventures like the Metaverse and AI to a personal inability to stop building new things.
He cites Augustus Caesar's redefinition of "peace" as a historical parallel for shifting industry paradigms from military conflict to economic stability.
Zuckerberg views AI as a fundamental technological shift comparable to the invention of computing, fundamentally altering how humans work and create.
He distinguishes between intelligence and consciousness, believing current AI trajectories can separate the two, making AI a tool rather than a sentient entity.
The company aims to democratize digital presence through the Metaverse, allowing people to feel physically co-located remotely, though this remains a long-term vision compared to AI adoption.
Zuckerberg notes that organizational focus is often scarcer than capital for large companies, requiring CEOs to actively manage the "main thing" to avoid distraction.