Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398

Demonstration of Codec Avatars Technology: Mark Zuckerberg and Lex Friedman conducted a remote conversation appearing as photorealistic, 3D avatars with spatial audio, designed to replicate human presence despite physical separation.

• The technology utilizes detailed body and face scans of both participants to build a computer model that collapses expressions into an efficient codec.
• This method transmits encoded expression data rather than full video streams, allowing for bandwidth efficiency while capturing subtle imperfections like freckles, asymmetry, and eye movements.
• The system successfully captured nuances often lost in traditional video calls, such as the subtle movement of eye corners and facial asymmetry, which are critical for non-verbal emotional communication.

Technical Specifications and Efficiency:

• Current creation of avatars requires a lengthy scanning process taking hours to capture extensive data.
• Meta's R&D goal is to reduce the scanning process to 2–3 minutes using a smartphone, capturing expressions and basic text to generate high-fidelity avatars.
• Future optimization may involve stitching lower-fidelity body movements with high-resolution facial tracking to balance compute demands with emotional expressiveness.

Hardware Roadmap and Mixed Reality:

• Meta plans to integrate these avatar capabilities into the upcoming Quest 3 headset, releasing in the fall at a price point of $500.
• The Quest 3 features a Qualcomm custom chipset (next-generation) and cameras that enable mixed reality (MR) by superimposing digital avatars onto the physical environment.
• The device is 40% thinner than previous models, offering improved comfort and a "pass-through" experience that is superior to the Quest Pro in MR capability.
• Hand tracking capabilities have advanced significantly, allowing for latency-free gesture recognition (demonstrated via a virtual piano tutorial) without the need for physical controllers.

Application Scenarios and Use Cases:

• Work and Meetings: The technology aims to solve remote work isolation by allowing users to feel physically present in virtual conference rooms, blending physical attendees with photorealistic digital avatars.
• Gaming: While games may eventually favor stylized aesthetics, the immediate utility lies in mixed reality fitness and social experiences where digital objects overlay physical space safely.
• Social Connection: The primary value proposition is replicating the intimacy of physical proximity, allowing users to "hang out" or share experiences (e.g., watching a sunset) remotely without travel.

Future Evolution and AI Integration:

• Creator AI Personas: Meta is developing tools allowing creators to build AI versions of themselves to interact with fans at scale, addressing the limitation of human time availability.
• AI Embodiment: Future meetings may include AI employees (e.g., an engineer named "Bob") embodied as realistic avatars alongside human holograms.
• Open Source Strategy: Meta released Llama 2 as open source; Llama 3 is currently in training but not yet ready for release, with a priority on integrating Llama 2 into consumer products first.
• AI Studio: A platform is being built to allow any user to create and customize AI personas, expanding the ecosystem of AI agents across Meta's social platforms.

Philosophical and Ethical Considerations:

• Identity and Perception: The technology raises questions regarding identity, as avatars may retain a "frozen" look from the time of scanning, potentially differing from a user's current physical appearance.
• Post-Mortem Interactions: Zuckerberg acknowledges the potential for grieving individuals to interact with AI versions of deceased loved ones but notes that this requires careful study of norms to avoid unhealthy attachment.
• Definition of Reality: The discussion frames the "real world" as a blend of physical and digital spaces, where photorealistic avatars make the digital experience as emotionally valid as physical presence.
• Regulation and Safety: Zuckerberg proposes that digital rules should focus on "damage" rather than physical laws, suggesting fewer restrictions on actions like virtual combat that would be illegal in the physical world, while addressing new risks like anonymity-induced toxicity.

Timeline and Rollout:

• The technology is currently in a research phase with a limited number of scanned users.
• A progressive rollout is expected over the next few years, moving from specialized research scenarios to mainstream integration in productivity and social apps.
• Widespread adoption of "smart glasses" as the primary form factor for mixed reality is projected to occur by the end of the decade.

Key Quotes and Sentiment:

• "I just need to work on upgrading my emotion-expressing capabilities of the underlying human."
• "Perfection isn't the thing that leads to immersion; it's like the little subtle flaws."
• "I think the real world is the combination of the physical world and the digital worlds coming together."
• "This is truly the most incredible thing I've ever seen... it feels like the future."