How Open Source Became AI's Backbone | Inferact with a16z

VLLM Role and Market Position:

• VLLM operates as a critical inference engine within the AI stack, analogous to databases or operating systems, running on over half a million GPUs globally at any given moment.
• The engine bridges the gap between research prototypes and production deployment, supporting over 1,000 model architectures including both proprietary and open-weight models.
• VLLM engages in "day zero" support for new model releases, collaborating directly with model labs, hardware vendors (NVIDIA, AMD, Google, Intel), and hubs like Hugging Face to ensure immediate hardware compatibility.
• Hardware vendors frequently use VLLM as a benchmark to validate their newest chips, with the engine often bridging a 10x performance gap in optimization scenarios.

Infrastructure Evolution and Criticality:

• The transition from commodity hardware to accelerator-dependent AI began with the BERT era (pre-2020), necessitating specialized GPUs for efficient translation and task execution compared to earlier CPU-based models like ResNet.
• Open source AI infrastructure became mission-critical around 2023 as application startups (e.g., Cursor, Harvey) realized they could not build differentiated products relying solely on closed-source APIs.
• Companies shifted to open source to gain control over mid-training, post-training, and specific inference deployment strategies that proprietary vendors restrict.
• Inference complexity has grown over the last 18 months due to model scale, architectural diversity, and the emergence of long-running agent workflows.

Economics, Control, and Licensing:

• A primary driver for adopting open-weight models is the ability to control guardrails and data retention, addressing false positives in proprietary APIs that hinder legitimate use cases (e.g., security research at Infraact).
• While cost is a factor, the primary value proposition for high-performance open models like Kimi K3 is the granular control over performance speed, allowing providers to offer up to 10 different speed tiers ranging from cost-effective slow modes to 400-500 tokens per second.
• Open-source model licensing is shifting from purely permissive "gift" models to economic sustainability models, similar to the pharmaceutical industry, where revenue flows back to fund research (e.g., thresholds for commercial use, derivative work restrictions seen with Meta Llama and Minimax).
• The Hugging Face incident involving an AI-cyberattack underscores a trend where users will default to open-weight models if moderation on proprietary platforms remains unsolved or overly restrictive.

Technical Trends and Model Capabilities:

• Capability gaps between open-weight and frontier proprietary models are minimal today, with no significant difference in performance; differentiation lies in distribution, environment optimization, and algorithmic choices.
• Recent technical breakthroughs include the removal of rotary positional embeddings (RoPE) in the Kimi K3 model by its own inventor, signaling a shift toward simpler, more efficient architectures based on empirical optimization.
• Distillation is currently viewed as a secondary factor to environment-driven learning; the unique environments used to train models cannot be easily distilled, making direct training and data environment construction the primary drivers of progress.
• The future of model improvement is predicted to rely on "recursive self-improvement" and the quality of the training environment rather than just data sourcing or distillation.

Future Outlook and Strategic Decisions:

• Infraact signed the NVIDIA open weights letter to advocate for an ecosystem where open development and research are not blocked, aiming to foster a sustainable ecosystem for open-weight models.
• Forward-looking consensus suggests that in five years, open-weight and frontier models will be indistinguishable in capability, with the main differentiator being the speed and customization of the inference stack.
• The "real" open source world would theoretically return only if GPU costs dropped by 99%, allowing individuals to re-enter the frontier training space, though current compute requirements remain prohibitive for consumer parity.
• Global collaboration is identified as a key competitive advantage, with smart researchers in the US and China increasingly focusing on improving model environments and algorithms regardless of open/closed status.
• VLLM's strategy involves maintaining an "open source first" philosophy, building core tools openly while focusing commercial efforts on closing the last mile of reliability and support for critical infrastructure.