OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

OpenAI released GPT-OSS, its first open-weight model since 2019, available in two sizes: a 120 billion parameter version and a 20 billion parameter version.

GPT-OSS utilizes a Mixture of Experts (MoE) architecture where only the top four experts are activated per token to optimize inference efficiency.

The model supports a 131,000 token context window achieved by applying YARN scaling during the pre-training phase rather than at inference time.

GPT-OSS is trained as a decoder-only transformer incorporating Grouped Query Attention (GQA), SwiGLU activations, Rotary Positional Embeddings (RoPE), and pre-normalized RMSNorm.

OpenAI employed the O200K Harmony tokenizer, a byte-pair encoding system with over 200,000 tokens, and trained the model on a trillions-of-tokens text corpus focused on STEM, coding, and general knowledge.

The model was released exclusively in a quantized format to enable deployment on consumer-grade hardware, with no unquantized version currently available.

GPT-OSS underwent extensive post-training for safety and alignment, though community members are exploring ways to strip these layers to access raw model capabilities.

Alibaba Cloud released the Qwen 3 family in April, featuring both dense models (ranging from 0.6B to 32B parameters) and Mixture of Experts models (up to 235B parameters).

Qwen 3 sparse models incorporate 128 total experts with 8 activated per token and utilize QKNorm to dynamically rescale query and key vectors, replacing the static QKV bias used in Qwen 2.5.

The dataset for Qwen 3 comprises 36 trillion tokens, double the volume of Qwen 2.5, combining multilingual data, STEM sources, and synthetic data generated by previous models.

Training occurred in three stages: a general phase on 30 trillion tokens across 119 languages, a reasoning phase on 5 trillion high-quality STEM/coding tokens, and a long-context phase extending sequence length to over 32,000 tokens.

Qwen 3 employs a four-step post-training pipeline including a "long chain of thought" cold start, a reasoning RL stage using GRPO on approximately 4,000 query-verifier pairs, "thinking mode fusion," and general RL.

"Thinking mode fusion" allows users to toggle between reasoning-heavy and standard modes within a single model by fine-tuning on mixed datasets of intermediate and direct outputs.

Qwen 3 implements inference-time YARN scaling (increasing rope base frequency by a factor of 4) to reach 128,000 token contexts without additional long-context retraining.

DeepSeek released V3, a 671 billion parameter MoE model where 37 billion parameters are active per token, and subsequently updated it to V3.1 with hybrid thinking modes and improved tool use.

DeepSeek V3 utilizes Multi-Head Latent Attention (MLA) to compress key-value pairs into a smaller latent space, offering superior memory efficiency compared to GQA for long-context tasks.

The V3 architecture was trained natively in 8-bit precision to significantly reduce training costs, and V3.1 adds a two-phase long-context training approach.

DeepSeek's long-context capabilities were developed via a staged fine-tuning approach, first reaching 32,000 tokens and then extending to 128,000 tokens.

Across the three model families, GPT-OSS applies long-context scaling natively during pre-training, DeepSeek uses staged post-training fine-tuning, and Qwen 3 relies on inference-time YARN scaling.

Qwen 3 is the only family offering both dense and MoE variants, whereas DeepSeek V3 and GPT-OSS are exclusively MoE architectures.

DeepSeek V3 remains the largest model by parameter count (671B), exceeding Qwen 3's largest MoE model (235B) and GPT-OSS's largest variant (120B).

Qwen 3's RL post-training achieved strong results using only 4,000 query-verifier pairs, a notably small dataset size for effective reasoning alignment.

All three models utilize similar foundational components (GQA/MoE, RoPE, RMSNorm) but achieve comparable benchmark performance through distinct architectural choices and training methodologies.

The specific engineering of datasets remains opaque across all labs, suggesting that data engineering is a primary competitive moat distinct from architecture.

Current model development is characterized by empirical findings rather than first-principles justifications for specific architectural choices like MLA versus GQA.