AI Engineer18 min
Post-Training Gemini 4 for Real Engineering Work — Melvin Johnson, Google DeepMind
Conference Presentation, Keynote, Product Demonstration
Rapid Iteration & Strategic Shift
- Google achieved a 10x improvement in model shipping speed, launching two frontier models (Gemini 4.5 Argon and Carbon) within 28 days following a 236-day gap without new frontier launches.
- The speaker notes this acceleration counters external narratives that Google "is cooked," citing a 10x increase in the velocity of delivering frontier capabilities.
- A new model element, Gemini 4 Hydrogen, was introduced to democratize Gemini 4 intelligence across all surfaces, including the API, Android, iOS, and Search.
Performance & Cost Metrics
- Gemini 4 Carbon ranks #1 on CursorBench and is 3.3x cheaper than the second-best model.
- Gemini 4 Carbon leads Perplexity's Wander benchmark for deep research queries, offering a cost advantage of 9x over the next best competitor.
- Gemini 4 Carbon achieves #1 on Zapier's AutomationBench while being 2x cheaper per task than the runner-up.
- Gemini 4 Hydrogen scores 60% on Terminal Bench v4 compared to 20% for Gemini 3.8 Flash, representing a 3x performance gain with 3x faster inference speed.
Post-Training Methodology: Hard Tasks
- Training focuses on "hard tasks" to avoid saturation, as model improvements render easy prompts zero-learning signal due to lack of variance.
- The team prunes easy "sweep-and-style" tasks from training mixtures to focus on multi-file repository challenges.
- Grading has shifted from binary unit tests to rubric-based evaluation generated from real commits to provide dense, multi-dimensional learning signals.
- Specific training examples include:
- Extracting commits from multi-file repos to generate on-the-fly rubrics.
- Instructing models to build CLI tools (e.g.,
grep) from scratch using specifications and hidden test suites. - Extracting engineering tasks directly from real production codebase usage.
Grading Integrity & Anti-Gaming
- Grading environments are hardened by isolating agents in sandboxes, removing git history/remote access to prevent information leakage.
- Test sets are held out until the agent submits a final solution to prevent "peeking" at tests during execution.
- Scoring analyzes full code diffs for correctness, code quality, and scope, penalizing workarounds and environmental tampering.
- Cheating detection is based strictly on observable model actions rather than internal thoughts; Karvin achieved a 0% cheating rate on Terminal Bench v4, a 5x reduction from previous iterations.
Model Honesty & Fabrication Reduction
- Internal evaluation data shows fabrication rates dropping from 8% in early hard evaluations to 3% for Hydrogen and 1.5% for Carbon.
- Strategies to reduce fabrication include filtering training trajectories for unsupported claims and grounding agent claims in verified action steps.
- The training loop incentivizes partial progress reporting over the fabrication of false completion status.
Dogfooding & Deployment Cycle
- Internal deployment relies on 100,000+ Google engineers using the model for complex, multi-day engineering tasks within Google's monorepo.
- Rapid iteration is driven by blind battles and Live Experimentation (LEs) where engineers route traffic to candidate models, providing real-world feedback.
- This feedback loop enabled a shift from quarterly releases (Gemini 3 era) to six releases in the last 16 weeks (Gemini 4 era).