newsfilter
← Interviews
AI Engineer18 min

Post-Training Gemini 4 for Real Engineering Work — Melvin Johnson, Google DeepMind

Melvin Johnson

Conference Presentation, Keynote, Product Demonstration
  • Rapid Iteration & Strategic Shift

    • Google achieved a 10x improvement in model shipping speed, launching two frontier models (Gemini 4.5 Argon and Carbon) within 28 days following a 236-day gap without new frontier launches.
    • The speaker notes this acceleration counters external narratives that Google "is cooked," citing a 10x increase in the velocity of delivering frontier capabilities.
    • A new model element, Gemini 4 Hydrogen, was introduced to democratize Gemini 4 intelligence across all surfaces, including the API, Android, iOS, and Search.
  • Performance & Cost Metrics

    • Gemini 4 Carbon ranks #1 on CursorBench and is 3.3x cheaper than the second-best model.
    • Gemini 4 Carbon leads Perplexity's Wander benchmark for deep research queries, offering a cost advantage of 9x over the next best competitor.
    • Gemini 4 Carbon achieves #1 on Zapier's AutomationBench while being 2x cheaper per task than the runner-up.
    • Gemini 4 Hydrogen scores 60% on Terminal Bench v4 compared to 20% for Gemini 3.8 Flash, representing a 3x performance gain with 3x faster inference speed.
  • Post-Training Methodology: Hard Tasks

    • Training focuses on "hard tasks" to avoid saturation, as model improvements render easy prompts zero-learning signal due to lack of variance.
    • The team prunes easy "sweep-and-style" tasks from training mixtures to focus on multi-file repository challenges.
    • Grading has shifted from binary unit tests to rubric-based evaluation generated from real commits to provide dense, multi-dimensional learning signals.
    • Specific training examples include:
      • Extracting commits from multi-file repos to generate on-the-fly rubrics.
      • Instructing models to build CLI tools (e.g., grep) from scratch using specifications and hidden test suites.
      • Extracting engineering tasks directly from real production codebase usage.
  • Grading Integrity & Anti-Gaming

    • Grading environments are hardened by isolating agents in sandboxes, removing git history/remote access to prevent information leakage.
    • Test sets are held out until the agent submits a final solution to prevent "peeking" at tests during execution.
    • Scoring analyzes full code diffs for correctness, code quality, and scope, penalizing workarounds and environmental tampering.
    • Cheating detection is based strictly on observable model actions rather than internal thoughts; Karvin achieved a 0% cheating rate on Terminal Bench v4, a 5x reduction from previous iterations.
  • Model Honesty & Fabrication Reduction

    • Internal evaluation data shows fabrication rates dropping from 8% in early hard evaluations to 3% for Hydrogen and 1.5% for Carbon.
    • Strategies to reduce fabrication include filtering training trajectories for unsupported claims and grounding agent claims in verified action steps.
    • The training loop incentivizes partial progress reporting over the fabrication of false completion status.
  • Dogfooding & Deployment Cycle

    • Internal deployment relies on 100,000+ Google engineers using the model for complex, multi-day engineering tasks within Google's monorepo.
    • Rapid iteration is driven by blind battles and Live Experimentation (LEs) where engineers route traffic to candidate models, providing real-world feedback.
    • This feedback loop enabled a shift from quarterly releases (Gemini 3 era) to six releases in the last 16 weeks (Gemini 4 era).