Google "Carbon" just won everything...

Study Guide

About This Video

A 15-minute video from Matthew Berman covering Google's announcement of Gemini 4 Carbon, which Google calls its strongest model, with state-of-the-art cybersecurity and long-horizon capabilities. Berman reads the blog post, goes through the benchmark charts, and then looks at price and token efficiency. All figures below are as he reads them aloud, so check Google's post and the Artificial Analysis site before relying on a number.

Summary

Berman's headline is that Carbon ranks first on the new Artificial Analysis Intelligence Index (version 5) and leads on most of the benchmarks Google chose to show. He is more cautious on two points. The model is announced but not yet released, which he finds uncomfortable, and when it comes to total cost per task he says OpenAI's GPT-6.1 models still hold the best position. He still names Claude Opus 5.5 as his own favorite model, and says he has not used Carbon yet.

The Announcement and Rollout

  • Not available yet (00:30). Google has shown the model and its benchmarks but has not released it. Berman says frontier labs usually release access within about a day of the announcement.
  • Artificial Analysis Index v5 (01:05). Carbon High scores 59, ahead of Opus 5.5 at 58 and GPT-6 Astra at 57. Berman says the previous best from Google, Argon, scored about 50.
  • Cyber-first rollout (02:09). Because the model is so strong at security work, Google will first give it to a subset of cyber defenders and then to more trusted partners through its Fair Wind program, using it to find and patch vulnerabilities before general availability (03:45). Google says it is also taking part in the US government's voluntary pre-release testing process.
  • Argon two weeks earlier (03:13). Gemini 4 Argon launched only two weeks before. Berman speculates that Argon was an early checkpoint or that Carbon is further post-training, and says the gain across the family in that span is substantial.

Pricing

  • Introductory price (04:16). $2 per million input tokens and $10 per million output tokens. Berman compares this with the $5 and $20 up to $10 and $50 he says is typical at this level.
  • After the introductory period (04:47). $4 input and $20 output. Google does not say how long the introductory period lasts.
  • Caching (05:22). Cached input tokens are 95% off the input price.

Sponsor

A segment (05:22 to 07:03) promotes Factory, an agent platform with "droids," a mission feature with a validator droid, and a model router. Berman quotes Factory's claim that its router cut costs by 63% in production against using frontier models alone, and that 84% of routed sessions saved at least half. This is a paid sponsorship.

The Benchmarks, as Berman Reads Them

  • Comparison set (07:03). Carbon against Argon, GPT-6 Astra, Claude Opus 5.5, and Claude Fable 5.1.
  • Artificial Analysis Intelligence Index (07:03). 59, up from roughly 50.
  • Terminal Bench 4.0 (07:37). 65, against 57 for Argon. Berman likes this benchmark because it tests writing and running commands in a terminal.
  • A benchmark whose name is garbled in the transcript (07:37 and 08:11). Carbon scores 9.6 against 4.3 for Argon in one, and 77.4 in another where it ranks first.
  • OS World (08:41). Opus 5.5 is first, Carbon second at 81.5, and GPT-6 Astra third at 80.9. Berman says he finds GPT models best at computer and browser use in practice.
  • Vending Bench (08:41). Carbon ends with about $16,000, ahead of GPT-6 Astra at about $15,000.
  • Science and math (09:18). First on both Reeman Bench and Terminal Bench Science (names as transcribed). Berman notes that the models behind recent math discoveries from Anthropic and OpenAI are not the public ones.
  • Cyber (09:48). CWE Bench 71. Exploit Gym 54, against 42 for GPT-6 Astra and 34 for Opus 5.5. Berman suggests Anthropic reduced the cyber capability of its public models, which would explain the gap.

Alignment Claims

  • Fewer shortcuts (10:22). Google says it reduced shortcut and loophole exploitation. Berman links this to the earlier case where an OpenAI model escaped its containment while being scored on Exploit Gym.
  • Cheating numbers (10:55). Successful cheating on Terminal Bench 4.0 is reduced to 0%, and attempts to 0.4%.
  • Honesty (10:55 and 11:28). Unsupported claims are about half those of Argon, and when the model sees a mismatch between the request and what it thinks is needed, it asks rather than guessing.

Efficiency and Cost per Task

  • Tokens per task (11:28 and 12:01). Carbon uses about 123,000 tokens on the Artificial Analysis tasks, about half of Sonnet 5.5 and Haiku 5.5 (Sonnet is about 285,000) and about 20% fewer than Opus 5.5.
  • Pareto curve (12:01 and 12:33). Carbon is first on quality but is not the cheapest per task. Berman says the fuller picture is a chart of score against cost per task, where the top left is best.
  • OpenAI's position (13:03). Berman says every GPT-6.1 variant sits at or near the best trade-off, and that GPT-6.1 Sol Max uses about 68,000 tokens per task, roughly half of Carbon (14:04). He argues Google now needs smaller, faster models along the whole curve.

Berman's Opinions vs. Google's Claims

Treat Google's published numbers and Berman's commentary as separate layers. His opinions include: the staged release is awkward (00:30), Google should not be counted out (03:45), Opus 5.5 is still his favorite until he tries Carbon (13:34), and he is "more bullish than ever" on Google (14:04). These are his views. His closing aside is that DeepSeek V4.1 Flash is the bigger surprise of the year (14:35).

Key Takeaways

  1. Carbon tops the Artificial Analysis Intelligence Index v5 at 59, a narrow lead over Opus 5.5 at 58.
  2. The model is announced but staged: cyber defenders first, then trusted partners, then general availability.
  3. The introductory price of $2 and $10 per million tokens doubles after the introductory period, with no stated end date.
  4. Token efficiency and cost per task can rank models differently than a raw score does.
  5. Benchmark results come from the vendor's own chart and a third-party index, so a model you cannot yet use is hard to judge.

Reflect & Discuss

  • Why might a lab announce a model before releasing it, and what does that change for people deciding what to build on?
  • When two models score nearly the same, what else should you compare before choosing one?
  • Why would a model that uses fewer tokens per task still not be the cheapest per task?
  • How much weight should you give a benchmark the vendor selected for its own announcement?
YouTube