They actually did it... (Kimi K3.1)

Study Guide

About This Video

A 25-minute video from Matthew Berman testing Kimi K3.1, a new open weights model he calls frontier class. He paid for a Kimi subscription, used the Kimi desktop app, ran a series of build tests on K3.1 Pro at high reasoning effort, and then walked through the published benchmarks, pricing, and independent cost data.

Summary

Berman finds Kimi K3.1 very good, especially at 3D work, and says it is close to the frontier but not near Opus 5.5 or GPT 6.1. It aced a hard Rubik's Cube test, struggled with a rope knot in Blender, and needed repeated nudges on simpler tasks. Its published benchmark scores are strong and its price per completed task is low. He is happy to see open weights models this capable.

The Tests, in Order

  1. 3D Rubik's Cube (00:58). Berman expected this test to be dead, but says open weights models usually fail it (colors vanish, rotations make no sense). Kimi produced a truly 3D cube with reflections, keyboard control, a timer, undo, adjustable turn speed, and a solver that pre-calculates its moves and plays them out with a confetti finish. He calls it an absolute pass, "one of the best."
  2. iPhone Duo in WebGL (04:02). Nicely rendered with good metallic reflections, but not accurate (the logo looked off and the split screen was missing). The run took 28 minutes 45 seconds. He sent a follow-up asking it to research real specs and measurements (05:01) and says he will show the result later; the transcript does not include it.
  3. Blender rope knot (07:15). A test of computer control, web search, and spatial awareness. The overall figure-eight shape was there, but it was not knotted properly or attached to the harness. Berman calls it a failure.
  4. Knot motion graphic (08:40). The first version (5.5 minutes, 28 seconds, no sound) could not teach the knot because overlaps were unclear. GPT 6.1 on medium effort made a clearer 54-second version where over and under were visible. A prettier second Kimi version still did not fix it.
  5. Fall Guys clone (12:33). After 48 minutes, "Bean Blitz" arrived with 12 competitors, sweepers, gates, bouncing pads, checkpoints, and touch controls. Colors were too pastel, with weak collisions and flickering, but decent for an open weights model.
  6. Website redesign (14:33). Followed brand colors and fonts but felt like generic AI design with incorrect spacing.
  7. YTP video (15:17). A five-minute parody video built with Python orchestration and Kimi's own video generation API (16:11).
  8. Benchmark graph (17:05). Needed several nudges. See the lesson below.

Lessons From the Tests

  • Short prompts reveal taste (13:02). Berman keeps prompts short so the model makes assumptions and shows its taste. When you have strong preferences (colors, animations, level design, technology), that is when to add detail (13:43).
  • Kimi needs nudging (16:59). It did not list specific benchmark names until asked, then admitted its numbers were invented placeholders (17:56), and only produced the final image after another prompt. The result mostly recolored Kimi's own published chart.
  • Safety warnings (09:02, 11:15). Kimi added a "not a substitute for instruction" warning to the climbing knot guide, and Berman notes ChatGPT does the same. He says Kimi behaves much like ChatGPT and Claude models.
  • Eyebrow text (01:42). Kimi uses a lot of it, a pattern he associates with the Claude family. He mentions distillation accusations against Kimi but leaves that for another video.

Model Specs (as Stated in the Video)

  • Open weights, described as 3 trillion parameter class (19:00).
  • One million token context.
  • Mixture of experts: 2.8 trillion total parameters with 104 billion active (spoken figures; verify against Kimi's blog post).
  • Multimodal: accepts images. Berman has not tested video, and thinks audio is likely supported.
  • Kimi claims strength in full project coding, parametric 3D modeling in Blender and CAD, and finance work, with hallucination down 65% (their claim, not verified by Berman).

Benchmarks Berman Highlights

  • Terminal Bench (20:41). Opus 5.5 at 66.4, Kimi K3.1 Pro at 65.7.
  • A second agentic benchmark (20:57). Kimi 51.7 versus Opus 5.5 at 46.3, so Kimi wins. The benchmark name is garbled in the transcript.
  • Terminal Bench Science (21:10). GPT leads and Kimi scores about half as much.
  • GDP value (21:26). Opus 5.5 at 1866 Elo, Kimi K3.1 at 1846.
  • OS World (21:55). Kimi at 79.6, in line with Opus and Fable. Berman personally finds GPT best at computer and browser use.
  • Artificial Analysis intelligence index (24:08). Kimi K3.1 Pro lands just behind Opus, Sonnet, and Meta's Muse model, and ahead of Fable 5.1 and GPT 6. Spoken scores are garbled, so check the live index.

Berman's read: Kimi crushes benchmarks, yet his own tests show it is not near Opus 5.5 or GPT 6.1, which makes him wonder whether it is partly based on those models (18:36). That is his speculation.

Pricing and Cost per Task

  • Regular speed (22:45). $2 per million input tokens, $10 per million output tokens, 20 cents per million cached input tokens.
  • High speed (23:07). $4 input, $20 output, with cached input unchanged at 20 cents. Two to three times faster.
  • Cost per intelligence task (24:45). This metric combines token price with how many tokens a model uses. Fable 5.1 sits at $7.63 per completed task, Meta's Muse Wonder Max at $4.54, and Kimi K3.1 Pro at $2.11.

Other Models Mentioned

  • Opus 5.5 and GPT 6.1 have recently become strong at 3D modeling, so the labs are converging on the same skills (02:55).
  • Meta's Muse model is excellent but weak at 3D, which Alexander Wang acknowledged, according to Berman (03:33). A Google model was also named as weaker; its name is unclear in the transcript.

Sponsor

CodeRabbit sponsored the video (05:44). Berman describes its security scanning as separating real risk from noise, showing blast radius, and carrying findings through to fixes.

Key Takeaways

  1. Kimi K3.1 is a very strong open weights model, near but not at the frontier, by Berman's testing.
  2. 3D rendering and physics were its best area; spatial reasoning about a knot was weak.
  3. It often needs follow-up nudges where other models would assume intent.
  4. Published benchmarks look excellent, but your own tests may tell a different story.
  5. Cost per completed task, not just token price, is the fairer comparison.
  6. Open weights releases at this level let you use the model directly, from US data centers, or from other cloud providers (19:12, 25:34).

Reflect & Discuss

  • Why might a model ace a famously hard test like the Rubik's Cube yet fail a rope knot?
  • When is a short prompt a better test of a model than a detailed one?
  • How would you compare models for your own work: published benchmarks, your own tasks, or cost per completed task?
  • What does it mean that the model first reported invented numbers, and how would you catch that?
YouTube