This is a short Two Minute Papers episode from Dr. Karoly Zsolnai-Feher covering DeepMind's disclosure of the architecture behind Gemma 4's multimodal capabilities. The framing device is a paradox: enormous frontier-scale open models can cost a fortune to run and still be blind, while a model small enough to sit on a laptop can see and hear. The episode's payload is the architectural explanation for how that is possible, followed by an advocacy pitch for the open model ecosystem that produced it.
What: Zsolnai-Feher opens by noting that AI models have become "almost comically large." He cites DeepSeek as reaching over 1.6 trillion parameters and says it is not even the biggest. Running models at that scale costs hundreds of thousands of dollars. Then comes the twist: show one an image, ask what it depicts, and it does not know. It cannot see.
Why: The point is to sever the intuitive link between size and capability. Parameter count buys language competence, not perception. Perception is an architectural question, not a scaling question.
How: He sets up the apparent impossibility deliberately. A model 99% smaller, "a speck of dust comparatively," that nonetheless has vision. He calls it wishful thinking, then reveals that it exists: DeepMind's Gemma 4, free, open, running locally, downloaded more than 300 million times.
What: The news hook is that DeepMind has published the architecture Gemma 4 uses to see, rather than only shipping the weights.
Why: Disclosure turns a product into a technique. It explains observed behavior that previously looked like magic, such as the model role-playing a character in a voice while simultaneously identifying objects in your video.
How: With the architecture public, the question the rest of the video answers becomes tractable: how do you add vision and multimodal reasoning to an unusually small local model without ballooning its size?
What: Zsolnai-Feher describes the standard design. A conventional AI is really several neural networks connected together. Feed it an image and you need a dedicated vision model. Give it ears and you need an audio encoder. One specialized part per task.
Why: His critique is pointed: "this is not really looking or listening." The main model never touches the raw signal. It receives a translation that a different neural network produced on its behalf. Every modality arrives pre-interpreted by a component with its own assumptions and its own parameter budget.
How: He notes that Gemma's smallest models still work this way. The architectural break happens when you move up to the 12 billion parameter model.
What: This is the core of the episode. DeepMind's scientists, in his telling, say throw all of that away. Instead, the system cuts the picture into small patches, then projects those pixels directly into the model's internal representation, retaining where each patch came from. Audio gets the same treatment, sliced into 40 millisecond chunks. Then, as he puts it, "you just pour all these tokens into your main transformer."
Why: Because there is no vision transformer standing between the pixels and the model, nothing pre-digests the image. The single main network is "forced to learn to be the eyes, ears and brain at the same time." He calls this one of the most important architectural ideas in Gemma 4, for two reasons: it removes hundreds of millions of specialist parameters, and it blurs the boundary between perception and thinking.
How: The mechanism is unification of representation. Image patches, audio chunks and text all become tokens in the same stream, handled by the same weights. The parameter savings come from deleting the encoders; the capability gain comes from perception and reasoning being learned jointly rather than in separate stages.
What: The result is a system that, in his words, punches way above its weight. It handles images, handles audio, and is "bloody smart," while remaining small enough that "this is something that you can own."
Why: He adds two closing points. First, the Gemma 4 ecosystem continues to receive improvements that make it faster and better. Second, and more consequential, publishing the technique means Gemma 4 is not only valuable in itself. It can help DeepSeek and other systems learn to see better and more efficiently.
How: That second point is the difference between releasing a model and releasing knowledge. Weights improve one model. A published architecture improves the field.
What: The episode ends with an unusually direct appeal. Do not take it for granted that amazing open models will keep coming. "These are gifts to all of us." He warns the gifts may stop as capabilities increase, and states plainly that it is not a law of nature that we get these models for free in the future.
Why: The reasoning is that openness is a policy choice made by organizations, reversible at any time, and increasingly costly to them as models get more capable.
How: He addresses open model builders directly, calling them heroes who help scientists, students and millions of other people do their work better, and argues the community has to come together to support open systems.
The episode's technical claim is narrow and clear: Gemma 4's larger models drop the separate vision and audio encoders, project image patches and 40 millisecond audio chunks straight into the main transformer as tokens, and let one network learn perception and reasoning together. That deletes hundreds of millions of specialist parameters and produces a locally runnable model that outperforms far larger ones on multimodal tasks. The broader claim is about the ecosystem. DeepMind published not only the model but the method, which raises the ceiling for every other open system, and Zsolnai-Feher closes by insisting that this generosity is a choice worth actively defending rather than an entitlement to assume.