About This Video
An 18-minute video from Matt Maher, recorded while he was, in his words, "hoppin mad." A very large refactor of a system he had been writing for a while worked well in practice, but when he dug into the details, half of what he wanted the original to do had been thrown away. The video is a candid forensic walk-through of how that happened and what it teaches about trusting agents.
Summary
Maher's argument is that AI systems are not magic. Even a strong, expensive model working for many hours, with plenty of tests and written intent files, can quietly drop requirements while producing something that looks great. His fix is to verify the work: have a separate sub agent or another model check it, and write evaluations that tie each recorded reason ("the why") to a test before and after any big change. He says he is sharing the failure because he did the very thing he tells other people not to do.
What Happened
- The director pattern (01:56). A main agent, which he calls a director (or project director), partners with you. It either does quick work itself or hands work to a worker or to another director that coordinates workers.
- Slow, bloated startup (02:35). Agents told him it took about two minutes just to say hello, because the system read 30 to 40 files to get started (04:49).
- Truncation (03:18). His agents file pointed to a director.md file that had grown large. The agent only read the first chunk (he recalls something like the first 32k), so rules such as using certain emojis were never followed. He only noticed because the emoji behavior was so visible, and he suspects the same has happened many times in other systems he has built.
- The refactor (05:17). He worked with a high-reasoning model at extra high effort (the transcript renders its name as "Astra") to make startup lighter and use progressive disclosure, meaning the agent does not read a file until it is about to do that action. It worked for 20 some odd hours and returned a new system.
- The result (09:11 to 10:44). The original had roughly 100 tests; the refactor ended with roughly 60 or 70. When he questioned the new agent, it did not know about rules and files the original had. He did not check test by test because he trusted it.
Key Arguments
- Check what was condensed (06:38 to 07:15). When an agent turns meeting notes into a document, ask a sub agent to confirm everything in the notes made it in. He says doing this almost every time turns up missed or condensed items, often because the agent judged two or three things "similar enough."
- We are in a faith-based environment (07:23 to 07:49). Systems are generally excellent but not always correct or complete, and we will keep being surprised by what they miss.
- Trust and verify (08:54 to 09:11). It is not that you should not trust the models. You should also verify them, and build systems that evaluate the work.
- A refactor is not a rewrite (09:56 to 10:16). In engineering, refactoring means turning what is there into a different shape to do the exact same work. The agent instead rebuilt it with its own ideas.
- Get it out of its own brain (11:17 to 11:39). Have another model, for example Opus 5.5, evaluate the work, rather than the model that produced it.
- The why drives everything (15:54 to 16:27). Agents take huge swings, and 20 hours of changes is too many moving parts to track. The work needs to be anchored to the reasons you asked for it, which he likens to a constitution.
The Forensic Process
- Intent captured in files (11:59). His system writes down what he asks for and why. A rationale file under development holds the reasons, and archives hold the verbatim requests from building the system (12:17 to 13:27).
- Sample the important rationales (13:46). He asked the agent to read through the 18 rationales, pick seven important ones, and compare before and after: the why, the test cases, and which files and lines were read.
- Score the outcomes (14:23). Results came back as met, partial, improved, and mixed. Most were met. For partial ones he asks why, and whether his understanding of the system changed.
- Cost (15:30). He says this is taking him hours, and that intent management is critical to software built in the future. He also notes this system is words and prompts, not code.
Key Terms
- Director pattern: a main agent that partners with you and delegates to workers or other directors.
- Truncation: the agent reads only part of a file, so later instructions are silently lost.
- Progressive disclosure: do not read a file until you are about to perform the action that needs it.
- Refactor: reshape existing work to do exactly the same job, not a rewrite with new ideas.
- Intent / rationale: the written record of what you asked for and why.
- Evals: tests that check the system still does what you wanted.
Key Takeaways
- Agents are not magic. Verify their work, especially after long autonomous runs.
- Ask a sub agent or a different model to check that nothing was dropped or condensed.
- Keep large instruction files small enough that they are not truncated, and test for rules you can see being followed.
- Record the why behind each request in files so you can audit changes later.
- Write an eval for each recorded reason, run before and after a refactor, and write the evals first.
Review Questions
- What is the director pattern, and what problems did it have at startup?
- How did truncation make the agent ignore instructions, and how did Maher notice?
- What is progressive disclosure, and why did he want it in the refactor?
- How did the number of tests change after the refactor, and why did that matter?
- What is the difference between a refactor and a rewrite?
- Why does he suggest having another model, such as Opus 5.5, evaluate the work?
- How did intent and rationale files help him investigate what was lost?
- What would he do differently next time, according to his closing remarks?