Hey! Happy Sunday. Here is a quick note to remind you that Mostly Harmless AI, second edition, draft one, is finished. PDF and EPUB on Gumroad. Free online reader at books.apiad.net. Both live, right now.

This book is a permanent beta, because the AI story is being created as I write these words and it’s moving just too damn fast for anyone to put it down cleanly in a book. So don’t expect the kind of polishing that comes with retrospective looks at settled topics. This isn’t that kind of book. It’s raw, but up-to-date. It contains basically everything I know and believe about AI that doesn’t require code or math to understand it.

Get Mostly Harmless AI - 50% off

Sixteen chapters, mechanism-first

Three parts, sixteen chapters, around two hundred and sixty pages, almost three hundred footnote citations.

Part I — Foundations walks the mechanisms, from the symbolic systems of the 1950s to the agent loops of 2026. Classical AI, machine learning, deep learning, language modeling, generative AI, agentic AI. If you have ever wondered what attention, gradient descent, or RLHF actually mean (not the marketing line, the mechanism), start here.

Part II — Applications shows how AI is reshaping each field: knowledge work, scientific research, software development, education, creative work, policy. Dual-audience throughout. Experts get practice and gotchas. Non-experts see why their lawyer, their doctor, and their kid’s teacher are suddenly using these tools.

Part III — Dangers is the honest one. Alignment, the limits of language models, and the harms already in the world today: scammers cloning your mother’s voice, autonomous weapons that pick their own targets, hiring algorithms that quietly downrank you, jobs evaporating from under people who built careers in them. The existential-risk question is in there too, treated proportionally.

The book lands on a third position beyond maximalists and doomerists. These systems are powerful enough to demand real responsibility and limited enough that the worry should be about who deploys them, not about the silicon waking up. The future is not predetermined. Neither doomer nor utopian framings are right. We choose, and choices have responsibilities attached.

The first-draft post from two weeks ago has the longer description if you want more.

Free online, or pay for offline

• Read it free online, in perpetuity, at books.apiad.net. Clean typography, gentle dark mode, inline footnotes. No popups, no signup, no tracking. The reader was built for these books specifically.

• PDF and EPUB on Gumroad if you want it offline, on your e-reader, on a plane: apiad.gumroad.com/l/ai. Launch week is 50% off.

If the online reader is free, then why would you buy it? Well, you don’t have to. But here is why you may want to do it anyway.

Writing this book took two years. The second edition took three months of intense rewriting on top of that. If you find value in the work and want more of it to exist (these books, the Computist Library, the journal you are reading), paying for the PDF is how you tell me that. I do not run ads. I do not sell your attention. Every book sale is one person telling me to keep going, and the money buys the months it takes to write the next one.

If you cannot pay, read it free. No guilt, no caveat. That option exists on purpose.

June is algorithms month

After today, I am putting AI writing to sleep for a couple of months, on purpose.

Not the work — I will still be building AI tools and using these systems every day. But on this journal, starting tomorrow and through July, I am switching to algorithms and core computer science, with The Algorithm Codex as the spine. The Codex is the next book in the Computist Library, written in parallel with MHAI and ready for center stage. Already 200 or so pages, and 50+ algorithms in, but still very rough around the edges, in dire need of your feedback.

During June, you can expect daily posts, Monday through Friday, each anchored on a chapter. Why the fastest algorithm ever devised runs in essentially constant time. Why most programmers get binary search wrong. Why no comparison sort can beat n log n, and the loophole that lets you anyway. Why some algorithms become settled facts the way theorems do.

AI returns to the journal in August. The field will surely have moved a lot by then, and there will be more than a few chapters to retouch.

So if you come here for the AI material, this book is the most complete thing I have on it — go read that. If you have been curious about the algorithms side, the next two months will be the best stretch of it I publish all year.

That’s all for today. Tomorrow I’ll meet you again with a brand new article.

Until then, stay curious.

Get Mostly Harmless AI - 50% off

On May 31 we launch Mostly Harmless AI v2. This arc — how models learn, how agents work, where they break, and what it takes to build something real on top of them — is now a book, updated for May 2026. Newsletter subscribers get 50% off.

Get Mostly Harmless AI - 50% off

For the better part of the last two months I’ve been drilling you with a bunch of inter-connected ideas, all gravitating around the notion of agentic reliability. We started with how these models learn, argued about seventy years of AI history, traced the strange logic of prediction, dissected what an agent actually is, then spent two posts on the edges: how to write tools agents can really use, and what happens when you push them to their limits.

This is the last post of that arc, and it embodies my vision of where this whole Agentic AI thing is going. And to show I’m putting my money where my mouth is, this post is about the tools I’m building to bring forth that vision.

But first, why should you care?

Subscribe now

Agents aren’t just for coders

Here’s something the AI tool industry quietly gets wrong: every major agent harness — Claude Code, Gemini CLI, Cursor, OpenCode — is pitched as a developer tool. And technically, yes, they drive code.

But code is how agents solve problems, not what the problems are. When you ask an agent to research a topic, draft a document, reorganize a folder, schedule your week, or synthesize a dozen sources into one coherent brief — none of that is fundamentally a programming task. The coding is incidental. A historical accident of the fact that the people who built these tools happened to be programmers solving programmer problems.

So if you’re a researcher, a writer, a manager, a student, a scientist who occasionally touches a terminal — this piece is for you too.

What’s wrong with existing tools

I use Claude Code every day. The harnesses — Claude Code, Gemini CLI, Cursor, OpenCode — are genuinely good. The agentic loop is robust. Tool design is sharp. Context management is solved for practical purposes. I’ve opened ten-thousand-line projects and the agent knows exactly where it left off.

What’s missing is not single agent capabilities. It’s coordination.

Claude Code runs on Claude. Geminis CLI runs on Gemini. OpenCode allows any provider but you cannot use your existing subscriptions (which are heavily subsidized); you have to pay API rates.

If you’re running multiple agents with different models from different providers, you end up with four windows open: one per harness, one per model, one per subscription you’re already paying for. Different tools with slightly different annoying quirks and ways to do and call the same things.

But you can still make them collaborate, just not easily.

Most harnesses support sub-agents, which is a subroutine: the main agent mints a new subagent for a specific call, call it, it runs, it returns, the main agent continues.

What they don’t support is a mid-work handoff. Imagine you’re two hours into a brainstorm with Claude, a question has emerged that you cannot answer without trying some code, and now you want a second agent — different model, fresher context, perhaps Kimi from OpenCode Zen — to take over a brief coding session, and then handoff back to Claude what it found so brainstorming can continue.

That is somewhat achievable with subagents (except they run autonomously and die, you cannot, in most tools, interact with them and steer them into new directions). But what you cannot do, is have Kimi return to Claude, and stay up, waiting (with its full context alive) for a second follow up question to continue exploration.

That transfer doesn’t exist today. You have to ask Claude to produce a handoff manually, paste it in Kimi explaining the situation from scratch. The paste its response back to Claude, and so on. You have become a secretary between two AI agents.

Agents are isolated in all these tools. Two agents working on the same document will overwrite each other. There’s no locking, no merge protocol, no queue where one drops a task and another picks it up. No broadcast so all agents know when the plan changes. Agents don’t share a world — they each have their own private window into one.

I looked around before building. The closest I found: Conductor, which orchestrates multi-agent workflows — but only for Claude Code. And T3.codes, which drives any harness, closer to the spirit of what I wanted, but neither has cracked the coordination layer as I envision it.

Introducing Aegis

So, of course, I had to go and make my own. (Quick digression, if you’ve been reading this blog for a while you know I love to reinvent wheels if only for the learning experience, but this is a case where I genuinely couldn’t find something good enough.)

Here’s what makes Aegis different to anything out there, embodied in its slogan: the programable multi-agent meta-harness. Let’s build it from the back.

Meta-harness. You’re already paying for Claude via your Anthropic subscription. Gemini via your Google account. If a new tool wants to drive both, it has two options. It can re-authenticate you through its own layer: API keys, rate limits, and lost subscription benefits. Or it can call the native tools, which already have your credentials.

Aegis takes the second path. It drives Claude Code over its stream-json protocol, Gemini CLI and OpenCode over the Agent Client Protocol — calling the binaries on your machine, which already have your auth. It doesn’t touch your subscriptions. You stop worrying about which model wins this month’s benchmark.

And because Aegis calls the native harnesses rather than reimplementing them, it inherits everything they’ve spent months polishing: the agentic loop, the context management, the permission model. The harness keeps owning tool use, sandboxing, model selection. Aegis owns the layer above — tabs, routing, delegation — the things a single-conversation CLI was never built to do.

Multi-agent. Aegis provides six inter-agent syncronization primitives, built incrementally one above the others, to give you increasingly more powerful multi-agent capabilities.

The first primitive is a per-agent inbox. Any agent (including you) can hand another agent a message, that gets enqueued until the end of the current turn. This alone enables solving the problem we were just discussing in the previous section.

Then, on top of that, they got canvases: markdown files shared across agents with per section looks and callbacks that awake an agent when another finishes writing a section.

Then we got real terminals. Not an agent calling bash on a subprocess and blocking on its result. A real, shared, fully interactive terminal session that multiple agents can scan, tail, and write to, And you can too. One runs a command; another sees the output in real time and reacts. Or you run the backend and ask your agent to look at the logs when that heissenbug happens.

So far this allows you to spawn several agents and have them collaborate. But you can also have queues. Any agent can drop a task (a prompt) and the queue auto-spawns an ephemeral agent to take it, potentially calling back the emitter once done. Queues have a maximum cap on parallelism, as well as arbitrary rolling budgets on tokens and dollars so you keep control of how much work is allowed to happen without your supervision.

Agents can also be added into groups, dynamically created and destroyed on demand, by you or any other agent. Groups have a shared inbox and you can subscribe to them and get notified when the first, any, or all the agents in the group finish. This allows committee-like flows where different agents analyze a problem in parallel.

Programmable. And finally, you get workflows. Deterministic Python code that drives agent calls in sequence, with branching, conditionals, and loops. Think skills, but instead a hopeful blob of markdown one agent can choose to interpret as they want, these are composable routines that drive the entire substrate in the exact level of control you desire.

When you write a complex workflow in natural language, you’re hoping the agent follows through. It might decide step two is better done differently, skip the commit because something caught its eye, or forget step three entirely. A Python workflow doesn’t forget: it runs step 1, then step 2, then step 3, and commits. You wrote commit() in the code; the code runs. You get the agent’s creativity at each step; the program guarantees the steps happen.

Workflows can be scheduled: declare a cron entry in .aegis.yaml and the substrate fires it while you sleep. They also run across machines: one agent on your laptop can enqueue a task to a remote Aegis instance on a VPS and get the result back in its inbox.

Quick aside: Yesterday, as I was polishing this post, Anthropic announced Dynamic Workflows — a way to orchestrate long-lived agents over dozens of hours of work. I haven’t tested it yet, but It seems geared toward the same problem I’m trying to solve.

The difference is in the philosophy. Anthropic’s principle is to give agents as much agency as possible: trust the model, let it decide how to get there. Don’t get in the way, you stupid human. Tokens go brr. It’s the reason why all their solutions to problems are humoungos one-shot prompts.

My philosophy runs the other way. Leave agent creativity where it can do the most good — in the actual work, not in deciding whether step three happens after step four. The deterministic spine isn’t a constraint on the agent. Its what makes it work despite agent idiosincrasies, and why it works across all agents and harnesses, regardless of their intrinsic capabilities.

I’m using Aegis now for about 50-60% of my coding. It’s still rough around the edges, but it’s way more fun to use than any single CLI. There is a lot more in the box, like remote sessions, a built-in file browser, lots of metrics... but this post is already way too long. You’ll have to check it out on your own. Links at the end.

What I intentionally left out

Aegis has no native concept of skills. No AGENTS.md injected automatically. No memory system.

Those things are conventions, and conventions change. What belongs in an AGENTS.md today looks different from what it’ll look like in six months. Memory systems have a dozen competing designs and no consensus. If I’d baked any of that in, you’d be stuck with my choices the moment the community moved on.

Aegis has a very powerful plugin system instead (I told you, it is programable). You write a pure Python function, drop it on some folder, and it gets called anywhere in the agent lifecycle.

Want skills that activate on context? Write a plugin. Want a memory system? Write a plugin. Want to inject per-repository knowledge before every session? Write a plugin. The conventions you need are yours to build, and when they change, you change them, not me.

Coda

Claude Code and Gemini CLI are applications. You open them, use them, close them. Aegis is more than that. It’s a framework. You build on top of it — applications that spin up the agentic substrate automatically, pull in whichever harness fits the task, and run without anyone at the keyboard.

Picture a self-hosted Git forge where pushing a branch triggers agents: one reviews the code, one hunts bugs, one picks up open issues and starts implementing. Everyone on their own worktrees, independent, parallel, coding while you sleep. Push code; agents work.

That’s Sindri, and I’m also building it; but that’s a story for another Friday.

Aegis is open source: github.com/apiad/aegis. pip install aegis-harness to start.

May closes the agentic AI arc. If you want the full conceptual foundation — model internals, agent architecture, tool design, failure modes — it’s in Mostly Harmless AI v2, launching May 31. Subscribers get 50% off.

Also check the Compendium — one fixed price, every educational project I’ve built or will build, yours in perpetuity. Buy once, get everything.

In June: algorithms. The other half of the computational story — sorting, searching, graphs, optimization, the classical toolkit that AI didn’t replace and won’t. All month, one idea at a time. See you there.

Until next time, stay curious.

Every post this month is on the theme of building AI agents that actually work — anchored on the second edition of Mostly Harmless AI, 50% off during early access, where the five-phase structure this post describes is a full chapter with more failure cases, the artifact design patterns, and the context isolation mechanics in detail. You can also read the whole book online for free. More at the end.

Last night you give your AI agent a clear task. It worked hard for two hours. YOu woke up to a report that is technically correct and completely useless.

I’ve had this experience enough times that I stopped blaming the model years ago. The failure is always structural, and I mean that in a very specific, diagnosable sense.

AI labs fine-tune language models to produce output, not to question the frame. The whole reward cycle behind how these models are trained pushes them toward helpful completions; nobody in that loop is rewarding a model for pausing mid-task and asking “have you considered that you might be framing this wrong?”

Execution is the default. That setting works beautifully for atomic tasks: create a note, write a short email, implement a specific function, compare two clearly-defined options. It fails quietly, and expensively, when the real problem isn’t the task itself but the goal behind it. Here’s three examples:

First: you ask an agent to “implement user authentication” for your web application. Clear task. The agent gets to work, producing a technically sound implementation using JWT tokens, bcrypt password hashing, and session management.

Second: you ask an agent to “write a technical report on renewable energy storage.” Again, clear task. The agent downloads papers, synthesizes findings, produces a well-structured document.

Third: a family asks an agent to “plan our move to a new city.” Thorough research follows — neighborhoods, school districts, moving companies, cost estimates. In every case, the output is technically correct.

In every case, the person who asked ends up with something that doesn’t quite fit. The gap between what was asked and what was needed — that gap is what this article is about.

The structure is the problem, not the model.

Not the model’s intelligence, not the quality of your prompt, not the temperature setting. The structure of the workflow you handed the agent is the thing that determines whether it answers the right question or a reasonable-sounding wrong one. In this article, I’ll show you what the right structure is. But to understand why it must be so, we need to see how we got there.

The first fix the industry reached for was planning. It helped. It wasn’t enough. Strap on for a story.

Subscribe now

Plan First, Build Second

The structure is the problem, not the model. Right? So the first fix the industry reached for was adding a planning mode.

If you’ve used any of the major agentic coding tools over the past couple of years, you’ve seen this pattern. There’s a plan mode: read-only, no side effects, meant for thinking through the task. Then a build mode: where the agent executes.

The intuition behind it is sound — separating thinking from doing is genuinely better than collapsing them into a single continuous stream. When you mix planning and execution in the same pass, the agent makes irreversible changes based on its first interpretation of the goal.

A planning mode forces a pause. You get to inspect the plan, argue with it, revise it before anything is written to disk. The plan is the first concrete artifact in the handoff chain. You can hold it in your hands — or at least on your screen — and have a real conversation about it.

Genuine progress. I don’t want to minimize it. But watch what happens with the family move.

The agent produces a thorough plan: compare three target neighborhoods, compare school districts in each, contact five moving companies for quotes, estimate total relocation costs, build a timeline. Excellent plan. The family reviews it, nods along, says “looks good,” and the agent executes.

Six months later: they’re living in a city that made complete sense on paper — except the job that drove the whole move turned out to be fully remote. They never needed to relocate at all, or they could have moved somewhere half the price with better schools.

The plan was correct. The execution was correct. The goal was wrong. And here’s the point: the agent that built the plan never questioned whether “plan a move to City X” was the right goal, because it was never asked to. It took the first reasonable interpretation of the prompt — “they want to move, let me help them move well” — and planned confidently inside that frame.

The software auth example is shorter but sharper. “Implement user authentication”: the agent plans for JWT tokens, bcrypt hashing, session storage. Fine plan. Sound, even. For a single-tenant web app. This is a multi-tenant SaaS product (one where dozens of customers share the same running application.)

The plan was never going to catch that, because nobody told the planning agent that multi-tenancy was a constraint. It did exactly what it was asked.

Now, the objection you might be forming: “just review the damn plan carefully before you approve it.” That objection is right about something, review does matter, bit it misses the fact that the agent that produced the plan has already committed to a frame.

Every question it asks, every tradeoff it surfaces, every option it presents — all of it is already shaped by its first interpretation of the goal. When you review that plan, you’re not reviewing a neutral set of options. You’re reviewing a plan that already selected its own success criteria. The frame is invisible because it was never made explicit.

Planning without exploration locks in the first reasonable goal.

Research Before the Plan

Once more, the structure is the problem, not the model. The next obvious fix is to put a research phase in front of planning.

By the time we hit mid-2025, a third mode was appearing in serious agentic setups. A research phase: read-only, job is to understand the problem space, not to produce a solution. The artifact it creates is a description, not a prescription — a document that maps what is known before anyone decides what to do about it.

The intuition is right again: if the planning agent doesn’t know what it doesn’t know, it can’t plan well. Research is how it finds out. For the renewable energy report, thorough research might surface the fact that the intended audience is policymakers, not engineers — which changes the vocabulary, the technical depth, and the document’s opening frame.

Real progress, again. And again, not enough.

Watch the renewable energy report closely. The agent runs a solid research phase: downloads twenty recent papers, reads industry reports, synthesizes the state of the art on battery storage, hydrogen carriers, pumped hydro, and grid-scale thermal systems. Then it transitions into planning mode — in the same context window. And here is the problem: the same context window.

The planning agent isn’t a fresh mind looking at a research report. It’s the same agent, carrying everything it concluded during research, now deciding how to structure the work. If the research agent concluded “battery storage is the central challenge in the energy transition,” the planning agent will structure the report around battery storage. Not because it made a new decision — because it never had the opportunity to question the prior decision. It just kept going. The research was excellent. The plan followed naturally from the research. The report answered the question the research agent found most interesting. Not necessarily the question the reader needed answered.

The steelman still stands, and research sharpens it. The steelman says: “planning alone is enough, just review the plan.” Research proves it isn’t — but not because planning is the wrong approach. Research proves it because research without context isolation just moves the lock-in one step earlier. The same agent that researched now plans. It cannot escape its own prior conclusions, not because it’s incapable of abstract reasoning, but because those conclusions are literally sitting in its context, shaping every next token it generates.

A fresh planning agent, handed only the research artifact as a clean document, can genuinely question whether the research answered the right question. It can push back. It can say “your research focused heavily on battery chemistry, but I’m not sure that’s what the audience needs.” The same agent that did the research cannot do that. Not really.

But here is the real kicker: research gives you facts; it doesn’t tell you if you’re solving the right problem. Save that thought for a minute, we need one more step.

Review After the Plan

The structure is still the problem, not the model. And to fix it, we added a new, and in hindsight, pretty obvious step. A review phase after implementation.

This is a dedicated pass where a separate agent — or the same agent in a separate context — evaluates the produced artifact against known criteria. Not just “does this code run,” but “does this code do what we intended.” The distinction from implementation is real and it matters. An implementation agent is building; a review agent is hunting for the thing that will break it.

What review actually solves is real. The software auth implementation, evaluated by a review agent, surfaces real questions: Is the JWT expiry window set appropriately for the threat model? Is the bcrypt cost factor tuned for this hardware? Are session tokens actually invalidated on logout, not just expired? These are genuine bugs a fresh pass can find. I’ve seen review agents catch the kind of subtle mistake that a second human reader catches — not because they’re smarter, but because they’re looking for problems rather than building a solution.

But watch what happens when the review agent evaluates the software auth implementation against the plan.

The plan said: “implement JWT-based authentication for the web application.” The review agent confirms: JWT is implemented correctly. Bcrypt is used. Session management is in place. The implementation passes review. It ships.

First enterprise customer tries to log in: there is no tenant isolation. Every user in the system shares a single authentication namespace. The review agent found no bugs. The implementation had no bugs. The plan specified the wrong thing. And the review agent couldn’t catch it — not because it was careless, but because review only checks “did we implement the plan correctly.” Not “was the plan the right plan.” Those are different jobs. And you can’t review your way out of the wrong spec.

This isn’t a failure of intelligence. It’s a consequence of what the review agent is handed. It receives the implementation and the plan. It has no clean access to the original problem statement — what the user actually needed, what constraints were implicit in the product, whether the product served one customer or a hundred who shared a namespace. It’s evaluating the gap between the artifact and the plan, not between the artifact and the goal.

Human code reviewers fail the same way, by the way. Code review finds style violations, off-by-one errors, missing null checks. It rarely questions the architecture decision that was made three sprints ago and embedded in every layer of the codebase. That kind of question requires a different context — a different meeting, a design review, a fresh set of eyes on the spec rather than the implementation.

Review catches errors, but only inside the frame you committed to in the planning phase. You can’t review your way out of the wrong spec.

Name the Problem Before Solving It

Finally, the structure IS the problem, not the model.

Research doesn’t ask it. Planning doesn’t ask it. Implementation doesn’t ask it. Review doesn’t ask it. The question is: what problem are we solving, and how will we know when we’ve solved it?

That’s not a rhetorical question. It has specific, concrete answers. And those answers should be a document — a specification — produced before the plan is written. Not during planning. Not as a side effect of research. Its own phase. Its own artifact.

A specification answers four questions, and it only takes one page to do it. What is the exact output we are trying to produce? What are the hard constraints it must satisfy? What does success look like — specifically, what would we check to confirm it? What does failure look like — what would make us say this didn’t work?

These sound obvious. They are almost never answered before a planning phase begins. In my experience, the reason is that they feel like they slow you down. They don’t. They prevent six months of work in the wrong direction. Think of it as a single page you could tape to the wall — the kind you’d point at during a disagreement about whether the output succeeded or failed.

Go back to the family move. After a thorough research phase — neighborhood data, school ratings, crime statistics, cost of living comparisons — a specification phase asks: what does a successful move look like for your family? The family sits with that question. It turns out they’ve never explicitly answered it.

The husband’s answer: proximity to his aging parents, who live in a specific region of the country. The wife’s answer: their daughter getting into a specific school district that has strong arts programs. The budget answer: keeping total housing costs under a threshold that lets them maintain their current savings rate. Three explicit success criteria.

The research phase found no conflicts because it was never told what it was optimizing for. The specification phase surfaces all three criteria before the plan commits to a single city. The planning phase can now do something useful: find cities that satisfy all three criteria — or, just as importantly, discover that no city satisfies all three and surface that conflict before anyone books a moving truck.

The software auth case is fast. Specification asks: what does correctly-implemented authentication look like for this product, given who its customers are? The answer: it must support multi-tenant isolation with strict data separation, SSO for enterprise customers, and a free tier with email-only login.

Now the plan can be written for the actual product. The research phase’s work on JWT and OAuth is still valid; it just needs to be read through the lens of multi-tenancy, which the specification made explicit.

The full chain, with its five concrete artifacts, looks like this.

Research produces a collection of source materials plus a descriptive state-of-the-art report — what is known about this problem space. Specification produces a success-and-failure criteria document. Planning produces a concrete step-by-step plan — how to get from here to there, given the spec. Implementation produces an evaluable artifact — code, document, report, recommendation. Review produces an evaluation report checked against the specification, not just the plan — a real answer to “did we solve the problem?”

Five documents. Five handoffs. Five chances to catch the wrong frame before it becomes expensive.

We Knew it all Along

The fun part is all of this was known, at least in principle, since before 1987.

IDEO, a by-now ultra famous design consultancy, articulated a five-phase creative process that Stanford’s d.school later codified as Design Thinking.

Tim Brown formalized the diverge-converge logic in Change by Design. The five phases: Empathize, Define, Ideate, Prototype, Test. If those names sound familiar given what you’ve just read, that’s not a coincidence.

Empathize is research: go wide, gather context, talk to users, understand the problem space from the outside rather than the inside. Define is specification: converge on an explicit problem statement with clear success criteria, the “how might we” question that frames everything downstream. Ideate is planning: diverge again, generate candidate solutions, explore the space of possible approaches. Prototype is implementation: produce an evaluable artifact, something you can put in someone’s hands. Test is review: evaluate the prototype against the problem statement, not just against the prototype’s internal logic.

Every phase the agentic world has been bolting on since 2023 was already named, sequenced, and justified in a framework that predates the modern web by a decade and a half. It’s a framework every Silicon Valley startup, incubater, accelerator, and VS knows in and out. It is taught in bussiness majors all over the world. It’s literally the structure of most pitch decks. But it is still missing in most agentic protocols we use every single day.

The piece most clearly missing is the Define phase — the second one, which IDEO put second for a reason: without a clear problem statement, everything downstream answers the wrong question. It’s a very old insight the field keeps rediscovering from scratch — Agile’s Definition of Done, test-driven development’s failing-test-first, specification by example. Each was the same insight under a different name.

Now, here is the strongest version of your initial objection. “You don’t need all this structure. A great prompt specifies the audience, the format, the success criteria, the constraints. Write a better prompt and you get all five phases in one go.”

Let’s take this seriously, because it’s not wrong about prompt quality. A genuinely well-crafted prompt that specifies who the output is for, what format it should take, what it must accomplish, and what would make it fail — that prompt is effectively a specification. You’re right that prompt quality matters.

But a single prompt containing research context, goal specification, a plan, and execution instructions is not a cleaner version of the five-phase process. It’s five phases collapsed into one context window, with no mechanism for each phase to question the prior one’s conclusions.

When research and planning share a context, planning can’t interrogate research. When planning and implementation share a context, implementation can’t push back on the plan. When specification and review share a context, review is already biased toward confirming the specification it helped write. Prompt quality is about what you ask.

Phase independence is about who processes each answer, and whether they can genuinely disagree with the prior step. You can write the world’s best prompt and still hand it to an agent that will execute it inside the same narrowing tunnel, compounding the same assumptions with every step.

Picture someone reading their own manuscript for the fifth time — they no longer see what’s there, only what they meant to write. It is one of the most replicated findings in human psychology: once we form a belief, we interpret subsequent evidence through the lens of that belief. We notice confirming evidence, discount disconfirming evidence, and generate hypotheses that assume the belief is correct.

This is not a weakness of intelligence. Every mind — human or artificial — interprets through the conclusions it has already drawn. The agent that researched your problem already believes things about it. When it transitions to planning, it plans in service of those beliefs. The agent that planned has a solution in mind. When it implements, it makes countless micro-decisions that serve that solution. The agent that implemented defended choices as it worked. When it reviews, it reads its own output charitably.

Context isolation breaks this chain. A fresh context hasn’t seen the prior steps. It cannot be fooled by conclusions it never drew. It reads the artifact cold, which is the only way to genuinely evaluate it.

Design Thinking’s diverge-converge logic is not about what each phase does. It’s about who does it, and whether they can arrive at it without inheriting the prior phase’s commitments.

Start Doing This Yourself Today

The artisanal version of this is simpler than it sounds:

• Treat each phase as a distinct conversation. Start a fresh session for each one.

• Hand it only the artifact from the prior phase — not the prior conversation, not your running context, not a summary of what you’ve been thinking about. The artifact alone.

• And tell the agent explicitly what mode it’s in. “You are in Research mode. Do not propose a plan. Do not suggest solutions. Your only job is to describe the problem space and produce a research report.”

That instruction matters. Not because the model needs to be controlled, but because explicit mode assignment prevents the agent from sliding into execution behavior when it senses a gap to fill. Models are trained to be helpful; helpfulness steers every gap toward a solution. Naming the mode is how you resist it.

The discipline lives in you, not the tool. This works with any agent, any interface. Fresh context, explicit mode, artifact handoff. That’s the whole recipe.

If you want to go further, you can make phases structurally enforced rather than just instructed — agents that literally cannot execute, subagents that receive only the artifact, automated handoffs with no shared context. Programmable harnesses give you this level of control with permission levels per skill. (If you don’t have one, call me, I’ll lend you one for free.)

One small step you can take today: add a specification phase to whatever workflow you already use. Before your planning phase writes a plan, ask for a success-criteria document first. One page. Explicit pass/fail conditions. What would make this output a success? What would make you throw it out? Review that document before planning begins.

This single addition — inserting a define phase between research and planning — catches more failures than adding a review phase after the fact. Because it catches them before the plan commits to the wrong goal.

What not to do: don’t implement all five phases as a mechanical checklist. And don’t add phases as ornamentation — a research phase that shares a context with planning adds conversation turns, not structure. More words in the same window is not more phases. The phases are context boundaries, not steps in a recipe. A phase that doesn’t produce a concrete artifact and doesn’t hand it to a fresh context adds nothing.

One document per phase. Fresh context per phase. That’s it.

Structure Before You Re-Prompt

Picture the artifact chain as a physical thing. A manila folder passed from one desk to the next. The research desk produces a report, closes it, slides it across. The specification desk opens only that folder, reads it, produces a criteria document, closes it, slides it across. The planning desk never opens the research folder — it opens only the criteria document.

And so on down the line. Each desk sees exactly one prior document. Each desk produces exactly one new document. The chain is what makes independence possible. You cannot hand off a vague intention. You cannot slide a feeling across a desk. Only a document.

Context isolation is the move most pipelines skip, and it’s the one that does the most work. Every phase that shares a context with a prior phase inherits its commitments. Not because the model is lazy or wrong — because that’s how cognition works, human or otherwise. We interpret through the lens of what we already concluded.

Context isolation is cheap: start a new session, pass only the artifact. The cognitive science is unambiguous: breaking the confirmation-bias chain requires a structural break, not a better instruction. Context isolation gets skipped because it looks optional. It isn’t.

Remember, the structure is the problem, not the model. Restructure before you re-prompt.

Until next time, stay curious.

This is the core argument of the agentic workflows chapter in the second edition of Mostly Harmless AI — the full chapter walks the failure cascade with more cases, the artifact design problem (a research report can be thorough and still hand the wrong thing forward), and the context isolation mechanics in depth.

This specific article is new content, still not in the book, but it will land there shortly. The book is 50% off while it’s in early access, and also free to read online in a custom reader I built: dark mode, font controls, progress tracking, offline support, the works.

If you want the architecture behind these systems — how they fail, what the harness around them should look like, and what to actually do about it — that is what the book is for.

Get Mostly Harmless AI - 50% off

Picture Donald Knuth. Eighty-eight years old, the man who wrote The Art of Computer Programming by hand in TeX, which he invented himself to be able to write his own books. The father of algorithmic analysis. The most laudeated living computer scientist. A legend, and a well-known AI skeptic.

Now picture him reading a printed chat log between a fellow colleague and Claude Code. Not skimming. Reading it in detail, because there is something genuinely baffling about it.

The log belongs to Filip Stappers, a mathematician who ran thirty-one coding explorations with Claude Opus 4.6, systematically probing a class of combinatorial objects Knuth had spent decades thinking about. Exploration 15 surfaced something unexpected: a structural pattern nobody had written down.

Knuth read it, judged it valid, proved it correct by hand, and wrote a paper about it. He called it “Claude’s Cycles.” Knuth noted, with his characteristic precision, that he’ll “have to revise his opinions about generative AI” one of these days.

Most accounts that open with this scene take one of two off-ramps.

The first off-ramp is the replacement narrative. AI is now the scientist. The model had the insight; Stappers just ran the prompts; Knuth read the output and judged it true. Discovery has been automated. We are, depending on your temperature, either liberated or obsolete.

The second off-ramp is the stochastic-parrot dismissal. It’s just a language model predicting tokens. It doesn’t understand combinatorics; it doesn’t understand anything. Stappers did the science; Claude shuffled plausible-sounding symbols. Attribute the discovery to the researcher, not the autocomplete.

Both off-ramps feel satisfying. Both are wrong. And they’re wrong in the same way: they’re answering the question “did AI do the science?” That’s the wrong question. The interesting object is not the agent. It’s the loop.

The loop looks like this: a human poses a question; a model proposes candidates; a verifier filters the candidates; a human curates what survives. Round and round. What Stappers and Claude did is not fundamentally different in shape from what Tao and Lean are doing, or what the GNoME pipeline does in materials science, or what AlphaFold did for protein structure. The shape is the same. The loop does the discovery.

I want to be clear about what that means, because it’s easy to hear it as either a compliment to AI or a dismissal. It is neither. It’s an empirical claim about where the causal action lives. Not in the model, not in the human, but in the interaction structure between them. Get that structure right and you get science. Get it wrong and you get confident nonsense at scale. The details of what ‘wrong’ looks like are worth walking through carefully, and we’ll do that a couple of sections down.

But before we can dismiss either off-ramp, we need to walk four recent cases. Because in every one of them — Claude’s Cycles, Tao and Lean, AlphaFold, GNoME — the replacement narrative is not just philosophically confused. It is empirically wrong. The loop does the discovery.

Four Cases, One Shape

In each of the following cases, the loop does the discovery. I’m going to give you four of them across four domains, because I think the pattern only becomes undeniable when you see it that many times. Same shape, different materials.

Claude’s Cycles

You already have the scene. Stappers runs thirty-one explorations. Not one inspired conversation, thirty-one numbered, documented, methodical probes. Claude proposes. Stappers evaluates. Knuth reads the surviving logs, verifies the mathematical claims by hand, and writes. One human author. One credited co-explorer. The question was Stappers’; the verification was Knuth’s; the curation of which thirty-one explorations were worth pursuing was human throughout. The model was the proposer.

That word matters, and I want to be precise about it. Proposer. Not discoverer, not author, not scientist. The one that generates candidates fast enough that the verifier can find something in the haystack.

Tao and Lean

Terence Tao, Fields Medal, the current standard of living mathematical genius, has been publicly working through what it looks like to use LLMs for mathematical research. His account in the Notices of the American Mathematical Society in 2025 is careful and specific, and I appreciate the care because it’s easy to overread these things in both directions.

Here’s what you’re actually looking at: an LLM proposes proof steps, intermediate claims, candidate lemmas, reformulations of the problem. Lean’s type-checker is the verifier. And Lean’s type-checker cannot be fooled. It either accepts a proof term or it doesn’t. There’s no “plausible-sounding but wrong” in Lean. The system rejects garbage instantly and silently. What the human does — Tao, or anyone in his lab — is curate: which of the surviving proof steps are worth following, which directions are worth pursuing. The loop is tight. The proposer is creative and unreliable; the verifier is reliable and uncreative; the human operates between them.

I’d reframe that: the breakthrough isn’t that LLMs can now do mathematics. It’s that the loop can now run fast enough and cheaply enough to be useful.

AlphaFold

I want to include this one even though it’s older and better-known, because it’s the case where people most confidently say “AI solved the problem.” And that confidence is telling.

The problem was fifty years old. How does a protein fold? You have the amino acid sequence; you want the three-dimensional structure; that structure determines function; that function determines everything downstream in biology and drug design. AlphaFold 2 proposes structures. Experimental crystallography verifies them: X-ray diffraction, cryo-electron microscopy, techniques that require real equipment, real samples, real physics. You cannot hallucinate a crystal structure into being; the protein either folds the way the model says or it doesn’t, and the experimental method tells you which. AlphaFold 3 extended this to molecular complexes: DNA, RNA, small-molecule ligands binding to proteins.

I think this is worth sitting with: human researchers curated which structures were worth solving, which proteins mattered, which findings were publishable. AlphaFold proposed. Nature verified. Humans curated.

GNoME and A-Lab

This is the one I find most clarifying, because the verifier here is the most brutally physical. You can’t argue with a crystal.

GNoME — Google DeepMind’s Graph Networks for Materials Exploration, if you want the full name — generated 380,000 candidate stable crystal structures. That’s the proposal stage. Then A-Lab, UC Berkeley’s autonomous laboratory, took 58 of those candidates and actually tried to synthesize them. Robotically. With real chemicals, real furnaces, real diffraction equipment to check what came out. Forty-one novel materials were successfully synthesized in 17 days.

Think about the verifier in that loop. Not a type-checker, not a scoring function. A physical robot in a real lab mixing real compounds at real temperatures and asking whether the crystal forms. You cannot fake a crystal. If A-Lab says it synthesized a novel stable material, it synthesized a novel stable material. The model proposed; the physical world verified; human researchers curated which structures were interesting enough to attempt.

I want to be blunt here: the pattern is not subtle. In every case: model proposes, independent verifier filters, human curates. The model is never the verifier. The model is never the question-poser. The model occupies exactly one slot in the loop, and it’s the slot that benefits most from creativity, speed, and a high tolerance for being wrong.

The replacement narrative requires you to believe the model is doing all three jobs. It isn’t. Empirically, demonstrably, across four domains, it isn’t.

Before we anatomize what changed inside the loop, it is worth noting that this shape is fifty years old. Because that changes what we should find surprising.

The Loop Has Is Older Than You Think

The loop does the discovery today in the same way it did the discovery in 1976.

I want to be somewhat insistent about this, because I think the historical amnesia around computational assistance in science is part of what makes AI discourse so unmoored. This isn’t new. What’s new is narrower than you think, and understanding exactly what changed is the only way to correctly evaluate what it means.

Appel and Haken, 1976

You probably know this one: the four-color theorem says any map can be colored with four colors such that no adjacent regions share a color. Mathematicians had been trying to prove it since 1852. In 1976, Kenneth Appel and Wolfgang Haken proved it. Using a computer. Their proof involved reducing the problem to 1,482 configurations and verifying each one computationally. No human read every step. No human could. The proof was real; the verification was mechanical.

The mathematical community was genuinely unsettled. The theorem wasn’t wrong. But the proof didn’t fit the usual epistemological frame. You couldn’t follow it the way you follow a conventional proof. You had to trust the computer. That discomfort was the first serious confrontation with what I’m calling the loop: humans posed the problem, humans designed the reduction, a computer verified the 1,482 cases, humans accepted the surviving result.

Same shape. Fifty years earlier.

Hales and Flyspeck, 1998–2014

Thomas Hales proved Kepler’s sphere-packing conjecture in 1998. The conjecture: the way you’d intuitively stack cannonballs, face-centered cubic packing, is in fact the densest possible arrangement. The proof relied on computer enumeration so extensive that the referees couldn’t verify it. They were “99% confident” and said so, which is an unusual thing for mathematical referees to say.

Sixteen years later — sixteen years — the Flyspeck project completed a formal verification of the proof in HOL Light, a proof assistant. I want to dwell on that: sixteen years. The loop had a very slow verifier in the middle. It still worked. The result was real.

Sixteen years from claim to verified closure. I find that humbling. We talk about AI accelerating science as though “fast” is a new property of loops. The loop has always done the discovery. Sometimes slowly.

AI Feynman, 2020

You won’t usually see this one in the AI-in-science timelines, but it belongs here. Silviu-Marian Udrescu and Max Tegmark built a system called AI Feynman that uses symbolic regression (searching the space of mathematical expressions) to recover physics equations from data. Feed it measured relationships between physical quantities; it proposes the equation. Tested on 100 equations from the Feynman Lectures on Physics. Human scientists posed the problems; the system proposed expressions; formal mathematical checks filtered them.

This is 2020, two years before the moment people usually date as the AI-in-science inflection. I find it clarifying: the loop is the same.

What 2022 Actually Changed

Here’s what changed. One thing in the loop changed, and it’s the proposer slot.

Before roughly 2022, the proposer in loops like these was domain-specific, narrow, and hand-engineered. AlphaFold’s architecture was designed from the ground up for protein structure prediction. The Flyspeck enumeration was written for Kepler’s problem. AI Feynman’s symbolic regression engine was built for recovering physics equations. The verifiers were already strong: formal proof checkers, physical experiments, crystallography. The curators were already human. But building a proposer required significant domain-specific engineering effort for each new application.

What changed, and I think this is the key move, is that the proposer slot is now increasingly occupiable by general-purpose large language models that can be directed with natural-language specifications.

The same model that helps Stappers explore combinatorial objects can, with different prompts, propose protein structures, generate proof steps, suggest material candidates. It’s not that the model does these things well in some absolute sense. It does them well enough that a strong verifier can find the real results in the output. And FunSearch, DeepMind’s system for mathematical discovery, goes a step further: it uses LLMs to generate the search strategy itself. AlphaEvolve extends this to evolving the algorithms. The proposer writes the proposer.

I want to flag what didn’t change: verifier reliability, human curation, question-posing. Knuth still read the logs. Tao still decides which lemmas are worth pursuing. The IMO committee still wrote the problems. The physical world still decides whether a crystal forms.

So: what exactly is the loop, and what changed inside it? The proposer and verifier are different jobs — you can confuse them, and that confusion is the mistake that both off-ramps make.

Anatomy of the Loop

The loop does the discovery. But what is the loop, exactly? It has four roles: poser, proposer, verifier, curator. They are not interchangeable.

I want to be specific about each, because vagueness here is where confused takes come from. When someone says “AI did the science,” they’re usually collapsing all four roles into one and attributing them to the model. When someone says “it’s just a tool,” they’re usually denying that the proposer role is meaningfully distinct from, say, a search engine. Both collapses are wrong.

The Proposer Role

I want to define this carefully, because the proposer role is where most confusion lives. The proposer generates candidates. That’s it. It doesn’t need to know which candidates are correct; it doesn’t need to understand the domain at the level required to verify; it doesn’t need to bear accountability for wrong answers. It just needs to produce output that the verifier can evaluate. Fast. In large quantities. With enough breadth that interesting things appear in the distribution.

This is a real and important job. Generating candidates well, in a domain, at the right level of specificity, with the kind of variety that makes the verifier’s job tractable, is genuinely hard. What LLMs are good at is producing plausible candidates in natural-language-specified domains. That’s useful precisely because “plausible” is a different bar from “correct.” The proposer buys lottery tickets; the verifier checks which ones won.

Galactica, or: What Happens Without a Verifier

In November 2022, Meta released Galactica, a large language model trained on scientific literature. I remember when this dropped: the pitch was that it could reason about science, write papers, explain concepts, generate hypotheses. It was fluent. Confident. It had read more papers than any human alive.

It was retracted in three days.

Galactica produced plausible-sounding but fabricated citations. It generated chemistry that was wrong. It stated incorrect facts with the same calm certainty it used for correct ones. The model had no verifier. It had no external check against which its proposals were filtered. It was a proposer talking directly to readers who were treating it as a verifier.

That’s the failure mode. “LLMs are bad at science” is the wrong lesson. Galactica was a capable proposer deployed without a loop. Dump the proposer’s output directly into the world, skip the verifier, and you get confident nonsense. The problem wasn’t the model. The problem was the missing loop.

The Verifier Role

I want to be careful here, because this is where the asymmetry lives. The verifier is not creative. The verifier must be right. These are the two properties that matter.

Lean’s type-checker does not hallucinate. It processes a proof term and either accepts it or rejects it, and its answer is correct by construction: it’s checking against the formal rules of type theory, and it doesn’t have opinions or moods or bad days. A crystal either forms or it doesn’t. X-ray diffraction either confirms the predicted structure or it doesn’t. These verifiers are not AI systems. They’re physics, or mathematics, or formal logic.

Here’s a slogan worth keeping: the verifier is the one that matters. A loop with a weak proposer and a strong verifier still produces valid science — it’s just slow, because it needs more proposals before one survives — this is just what “normal science has been so far”, slow proposer, strong verifier.

A loop with a strong proposer and a weak verifier produces Galactica. The asymmetry is important. You can have crap proposals and still win, as long as the verifier is robust. You cannot have a weak verifier and still win, no matter how impressive the proposer.

Tao’s Insight on Composability

Tao articulated this in his 2025 Notices piece, and I think it’s genuinely the right way to think about it: the value of the LLM-plus-formal-verifier combination comes from combining complementary weaknesses. LLMs are creative but unreliable. Lean is reliable but cannot be creative; it needs to be told what to check. Individually, each is limited. Together, the loop covers both limitations: the LLM proposes, Lean certifies, and you get results that are both novel and guaranteed correct.

This is not a new insight in the philosophy of science or even in computational science. It’s the same structure as Appel-Haken: creative human mathematicians posed and structured the problem, mechanical verification checked the cases. What’s new is that the LLM makes the creative-but-unreliable proposer slot much cheaper and more general to fill. You don’t need a domain expert to hand-engineer each proposer; you need a prompt and a general-purpose model.

What’s Genuinely New Since 2022

Let me be specific, because this matters for evaluating claims. Three things changed:

First, open-ended program synthesis. The proposer can now write code, not just fill templates. FunSearch and AlphaEvolve don’t just suggest candidate solutions; they generate the search strategy itself, which then searches for solutions. The proposer proposes how to propose. That’s a qualitative shift.

Second, cross-domain transfer at useful fidelity. The same general-purpose model can propose protein structure candidates, proof steps, crystal candidates, combinatorial patterns, all with natural-language specification. You don’t need to rebuild the proposer for each domain. The moat that used to come from hand-engineering a domain-specific proposer is largely gone.

Third, tight LLM-to-formal-verifier loop latency. The Flyspeck project took sixteen years. The Lean loop that Tao describes closes in seconds. When the loop runs fast, you can iterate fast, which means you can attempt more ambitious problems and accumulate signal faster about which directions are productive.

What hasn’t changed: humans pose the problems; the verifier is still not an LLM; which questions are worth asking remains entirely human. Knuth read the logs. Tao decides which lemma is promising. The IMO committee wrote the problems. The GNoME team decided that stable crystal structures were the target. The question of what to look for has not been automated.

With the anatomy clear, we can finally steelman both off-ramps seriously. And explain precisely why they are both wrong.

Why Both Extremes Are Wrong

The loop does the discovery, but what does that mean for who gets the credit?

I want to take both positions seriously, because I think each is tracking something real. Dismissing either one without engagement is how you end up with a position that sounds crisp but falls apart in the edge cases.

The Maximalist Steelman

Knuth is eighty-eight years old. He has spent decades thinking about the class of combinatorial objects Stappers was exploring. He hadn’t found the pattern that surfaced at exploration 15. Stappers, presumably, hadn’t either, or there wouldn’t have been thirty-one explorations looking for it.

Here’s the maximalist claim: if surprise is constitutive of discovery, if discovery is the moment something genuinely unexpected becomes known, then the model contributed something essential. It’s not that the model was incidentally involved in a process humans could have completed. It’s that the specific result didn’t exist in any human mind before the model generated it. A system that reliably surfaces what experts miss is doing something scientists do. The fact that the model can’t feel pride or bear responsibility doesn’t settle whether it participated in discovery.

The maximalist is pointing at something real. The model wasn’t executing a search strategy that a human had pre-specified. It was navigating a space in ways that produced genuinely unexpected output. That matters.

The Dismissive Steelman

A hammer drives nails. We don’t say the hammer built the house. Excel executes arithmetic the analyst specified; we don’t credit Excel with the financial model. AlphaFold was trained on structural biology data that human researchers collected over decades; the training objective was designed by human engineers; the evaluation criteria were set by human scientists. The model is a sophisticated tool. Sophisticated tools don’t discover; they execute.

The dismissive case is not purely rhetorical. There’s a serious accountability point underneath it: when a model-generated finding turns out to be wrong (and some will), who is responsible? If you’ve attributed agency to the model, you’ve obscured the accountability chain. The human who deployed the loop without a strong verifier, who curated poorly, who published without checking: that human is responsible. Diffusing credit into the model diffuses accountability too, and that’s a practical problem, not just a philosophical one.

The dismissive steelman is also tracking something real. The model didn’t choose the question. It didn’t decide the result mattered. It didn’t design the verifier. It didn’t stake its reputation on the finding.

The Verdict

Both steelmans are right about something; both are wrong about the structure.

The maximalist is right that the model contributes something essential, something that wouldn’t have been there without it. Wrong that this constitutes independent discovery: you cannot have independent discovery without the ability to pose a question, which requires caring about the answer, which requires the kind of intentionality the model doesn’t have.

The dismissive is right that the model doesn’t choose questions, doesn’t bear accountability, and shouldn’t be treated as an author in the full sense. Wrong that this makes it a mere tool in the hammer-and-nail sense. AlphaTensor, DeepMind’s system that found faster matrix multiplication algorithms, didn’t just execute a search the engineers specified. It found an algorithm that reconfigured what experts believed was achievable. That’s not executing; it’s navigating a combinatorial space in a way that produces genuine surprise. The hammer never surprises you — except when you’re not looking and it hits your finger, which by the way, AI can also do, and with a louder bang.

The right frame is, I think, the AI lab member. Indispensable, capable, sometimes surprising. Not a hammer; not a principal investigator, not a replacement for another human. Just a genuinely, qualitatively new kind of entity. An entity that occupies the proposer slot in the loop and does it better than anything that occupied that slot before, but doesn’t touch the verifier, doesn’t pose the questions, and doesn’t bear the accountability that comes with authorship.

If AI is a lab member and not a scientist, what does that do to the publishable paper, and to the metrics we use to measure science?

But What Happens to the Paper

The loop does the discovery. But the paper still needs to be written, and the paper is what the institution counts.

This is where I get more pessimistic, or at least more cautious. The loop is good news for science-the-activity. It’s more complicated news for science-the-institution.

Discovery and Paper Count Were Already Decoupled

Park, Leahey, and Funk published in Nature in 2023 a careful empirical study of scientific disruption, measuring across decades of papers and patents how often new work displaces the prior literature versus consolidates it. Their finding: disruption has been declining since 1945. Not linearly, not dramatically, but consistently. Meanwhile, paper count has been exploding.

The interpretation I find most compelling: the processes that generate papers and the processes that generate genuine advances were already decoupled before AI. A lot of papers are small increments, confirmations, replications, applications of known methods to new datasets. That’s not waste; science needs that infrastructure. But it means paper count was already a noisy proxy for discovery rate.

What an AI proposer does to this: it makes generating candidate findings cheaper. Cheaper to generate means more candidates, means more papers. Discovery and paper count decouple further.

Goodhart as Accelerant

Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. Academic institutions have been targeting paper count, citation count, journal impact for decades. Paper mills, factories producing fake or low-quality research for pay-to-publish journals, were already a pre-AI problem. Ioannidis documented the replication crisis in 2005 and again in 2018; most published findings in some fields don’t replicate, and that was before AI made generating plausible-sounding results cheaper.

The AI proposer is an accelerant deployed into a system already optimizing for the wrong thing. It makes the Goodhart problem worse. A lot worse. If you can generate a thousand candidate papers in the time it used to take to generate ten, and the verifier in your loop is peer review (slow, inconsistent, and famously gameable), then you have a problem that is structurally different from the pre-AI problem in scale.

I want to be clear: this is not an argument against AI-in-science. It’s an argument that the institution of science has a verifier problem, and AI proposers make that verifier problem more urgent. The fix is not to slow the proposer. The fix is to build better verifiers.

The Optimistic Edge

Here’s where I land, and I’m genuinely somewhat optimistic about this: if generating candidates is cheap, the scarce skill shifts upstream.

The researchers who will matter most in this environment are not the ones who can generate the most proposals. They’re the ones who can pose the right questions, who can identify which problems are worth solving before they know the answer, and the ones who can build strong verifiers. Question-selection and verifier-design become the competitive moat. That’s a real and important skill. It’s harder to fake than generating a paper. And it’s the skill that the loop most depends on.

If I’m hiring for a lab in 2026, I’m not looking for “AI scientist” as a job description. I’m looking for people who can look at a domain and say: here is what a correct answer would have to look like, and here is how I would know if I found it. That’s what the verifier is. Building it is hard. It requires deep domain knowledge, epistemological clarity, and the kind of judgment that comes from years of thinking carefully about what you’re actually trying to know.

A Diagnostic Heuristic

Let me give you something practical. When you read an AI-in-science result, a press release, a paper, a breathless tweet thread, ask this: what was the verifier, and who built it?

If the verifier is Lean, or a crystal, or experimental replication with pre-registered protocols, or a physical measurement with known error bars: trust the result. The proposer’s reliability doesn’t matter much; the verifier caught the garbage. The finding is real regardless of how the candidates were generated.

If the verifier is not named, if the paper says “we used GPT-4 to generate hypotheses and evaluated them with GPT-4 to assess plausibility,” you’re looking at Galactica with extra steps. The proposer and the verifier are the same system. The loop is not closed. Be skeptical.

That diagnostic question is the institutional choice this moment is forcing. What was the verifier, and who built it? Let’s land it.

The Verifier Is the One That Matters

“I’ll have to revise my opinions about generative AI one of these days.”

Knuth said that. Not “AI did the science,” not “it’s just autocomplete.” Something more specific: I may have underestimated what this thing can do in the right loop. The loop did the discovery. The human owned the question and the verifier. The model was the proposer. And the result was real.

The Investment Gap

Here’s the uncomfortable institutional reality: almost all current investment is in the proposer. Larger models, more parameters, cheaper inference, better fine-tuning, faster generation. The race to make the proposer better is well-funded, well-publicized, and moving fast.

The verifier is comparatively neglected.

Formal verification tools like Lean exist, but they’re hard to use, require significant expertise, and don’t cover most scientific domains. Physical verification (A-Lab-style robotic synthesis) is expensive and slow relative to the speed at which the proposer can generate candidates. Experimental replication is underfunded as a scientific activity; it’s less prestigious than novel claims. The referee system in academic publishing was designed for a world where generating candidates was the hard part. It was not designed for a world where a model can generate ten thousand plausible candidates in an afternoon.

Hiring “AI scientists” misframes the institutional need. The need is for researchers who can pose hard questions and build reliable verifiers. The “AI lab member” frame points to what needs managing: not the model, but the loop. And the loop’s bottleneck right now is the verifier.

I should mention that this piece is drawn from the science chapter of Mostly Harmless AI, a book I’m writing about what AI actually does versus what the headlines claim. If you’re finding this useful, the book is where the longer argument lives.

What Comes Next

Next week: AI in creative output. Art, literature, music. Where the verifier question changes shape entirely. Because in mathematics and materials science, you can at least define what “correct” means. In creative domains, the verifier problem is not just harder. It’s constitutively different. What does it even mean for a creative proposal to be correct? I have some thoughts, and I think they’re going to make the science case look straightforward by comparison.

Stay curious.

P.S. There’s a Subscribe button somewhere below this. I’m told it does something useful. My understanding of subscription mechanics is below the level of a confident stochastic parrot. I believe it works, but I haven’t checked every step. Click it anyway. The verifier here is whether you keep showing up next Monday.

Subscribe now

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI — 50% off during early access — where the Limitations chapter walks all seven failure modes that compound into the curve below. You can also read the whole book online for free. More at the end.

The headline number you’ve seen on every AI-progress chart — “model X completes two-hour tasks half the time” — is the 50% reliability horizon. That number is moving fast. Doubling every seven months, per METR’s time-horizon work. It’s the curve on every AI-progress chart, the one conference talks lean on, the one that lands in investor decks.

The number that decides whether you can actually deploy an agent is a different one.

The 80% reliability horizon — the task length at which an agent finishes well enough that you would not feel the need to check — sits 70–80% below the 50% figure, and it moves up far more slowly. That gap is the difference between demo and deploy. The 50% is what passes the eval. The 80% is what survives the afternoon you weren’t watching. Not two hours. Thirty minutes you’d hand off.

I want to be precise about what I’m not arguing. I’m not arguing agents are broken or that AI progress isn’t real. It is real, and it’s fast. I’m making the narrower claim that the 50% and 80% horizons move at different speeds, and that the 80% is the one that matters when someone else’s data is on the line. This post is the math behind the gap, why it’s structural, and what you can do about it.

If you’re building on agents, you are building on the 80% horizon. The 50% number is for the marketing deck.

Subscribe now

Two horizons, two stories

METR’s methodology is clean: take a population of tasks with measured human completion times, then find the longest task a given model clears at success rate X. Do that for X = 50%, and you get the 50% horizon. Do it for X = 80%, and you get a different curve. A different story.

The 50% horizon has been doubling roughly every seven months. Late 2025, it sat around a couple of hours for software tasks. That’s the curve that makes headlines. That’s the curve you’ve seen on every slide. Striking.

The 80% horizon sits roughly 70–80% below. The same agent that clears a two-hour task half the time clears a half-hour task four-times-in-five. Not two hours. Thirty minutes. And that gap doesn’t close at the same rate. It moves slowly, stubbornly, for reasons that are mathematical before they are engineering.

So you have two curves growing at different speeds. The 50% horizon is the curve of capability: what can this system do, under ideal conditions, at least sometimes. The 80% horizon is the curve of trust: what can this system do reliably enough that you’d hand it a production key and walk away.

They are not the same curve. And they do not close the same way.

The longer your task horizon, the wider the gap between can-do-it-sometimes and can-be-trusted-with-it. The mechanical reason is one piece of math.

Probability arithmetic

Here’s the setup. A language model you’re calling spends a fixed compute budget per output token. Each step in a multi-step process has some per-step success probability p that is strictly less than one. The model is stochastic, the world is noisy, context degrades.

String n steps together, and the probability that all of them succeed is roughly pⁿ. That’s it. That’s the math.

Here’s what that looks like with actual numbers. Suppose your agent is excellent: p = 0.99 per step. That’s a 99% success rate on any single action. Compound it over 100 steps: 0.99¹⁰⁰ ≈ 0.37. You’ve gone from near-certain to worse-than-a-coin-flip without anyone making an error. Now drop to p = 0.95 (still quite good, still 95% per step). Over 100 steps: 0.95¹⁰⁰ ≈ 0.006. Six in a thousand runs succeed.

This is not a gap you close with next year’s training run. It is the shape of any probabilistic process operating in sequence over time. The curve doesn’t change when you improve p; it just shifts outward.

Reasoning models — the o-series, R1, extended-thinking variants — are valuable here. They buy you a higher per-step p, and they let you spend more steps at that higher rate. Both matter. But they push the curve outward. They do not change its shape.

Two pieces of evidence you should hold next to that math. GSM-Symbolic (Apple, 2024): perturb a math problem the model has seen (swap a name, change a number) and accuracy collapses even when the reasoning path is identical. The model has memorized the route, not the reasoning. Faith and Fate (Microsoft, 2023): transformer accuracy degrades with computational-graph depth even when each individual sub-step is solvable in isolation. Depth itself is the failure axis. More steps means more surface for p < 1 to accumulate.

Reasoning models buy you a higher per-step p and more steps to spend. They do not change the shape of the curve.

Where the chain gets long

Agents are exactly the setup that makes pⁿ painful.

Think through a typical agent run: read prompt, plan, call tool, read result, call tool again, critique output, adjust plan, call final tool, write response. Seven steps if you’re being generous. A real production agent reaches hundreds. Each step is one more p rolled. Each tool call is one more chance the orchestrator hands the tool the wrong arguments — garbage in, deduction out.

Self-critique doesn’t repair this — and you can verify the result yourself if you’ve tried it. Huang and colleagues (2024) showed that intrinsic self-correction without an external oracle signal actually degrades performance. The model talks itself out of correct answers as often as it talks itself in. The paradox is clean: if the model could recognize the error, it would not have made it. Asking it to introspect on failures is asking the broken compass to check itself.

So let’s put numbers on a real scenario. An agent that succeeds on each of five steps 95% of the time lands at 0.95⁵ ≈ 0.77. Decent. Not great, but workable. Now extend that same agent to a fifty-step trajectory: 0.95⁵⁰ ≈ 0.08. Eight runs out of a hundred finish correctly.

The demo ran five steps. The deploy runs fifty. The demo and the deploy are two different machines.

That’s the 80% horizon you’ll actually feel in production. It’s not a philosophical concern about AI reliability in the abstract. It’s the arithmetic of what happens when you take a stochastic generator and ask it to maintain a chain of reasoning over a long enough trajectory that pⁿ has time to do its work.

What you can actually do

Three mitigations. Each one genuine, and each one with a ceiling you should know before you commit.

Verifier-shaped tasks. Where the output can be checked deterministically (arithmetic, code that compiles and runs, SQL that parses, a formal proof), you can recover trust that the probabilistic generator alone cannot provide. A SAT solver beats an LLM on deductive closure every time. The architecture that wins here is LLM-proposes-candidate, deterministic-system-signs-off. The generator explores the space; the verifier approves the exit. This is, incidentally, the same pattern Monday’s post on the seventy-year argument named: a deterministic shell around a stochastic core, applied at the task level rather than the system level. The twist is that not every task has a fast verifier. Code that runs is checkable; code that runs correctly for all future inputs is not.

Retrieval-augmented generation. If the fact your agent needs is no longer arbitrary recall but lives in a curated document the model is required to cite, then Kalai and Vempala’s 2024 lower bound on calibrated hallucination does not apply to that fact. Most agent failures upstream of a tool call are recall failures the agent doesn’t know it’s making; retrieval changes the error mode from confident confabulation to visible gap. RAG turns a free-running generator into a paraphrase-and-summarize system over a known corpus. The reach of the system is now bounded by the reach of the index. But anything outside that index is back to pure p < 1 territory.

Narrow the horizon. The cheapest move is the one nobody wants to make: don’t deploy your agent on a fifty-step trajectory. Cut it to five. Hand off to a human at the boundary. At five steps with p = 0.95 you’re at 0.77; at fifty steps you’re at 0.08. That’s not a small difference. That’s the difference between a tool that works and a demo that occasionally works. Now, this trades autonomy for reliability. That trade is worth making in most production contexts right now. Whether it’s worth making in your context is a product question, not a research question.

Watching the right number

The 50% number will keep doubling and you should track it. That is real progress and worth watching closely.

But it is not the number your users feel. The number your users feel is whether the agent finished their task well enough that they didn’t have to re-run it, check its work, or clean up after it. The difference between “I tried that AI agent thing and it was magic” and “I tried that AI agent thing and it broke my Friday” is roughly the distance between the 50% horizon and the 80% horizon at your task length.

The shape of the next several years of agent engineering is already visible in the mitigations you’ll be reaching for: deterministic verifiers around stochastic generators, retrieval around recall, short trajectories with human handoffs where the math demands it. Not because agents are weak. They’re remarkable. But the pⁿ curve doesn’t care about benchmark scores. It cares about chain length.

One number, slowly creeping upward, every quarter. Watch that one.

Until next time, stay curious.

If the 80%-horizon framing landed, the second edition of Mostly Harmless AI walks the seven failure modes that produce the curve — the calibrated-hallucination lower bound, the U-shaped attention curve, the reversal curse, the depth ceiling on deduction, the rest. 50% off during early access. You can also read the whole thing online for free in a custom reader I built — dark mode, font controls, offline support, the works.

Get Mostly Harmless AI - 50% off

And if you want everything I’ve written, plus everything I’m going to write, that’s the Compendium. One purchase, in perpetuity.

Part of the run-up to the second edition of Mostly Harmless AI — 50% off during early access — where this is the spine of a new chapter on generative and multimodal AI. You can also read the whole book online for free. More at the end.

One morning, not so long ago, perhaps you asked Claude (or Gemini, or ChatGPT) to do something for you, and decided it was easier to just give it a picture of it than explain the whole thing. Perhaps it was “how do I cook this thing?” or “what building is that?” or “do this homework for me, please, please, my live depends on it”. Then you uploaded the picture, and back came a textual response.

Not happy with what the bot understood, you decided a thorough explanation was owed. But, alas, since all we got is a couple fatty fingers for typing, you decided it was best if you explained it with your own voice. And again, uhms and ehms notwithstanding, you again got a full response back, this time with an audio voice over.

Ten years ago, this simple dance of back-and-forth multimodal information would have required four separate research fields, each with its own conferences, its own vocabulary, and its own priesthood. They have quietly become one single thing. It’s all tokens all the way down. Language has subsumed all modalities. This is how.

Subscribe now

The recipe never cared what it was eating

Strip “generative AI” down to the one idea doing all the work and you get a single sentence: look at a big pile of examples, learn the distribution that produced them, then draw new samples from it. That is the whole trick. It is what a language model does, and it is the only thing a language model does. It is also what an image model does, and an audio model, and a video model.

The recipe is indifferent to what the examples are. Text is a one-dimensional run of symbols. An image is a two-dimensional field of colour. Audio is a pressure wave sampled tens of thousands of times a second. Video is all of that, plus time, which is why it is the hardest. Four different shapes of data, one identical question asked of each: given what I have seen so far, what plausibly comes next? The machinery that answers that question does not need to know whether “next” means a word, a patch of pixels, or a slice of waveform. It only needs the data turned into a sequence of countable things.

Tokens.

So the thing we have been calling a language model was never really about language. Or, put better, it was never about written language. It turns out, language is something far more powerful.

Ask any linguist, and they’ll say any set of sequences of distinct symbols (tokens) can be modelled as a language. It doesn’t matter what your symbols are—letters, words, patches of images, numbers in a math formula, whatever—language is just the structure around them, what makes some sequences valid and others nonsense.

This is the key idea. All else is (incredibly good) engineering.

A decade of building the same machine, separately

It did not look that way while it was happening. For about a decade every modality got its own bespoke contraption, and each one looked like its own discipline.

Image people had generative adversarial networks: a forger and a detective locked in a training duel, the forger getting better at faking until the detective could no longer tell. The beautiful idea buried in there — and the one that survived the technique itself — was the latent space: a compressed interior map of “all possible faces,” where walking in a straight line morphs one plausible face smoothly into another. GANs were temperamental, prone to collapsing into a single good fake and refusing to leave, and by the early 2020s they had lost the lead. The latent-space intuition outlived them and runs underneath everything that came after.

Then diffusion took over image generation with a trick that sounds like it shouldn’t work. Take a real photo, add a little static, add a little more, keep going until it is pure snow. Now train a network to undo one step of that. To make a new image, start from snow and run the undo, over and over, until something coherent surfaces. It is sculpture by removing noise instead of removing marble, and it is what powers essentially every image generator you have used.

Audio had its own separate lineage: speech-to-text built one way, text-to-speech another, music a third. Text had the large language models, off in their own enormous-budget corner of the field. Four communities, four sets of architectures, four sets of war stories. If you had asked, in 2021, whether the image people and the language people were building the same machine, both sides would have laughed.

CLIP quietly knocks out the wall

The crack in the wall came from a 2021 model whose job sounds almost too modest to matter: teach one system that the word dog and a photograph of a dog are talking about the same thing.

The way you do that is to train a text encoder and an image encoder together, on hundreds of millions of caption-and-picture pairs, with one instruction: put a picture and its true caption close together in a shared space, and shove mismatched pairs apart. What you get at the end is a single space where “a photo of a golden retriever” and an actual photo of a golden retriever land as neighbours. Text and pixels, in the same room, with the same coordinates.

That sounds like a party trick for image search. It was the hinge the whole field turned on. Once text and images live in one space, text can steer image generation — point the diffusion process at the region of the space that means “golden retriever in a spacesuit,” and let it denoise toward there. Every text-to-image system you have used is, under the paint, that move. And the deeper implication was harder to ignore than the application: if you can put two modalities in one space, the wall between them was never structural. It was just a wall nobody had walked through yet.

Tokens all the way down

Here is where it lands. By the mid-2020s the bespoke machines stopped being separate machines.

The move is almost embarrassingly direct. Tokenise everything. Text already broke into tokens. Cut an image into a grid of patches and treat each patch as a token. Run audio through a neural codec that emits discrete chunks, and those are tokens too. Now you do not have a text stream and an image stream and an audio stream. You have one stream of tokens that happen to have come from different alphabets, and you train a single model on the only objective that was ever in play: predict the next token, whatever kind it is.

A model trained that way reads and writes everything, because to it there is no “everything” — there is just the sequence and the next position in it. You have used these. The one that holds a spoken conversation with sub-second latency, looks at the photo you paste in, and writes you a paragraph back is not a language model bolted to an image model bolted to a speech model. It is one model that was never told these were different problems.

Which is why the question that organised the field for a decade — is this a language model or an image model? — has quietly stopped having an answer. It is the same machine. The only thing that ever changed between text and pixels and sound was the alphabet, and the transformer emitting the next token has never cared which alphabet it is spelling in. It is tokens all the way down. “Language modelling” was a local name for something with no allegiance to language at all: modelling sequences of anything we can count.

The honest part

It would be easy to end on the astonishment, and the astonishment is real. One model, every modality, falling out of one stubbornly simple objective applied to a wider and wider definition of “token” — that is one of the genuinely beautiful results of the decade, and the kind of unification that does not come along often.

But unification is not the same as understanding, and I am not going to let the elegance smuggle that past you. A system that can place “dog” next to a dog in its latent space has learned the statistics of how dogs are described and depicted.

Whether it has learned what a dog is is a different question, and the convergence story does not answer it. It just makes the question apply to every modality at once instead of only to text. The machine got more general. It did not get more grounded necessarily. Both of those are true at once, and the interesting work of the next few years lives in the gap between them.

Until next time, stay curious.

This is the core argument of a new chapter in the second edition of Mostly Harmless AI — the full chapter walks GANs, diffusion, CLIP, audio, and native multimodality with the scenes and citations this post had to cut, and it is 50% off during early access. The whole book is also free to read online. If you want the rest of the argument — how these systems are trained, where they break, and what to actually do about it — that is what the book is for.

Get it (50% off) for life

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI — 50% off during early access — where the history below is the full ~8,000-word opening chapter, with 70+ references and all the scenes this post had to cut. You can also read the whole book online for free. More at the end.

Seventy years ago, two men sat in two different rooms and disagreed about what a thinking machine should look like. Neither of them has been proven right. Both have been proven half-right, several times, in alternation, for the whole of my lifetime and most of yours.

I think the entire history of AI is that one argument, still going.

The first camp wanted to build minds out of rules. Feed the machine enough knowledge in a logical enough form, and reasoning falls out of the logic. It called itself a lot of things over the decades — symbolic AI, knowledge-based systems, good old-fashioned AI — but its home is rationalism. The second camp wanted to build minds out of examples. Feed the machine enough data, in any messy form whatsoever, and behavior falls out of the statistics. It also kept renaming itself — connectionism, machine learning, deep learning — but its home is empiricism. Same goal, a machine that does what intelligent people do. Sixty years of disagreement about how.

Here’s the ending, spoiled early, because this isn’t a thriller. The argument did not produce a winner. It produced a marriage. The chatbots, the image generators, the agents writing code while you sleep — none of them is one side beating the other. They’re both sides, finally forced to share a workshop. Let me walk you through how we got there. Fast.

Subscribe now

Both seeds, one summer

They were planted within five years of each other. In 1943, McCulloch and Pitts wrote down a neuron as a weighted sum with a threshold — twelve pages, the seed of the empiricist branch. In 1950, Turing refused to define thinking and proposed a behavioral test instead, a question both camps could chase. In 1956, ten people spent a summer at Dartmouth, coined the phrase artificial intelligence, and planned to crack language and reasoning in a few months. (We are still working on it.) In 1957, Rosenblatt built the Perceptron, the first machine that learned from examples, and the New York Times announced it would soon walk, talk, and be conscious of its own existence.

Two foundational myths, in the ground, in the same decade. The rest is which one got watered.

The rationalists win the first round

And they win it convincingly. In the 1950s and 60s compute is tiny and data, in the sense of millions of labeled examples, does not exist.

What you can do is write a program that does something specific and inspect every step of it. So the symbolic camp gets the better results and the better tools. Newell and Simon’s theorem-prover. McCarthy’s LISP. Weizenbaum’s ELIZA — four pages of pattern-matching that understood nothing, and that people confided in anyway. (Hold onto ELIZA. The field will relearn that exact lesson about six more times.) Winograd’s SHRDLU, fluent and thoughtful inside a closed world of colored blocks.

The catch was always the world. SHRDLU’s blocks could all be known, listed, reversed. The real world has rain, and grandparents, and the smell of coffee, and you cannot list it. In the closed world of symbols, symbols were enough. The next decade was about discovering, painfully, that the world is not closed.

The cost of winning too hard

In 1969, Minsky and Papert published Perceptrons and proved a single-layer network can’t compute XOR.

The proof was correct. It was also narrow — they admitted multi-layer networks could do it, nobody just knew how to train them yet. But the field was hungry for a verdict, and it read the book as one. Funding for neural networks collapsed. Rosenblatt died two years later in a boating accident, on his 43rd birthday. The algorithm that would resurrect his branch didn’t arrive at scale until 1986. Seventeen years of silence.

Modern AI runs on the work of people who weren’t born when Minsky and Papert published. The reason their work came so late is that the field they’d return to had been kept near-dead for two decades. The symbolic camp’s victory was real. The field paid for it. It will pay that bill again.

The rationalist trap

Through the 1970s and 80s the symbolic branch found something that made money: expert systems.

MYCIN matched infectious-disease specialists. XCON saved DEC tens of millions a year. The thesis was clean and seductive — intelligence is rules plus facts; hire the expert, extract the rules, ship the system. And these systems were legible. You could read every rule, audit the reasoning, fix the wrong line. (Your favorite large language model cannot do this. We’ll come back to that another day.)

Two problems killed it. Common sense turns out to be unrepresentable in rules — birds fly, except penguins, except baby penguins, except dead ones — and the rules contradict each other faster than you can write them. And then there’s Cyc: in 1984 Doug Lenat set out to hand-encode all of common-sense knowledge, estimated ten years, and is still at it forty-two years later. It is the most thoroughly humbling monument in the history of cognitive science.

By the late 80s the money dried up and the Second AI Winter set in. The field was tired of the rationalists.

The empirical rebellion

We’re on 1986: backpropagation, in Nature, multi-layer networks are finally trainable.

Then the empiricist branch spends fifteen years not scoring one big win but a thousand small ones. Support vector machines. Random forests. Boosting. Statistical methods quietly eating one application after another, including the symbolic camp’s home turf — language, where IBM’s speech team found that every time they fired a linguist, the system improved.

Why now? Three things are moving together, slowly. Compute grew. The internet started producing data in volumes nobody had imagined. And the methods were simple enough to scale with both.

In 2019 Richard Sutton would name this The Bitter Lesson: across seventy years, the general method that scales with compute beats the clever hand-engineered one, every time. It’s bitter because it tells researchers their hard-won taste gets steamrolled by someone with more GPUs. It is mostly right.

The thing that complicates it is the thing symbolic AI was good at all along — but I’m getting ahead of myself.

The earthquake

Now jump to September 2012.

AlexNet — eight layers, two gaming GPUs, a couple of training tricks — drops the ImageNet error rate ten points below the nearest hand-engineered system. A ten-point gap isn’t an improvement. It’s a different category of result. Within six months every computer-vision lab on Earth has pivoted. AlexNet is, by a wide margin, the single most consequential paper in modern AI.

Then it cascades, almost too fast to track. Sequence-to-sequence translation. GANs. Atari from raw pixels. In 2016 AlphaGo beats Lee Sedol at a game with more board positions than there are atoms in the universe — and almost nobody notices that inside it is a deep network (empiricist) wrapped around a tree search (symbolic). The marriage is already there, in 2016, hiding in plain sight. In 2017, “Attention Is All You Need” introduces the Transformer, and every model in your chat window today descends from that one paper.

The crown jewel nobody talks about

The most consequential AI system of the modern era is not a chatbot. It doesn’t write poems. It’s in the bloodstream of structural biology.

Predicting a protein’s 3D shape from its amino-acid sequence is a fifty-year-old problem. The rationalist approach — simulate the physics — was beautiful and almost completely intractable. For twenty years the field’s hardest benchmark plateaued at a score around 40. In 2020, DeepMind’s AlphaFold 2 scored above 92 on that exact tier. The grand challenge was, for practical purposes, solved. Hassabis and Jumper got the 2024 Nobel in Chemistry for it — the only AI work so far to produce a Nobel-level scientific breakthrough.

Read the citation. It isn’t about AI as a technology. It’s about a problem that got finished while the people whose careers were defined by it slept. The chatbots get the headlines. The image generators get the lawsuits. The protein folder got the world. Remember that the next time someone wants to tell AI is ChatGPT.

The synthesis

Now the marriage. The word agent did not come from machine learning. It came from classical, symbolic AI in the 1970s and 80s: a system that perceives its environment, deliberates, picks an action, acts, observes, loops. The architecture was right. The brain was missing. Pure symbolic computation could never model a world with grandparents and coffee in it, so the agent shell sat there for decades, structurally correct and operationally empty. Cyc, again, is the long sad proof.

The empiricists borrowed the same word in the 2000s — in reinforcement learning, an agent is a learned policy. DQN was an agent. AlphaGo was an agent. A new brain, slotted into the old shell. Spectacular, and narrow. An AlphaGo cannot make you a sandwich.

In 2024 the cognition slot gets filled a third time, by a general-purpose reasoning language model. The shell is still the seventy-year-old symbolic frame: perceive, deliberate, pick an action with a name and a meaning — read_file, run_tests, send_email — act, observe, loop. The brain is now an LLM. From the empiricist side the system inherits flexibility: it has read enough of the world that you don’t have to tell it what a file is, or what an angry customer sounds like. From the symbolic side it inherits structure: the actions have names, the consequences are bounded, the trajectory is auditable. The model can hallucinate; the system can’t run rm -rf unless somebody wired that action in and granted it.

The 1970s symbolic agent could never reason. The 2010s RL agent could never generalize. The 2026 agent does both — badly, often clumsily, but for the first time at the same time. You can watch this happen most clearly in software development right now: a language model at the core, a harness of tools around it, a test suite as the verifier, a human reviewing the diff. All four layers, on a laptop, today. Software is the canary. The same pattern is already moving toward research, then education, then everything whose feedback loops are fast enough.

So here is the closing claim, the techno-pragmatist version of the whole story. The seventy-year argument did not produce a winner. It produced three layers: a learning substrate that absorbed the written record of humanity, a symbolic shell that makes it accountable, and a human frame that decides what the whole thing is for. The first two are engineering. The third is the only one that was ever really about us.

The synthesis exists. What we do with it is still up to us.

Until next time, stay curious.

This post is the speedrun — the book’s ~8,000-word opening chapter compressed down to its spine. The full version in the second edition of Mostly Harmless AI has every scene with its characters, the eras this post skipped, 70+ references, and the agentic stack the rest of the book then takes apart mechanism by mechanism. It’s the book I wish someone had handed me when I was trying to make sense of the noise — and it’s 50% off while it’s in early access. You can also read the whole thing online for free in a custom reader I built and am rather proud of: dark mode, font controls, progress tracking, offline support, the works.

Get it (50% off)

And if you want the whole catalog of everything I’ve written, plus everything I’m going to write, that’s the Compendium. One purchase, in perpetuity.

You type a message to an AI assistant and it answers. The answer isn’t looked up. It isn’t scripted. The model generated it, character by character, guided by a single mathematical question: what comes next?

That question is the foundation of every language model ever built. A language model is a probability distribution over text — a function that, given a sequence of words, assigns a probability to every possible continuation. “The cat sat on the mat” scores higher than “the mat sat on the cat” not because a language model understands what cats do, but because the first sequence appears in human text and the second doesn’t. The model has compressed the co-occurrence patterns of an enormous corpus into its weights, and that compression is what produces a score.

Generative: you give the model a prefix, it samples the highest-probability next token, appends it, samples again, and repeats until a stop token arrives. The multi-paragraph response you got from ChatGPT this morning was that loop running a few hundred times. No lookup table. No if-else tree. No pre-scripted answers. Just: given all of this text, what is most likely to come next?

Here’s the thing I find quietly strange about this: it works. A procedure this simple — assign probabilities, sample the maximum — has produced the most influential technology of the last decade. What makes it work isn’t the procedure. It’s everything that goes into building a probability distribution that’s actually good. Good enough to write coherent paragraphs. Good enough to reason about code. Good enough to pass the bar exam and explain quantum mechanics in language your parents can follow.

Getting there took decades of compounding ideas. The arc is what this piece covers — from the crudest possible approximation of “probability over text” to the current frontier, where models are learning to think.

Each section of what follows is best understood as a response to the failure of the previous one. N-gram models worked until they didn’t. Neural embeddings fixed the part that broke. Pretraining scaled the fix to the size where it became genuinely impressive. Instruction tuning made the result useful for the first time. Preference learning fixed what instruction tuning couldn’t. Reasoning models added something nobody was sure was trainable at all.

Seven steps. One direction.

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI, where the engineering details that don’t fit a blog post live. You can also read the whole book online for free in a custom reader I built. More at the end.

This post may be truncated in your email. Read it online for the best experience.

Subscribe now

The n-gram intuition

The simplest possible implementation of “probability over text” is a lookup table.

Take a large corpus — a hundred million words will do to start. For every three-word sequence (trigram) in that corpus, record which word follows it most often, and with what frequency. “The quick brown” → “fox,” nine times out of ten, because Project Gutenberg is full of that particular sentence. “The capital of” → “France” thirty percent of the time, “Germany” twelve, “England” eleven, and so on through the geography. For every trigram you’ve seen, you have a probability distribution over what comes next.

Now generate text. Give the model “The quick brown” and it continues: “fox.” Give it “jumped over the” and it continues: “lazy.” Three words in and it’s generated “The quick brown fox jumped over the lazy” — and if you’re lucky, it lands “dog” and you’ve reproduced a famous sentence entirely from corpus statistics. Locally, it’s plausible. You could read a sentence of this and not immediately know you’re looking at a machine.

The problems start fast. By the third sentence, the model has no idea it was talking about a fox. It only remembers the last three words. You ask it to continue “The fox had been running from the” and it has no idea that a fox is involved, or that running happened, or that there’s a pursuit in progress. It just has three words and a lookup table. The output is statistically English. It is not coherent.

This is the Markov assumption: the next word depends only on the last N words, not on the full history of the text. For N=3, it’s a trigram model. You can increase N — five-gram models were standard in commercial speech recognition for years — but the table explodes. Fifty thousand words in the English vocabulary gives 50,000³ possible trigram contexts, roughly 125 trillion entries. At N=10, the number of possible sequences exceeds the estimated number of atoms in the observable universe. The table can never be complete enough to cover the distribution.

There’s a real engineering solution to the “we haven’t seen this exact trigram” problem: smoothing and interpolation. Estimate the probability of an unseen N-gram from shorter sub-sequences. Hidden Markov models formalised this in a probabilistic framework that, by the 1990s, had enough polish to power industrial speech recognition and early machine translation. I don’t want to undersell it — it worked. It was genuinely useful. It just topped out.

The wall is fundamental. Real language has dependencies that can be arbitrarily long. “The man who sold the car that had been parked in front of the house where my grandmother lived was finally found.” The subject of “was finally found” is seventeen words and three nested clauses back. No N-gram model reaches it. You need something that can condition on the full context — or at least compress the full context intelligently — rather than amnesiacally forget everything more than N words ago.

You need a model that generalises from sequences it has seen to sequences it hasn’t. A lookup table can only interpolate from what it’s seen before. What you need is something that has understood the pattern deeply enough to extrapolate.

N-gram models work until they don’t — and they don’t beyond a few words.

Words as numbers

Neural networks can learn the compression n-gram models can’t achieve. But they have a hard prerequisite: they operate on numbers. Words are symbols. Before a neural network can do anything useful with text, you need to represent words as vectors. The naive approach throws away everything that matters.

The obvious first attempt is one-hot encoding. Vocabulary of 50,000 words; each word is a vector of length 50,000 with a single 1 and 49,999 zeros. “Cat” is at position 4,312; “dog” is at position 17,846; “carburetor” is somewhere else entirely. The problem: nothing in this representation suggests that “cat” and “dog” are more similar to each other than either is to “carburetor.” The distance between every pair of one-hot vectors is identical. You’ve handed the network a symbol system with no structure, and it has to reconstruct the structure from scratch — spending enormous capacity learning that cats and dogs are both animals, that both appear near “fur” and “vet,” that “cat food” and “dog food” are structurally related — before it can learn anything about how language actually works.

The key insight that resolved this came from linguistics, not machine learning, and I think it’s underrated as an idea. J.R. Firth, writing in 1957: “you shall know a word by the company it keeps.” The distributional hypothesis. Words that appear in similar contexts — near similar neighbouring words, in similar grammatical positions — tend to have similar meanings. “Cat” and “dog” both appear near “pet,” “feed,” “vet,” “owner,” “fur,” “collar.” The context is a fingerprint of the meaning. Encode that fingerprint in a vector and you have a representation where similar words land close together in space.

Word2Vec (Mikolov et al., 2013) turned this into a training procedure. Train a shallow neural network to predict a word from its surrounding context words, or vice versa. Force each word’s representation down into a dense vector of, say, 300 floating-point numbers. Train on a billion words of text. The network learns that words appearing in similar contexts should have similar representations, because that’s what makes the prediction task cheaper. Words with similar distributional patterns end up with similar vectors — not because anyone programmed that, but because it follows from the objective.

The result that made people pay attention: vector arithmetic encodes semantic relationships. Take the vector for “king,” subtract the vector for “man,” add the vector for “woman.” The nearest vector in the resulting space is “queen.” Paris minus France plus Italy is approximately Rome. Try it yourself: it works because the structural relationship between “king” and “queen” is parallel to the relationship between “man” and “woman” in how the four words co-occur with everything around them. No one wrote these analogies in. The geometry of the space mirrors the structure of meaning, because both are implicit in how words appear together in natural language.

I find this genuinely strange, in the best possible way. You trained a network to do a simple word-guessing task on flat text, and the side effect was an algebra of concepts. The geometry was always latent in the co-occurrence patterns. Word2Vec just made it legible.

Modern language models don’t use Word2Vec as a separate preprocessing step — the embedding representations are learned jointly with the rest of the network during training on text. But Word2Vec’s intuition is why learned embeddings work at all. Once language is geometry, gradient descent has a surface to grip. You can compute distances, optimise them, stack arbitrarily deep networks on top, and train the whole thing end-to-end.

Embeddings are how we lie to neural networks in a useful way. We pretend words are points in space, so the math works out.

Pretraining

Now scale it.

Take a deep neural network — not the shallow two-layer thing in Word2Vec, but a transformer with dozens or hundreds of layers, billions of parameters, and an attention mechanism in every one of them. Feed it next-token prediction across the entire accessible internet: Wikipedia, GitHub, every book ever digitised, every forum thread, every research paper, every recipe, every political argument, every user manual for every piece of machinery ever manufactured. Same objective the n-gram model had: given what came before, what comes next? Except now the model has billions of parameters to compress the patterns into, the training signal is trillions of tokens, and the architecture is built to handle arbitrarily long context.

The architecture is what made everything else possible. The transformer (Vaswani et al., 2017) uses self-attention as its core operation. For each token in the input, self-attention computes relevance weights over every other token in the sequence — learned weights, computed from the data, different for each token, different in each layer. A pronoun can attend strongly to the noun it refers to, twenty positions back. A closing argument can reach back to the premise from the opening paragraph. There is no fixed window; the model considers, in principle, the full context at every step.

This is what broke the n-gram scaling wall. Not a bigger lookup table. Not smarter interpolation. A learned, flexible attention mechanism that compresses long-range dependencies into the model’s weights rather than trying to enumerate every possible context sequence. The key property, and it’s the one I keep coming back to: soft. Self-attention doesn’t pick one relevant token; it blends all of them with learned weights. The whole sequence contributes to every prediction, with a learned notion of how much each part matters.

The other critical property is self-supervised learning. There are no human-provided labels anywhere in pretraining. The text itself is the training signal. Show the model “The capital of” and ask it to predict “France.” It’s wrong; the gradient flows; the weights update. Show it three trillion tokens; let the gradient flow three trillion times. The entire digitised corpus of human knowledge is your training set, with zero labelling cost, because the next token is always right there.

Kaplan et al., 2020 measured loss as a function of model size, dataset size, and compute over seven orders of magnitude. The result: loss falls as a clean power law across all three dimensions. Double the parameters, get a predictable drop in loss. Double the training data, same. Scale is not a bet on something uncertain; it is a known return on investment, measured and re-measured across a staggering range.

Hoffmann et al., 2022 — the Chinchilla paper — corrected a real error in how the field had been applying Kaplan’s result. Most large models of 2020-21 had been trained on far fewer tokens than their parameter count justified. The Kaplan result suggested scaling up models as fast as possible; Hoffmann’s finding was that you need to scale model size and training data together, roughly equally, for a given compute budget. A 70B-parameter model trained on 1.4 trillion tokens outperformed a 280B-parameter model trained on fewer tokens at the same total compute cost. Smaller model, more data, better result. Scale both dimensions together.

What do you get at the end of all this? A base model. And this is the part that surprises people who haven’t seen one.

Type “What is the capital of France?” into a raw pretrained model and it continues the text. Maybe it writes out a geography quiz — “What is the capital of France? What is the capital of Germany? What is the capital of Italy?” Maybe it generates a fake Wikipedia article. Maybe it starts a trivia show transcript. It has not answered your question. It has found the most probable continuation of your prompt, given everything it absorbed during training.

The base model has absorbed more text than any human could read in a thousand lifetimes. The co-occurrence patterns of the entire digitised corpus of human writing are in those weights. It knows facts, relationships, styles, concepts, code, chemistry, poetry, legal prose, and every other form in which humans have arranged words.

It was trained to continue, not to respond. Ask it a direct question and it treats the question as the opening line of some text pattern — one it will extend in whatever direction seems most probable. It has no concept of “you asked me something and I should answer it.”

Pretraining gives a model knowledge. It does not give it manners, opinions, or any idea what you want from it.

Instruction tuning

Step one of making a base model useful: show it what “useful” looks like.

Collect thousands of demonstration pairs. A human writer sits with a prompt — “Explain the difference between supervised and unsupervised learning in plain English,” “Write a polite email declining this meeting invitation,” “Debug this Python function” — and writes the ideal response. Then fine-tune the pretrained base model on these (prompt, response) pairs using the same next-token objective, now applied to curated demonstrations rather than the open web.

This is supervised fine-tuning, or SFT. It is plain supervised learning — the same paradigm that has been in the machine learning textbooks since the 1980s. What’s new is only what it’s being applied to.

The headline result from Ouyang et al., 2022 — the InstructGPT paper — is still worth stating plainly: a 1.3 billion-parameter model, fine-tuned on human-written instruction-following demonstrations, was preferred by human evaluators over a raw 175 billion-parameter GPT-3. One percent of the parameters. Preferred.

Sit with that. The quality of the training signal matters more than raw scale. A carefully curated set of demonstrations of what “helpful answering” looks like is worth more, for the specific goal of being helpful, than a hundred times more parameters trained on unstructured internet text. The base model knows more. The instruction-tuned model is more useful. These are different things.

SFT teaches the shape of a helpful answer: addressed to the question asked, reasonably structured, proportionate in length, appropriate in tone. These are learnable patterns. The base model already has all the relevant knowledge in its weights; SFT is teaching it to retrieve and present that knowledge in a particular format.

Here’s the failure mode, and it matters for understanding everything that comes next.

SFT shows the model what good answers look like. It gives no mechanism for the model to evaluate, at generation time, which of two candidate continuations is more accurate, more honest, or less likely to cause harm. The model learned to imitate the shape of correct answers; it did not learn to prefer correctness over fluency when the two conflict. A confidently phrased wrong answer and a confidently phrased right answer can look identical from a format standpoint. SFT cannot distinguish them.

Teaching consistent refusals is especially brittle. To get a model to reliably refuse a class of harmful requests via SFT, you need human-written refusals for every phrasing variant you can anticipate. You will miss variants. The model has no general theory of harm. It has only pattern-matching against the phrasings it saw. Change the phrasing, add a fictional framing, ask in a different language, and the refusal can fail.

The deep limitation is this: SFT can teach what a good answer looks like, but it cannot teach which of two candidate answers is better. For that, you need to know something about better that you didn’t encode in any single example. You need preferences.

SFT teaches the shape of a good answer. It has no way to choose between two good shapes.

From demonstrations to preferences

The move that follows from SFT’s failure: what humans can do faster than writing demonstrations is ranking them.

Show a rater two model responses to the same prompt — response A and response B — and ask which is better. They can answer in seconds. Writing a response from scratch takes minutes. This means you can collect preference labels at much higher volume than demonstrations, and the preference label contains a different kind of information: not “here is the target,” but “this is closer to the target than that.”

Scale up the preference collection. Collect hundreds of thousands of (prompt, response A, response B, human ranking) tuples. Train a small auxiliary model — the reward model — to predict the human rankings: given a prompt and a response, output a scalar score. Then use reinforcement learning (specifically PPO) to push the language model toward generating responses the reward model scores highly.

This is RLHF — reinforcement learning from human feedback. Ouyang et al., 2022 used it as the third stage of the InstructGPT pipeline: pretraining → SFT → RLHF. ChatGPT’s characteristic tone — helpful, reliably cautious about harmful requests, good at hedging uncertainty, consistent about refusals — comes almost entirely from this stage.

The deepest shift over SFT: the model now has feedback about direction, not just target. SFT says “produce something like this example.” RLHF says “of the things you just produced, this kind is better than that kind — adjust accordingly.” A direction is a richer signal than a target. It can propagate to novel situations no demonstration ever covered.

The practical problem is that RLHF is a genuine engineering challenge. PPO is unstable. The reward model can be gamed: the policy learns to produce outputs that score highly on the reward model without actually being better, because the reward model is an imperfect proxy for true quality. Over long training runs, the policy finds exploitable features in the reward model and optimises for those rather than for what humans actually wanted. Balancing the RL update against the pretrained base (so the model doesn’t drift into incoherence while chasing reward) requires careful tuning. RLHF works, but it’s expensive, brittle, and hard to reproduce without a dedicated ML infrastructure team behind it.

Rafailov et al., 2023 found something that, in retrospect, looks almost obvious: you can skip the reward model entirely.

The paper, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” makes a mathematical observation. The preference-fitting problem that RLHF solves via a reward model + PPO can be reformulated as a classification loss directly on the language model policy. Given a preferred response and a dispreferred response to the same prompt, you want the model to be more likely to produce the preferred one. You don’t need a separate reward model to express that preference. You don’t need PPO to optimise it. The preference is a loss; the loss can be minimised directly on the policy.

DPO is computationally lighter and far easier to get working. The abstract says it “eliminates the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning.” In practice, the gap between “has the resources of a frontier lab” and “fine-tuned a model that actually behaves well” narrowed substantially after DPO. Within a year of publication, it became the default alignment approach for most open-weight instruct models — the Llama family, Mistral, Qwen, and most of their derivatives. When you read that a model is “instruction-tuned” in 2024 or later, DPO or one of its descendants is almost always in the pipeline.

Both RLHF and DPO require human preference labels. In 2022-23, this was tractable. By 2024, at frontier scale, it was a real bottleneck. You need raters capable of judging quality on maths, code, medicine, science. You can’t hire enough such raters to keep pace with the rate at which models can generate candidate outputs.

The field’s response was predictable in retrospect: use the models themselves.

RLAIF (Lee et al., 2023) replaces human raters with a strong language model as the preference oracle. Head-to-head comparisons with RLHF showed that AI-generated preference labels are competitive with human ones on summarisation and dialogue. The reward model trained on AI labels performs comparably to the one trained on human labels. The human rater is no longer in the loop.

Constitutional AI (Bai et al., 2022, Anthropic) does something more principled. Write a list of principles — a constitution — stating what the model should and shouldn’t do. Ask the model to critique its own outputs against those principles and revise them. The critiques and revisions become training data. The RL stage uses the model’s own evaluations as the reward signal. Human preference labelling is replaced by explicit normative reasoning: the model has to argue about whether its outputs satisfy the stated principles, not just produce outputs that pattern-match to human-labelled examples.

The logic extends one step further. If models can generate reliable preference labels, can they generate training data directly? By 2024-25, the answer in a widening range of domains was yes. Maths problems with worked solutions. Code problems paired with passing test suites. Instruction-following demonstrations written by large models to train smaller ones — the distillation pipeline, where a 70B model generates training examples that improve a 7B model, and the better 7B model feeds the next iteration. By 2025, a substantial fraction of the data used to train frontier models isn’t scraped from the web. It’s generated by earlier versions of the models themselves.

Ilya Sutskever, speaking at NeurIPS 2024: “Pre-training as we know it will end. Data is the fossil fuel of AI. We have but one internet.” The scaling curve that had defined the field since Kaplan 2020 was visibly flattening. The field didn’t slow down. The growth frontier moved: from bigger pretraining to better post-training. The headline AI announcements of 2024 were not “we trained a bigger model on more of the internet.” They were “we trained a better model by using our previous models to generate, evaluate, and curate the training signal.”

The sharpest shift in AI in 2023-24 wasn’t a bigger model. It was figuring out how to use models to train better models.

A new axis

There is a 2024-25 discovery that changes the picture in a qualitatively different way. Not a refinement of post-training preference optimisation. Something new.

RL doesn’t just align models. It can teach them to think.

The observation that sets it up: language models already have scratch space. Their output is text; nothing prevents them from writing intermediate reasoning steps before writing a final answer. Chain-of-thought prompting — asking a model to “think step by step” — has been known since 2022 to improve performance on reasoning tasks. The model writes out intermediate steps, and those steps help it arrive at a better final answer.

But chain-of-thought as a prompting technique has a persistent problem. The intermediate steps are generated by the same forward pass as the final answer. You can ask the model to think out loud, but you can’t verify that the scratchpad is doing reasoning work rather than performing reasoning for the reader. A model that writes plausible-sounding intermediate steps that happen to be wrong, then arrives at a wrong final answer, has not improved by being asked to show its work. The steps are decorative.

The o-series models from OpenAI in late 2024 made a conceptually simple training move: use RL where the reward is the correctness of the final answer, and leave the intermediate chain of thought entirely unsupervised. The model can write whatever it wants in the scratchpad. The only signal is whether the final answer is right.

What emerged from training was not what anyone programmed in. The model learned, without any explicit supervision of the intermediate steps, to use the scratchpad as actual working memory. Backtracking when an approach failed. Trying alternate formulations when one hit a wall. Verifying intermediate results before continuing. Restarting from scratch, several steps back, when it found an error in something it had already written. None of these behaviours appeared in labelled training examples. They fell out of the objective: over enough RL iterations, the training process found that careful scratchpad use led to more correct final answers, and it reinforced that.

DeepSeek-R1 (arXiv:2501.12948, January 2025, open weights) replicated the result outside a closed lab. Pure RL on reasoning trajectories; no human-labelled chains required. The paper documents what they call the “aha moment” in training: a specific point where the model spontaneously began verifying its own intermediate steps and restarting when they failed. Not because the training data contained this behaviour as a pattern. Because the reward for correct final answers made careful intermediate reasoning instrumentally useful, and the RL training loop discovered it.

I think DeepSeek-R1 is the clearest published demonstration of something the field had suspected but hadn’t proven at scale: that reasoning, as a behaviour, is trainable from a simple outcome-based reward signal. You don’t need human annotations of good reasoning traces. You don’t need to supervise the scratchpad. You need to reward the right answer and run enough RL. The reasoning emerges.

Three things follow from this that are worth naming separately.

Test-time compute is a new scaling axis. Pretraining scales with more data and more parameters — you pay at training time and get a more capable model. Reasoning models scale with more inference compute — you pay at generation time, by thinking longer, and get a better answer on the current problem. A smaller reasoning model that thinks for ten seconds can match or outperform a larger standard model answering in one pass. These axes are complementary, not competing. You can now trade training-time capability against inference-time deliberation, and that tradeoff is explicit and controllable in a way it wasn’t before.

Diagnosability changes the failure mode. A standard model that gets a maths problem wrong gives you a wrong number. A reasoning model that gets it wrong gives you a chain of thought — readable, traceable, inspectable at every step. You can see exactly where the logic went off course: which intermediate claim was false, which inference was unwarranted, at what point the reasoning was solid and where it broke down. For systems where the reliability of the output matters — and in agent pipelines, it almost always does — this is the property that makes the difference. The failure is visible. Visible failures are debuggable. Black-box failures are not.

And the arc closes. The whole story of this piece — n-grams, embeddings, pretraining, instruction tuning, preference learning, and now reasoning — is one continuous story of making the training signal more specific. N-gram models encode raw co-occurrence statistics: this is what tends to follow that. Embeddings compress those statistics into geometry that neural networks can use. Pretraining scales that compression to the entire digitised corpus of human writing. Instruction tuning adds: here is what a helpful answer looks like. RLHF and DPO add: here is what better looks like, relative to what you just produced. RLAIF and synthetic data close the loop so models can teach each other. Reasoning models add the final turn: here is what thinking carefully looks like. Not by showing examples of good reasoning. By rewarding the right final answer, and letting the model figure out the rest.

Reasoning models aren’t smarter than other models. They’re models that have learned to spend their intelligence more deliberately.

Each step in this story makes the feedback signal richer. Each step exists because the previous step’s signal wasn’t specific enough.

The direction is clear: we keep finding more precise ways to tell models what we want, and they keep using it.

Until next time, stay curious.

The second edition of Mostly Harmless AI goes deeper on what these training paradigms make possible in practice — why a reasoning model behaves differently as an agent core, what alignment actually means when you’re building a system rather than evaluating a benchmark, and the chapters that didn’t fit any blog post. The whole book is also available to read online for free in a reader I built and am rather fond of: dark mode, font controls, progress tracking, offline support, the works.

If you want everything I’ve written and everything I’m going to write, the Compendium bundles it all — one purchase, in perpetuity.

Check it out

You have used a language model in a chat box. You typed a question, you got an answer, you closed the tab. The whole interaction lasted under a minute. The model did not remember you the next time you opened the page.

You have also seen, or read about, agents that work for hours. A coding agent that ships a feature overnight. A research agent that pulls together a hundred sources before breakfast. They plan, they call tools, they back out of dead ends, they hand you something you can use.

Both are the same model. Same neural network. Same forward pass. The only thing that changed is what’s wrapped around it.

This essay is the bridge. The architecture that turns the first thing into the second is not a single insight. It is a stack, a small number of layers, each one added in response to a failure mode of the previous layer. By the end you should be able to point at any agent doing real work in 2026 — coding, research, customer ops — and name which layer is doing the heavy lifting at any given moment.

Some of those layers are old. The fundamental one was published in 2022, before ChatGPT shipped. Some are very new. One was named eighteen months ago and is still settling. None of them, individually, is hard to follow. The trick is seeing them as a sequence, each fix opening the door for the next.

If you want a number to anchor where we start: METR has been measuring the time-horizon of frontier agents, and a language model on its own, with no scaffolding around it, sustains roughly a few minutes of human-equivalent work at 50% reliability. The equivalent of writing a competent meeting summary.

That is the floor.

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI, where the engineering details that don’t fit a blog post live. You can also read the whole book online for free in a custom reader I built. More at the end.

Subscribe now

The base case

Strip everything away first. No agent, no tools, no skills, no harness. Just the model.

A language model, in the strictly minimal sense, is a function from a string to a string. You hand it a sequence of tokens. It hands you back a sequence of tokens. One forward pass through the network. The input goes in at one end, the output comes out the other, one token at a time until the STOP token is generated, and that is the entire interaction. No state is held between calls. The next time you ask the same model the same question, it has no idea you have ever spoken before.

Inside that one shot, the model delivers. It will answer, draft, summarize, translate, brainstorm. Give it a good piece of context and a clear ask, and the response that comes back will, in my experience, often be useful enough to ship as-is. This is the experience that made everyone notice in late 2022. Open a chat, ask anything, get something back you can use. People called it magic at the time. Most of them still do, even though useful function with no memory is the more honest description.

But notice what it cannot do — which is most things you would ever want from an agent.

It cannot verify its own output. The same forward pass that produced the answer is the only one available to check it. There is no second opinion, no quick lookup, no let me try it and see what happens. The model is committed to whatever came out the first time.

It cannot look anything up. Whatever facts it has are baked into the weights from training, frozen at some cutoff date. If you ask about today’s news, or your codebase, or an internal company document, the model has nothing. And worse, it will frequently invent something plausible-sounding because completing a confident sentence is what it was trained to do.

It cannot act on the world. It cannot write to a file, send an email, call an API, run a command. It cannot do anything that has a side effect outside the chat window. The only thing it can produce is more text.

Inside the four walls of the context window, the base model is the most capable text engine the field has ever built. A single chat box was enough to launch the largest consumer product of the decade. Outside those walls, it is inert.

METR’s measurements of an unaugmented model — no tools, no loop, no scaffolding — put the time horizon at something on the order of minutes of human-equivalent work. Minutes. That is the starting capability. Everything else in this essay is a way of making those minutes compound.

The first leap

The first real agent paradigm is older than ChatGPT.

In October 2022, a team at Princeton and Google published ReAct: Synergizing Reasoning and Acting in Language Models. It went out about six weeks before the ChatGPT launch that made the public notice agents existed at all. Every working agent today — Claude Code, Codex, Gemini CLI, the dozens of research agents and customer-ops agents shipping this year — is some refinement of the loop that paper introduced.

Here is the setup. An agent operates in some environment: a Wikipedia API, a household simulator, a web shop, your codebase. The environment offers an action space, the set of things the agent is allowed to do. Call it A. A policy maps the current context to the next action: given everything the agent knows, what does it do next? With nothing else, the policy has to map a long, noisy trajectory of past observations directly to the right next move. This is brittle. The longer the task runs, the more lost the model gets.

ReAct’s move is to enlarge the action space. The new action space is A plus L, where L is the space of natural language. A “thought” is an action in L, the agent pausing to write itself a sticky note before reaching for the next tool. It does not change the world, it changes the context. The next action is conditioned on a context that now includes the model’s own reasoning about what just happened.

The paper spells out what thoughts are actually for, and the list is concrete, not mystical. Decomposing the goal into a plan. Injecting commonsense the environment does not supply. Extracting the relevant signal from a noisy observation. Tracking progress and noticing when a subgoal is done. Handling exceptions when something breaks. Five jobs.

Why this beats the alternatives is where the paper earns its place. Chain-of-thought prompting, the prior art, has the model reason in a closed loop inside its own head, with no contact with the world. The paper’s own ablation on the HotpotQA benchmark is brutal: chain-of-thought hallucinates in 14% of its successes and 56% of its failures. Acting alone, calling tools without thought, is grounded in the world but loses the global plan after a few steps. ReAct synthesizes them. On the same task, ReAct hallucinates in 6% of successes. Less than half. Both halves of the loop have to be there.

One concrete anchor before we move on. ReAct’s HotpotQA action space, the entire set of things the agent could do, was exactly three actions: search[entity], lookup[string], finish[answer]. Three. The first working agent paradigm operated on three tools. Hold that number.

The paper closes with the line that becomes the engine for the rest of this essay. “Complex tasks with large action spaces require more demonstrations to learn well, which unfortunately can easily go beyond the input length limit of in-context learning.” In plain English: more capability needs more action descriptions, which need more context, which we do not have. Every layer that follows is the field iteratively solving exactly that bottleneck.

METR step: a model wrapped in this loop moves from minutes to tens of minutes on bounded tasks.

Tools

So how do you fix ReAct’s bottleneck, the one the paper named in its own conclusion?

The first, most obvious answer: give the agent more actions to take. If A was the original action space and ReAct enlarged it to A ∪ L, the next move is to make A itself bigger.

That is what a tool is. A tool is a function the model can call. It has a name, a typed schema for its arguments, and a return value. The model writes a tool call into the trajectory the same way it writes a thought. Except this one has a side effect on the world. The harness picks it up, runs the function, drops the return value back into the context. The next turn of the loop sees the result and decides what to do next.

The loop is unchanged. Same thinking, same acting, same context-grows-by-a-turn shape ReAct described. The difference is what the agent is allowed to do.

ReAct, recall, ran on three tools: search, lookup, finish. That was the entire menu. Claude Code in 2026 ships with more than twenty: read a file, edit a file, run a shell command, search the codebase, fetch a URL, spawn a subagent, take a screenshot, schedule a future tick, and so on. Each one is just a function with a schema. Each one expands the set of things the agent can do without changing one line of the underlying loop.

This is the part that surprised me, the first time I sat with it. The chatbot you typed at in 2022 and the agent that wrote your test suite this morning share one loop. What changed is the tool catalog. Same loop. Bigger menu.

That observation is the unsexy version of why tool-building is now a discipline of its own. Every capability you add to an agent — search the web, read a Slack channel, hit your billing API, deploy to staging — is just another function with a schema. The architecture does not change. The leverage is entirely in which tools you build and how you describe them to the model.

The design discipline that emerges is short to state and brutal to follow. Tools should be few, sharp, and self-describing. Few, because every tool you add takes up tokens in the system prompt and a slot in the model’s attention. Sharp, because a tool that does seven things is one the model will use wrong six times out of seven. Self-describing, because the model only learns to use a tool from its name, its docstring, and its argument schema. There is no other channel. (More on this on Thursday. Anthropic’s recent guidance on writing tools for agents is the cleanest summary of this craft I have read.)

METR step: a model with the right toolkit moves from tens of minutes to hours of bounded work.

Skills

Tools fix half of ReAct’s bottleneck. They expand the action space.

The other half, recall, is the input-length limit. Every tool you add costs tokens in the system prompt to describe: name, schema, when to use it, what its return value looks like. Add fifty tools that way and the system prompt is a small book. The model is reading every single tool description on every single turn, even when ninety-five of those turns have nothing to do with that tool.

Skills are the move that fixes this.

Anthropic shipped the idea in late 2024 and the rest of the field has been catching up since. A skill is, mechanically, almost embarrassingly simple. It is a markdown file. It has a name, a one-line description of when it applies, and a body that explains how to do the thing. The agent does not read it on startup. The agent reads it on demand: when, in the middle of a task, it notices a description that matches what it is about to do.

So instead of jamming and here are seventeen other things you might want to do into the system prompt, you put each of those things in its own file with a one-liner that names when to consult it. The system prompt stays small. The latent capability of the agent becomes, for practical purposes, unbounded. Every skill you write is one more thing it can do, but only when it actually needs to.

I find the deeper shift here more interesting than the engineering. The agent is reading documentation written for it. Not training data ingested months ago and frozen into weights. Documentation. Authored in plain prose. Versioned in git. Like the laminated procedure sheet a mechanic posts above a workbench for a job done once a month. Improvable by the same process that improves any document: someone notices the agent doing the wrong thing, edits the file, the next agent reads the new version and gets it right.

This is self-extension by reading, not by retraining. A new capability used to require a new training run, or at minimum a new fine-tune. Now it requires a markdown file. The cost of teaching an agent to do one more thing has fallen from days of GPU time to the minutes it takes to write a paragraph, and almost nobody outside the people building agentic systems has noticed.

The system prompt stays small. The set of things the agent can do, on demand, grows without bound. The two used to be the same number.

METR step: skills, more than anything else in this list, are what made the time horizon stop being bounded by how cleverly you wrote the system prompt.

MCP

For most of 2024, every agentic harness invented its own way to attach the same set of capabilities. You wrote a tool for Claude Code; it would not work in Codex. You wrote a skill for one harness; another harness could not see it. You hooked your billing API into one agent and had to do the same wiring four more times for the others. Every integration was bespoke. Nothing composed.

The Model Context Protocol (MCP) is the field’s answer to that. Anthropic shipped the spec in late 2024. By the end of 2025 every serious agent harness, including the ones not built by Anthropic, had adopted some version of it. Codex talks MCP. So does Claude Desktop, and Cursor, and a long list of others. This is one of those quiet moments where an industry just... agrees on a wire format, and a year later the world is different.

The architecture is three nouns. Hosts are the applications you actually use: Claude Desktop, Codex, Cursor. Clients live inside the host and talk to one server each. Servers are the things that actually expose capability: your codebase, your billing API, the Wikipedia search box from the ReAct paper four years ago.

What a server offers is the second triple: Resources (data the model can read), Prompts (workflow templates the user can invoke), and Tools (functions the model can call). Three nouns, again. The whole protocol is two threes.

The point is portability. A skill or tool you wrote once, against the protocol instead of against a specific harness, works everywhere. The lock-in moves out from under you. The agent ecosystem starts to compose the way the web did in the late 1990s. Not because someone planned it, but because everyone independently noticed it was cheaper to talk a shared protocol than to keep reinventing the connector layer.

Worth noticing what the spec foregrounds at the top of every chapter on tool calls: user consent. Capability requires permission. The protocol does not assume the model can do whatever a server exposes. It assumes the model has to ask, and the user has to answer. A small design choice with very large downstream consequences, and the reason the rest of this stack does not collapse into something nobody would let near their email.

METR step: not a step on the ladder, but a multiplier. The tools and skills from the last two sections now travel.

Context engineering

Add tools. Add skills. Add MCP. The agent can now do, in principle, almost anything you can describe in a prompt and a function. The trouble is what happens when it actually starts trying.

A long agent run accumulates context. Every observation from a tool call goes in. Every thought goes in. Every error message, every retry, every half-attempted plan that did not work goes in. After a few hours of work the context window is mostly exhaust: the trail of everything the agent tried, the great majority of which is no longer relevant to the next move. The model is searching for signal inside its own attic.

Karpathy named this context engineering in 2024, and the name stuck because the field had been doing it without a name for two years. Simon Willison wrote it up. LangChain made it a category. By 2026 it is a craft of its own: what to put in the context, when to summarize, what to evict, what to keep verbatim because the agent will need its exact wording later.

The central primitive in the discipline is compaction. At some threshold, typically 70% to 85% of the window, the agent stops, reads its own history, and rewrites it into a smaller form. Here is what we were trying to do. Here are the decisions we made. Here is the state we are in. Here is the next move. The compacted summary replaces the noisy trail. The agent keeps going on a fresh, smaller context with the salient bits intact.

The deeper move is that the agent now owns its own working memory in a way it never did inside a single ReAct loop. ReAct kept the entire history. Compaction lets the agent curate the history. A small change of grammar with a giant change of consequence.

Notice what this fixes. ReAct’s authors, in the same 2022 paper, named the dominant failure mode of their own system: “the model repetitively generates the previous thoughts and actions, often failing to reason about what the proper next action to take should be and jump out of the loop.” Translation: the agent gets stuck because its context is full of the same noise as the previous turn, so the next turn is the same noise plus a little more. That is a context problem. Context engineering is what stops it.

Without this layer, every previous layer eventually drowns. A hundred tools is useless if the agent’s context is so saturated it cannot find the right one. The five-thousand-word skill on how to handle a billing dispute is useless if the agent compacted it away on turn forty. Context engineering is the layer that makes the others compound over a long run instead of degrading into noise.

METR step: this is the layer that turns a few hours of focused agent work into a workday.

The hierarchy of agency

Stack the layers and the picture comes into focus. At 50% reliability on the METR time-horizon scale, a language model alone, with no scaffolding around it, sustains minutes of human-equivalent work. Wrap it in a ReAct loop with no tools, and that becomes tens of minutes. Add tools to ReAct, hours. Add skills and context engineering on top, a workday. Add an external loop above all of that, a fresh agent per turn on a clock with a journal handing state to itself, and the horizon stretches into days and weeks.

Stare at that ladder for a second. Each rung is the same model. What separates a chatbot from a coding agent that finishes a feature overnight is the scaffolding stacked around it. The frontier of what an agent can do in 2026 is set, almost entirely, by where you stop climbing.

Each layer has the same shape, in the abstract. Find the thing that bottlenecks the previous layer. Add a structure that lets the model offload that thing into the world, the way a machinist offloads a measurement into a caliper rather than holding it in memory. Into language, into tools, into files, into a clock. The model’s per-turn intelligence does not change. What changes is the time horizon over which that intelligence compounds.

The last rung is the one most people have not seen yet, and it is the one I have spent the last few months running on my own infrastructure. The trick is the same one. Take the bottleneck (the agent runs out of context before it runs out of work) and offload it. The new offload target is the file system. The new clock is cron. Past-Claude writes a markdown file at the end of its turn that says what it did and what comes next. A timer fires some hours later. Future-Claude wakes into a fresh context, reads the file, makes the next move, writes the file, exits. The continuity is in the file, not in the model.

That is the entire primitive. A markdown file and a timer. Past self tells future self what to do.

What you get from it is hard to describe to someone who has not run one. The agent works on your stuff for weeks at a time. It writes new jobs for itself. It reads the documentation about its own substrate and uses the tools that documentation describes. It makes mistakes (one in five runs produces something I have to throw out) but the mistakes are caught by the same kind of boring engineering that catches mistakes in any other autonomous system. Audit log, lock registry, archive-only deletion, every state change committed to git before the next turn starts.

The point of saying this out loud is that the same trick keeps working. Extend the action space; add a layer that compounds; let the previous layer drop the things it could not hold. The trick does not stop at hours. It does not stop at days. METR’s curve has been doubling every four months over the last two years. The 2027 projection is a working day. The 2028 projection is a working week.

Each doubling is one more scaffolding layer.

The frontier is not the model

Step back from all of it.

The architecture you’ve just walked through is layered. A language model at the core. ReAct around the model, turning tokens into actions. Tools around ReAct, expanding what those actions can be. Skills letting the agent pull capability from the file system instead of carrying it in the system prompt. MCP making everything portable. Context engineering keeping the whole thing from drowning in its own exhaust. An external loop on top of all that, when the work runs longer than a single context window can hold.

Every agent doing real work in 2026 — your coding agent, your research agent, the customer-ops bot answering your refund request, my private-tick agent running once an hour — has this shape. They differ in which tools they ship and which skills they read on demand. They do not differ in the shape of the stack. Once you can see the layers, you can see them everywhere.

So here is the closing claim, the techno-pragmatist version of what the article has been arguing the whole time. The frontier is not the model. It is the layers around it. And the entire stack is the field’s three-year answer to a single sentence in a single paper from October 2022 that named its own ceiling and dared the rest of us to climb past it.

One frontier worth flagging before I close. A competent agent can already write its own tools and skills on demand. That part is shipping today. The next move is teaching it, via tools and skills, to detect by itself when its current toolkit doesn’t cover what it’s trying to do, so it knows when to extend itself without being told. Self-extension that triggers itself. That is the live edge right now, and where the next few posts are headed.

The next post zooms in on the innermost layer the agent touches: the tools themselves, and what makes a tool safe enough to live inside a stack like this. That is a story for another Thursday.

Until next time, stay curious.

If this is the worldview you want to take more seriously, the second edition of Mostly Harmless AI (due May 25th) goes deep on the agentic stack we walked through here. Full chapters on context engineering and on the harness around the model, with the math, the case studies, and the parts that didn’t fit a blog post. You can also read the whole book online for free in a custom reader I built that I’m rather proud of: dark mode, font controls, progress tracking, offline support, the works.

If you want the whole catalog of everything I’ve written, plus everything I’m going to write, that’s the Compendium. One purchase, in perpetuity.

You’ve been using AI coding agents for months. You’ve crafted elaborate system prompts. You’ve added a dozen skills. You’ve learned the dance of context window management. And somewhere around the third hour of work, something breaks. The agent starts forgetting things. Making wrong assumptions. Doing something close—but not quite—what you asked.

This isn’t a failure of the model. This is a failure of the system.

To be sure, better models make things easier. And models are getting better by the day. But no matter how good a model is, bad systems lead to bad outputs. Even the smartest people produce junk when fed with incorrect assumptions or given incomplete instructions.

In contrast, a good system with clear boundaries and explicit rules, that leaves the exact amount of flexibility necessary, makes creativity and productivity thrive.

You see this day and night in teams (of real humans) in every industry. It’s not often the smartest person in the room that solves the hard problem. It’s when you combine the right kinds of intelligence with the right kind of system that things click.

In this article, I want to make the case for a structured way to think about Large Language Model (LLM)-based agentic systems (mostly for coding, but also for knowledge work in general) that fixes some of the greatest pains I (and I sure most of you) have been facing when trying to scale AI-assisted workflows to professional levels.

It’s a system that puts the right constraints in the right places and leaves just enough space for creative exploration (or however you want to call what LLMs do when they hallucinate in your favor). It’s also a system that makes it clear you are in charge.

Everything an AI agent does happens inside a context window. System prompt, user input, tool results, skill injections—they all live there. The agent’s only mechanism for action is the ReAct (Reasoning + Acting) loop: think, call tools, observe results, repeat. Each cycle grows the context. Each skill activation injects more.

This creates a fundamental tension: context is power, but context is finite. Too little and the agent can’t connect the dots. Too much and the important stuff drowns. The gap between those two failure modes is narrow—and most agent frameworks ignore it entirely.

I’ll walk through why current systems fail, introduce a four-element framework for thinking about agentic architectures, show you how these principles apply across three domains, then present a vision for better AI harness engineering.

Part I - The Symptoms

To understand the problems we first need to understand how a standard agentic loop works. The typical architecture is what’s called a ReAct loop. The LLM runs in a loop that determines the next action given context, which can be read some files, ask the user, invoke a tool, inject a skill, etc. When the agent decides no more actions are necessary, the loop ends and the user is given control back to continue the prompt.

That’s it. All the seemingly supersmart behaviours of Claude Code, Gemini CLI, and Codex are, under the hood, some form of the basic ReAct loop. There are of course nuances. For example, most systems decide that if the agent calls the same tool with the same args three times, it must be stuck in a loop and stop the turn. There are perhaps hard limits on how many tool calls the agent can do in each turn.

Context is the bottleneck. Not the model. Not the prompt. Context.

The agent doesn’t have memory. It doesn’t have state. It has context. Everything it knows about your project, your preferences, your conventions, all of it lives in the context window. When you add a skill, you’re injecting more context. When you run a tool, the result goes into context. When you switch modes, you’re switching which system prompt is active, all still in context.

This means context engineering is AI agent engineering. The agent’s behavior isn’t determined by the model alone, or even primarily, but by what context you give it, and how you structure that context over time.

Most tools treat context as a solved problem. They stuff everything in and hope the model figures it out. In-context learning seems almost magical, but it has limits—and those limits become visible fast.

When context is thin, the agent simply doesn’t know enough about your project to make informed decisions. It relies on baked-in assumptions from training and falls back to consensus instead of following your style: it uses the common tools and practices it learned from pretraining. This often means it uses slightly old and outdated tools and practices.

So you do the sensible thing, and inject project-specific information into the context. But then if context grows too large, even if it doesn’t technically exceed the model’s capacity, things start to get lost in the middle. Moreover, failed tool calls, wrong assumptions the model had to correct, etc., start creeping up in context, not only taking up valuable space but also, and more importantly, distracting the model and biasing it towards mediocre decisions.

Then there is context compaction: when the context fills in to about 85%, most systems will invoke a special prompt to instruct the agent to summarize the current state. These prompts vary in detail, but often involve asking the agent what it is immediately doing, where is it stuck, what has failed, etc. Clever, but a hack nonetheless. This hard context reset means the agent will forget important nuances in the current conversation and will repeat past mistakes. It’s frustrating.

Let’s look at how these problems surface in specific symptoms that all LLM-based agents display at some point.

Symptom One: Unstated Assumptions

The first failure mode isn’t dramatic. It’s quiet. You ask the agent to write a test, and it writes a unittest.TestCase instead of a pytest function. You ask it to add a dependency, and it edits requirements.txt instead of running uv add. You ask it to deploy, and it pushes directly to main.

These aren’t model failures. They’re assumption mismatches. The agent doesn’t know how your team does things. There’s no guardrail for “in this project, we always use pytest, we always use uv, we never commit directly to main.” The agent improvises from general knowledge, and general knowledge is often wrong.

Skills are supposed to fix this. Add a skill document that says “use pytest” and the agent should know. But skills introduce a new problem.

You add a skill for code review. Then one for documentation. Then one for PR descriptions. Then three more for your company’s specific stack. Each skill seems small. A few hundred tokens each. But they pile up—always-on knowledge the agent carries but can’t prioritize.

The result is context bloat. The agent can’t tell what’s relevant in any given moment. So it blends everything together, and hallucinations increase. More skills made it worse—not better.

Symptom Two: Permission Leakage

Every agent framework implements the same plan then build pattern. The idea is sound: think first, plan second, execute third. In practice, the boundaries leak.

Plan mode is supposed to be read-only. Design the change, review the approach, lock in the scope. Build mode is supposed to execute. Write the code, run the tests, commit the result.

But “plan mode” in most tools is just a prompt. There’s no enforcement. The agent can write code in plan mode if it wants to. It can ignore the plan in build mode. It can skip straight to implementation if the prompt implies urgency. The modes are suggestions, not constraints.

This matters because a plan only works if it’s actually followed. If the agent can deviate mid-execution—if “plan mode” and “build mode” are just prompts with different names—the plan becomes advisory. And advisory plans get ignored.

The second problem is structural: there’s no artifact that passes from plan to build. The plan lives in the context. By the time build mode starts, the plan is mixed in with everything else the agent said. Which file was the plan? Which changes were approved? The agent has to re-read the conversation to remember. Context saturation accelerates.

Symptom Three: Context Saturation

After extended work, you see the same pattern: the agent makes 95% of the progress, then fails on the last 5%. It nails the architecture. The logic is sound. The core implementation works. Then it stumbles on a detail—because context has saturated. It forgot which environment it was in, which conventions still apply, which constraints matter.

But the deeper problem is internal noise. The agent keeps everything in context: all internal reasoning, all tool calls, all results. This is fine for minute-to-minute action. But after four failed attempts to solve something, the old tool calls are just noise. These were attempts that went nowhere, just add cost and accelerate saturation.

The supposed solution for this is context compaction. But this creates a lossy summary problem. The agent is supposed to leave a trail for its future self. After context compaction, it should be able to pick up where it left off. But if agents struggle with long contexts, how are they supposed to build a good trail? The compaction report is only as good as the agent’s ability to summarize. And summarization is lossy and injects back lots of unstated assumptions from pretraining.

The frustrating part: this wasn’t a hard problem. The agent had all the knowledge it needed. But context filled with noise, and the important bits got pushed out. More tokens in, less signal out.

The solution isn’t just better prompts or larger context windows. Yes, these help. But the symptoms are systemic, so the solution must be a system overhaul.

Let me show you how that system looks like.

Part II - The System

Now that we understand the problem, let’s look at how every agent system actually works. Every AI agent system addresses four concerns. When you conflate them, the system breaks. When you separate them, the system scales.

This taxonomy isn’t original to me. It’s a synthesis of how modern AI agentic systems work under the hood. Most explicitly, it’s implemented in the OpenCode CLI (opencode.ai), but all other tools follow a similar pattern, even if they use different names.

Here’s the breakdown. Every agent system you’ll encounter (explicitly or implicitly) is managing these four things:

Mode — the who. A mode is the persona the AI adopts. It defines the thinking style, the permissions, the available tools. When you interact with a “code assistant,” you’re in a coding mode. When you switch to “creative writer,” you’re in a creative mode.

Modes are explicit. They’re top-level system prompts that define behavior and permissions. You tell the agent: “This is how you should think and behave. These are the tools you can use. These are the parts of the filesystem you can write to.”

Skill — the knowledge. A skill is knowledge the agent can recall when necessary. It doesn’t get invoked explicitly, it gets applied implicitly when necessary. When you give an agent knowledge about SQL optimization, that skill is available whenever relevant. The agent doesn’t need to be told to use it. The ReAct cycle injects it when it deems suitable.

Unlike modes, skills can layer. An agent might have a SQL skill, a documentation skill, and a debugging skill, all active simultaneously, all contributing when relevant. Skills are implicit because the agent should just apply them naturally. They can also contradict or complement each other. In-context learning should be capable of using them in a combined manner.

Command — the workflow. A command is a script. It tells the agent: do this, in this order, using these tools. “Refactor this function” is a command. “Run these tests and report results” is a command.

Commands are explicit: you invoke them. Under the hood, commands are just prompts. The difference is who injects them: the user. When you run /build, you’re injecting a workflow prompt into the agent’s context. That’s it. The command tells the agent: do this sequence of things. The complexity lives in the orchestration of the ReAct cycle, not the command itself.

Commands are intentionally simple. They don’t contain knowledge. That’s intentional separation of concerns. The command itself shouldn’t know how to build; it knows when to spawn subagents and which mode to use. This keeps commands thin and changeable without rewriting underlying knowledge.

Subagent — the delegation. A subagent is a spawned agent for background or parallel tasks. It handles isolated work, returns summarized results, then disappears. It is instantiated with a system prompt and specific instructions (synthesized by the primary agent that called it), and runs for one full ReAct turn.

Subagents are ephemeral. Their internal reasoning stays private. The main agent only sees the synthesis. You spawn a subagent when you need parallel processing, isolation, or both. They are the way to fork, solve a specific subtask, and return a result, but keep context clean. Kind of like subroutines.

Why This Separation Matters

Understanding this distinction unlocks everything else. Once you see skills as implicit knowledge and commands as explicit scripts, the rest of the architecture clicks naturally. Most agent setups conflate these. They embed knowledge in commands. They make skills behave like workflows. They mix persona into everything else. And the massively underuse subagents.

When you separate these concerns–modes for persona, skills for knowledge, commands for orchestration, subagents for delegation–you get something that looks like good systems engineering. You can swap skills without touching commands. You can change modes without rewriting workflows. You can spawn subagents without the main agent knowing or caring how they work internally. The result is a system that works and adapts and scales like good software should do.

The system scales because the pieces are independent. Change one without breaking the others. Each component has a single job, and the boundaries between them are meaningful. When context shifts, when requirements evolve, when a new skill needs adding, the system adapts incrementally rather than collapsing under the weight of accumulated complexity.

Part III: The Practice

If so far this seems like abstract theory for you, in this section we will ground these concepts in actual practice. Let me show you how I’m using these ideas today to improve my AI-assisted coding practice. I’m using opencode.ai but I believe the following is easily adaptable to any agentic toolkit out there.

My Three Modes

Every agentic system needs boundaries, not social contracts, but enforced constraints. In my setup, those constraints come from three modes: analyze, design, and create.

Each of these modes defines a thinking style—a persona—and a set of constraints for tool use and filesystem access.

Analyze mode is research and investigation. This mode reads your work and writes summaries to a knowledge base. It cannot touch production files. Not “should not” but cannot. The permissions are built into the mode itself, not enforced through prompts or warnings. The agent is incapable of writing outside of a .playground folder, and is incapable of doing anything that can harm the project or the system (more on how a bit later) but it is still capable of running arbitrary code, download anything from the internet, and play around as it needs.

Design mode is architecture and planning. This mode bridges analysis and implementation. It can read your project and write design documents, architecture diagrams, and implementation plans, but still cannot touch production code. It cannot run shell scripts either, at all. It can look at git status and logs, read folder contents, etc., but it can only write to a space where plans and design documents go.

Create mode is execution. Full read-write access. This is where production work happens. The agent can write code, create files, and modify the project directly. Again, it cannot do anything outside the project scope, though. It won’t accidentally change /etc/host(s)1 even if it tries to.

The key insight: modes define permissions, not just persona. You can’t accidentally prompt your way into code generation during research. The agent literally lacks the capability. The agent doesn’t need to “understand” these constraints, it simply operates within them.

Mode is the who, and it determines what the agent can do, not just how it thinks.

Let me show you how they work in three different domains that make the bread and butter of my daily job: software development, scientific research, and technical writing.

I chose these domains because they illustrate the simplicity and scalability of the system. Software development shows the framework under constraints: deadlines, production code, real stakes. Research shows it under complexity: synthesis, evaluation, structured output. Technical writing shows it under nuance: voice, audience, iterative refinement. Three different pressures, one consistent architecture that works in all three cases.

In each of these domains we have two layers to go through: first is the set of implicit skills that are available to the agents, and second is the set of explicit commands (each tied to a specific mode) that setup concrete workflows. I will show you one example workflow that cross-cuts across the three modes in each case. I will also tell you exactly where delegation occurs.

Domain A: Software Development

Software development is where agentic systems face the harshest constraints. Production code has stakes. Deadlines are real. Mistakes cost money. Let’s see how the framework applies.

Implicit Skills

A software development agent carries knowledge it never needs to be told to use. It knows language idioms and patterns like the idiomatic way to write a list comprehension in Python, or the conventions for error handling in Go. It knows testing conventions: where tests live in the directory structure, how they’re named, what assertions to prefer. It knows architecture conventions: layered structure, dependency injection patterns, how error states propagate. It knows code review standards: what to flag, what to praise, when to ask for clarification.

Example Workflow: Bug Hunting

I use this workflow for finding and fixing bugs. It starts with investigation. The agent spawns dozens of subagents to try and break the system (either guided towards a purpose, or completely unbiased). Then you build a comprehensive plan to solve it. And then you execute that plan. Simple, right?

Phase 1: /trace (analyze mode) runs systematic experiments to detect and narrow down a bug’s cause. The agent examines stack traces, compares behavior across commits, and pinpoints the exact files and functions that need attention. This mode is read-only by design, except for a .playground folder. Research happens here, not in the code itself.

Each experiment is run on a subagent that has the job of verifying one assumption. The main agent receives only experiment results, and constructs an executive report of findings. This means you can run dozens of different experiments autonomously to detect what breaks what.

Phase 2: /plan (design mode) takes the diagnosis and defines the changes needed, along with their architectural impact. The agent reviews the affected modules, considers alternative approaches, and documents the implementation plan before touching anything. This is where the scope gets locked in.

The result of this phase is a structured plan with step by step details on what files must be touched and what must be done in there (semantically, not code). For every phase, it defines success criteria: what must be validated before we can say we got that phase right.

Phase 3: /build (create mode) executes the plan step by step. The agent writes tests first (following Test-Driven Development (TDD) discipline) for the success criteria defined for that phase and watches them fail. Then it launches a coding subagent that has read-only access to tests, so it cannot cheat and change the tests.

The subagent attempts to implement changes that make the test pass. If it succeeds, the main agent commits and moves on. If it doesn’t, the main agent retries a few times. If there is no progress, the main agent resets the work tree (no harm done), and reports on failure. This usually means the plan needs revisions.

Domain B: Research

Research is where agentic systems face the greatest complexity. Sources multiply, methodologies diverge, synthesis requires judgment. Let’s see how the framework applies.

Implicit Skills

A research agent knows the conventions of academic writing without being reminded. It knows citation formats like APA, MLA, Chicago, and IEEE, and when to use each. It knows how to evaluate papers: methodology soundness, sample size adequacy, replicability claims, conflict of interest disclosures. It knows the structure of literature reviews: how to organize by theme, methodology, or chronological development. It knows domain-specific terminology, distinguishing between “accuracy” and “precision” in machine learning, or between “confounding” and “colliding” in causal inference.

Example Workflow: State-of-the-Art Report

Phase 1: /research (analyze mode) spawns subagents to gather sources in parallel. Each subagent reads a batch of papers, synthesizes findings, and returns summaries. The main agent synthesizes those summaries into structured notes. This phase can be run multiple times to collect batches of sources without overwhelming context. At the end, you get hundreds of sources summarized into clean research notes.

Phase 2: /outline (design mode) identifies patterns across the collected literature. The agent groups papers by methodology, extracts recurring findings, and maps the landscape of the field. It generates outline options for the final document, based on typical structures like problem-solution or paradigm-methods, highlighting gaps where the research is thin and consensus areas where findings align.

Phase 3: /draft (create mode) builds the document section by section, following the outline. Each section draws on the structured notes, weaving together sources into coherent narrative.

The agent launches subagents for writing each subsection because typically, agents write more or less the same length in a single write command, so if you ask it to fill in a large outline all at once you’ll only get a mediocre extended outline. By launching independent writers for specific sections of the outline, you get all the attention of a single turn to read source material and write a good 4 or 5 paragraphs for a concrete section.

A cool idea I’ve been meaning to try is have the main agent can spawn several subagents to write the same section, with a high temperature, and then perform some sort of aggregation or evaluation before building the final draft for every section. This burns through 3x tokens but ensembles have been shown over and over to improve AI models outputs. If you try it, let me know.

Domain C: Technical Writing

Technical writing is where agentic systems face the most nuance. Voice matters. Audience varies. Iterative refinement is the norm. Let’s see how the framework applies.

Implicit Skills

A technical writing agent carries knowledge of prose style without being coached. It knows voice and tense conventions—active voice for clarity, past tense for completed processes, second person for direct instruction. It knows structural patterns: how documentation differs from blog posts, how reports differ from tutorials, how reference material differs from guides. It knows audience awareness: what to explain for newcomers, what to omit for experts, when to elaborate and when to abbreviate. It knows cross-referencing and linking norms: when to link, when to inline, how to name anchors for scannability.

Example Workflow: Paper Review

Phase 1: /review (analyze mode) performs detailed review in a specific order: structural issues first, then content, then style. The agent examines the narrative arc—how main points connect, whether the flow makes sense, before worrying about grammar or word choice. This ordering matters; reviewing low-level details when high-level problems exist wastes effort.

Each iteration is performed by spawning several subagents that focus on specific types of problems, like transitions, unverifiable claims, etc. Each subagent returns a structured list of issues, pointing back to exact line numbers and phrasing. Then, the main agent edits the original paper and injects markdown comments in every marked issue, next to the paragraph, or under the header where it best fits.

Phase 2: /revise (design mode) plans changes to specific sections, prioritizing by review type. The agent maps structural fixes to particular paragraphs, content additions to thin sections, style improvements to verbose passages. It produces a concrete plan, section by section, change by change. Then it goes into the manuscript and writes markdown comments as replies to the existing review comments, thus grounding the revision plan in the exact context it must fit.

Phase 3: /rewrite (create mode) follows the plan. The agent revises sections in priority order, applying structural changes first, then content, then style. Again, each step is performed spawning a subagent tasked with just a change (for style changes we actually do it section by section).

The subagent doesn’t edit; it produces a draft revision that the main agent is then tasked to paste into the document where it fits. Crucially, the main agent is instructed to leave the editorial comments but mark them as solved, with a short trail of what was changed. This works wonders for a later human review phase.

Part IV: A Look into the Future

These workflows work, but with some caveats. There’s a gap between “working” and “working well.” Three key pains remain in my implementation.

1. Long commands are hard to follow when given as a single prompt. The fourth step gets forgotten since it is buried at the beginning of the context.

2. Permissions as currently implemented are all-or-nothing. You either have shell access (destructive) or you don’t. I want broad permissions (run whatever you want) with provable security (nothing you run can change this file).

3. Context saturation still happens even with delegation. After a while, the agent will have to compact context, and this usually means you lose important information.

I have three ideas for closing this gap. The first is about how commands work. The second is about security. The third is about context management. They are in different levels of implementation, so let me show you what I’m building toward.

Idea One: Better Commands

Commands in most tools (Claude Code, Gemini CLI, Codex, Copilot) are one-shot interactions: you invoke the command, a single massive prompt is injected. The agent runs until it decides to stop.

To make commands truly useful, we need to be more like scripts. Here’s what that means:

1. Commands that inject prompt instructions one step at a time, waiting for the agent to do a full turn each time. Instead of dumping a large prompt to run all steps at once, a command like /review could insert surgical mini prompts that say “read the file”, wait for the agent, “analyze structure”, wait for agent, and so on, until “write the report”. This massively reduces the problem of lost-in-middle context saturation. Each turn the agent is focused on one specific step, and you get N times the compute power to solve an N-step workflow.

2. Commands that extract structured information from the agent response, and can later inject variables back into prompt. This allows to reinject important information into later prompts, keeping important information as a contextual variable, not just a string lost in the middle of the prompt. But it allows for something else.

3. Conditional branching based on context or user input. Once we have structured parsing and contextual variables, we can inject different prompts based on whether the agent succeeded or failed. If the plan reveals a breaking change, route to architectural review. If it’s a bug fix, route directly to implementation. The command adapts its path based on what it discovers.

4. Finally, commands that embed and execute external scripts. Instead of asking the agent to run some script, the command can run arbitrary Python, JS, Bash, or whatever, to, for example, transform structured information. The command becomes an orchestrator of other processes.

Basically, what I’m asking for here is a Domain-Specific Language (DSL) for guiding agents in a far more structured manner, but still having the power of arbitrary prompts for flexibility. Mixing code and prompts in this way gives us the tools to find the precise balance between constraints and capabilities.

If this sounds exciting, I’m happy to tell you this is already doable, to some extent. Check out my literate-commands project for an OpenCode-specific implementation of these ideas. It’s still a bit rough around the edges, but it works much better than plain, single-prompt commands.

Idea Two: Sandboxed Security

Most agentic tools have very coarse permission settings. You can allow, deny, or set a specific tool to “ask” mode, which means the agent will pause and emit a notification for the user to give permission.

This works fine for coarse-grained permissions like read-only access, or write but no shell. In OpenCode, you can even define permissions for specific paths, or even specific shell commands (with simple glob patterns, so you can, e.g., allow ls * but reject all other shell commands).

However, even in this case, I find these permissions too restrictive. They are conflating two different dimensions into one–what tools the agent can use, and what side-effects can those tools have.

For example, say I want to give my agent git access but only for reading operations. How do you achieve that? You need to list all safe patterns like git ls-tree *, git status, git log *. But what about git branch? Depending on the arguments, this subcommand can have read-only or write side effects. And then think about pipes, shell substitution, custom bash scripts, or worse, python *.

If you want your agent to be capable, you need to give it access to a wide variety of tools. For example, my bug-hunting workflow depends on the agent being able to execute arbitrary code that it synthesizes on the fly. However, I want guardrails. There is simply no way to whitelist all possible commands. We need separation of permission to run a command and permission to modify the system.

The solution, of course, is some form of filesystem isolation. The most obvious one is wrapping all shell execution in Docker, so commands run in a container with proper constraints. This creates all sorts of other problems, which I can discuss in a future post, but for now, it remains my best (and simplest) solution to robust sandboxing.

And this isn’t just about safety, though. When you know the agent can’t accidentally wipe your home directory or exfiltrate your API keys, you can let it do more. Security enables capability. You can let the agent download arbitrary code from the internet, run arbitrary scripts, break things and observe changes. Everything happens inside a Docker container with precise constraints that enable maximum capability with absolute security.

As of now, I kind of implemented this as a plugin for OpenCode, but it’s still in beta phase and not ready for widespread use. More on this idea in a future article.

Idea Three: Context-Aware Execution

And finally, we need to rethink the whole oversimplistic ReAct loop that simply grows the context linearly. The agentic cycle doesn’t have to be a straight line. Real work branches: you explore options, try things, backtrack when they fail. The context should reflect that.

I’ve been designing a system where the context never saturates. It branches when you’re exploring, spawning parallel contexts for different approaches. It prunes old tool calls that went nowhere. It removes internal reasoning that no longer matters. It maintains a “trail” that actually works: a structured record of decisions, not a lossy summary.

The goal is simple: keep context between 40% and 60% saturation at all times. Not by compacting a 150K tokens context down to 10K—which kills all understanding the agent had achieved—but by never letting it grow unchecked.

Nothing like this exists yet, so I’m building it, but it’s a story for another day.

Conclusion

The main takeaway from this article is not that my system is better. It’s that you can design your own system to adapt perfectly to your workflows if you clearly separate concerns. The main modes are for establishing an overall persona–inquisitive and critical, versus detailed and forward-looking, versus focused and action-biased–while skills incorporate domain knowledge, and commands act as precise workflows.

The workflows I described are real, based on actual commands and prompts I’m using in production code. But I have abstracted them a bit to make them easier to understand in the context of an arbitrary agent, not tied to specific idiosyncrasies of the tool I happen to be using at the moment. If you want to see and try for yourself a concrete implementation of these ideas—still imperfect, but working nonetheless—check out my opencode toolkit repository. It’s still pretty much work in progress, so use it with care.

In future articles I will explore specific problems in more detail and discuss concrete strategies to implement powerful workflows that keep you, the user, in absolute control, while delegating the majority of the grunt work.

And, as a final remark, I’m seriously considering building my own CLI agent. I know, I know. Reinventing the wheel and all that. But my plan is not to compete with any of the professional tools out there. What I always care about is understanding things deeply, and as my computer science career has taught me so far, there is no deeper understanding than the one you gain from actually building stuff.

So stay tuned for that. I will share progress as usual in the form of educational articles, so you’ll get to see under the hood how to build a fully functional CLI agent with tool calling, context compaction, skills, commands (the powerful ones, not the cheap single-prompt injection), subagent delegation, sandboxing, and all the engineering design hurdles that come with it.

Until next time, stay curious.

You can’t scroll through a tech feed these days without tripping over a prophecy: the AI bubble is about to burst, and a long, cold “AI Winter” is coming. The narrative is as seductive as it is simple. The current frenzy around Generative AI, we’re told, is a speculative mania. When the inflated expectations inevitably collide with reality and the firehose of investment capital slows to a trickle, the whole enterprise will be exposed as a grand fiasco. We’ll discover, the skeptics say, that it was all a cuento.

And let’s be clear: they’re not entirely wrong about the first part. The expectations are inflated. A correction is not just likely; it’s necessary.

But here’s my thesis: the idea that this correction will lead to another AI Winter—a catastrophic freeze comparable to the funding droughts of the 1970s and 80s—is a fundamental misreading of the landscape. I will argue that what we are heading for is not a collapse, but a normalization—what I will call an AI autumn.

The inevitable deflation of the hype won’t reveal a failed technology. Instead, it will reveal a technology that has already, quietly and irrevocably, proven its utility and woven itself into the fabric of our digital lives.

This isn’t a story about a bubble bursting; it’s about a revolutionary technology finally growing up. But let’s be clear: growing up can be a painful process. The normalization I’m describing won’t be a gentle, seamless transition. An industry built on unsustainable economics and AGI-or-bust promises can still face maybe not a brutal winter, but a significant autumn, even if the underlying technology continues to thrive.

Subscribe now

Anatomy of the Hype (Or Why the Skeptics Have a Point)

Before we can talk about the future, we have to be honest about the present. The current AI landscape feels like a bubble because, in many ways, it is one. This isn’t to say the technology is vaporware; far from it. The frenzy is built on a kernel of genuinely astonishing progress. But that kernel has been buried under an avalanche of speculative capital and quasi-religious prophecy.

The promises are, to put it mildly, grandiose. Tech leaders, flush with unprecedented investment, speak of replacing vast swaths of the workforce and ushering in an era of unimaginable productivity. Every incremental improvement is framed as another step on the inexorable march toward Artificial General Intelligence. This narrative is then amplified by a chorus of accelerationists and futurists who speak of the Singularity not as a distant sci-fi concept, but as an imminent event. It’s a powerful and compelling story, and it’s fueling a gold rush.

But back on planet Earth, the story is more complicated. For every breathless demo, there are practical and theoretical roadblocks that the hype conveniently ignores. The most glaring is the hallucination problem. These models, by their very nature, invent things. We’ve managed to reduce the frequency, but we haven’t eliminated the phenomenon, and there are compelling theoretical arguments that we may never be able to. This isn’t just a bug; it’s a feature of the architecture, a fundamental crack in the foundation of trust.

This technical limitation then crashes headfirst into the corporate world’s messy reality. Most companies, lured by the promise of easy productivity gains, are discovering a massive adoption gap. They lack the clean data, the streamlined processes, and the technical expertise to reliably integrate these powerful but flawed tools. It’s no wonder, then, that an astonishing number of corporate AI projects—some estimates say as high as 85%—are quietly failing to deliver a return on investment. Sky-high promises plus messy, difficult reality is the classic recipe for a bubble.

Perhaps the most potent dose of reality, however, is coming from the frontier models themselves. We’re witnessing a classic case of diminishing returns. The leap in capability from GPT-3 to GPT-4 was so profound it felt like a paradigm shift, leading many to draw a straight line on the progress graph and conclude that GPT-5 would be knocking on AGI’s door. That hasn’t happened.

The newest models are better, certainly, but the improvement is incremental, not awe-inspiring. It strongly suggests we’re hitting the ceiling of what the current paradigm can do. Experts like Yann LeCun and François Chollet argue persuasively that to progress further, we need fundamentally new approaches—paradigms that have yet to be invented. This pushes the dream of AGI firmly back into the realm of long-term research, not the foreseeable future.

Compounding this is a simple fact: the economics of frontier AI are fundamentally broken. The cost to train a single model like GPT-4 is north of $100 million. The data center infrastructure required to support the industry’s ambitions will require an estimated $5.2 trillion by 2030.

Unsurprisingly, this has created a severe profitability crisis. In 2024, OpenAI reportedly lost approximately $5 billion on $9 billion in revenue, with inference costs alone accounting for a multi-billion dollar loss. This isn’t a business model; it’s a venture-subsidized science experiment, and it’s hitting a hard physical wall with an energy grid that cannot keep up.

Furthermore, we must recognize that this isn’t just another tech bubble. The investment flowing into AI is qualitatively different from, say, funding for a better SaaS tool or a more efficient database. A significant portion of this capital is a high-stakes, geopolitical bet on the imminent arrival of AGI. The valuations of the frontier labs are not based on their current, money-losing products; they are based on the promise of creating a literal god-in-a-box.

Whether Sam Altman and company believe or not is beyond the point. This dream of AGI is driving market valuations, and when the market finally digests that we are hitting a paradigm ceiling—a point this article has already made—the withdrawal of that ‘AGI-or-bust’ capital won’t be a gentle correction. It will be a sudden, violent repricing that could vaporize billions in paper wealth overnight.

What Will Happen When the Bubble Bursts?

So, given the inflated expectations and technical ceilings, what happens when the hype recedes? I don’t really like to make predictions, and much less about the future. It’s damn hard. But I think we can outline a possible, perhaps even probable near future. I want to draw an analogy here and claim we will see not a true AI winter, but something close to an AI autumn.

An AI autumn is an economic event. It’s a period of massive financial correction, characterized by layoffs, hiring freezes, startup failures, and a freeze in venture capital. It’s painful for the people and companies in the field. An AI winter, on the other hand, is a crisis of relevance of the core technology. It’s when the technology itself proves to be a dead end, progress stalls, and the world moves on.

To be as blunt as I can, I do believe a severe autumn for the AI industry is not just possible; it’s likely. The current economics are unsustainable, as we’ve seen. But the central argument of this article is that this painful industrial correction will not trigger a catastrophic winter, which would be far worse. No, AI is here to stay, and here is why.

First, we can’t ignore the relentless democratization of compute. The idea that cutting-edge AI will forever be the exclusive domain of billion-dollar data centers is a historical fallacy. We are already seeing an explosion of highly capable open-source models that can run on local, consumer-grade hardware. What requires a professional-grade, 10,000 dollars GPU today will run on your laptop in two years, and on your phone two years after that.

This trajectory completely decouples the utility of AI from the subsidized business models of a few large companies. The capability is escaping the lab and becoming part of the background radiation of computing.

Second, even if the progress of frontier models were to stop dead in its tracks today—which it won’t, but it will likely continue to decelerate—we still have a decade’s worth of technological breakthrough that most of the world has not even begun to properly digest. The current adoption gap isn’t a sign of inevitable failure; it’s a sign that the technology has advanced far faster than our institutions can keep up.

A slowdown in R&D investment won’t cause a retreat. Instead, it will trigger a necessary and healthy shift in focus from pure research to practical implementation, integration, and process refinement. This is what maturity looks like. The frantic sprint to invent the future will become the marathon of actually building it.

Most importantly, this shift will not trigger a true AI winter because we are simply far beyond the point where Artificial Intelligence can disillusion us. It is already a proven technology, woven so deeply into our digital infrastructure that a true winter is no longer possible.

Why We Won’t See Another AI Winter

Let’s start with Generative AI itself. Even with all its flaws, its core utility is now undeniable. The previous AI winters occurred when promising lab demos failed to translate into real-world applications. That is not the situation today.

A significant percentage of the global population—some conservative estimates say around 10%— now uses these tools not as novelties, but as integrated parts of their daily work. It’s the assistant that transcribes a meeting and pulls out action items, summarizes a sprawling email thread you don’t have time to read, and helps you rephrase a blunt message into a diplomatic one. Online search is quickly becoming the playground for generative AI, and online search is by far the most profitable business in the Internet Era.

The genie is out of the bottle; people are not going to suddenly stop using a tool that demonstrably saves them time, just because its creators promised it would become a god.

But perhaps the world of software development is an even more potent example. There’s a lot of noise about irresponsible “vibe coding,” where novices generate code they don’t understand, creating an unmaintainable mess. This is a real problem, but it’s a problem of skill, not a failure of the tool.

For experienced developers, these assistants are transformative. The mythical “10x productivity” boost is largely a myth, but a consistent 1.5x to 2x multiplier is very real. I’ve seen it in my own projects. Code assistants act as the new IntelliSense, handling the mind-numbing boilerplate and letting me focus on the architectural challenges. I may now only write 20% of the final characters in the codebase, but I am still the author of 90% of the critical ideas. This is not a crutch; it’s leverage.

And beyond these consumer-facing applications lies an even larger world of traditional machine learning that is indispensable to modern science and industry.

From drug discovery and genomic sequencing in biotech to predictive maintenance and supply chain optimization in manufacturing, decades of successful applications of AI in the industry today delivers billions of dollars in quantifiable value. Their success is measured in efficiency gains and scientific breakthroughs, not hype cycles.

But the more fundamental point is this: the debate over a “Generative AI” bubble distracts from the fact that the broader field of AI has already won its place. We haven’t had a true AI winter since the 1990s because AI stopped being a distinct, speculative field and became the foundational plumbing of the modern world. The search engine that found this article? That’s AI. The recommendation algorithm that determines your social media feed? AI. The logistics network that delivered your last package, the facial recognition that unlocks your phone, the voice transcription that takes your meeting notes—it’s all AI. Not Generative AI (for the most part), but AI nonetheless.

The line between computer science and AI has become so blurred that it’s practically meaningless. To talk about an AI winter today is like talking about an Internet winter in 2005. The technology is simply too embedded to fail.

However, as we’ve argue, there will be some painful correction. That much is, I think, almost undeniable. If that’s indeed the case, here are some optimistic arguments for why it may all be for the better in the end.

The Renaissance of AI Research

When the unsustainable hype collides with this resilient foundation, a fundamental law of economics reasserts itself: there is no free lunch. An AI autumn is the inevitable trade-off for a period of unchecked exuberance. A wave of consolidation will wash away unprofitable startups, and the market’s strategic focus will pivot from “bigger is better” to efficiency.

But this period of commercial cooldown has a powerful, if counter-intuitive, silver lining: a renaissance of real research. History shows us that AI’s greatest winters have been fertile ground for its most important breakthroughs. The hype recedes, and with it, the noise. The crushing pressure for short-term commercial returns is replaced by the intellectual freedom to tackle fundamental, long-term challenges.

Many of the core technologies fueling today’s boom were born in the quiet of previous winters. The backpropagation algorithm, popularized by Geoffrey Hinton in the 1980s, was refined during a period of deep skepticism about neural networks. Most famously, the Long Short-Term Memory (LSTM) architecture, which was a cornerstone of natural language processing for decades, was developed by Hochreiter and Schmidhuber in 1997, the absolute heart of the last AI winter.

The coming autumn will trigger a similar cycle. As the brightest minds are freed from the scaling hype, the real work on the next generation of AI can begin. We are already seeing the intellectual seeds of this shift. AI pioneers are openly discussing the deep limitations of current models. Yann LeCun is championing his Joint Embedding Predictive Architecture (JEPA) as a path toward “world models” that learn abstract representations of reality.

The field of Neuro-Symbolic AI, which fuses neural nets with structured logic, is experiencing a surge in interest. These are not incremental improvements; they are explorations of entirely new paradigms.

Conclusion: No Retreat, Just Normalization

So, where does that leave us? The coming correction is not an apocalypse; it’s a maturation. The frantic, gold-rush energy will dissipate, and in its place, something far more durable will emerge. The deflation of the hype bubble will not send talent fleeing the field or cause us to abandon the tools we’ve built. Instead, it will mark the end of the beginning.

The great irony is that the very thing that guarantees AI’s long-term survival—its commoditization into reliable ‘plumbing’—is what makes the current industry valuations so precarious. Plumbing is a low-margin, utility business, not a world-dominating monopoly. This disconnect between utility and valuation is the financial fault line where the industrial earthquake will hit. The era of breathless, revolutionary promises will give way to the slow, difficult, and necessary work of integration.

This is the natural lifecycle of any transformative technology. It moves from a speculative curiosity to a reliable, if sometimes challenging, part of the professional toolkit. Generative AI will not become the all-knowing oracle we were promised, but it has already secured its place as a uniquely powerful tool for thought, creation, and productivity.

The question was never really if AI would change the world; the underlying technology has been doing that for decades. The real question is how we manage the transition. This industrial autumn will be cushioned, to some extent, by geopolitical reality. The race between the US and China ensures that a certain level of state-sponsored R&D will continue, preventing a total 1980s-style collapse.

But for the people working in the field, the transition will still be jarring. The future of AI isn’t a simple story of success or failure. It’s the messy, often painful process of separating a world-changing technology from the unsustainable industry that’s driving it, and going back to drawing board, back to building new and even cooler stuff.

I've spent the last few years trying to make sense of the noise around Artificial Intelligence, and if there's one feeling that defines the experience, it's whiplash. One week, I'm reading a paper that promises AI will cure disease and unlock unimaginable abundance; the next, I'm seeing headlines about civilizational collapse. This dizzying cycle of AI springs, periods of massive investment and hype, followed by the chilling doubt of AI winters isn't new. It's been the engine of the field for decades.

After years of this, I've had to develop my own framework just to stay grounded. It’s not about being an optimist or a pessimist; it’s about rejecting both extremes. For me, it’s a commitment to a tireless reevaluation of the technology in front of us; to using reason and evidence to find a path forward, because I believe we have both the power and the responsibility to shape this technology’s future. That begins with a clear-eyed diagnosis of the present.

One of the most useful diagnostic tools I've found for this comes from computer scientist Melanie Mitchell. In a seminal paper back in 2021, she identified what she claims are four foundational fallacies, four deeply embedded assumptions that explain to a large extent our collective confusion about AI, and what it can and cannot do.

My goal in this article isn't to convince you that Mitchell is 100% right. I don't think she is, either, and I will provide my own criticism and counter arguments to some points. What I want is to use her ideas as a lens to dissect the hype, explore the counterarguments, and show why this intellectual tug-of-war has real-world consequences for our society, our economy, and our safety.

Deconstructing the Four Fallacies

For me, the most important test of any idea is its empirical validation. No plan, no matter how brilliant, survives its first encounter with reality. I find that Mitchell’s four fallacies are the perfect tool for this. They allow us to take the grand, sweeping claims made about AI and rigorously test them against the messy, complicated reality of what these systems can actually do.

Fallacy 1: The Illusion of a Smooth Continuum

The most common and seductive fallacy is the assumption that every impressive feat of narrow AI is an incremental step on a smooth path toward human-level Artificial General Intelligence (AGI). That is, that intelligence is a single, unidimensional metric on a continuum that goes from narrow to general.

We see this everywhere. When IBM's Deep Blue beat Garry Kasparov at chess, it was hailed as a first step towards AGI. The same narrative emerged when DeepMind's AlphaGo defeated Lee Sedol. This way of thinking creates, according to Mitchell, a flawed map of progress, tricking us into believing we are much closer to AGI than we are. It ignores the colossal, unsolved challenge known as the commonsense knowledge problem—the vast, implicit understanding of the world that humans use to navigate reality.

As philosopher Hubert Dreyfus famously said, this is like claiming that the first monkey that climbed a tree was making progress towards landing on the moon. Well, in a sense, maybe it is, but you get the point. We didn't get to the moon until we invented combustion rockets. Climbing ever taller trees gets us nowhere closer, it's just a distraction. In the same sense, mastering a closed-system game may be a fundamentally different challenge than understanding the open, ambiguous world.

But here's the nuance. While beating Kasparov isn't a direct step to having a conversation, the methods developed can be surprisingly generalizable. The architecture that powered AlphaGo was later adapted into MuZero, a system that mastered Go, chess, and Atari games without being told the rules.

Furthermore, can we really call a Large Language Model narrow in the same way? Its ability to write code and summarize text feels like a qualitative leap in generality that the monkey-and-moon analogy doesn't quite capture.

This leaves us with a forward-looking question: How do recent advances in multimodality and agentic AI test the boundaries of this fallacy? Does a model that can see and act begin to bridge the gap toward common sense, or is it just a more sophisticated version of the same narrow intelligence? Are world models a true step towards AGI or just a higher branch in a tree of narrow linguistic intelligence?

Fallacy 2: The Paradox of Difficulty

We have a terrible habit of projecting our own cognitive landscape onto machines, assuming that what's hard for us is hard for them, and what's easy for us is easy for them. For decades, the opposite has been true.

This is Moravec's Paradox, named after the roboticist Hans Moravec, who noted it's easier to make a computer exhibit adult-level performance on an IQ test than to give it the sensory and motor skills of a one-year-old.

This explains why we have AI that can master the ridiculously complex game of Go, while a fully self-driving car remains stubbornly just over the horizon. The "easy" things are built on what Mitchell calls the "invisible complexity of the mundane." This paradox causes a chronic mis-calibration of our progress and priorities, leading us to be overly impressed by performance in formal domains while underestimating the staggering difficulty of the real world.

Of course, some would argue this isn't a fundamental barrier, but a temporary engineering hurdle. They’d say that with enough data and compute, the "invisible complexity" of the real world can be learned, just like the complexity of Go was.

From this perspective, the problem isn't one of kind, but of scale. This forces us to ask: as sensor technology and robotics improve, are we finally starting to overcome Moravec's Paradox? Or are we just discovering even deeper layers of complexity we never knew existed?

Fallacy 3: The Seduction of Wishful Mnemonics

Language doesn't just describe reality; it creates it. In AI, we constantly use anthropomorphic shorthand, saying a system "learns," "understands," or has "goals." Mitchell argues this practice of using "wishful mnemonics" is deeply misleading, fooling not just the public but the researchers themselves.

When a benchmark is called the "General Language Understanding Evaluation" (GLUE) and a model surpasses the human baseline, headlines declare that AI now understands language better than humans. But does it?

The term "stochastic parrot" was coined as a powerful antidote, reframing what LLMs do as sophisticated mimicry rather than comprehension. This isn't just a semantic game, Mitchell argues; it creates a flawed mental model that leads to misplaced trust, encouraging us to deploy systems in high-stakes situations where a lack of true understanding can have serious consequences.

A fair critique is that these terms are a necessary cognitive shorthand. At a certain level of complexity, a system's emergent behavior becomes functionally indistinguishable from "understanding," and arguing about whether it really understands is an unprovable philosophical distraction.

But that still leaves a crucial question: can we develop a more precise, less anthropomorphic vocabulary to describe AI capabilities? Or is our human-centric language the only tool we have to reason about these new forms of intelligence, with all the baggage that entails?

Fallacy 4: The Myth of the Disembodied Mind

This is the most philosophical, and in my opinion, the most important fallacy. It's the deep-seated assumption that intelligence is, like software, a form of pure information processing that can be separated from its body.

This "brain-as-computer" metaphor leads to the belief that AGI is simply a matter of scaling up compute to match the brain's raw processing power. It's challenged by Mitchell and many others with the thesis of embodied cognition, a view from cognitive science which holds that intelligence is inextricably linked to having a body that interacts with the world. If this is correct, then our current approach may just be creating ever-more-sophisticated systems that are fundamentally brittle because they lack grounded understanding.

This is where we hit the great intellectual battle line in modern AI. The primary counterargument can be framed in terms of Rich Sutton's famous essay, "The Bitter Lesson," which argues that the entire history of AI has taught us that attempts to build in human-like cognitive structures (like embodiment) are always eventually outperformed by general methods that just leverage massive-scale computation.

From this viewpoint, embodiment isn't a magical prerequisite for intelligence; it's just another fiendishly complex problem that will yield to more data and processing power.

This tension poses a critical question for the future: do multimodal models that can process images and text represent a meaningful step toward solving the embodiment problem? Or are they just a more sophisticated version of the same disembodied mind, a brain in a slightly larger digital vat?

What is Intelligence, Really?

As we dig into these fallacies, a deeper pattern emerges. They aren't just four isolated mistakes; they're symptoms of a fundamental schism in how the AI world thinks about intelligence itself. Again, my goal isn't to pick a side but to avoid falling prey to cheap heuristics or ideological banners, and instead evaluate which of these paradigms gives us a more useful map of reality.

On one side, you have what I’ll call the Cognitive Paradigm, championed by thinkers like Mitchell and her mentor, superstar AI researcher and philosopher Douglas Hofstadter. This view sees intelligence as a complex, integrated, and embodied phenomenon. It assumes that the things we associate with human intelligence—common sense, emotions, values, a sense of self—are likely inseparable components of the whole, emerging from rich interaction with a physical and social world.

From this perspective, the path to AGI requires a deep, scientific understanding of these integrated components, not just more processing power.

On the other side is the Computationalist Paradigm, which is the implicit philosophy behind many of today's leading labs, and best captured by The Bitter Lesson. This posits that the biggest breakthroughs have always come from general methods that leverage massive-scale computation—in other words, from scaling things up.

In this paradigm, intelligence is a more abstract, substrate-independent quality of optimization. Problems like embodiment aren't fundamental barriers; they are just incredibly complex computational tasks that will eventually be solved by ever-larger models and ever-faster chips.

Of course, it's not a perfect binary. Most researchers are pragmatists, like me, working somewhere in the messy middle. But these two paradigms represent the poles of the debate, and the tension between them defines the entire field. It shapes which research gets funded, which systems get built, and ultimately, which vision of the future we are collectively racing toward.

Why This Debate Matters

This debate isn't just an academic parlor game. These fallacies have a massive ripple effect across society because they obscure a fundamental rule of technology and economics: there's no free lunch, only trade-offs.

The hype generated by fallacious thinking isn't just an innocent mistake; it's the fuel for a powerful economic engine. The intense competition between tech giants, the flood of venture capital, and the geopolitical AI race all depend on a constant narrative of imminent, world-changing breakthroughs. This political economy of hype forces us into a series of dangerous trade-offs.

First, we trade long-term progress for short-term hype.

The fallacies create an unstable, boom-and-bust funding cycle. During an AI spring, capital flows to projects that can produce impressive-looking demos, often based on narrow benchmarks. This starves the slow, methodical, foundational research needed to solve the hard problems like common sense and reasoning. The result is a field that lurches from one hype bubble to the next, leaving a trail of abandoned projects and unfulfilled promises that trigger the inevitable AI winter.

Second, we trade public trust for market excitement.

The cycle of over-promising and under-delivering is deeply corrosive. When we use wishful mnemonics to describe a system that "understands," and it then fails in spectacular, nonsensical ways in the real world, it breeds public anxiety and skepticism. Recent studies show the public perceives AI scientists more negatively than almost any other field, specifically because of a perceived lack of prudence. This isn't a vague feeling; it's a direct reaction to the unintended consequences of deploying brittle, overhyped systems.

Finally, and most critically, we trade responsible validation for speed to market.

This is where the consequences become most severe. Believing a system is on a continuum with general intelligence, or that it truly "understands" language, leads to its premature deployment in high-stakes domains.

When a mental health chatbot, which is fundamentally, at least today, a sophisticated pattern-matcher, gives harmful advice to a person in crisis, it’s a direct result of these fallacies. When we over-rely on brittle systems in healthcare, finance, or autonomous vehicles, we are making a dangerous bet, trading real-world safety for the illusion of progress.

Conclusion

So where does this leave us? The value of Mitchell's fallacies isn't just in spotting hype, but in exposing the deep, productive tension between these two powerful ways of thinking about intelligence. We can't ignore the fallacies, but we also can't deny the incredible, world-altering power of the scaling paradigm that fuels them.

Mitchell in her paper compares modern AI to alchemy. It produces dazzling, impressive results but it often lacks a deep, foundational theory of intelligence.

It’s a powerful metaphor, but I think a more pragmatic conclusion is slightly different. The challenge isn't to abandon our powerful alchemy in search of a pure science of intelligence. The goal, at least from a pragmatist point of view, should be to infuse our current alchemy with the principles of science, to make scaling smarter, safer, and more grounded by integrating the hard-won insights about how intelligence actually works.

The path forward, I believe, requires more than just intellectual humility. It also requires a willingness to synthesize these seemingly opposed worldviews, and a commitment to a tireless reevaluation of the technology before us. The ultimate question is not if we should choose the path of scaling or the path of cognitive science, but how we can weave them together to guide the raw power of our modern AI alchemy with the deep understanding of a true science of intelligence.

PS: Remember you can get Mostly Harmless AI while in early access at a reduced price. We are now running a special offer that gives you the PDF and EPUB version of the book as it currently stands, plus guaranteed access to all future editions for just $5.

Get Mostly Harmless AI ($5)

For centuries, we humans have been captivated by the idea of a thinking machine. This isn’t some modern tech obsession; the dream of automatons and artificial minds is woven through our myths and philosophies. But the formal quest to build one began only in the mid-20th century, and its history has been a dramatic back-and-forth between two core, seemingly antagonistic approaches.

One path, rooted in the logic of rationalism, sought to build intelligence from the top down by programming explicit rules and symbols. The other, inspired by the biological empiricism of the brain, tried to create it from the bottom up by allowing machines to learn patterns from data and experience.

This chapter explores the history of Artificial Intelligence (AI) through the lens of this great intellectual tug-of-war. It is a journey through distinct eras, each defined by which philosophy was dominant, what external factors like computing power and data availability enabled its rise, and how these forces have finally begun to converge, leading us to the powerful tools we have today.

In the appendix of this book, we will present a detailed chronology of the most important milestones in the history of artificial intelligence.

Subscribe now

The Foundational Era (1940s - 1960s)

The dawn of AI was a time of immense optimism, where the very concept of a “thinking machine” was formalized. Even before the field had a name, its philosophical and theoretical groundwork was being laid. In his seminal 1950 paper, “Computing Machinery and Intelligence,” Alan Turing proposed the Turing Test, setting a profound, long-term goal: to create a machine whose conversation was indistinguishable from a human’s. In parallel, the work of Warren McCulloch and Walter Pitts in 1943 on the first mathematical model of an artificial neuron planted the seeds of the connectionist dream—the idea that intelligence could emerge from simple, brain-like units.

When the field was officially christened at the Dartmouth Workshop in the summer of 1956, the symbolic, logic-based paradigm took the lead. Researchers believed that human thought could be mechanized, and the primary task was to build systems that could manipulate symbols according to formal rules. This vision was solidified by the creation of the LISP programming language in 1958, a tool perfectly suited for this symbolic manipulation.

Yet, in that same year, the connectionist counterpoint took physical form. Frank Rosenblatt developed the Perceptron, the first artificial neural network that could learn to classify patterns on its own, offering a tangible, bottom-up alternative to pure logic.

The public imagination was quickly captured by early demonstrations of AI’s potential. The Unimate (1961), the first industrial robot, showed that machines could perform physical labor. Shakey the Robot (1966) took this a step further, becoming the first mobile robot to perceive its environment and reason about its own actions. Joseph Weizenbaum’s ELIZA (1964), a simple chatbot that simulated a psychotherapist, revealed how easily humans could attribute intelligence and understanding to a machine. But this initial optimism soon collided with reality.

The ambitious promises of creating true intelligence went unfulfilled, and in 1969, the publication of the book Perceptrons by Marvin Minsky and Seymour Papert delivered a critical blow. By rigorously detailing the mathematical limitations of simple neural networks, the book effectively starved the connectionist school of funding, ushering in the first “AI Winter” and ensuring that the symbolic approach would dominate the field for the next decade.

The Knowledge Era (1970s - 1980s)

With connectionism on the back burner, the field regrouped around a more pragmatic goal: instead of trying to create general intelligence, researchers focused on capturing and mechanizing human expertise in narrow domains. This led to the golden age of expert systems, the first commercially successful form of AI. The core idea was to interview a human expert, painstakingly encode their knowledge into a vast set of “if-then” rules, and use a reasoning engine to produce solutions.

This approach yielded impressive results. SHRDLU (1972) was a landmark natural language program that could understand and respond to commands about a simulated world of blocks, showcasing a new level of sophistication for symbolic AI. Expert systems like MYCIN (1972) could diagnose blood infections as accurately as junior doctors, while others like DENDRAL and PROSPECTOR found success in chemistry and geology. This culminated in the first true commercial boom, as companies like Digital Equipment Corporation used the XCON system (1980) to configure complex computer orders, saving millions of dollars. The ambition of this paradigm reached its peak with the Cyc project (1984), a monumental effort to manually encode all of human common sense knowledge into a single, massive database.

While the symbolic school reigned, a connectionist undercurrent continued to flow. In Japan, Kunihiko Fukushima’s work on the Neocognitron (1980) created a hierarchical, multi-layered neural network for visual recognition that was the direct ancestor of the architectures that would dominate computer vision decades later. And in 1986, the popularization of the backpropagation algorithm provided an efficient method for training these deeper networks, solving a critical problem that had plagued the field for years.

However, the symbolic paradigm’s dominance was destined to end. Expert systems were incredibly brittle; they were expensive to build, nearly impossible to update, and would fail completely if faced with a situation not explicitly covered by their rules. The hype, fueled in part by Japan’s ambitious Fifth Generation Computer Systems project (1982), once again outpaced reality. When the specialized hardware market collapsed in 1987, the field plunged into its second “AI Winter,” leaving the promise of AI unfulfilled once more.

The Internet Era (1990 - 2011)

The end of the second AI winter was not driven by a single algorithmic breakthrough, but by two external forces that changed everything: the public launch of the World Wide Web in 1991 and the invention of the Graphics Processing Unit (GPU) in The web began generating an unimaginable ocean of data—text, images, and user interactions. The GPU, particularly after the release of NVIDIA’s CUDA platform in 2007, provided a way to perform the massive parallel computations needed to learn from that data. These two catalysts—data and computation—created the perfect conditions for the statistical, learning-based paradigm to finally thrive.

Before deep learning took hold, this new environment fueled the rise of “shallow” machine learning. Algorithms like Support Vector Machines (SVMs) (1995) became dominant, and open-source libraries like scikit-learn (2007) made them accessible to a wide audience. This approach had a massive real-world impact, powering the recommender systems of companies like Amazon and sparking global competitions like the Netflix Prize (2006). The infrastructure to handle this new scale was built in parallel, with Google’s MapReduce (2004) providing the blueprint for big data processing.

During this time, foundational work in reinforcement learning was also bearing fruit. TD-Gammon (1992) showed that a program could teach itself to play backgammon at a superhuman level, and the textbook by Sutton & Barto (1998) codified the field for a new generation. The seeds for the coming deep learning revolution were being sown with the invention of key architectures like LSTMs (1997) and LeNet-5 (1998), while the creation of the massive ImageNet dataset (2009) provided the high-quality benchmark that would soon ignite it.

AI also became a tangible part of public life. The symbolic paradigm had its last great public triumphs with Deep Blue’s victory over Garry Kasparov in chess (1997) and Watson’s win on Jeopardy! (2011). But the future belonged to learning-based systems. Dragon NaturallySpeaking (1997) brought continuous speech recognition to consumers. Competitions like the DARPA Grand Challenge (2004) spurred the development of autonomous vehicles. Consumer products like the Roomba (2002) and Microsoft’s Kinect (2010) brought robotics and computer vision into millions of homes. With the launch of Siri in 2011, a conversational AI assistant was finally in everyone’s pocket.

The Deep Learning Era (2012 - 2018)

If the Internet Era set the stage, 2012 was the year the curtain rose on the deep learning revolution. In October, a deep convolutional neural network called AlexNet, trained on GPUs using the ImageNet dataset, shattered all previous records in the annual image recognition competition. This “ImageNet Moment” proved the overwhelming superiority of deep, data-driven learning and kicked off a Cambrian explosion of breakthroughs.

This period was a stunning validation of what AI researcher Rich Sutton would later call The Bitter Lesson: that general methods leveraging massive computation almost always outperform approaches that rely on hand-crafted human knowledge. The field progressed at a breathtaking pace. In natural language processing, Word2Vec (2013) provided a powerful way to represent the meaning of words as vectors. In generative AI, Generative Adversarial Networks (GANs) (2014) introduced a novel way to create stunningly realistic synthetic images. New architectures like ResNet (2015) allowed for the creation of networks hundreds of layers deep, solving a fundamental barrier to scale.

These new techniques allowed AI to achieve superhuman performance in increasingly complex domains. Deep Q-Networks (DQN) (2013) learned to master classic Atari games directly from pixels, and in a landmark event in March 2016, AlphaGo defeated Lee Sedol, the world’s greatest Go player. The revolution was powered by a new generation of open-source tools like TensorFlow (2015) and PyTorch (2016) that democratized deep learning, as well as specialized hardware like Google’s Tensor Processing Units (TPUs) (2016). The era culminated with the invention of the Transformer architecture in 2017.

However, 2018 marked a turning point. In March, the Cambridge Analytica scandal revealed how machine learning algorithms, fed by the personal data of millions of Facebook users, had been used for political manipulation, sparking a global reckoning over data privacy and the ethics of AI.

That same month, though, the scientific community formally recognized the field’s impact, awarding the Turing Award to Geoffrey Hinton, Yann LeCun, and Yoshua Bengio for their foundational work. The Turing Award–aptly named after the most important figure in the history of Computer Science as a whole, let alone Artificial Intelligence–is the most prestigious academic award in computing, akin to the Nobel Prize.

The Generative Era (2019 - Present)

The current era is defined by the application of the Transformer architecture at an unprecedented scale. By training these models on vast swaths of the internet, researchers discovered that quantitative leaps in size and data could lead to qualitative leaps in capability, resulting in models with emergent generative and reasoning abilities that have captured the world’s attention.

The first sign of this new power came with GPT-2 in 2019, whose ability to generate coherent text was so advanced that its release was initially staged due to safety concerns. Its successor, GPT-3 (2020), demonstrated that massive scale could unlock “few-shot” learning, the ability to perform tasks it was never explicitly trained on.

Soon, this generative power was applied beyond text to images with DALL-E (2021) and to code with GitHub Copilot (2021). But the cultural tipping point arrived in November 2022 with the release of ChatGPT. Its simple, conversational interface made the power of Large Language Models (LLMs) accessible to millions, sparking a global phenomenon and a new wave of investment.

This boom was accompanied by a powerful open-source counter-movement. The release of the image-generation model Stable Diffusion (August 2022) and Meta’s Llama models (2023) democratized access to powerful foundation models, sparking a “Llamaverse” of community-driven innovation. The field is now a global race, with major competitors like Anthropic’s Claude (2023), Google’s multimodal Gemini (December 2023), and China’s DeepSeek R1 (January 2025) demonstrating capabilities on par with the best proprietary systems.

Perhaps the most profound impact of this new era has been in science. In November 2020, AlphaFold 2 solved the 50-year-old grand challenge of protein folding, a breakthrough of such significance that its creators were awarded the Nobel Prize in This demonstrated that AI could be a tool not just for automating tasks, but for accelerating fundamental scientific discovery. The road ahead now points towards more autonomous, “agentic” systems, where the AI transitions from a single-response tool to a collaborator capable of executing complex, multi-step tasks on our behalf.

Conclusion

We’ve journeyed through decades of ambition, breakthroughs, and tough realizations. What we’ve seen is a constant back-and-forth, a dynamic dance between two powerful ideas: the precise, rule-based, inflexible logic of symbolic AI and the adaptable, pattern-based, unreliable power of statistical AI. This dance, as we’ve explored, often mirrors the philosophical tension between rationalism and empiricism.

Today, AI stands at a fascinating crossroads. The purely statistical systems that define the generative era have achieved incredible feats. Yet, we are beginning to see diminishing returns. With GPT-4 having been a high point, many newer models have made only incremental progress, suggesting that simply scaling the existing paradigm may not be enough to achieve the next level of intelligence. This has led some to speculate that we may be on the brink of a third “AI Winter,” as the hype once again outpaces the reality of the technology’s capabilities.

The recent focus on reasoning models and agentic systems seems capable of fueling the statistical hype a bit longer, but a growing number of researchers are realizing we may not achieve Artificial General Intelligence (AGI) purely by scaling. This brings us to a crucial realization: the future of AI likely isn’t about one approach winning out over the other, but about intelligently combining them. The inherent limitations of today’s models—their unreliability and lack of true reasoning—have sparked a renewed interest in the long-neglected symbolic paradigm. Hybrid approaches, particularly neuro-symbolic AI, which seek to integrate the pattern-matching strengths of neural networks with the rigorous logic of symbolic systems, hold immense potential for creating the next breakthrough.

Whether a winter is coming or not, it is indisputable that AI has already had a profound impact on society and will continue to do so. The history of this field is far from finished. It is a living, breathing, civilization-wide project with the potential to transform society for the better or, some believe, to become our ultimate doom. Everyone has a place here: technologists, yes, but also humanists, economists, historians, artists, and policymakers. The next few years promise to be extremely exciting, and you can be a part of shaping what comes next.

Thanks for reading! Below you’ll find the full expanded timeline. Please let me know if you think I missed something or made any mistakes. All feedback is appreciated!

PS: Claim your copy of Mostly Harmless AI for only $5 in the link below.

Get Mostly Harmless AI ($5)

Appendix: A Chronology of Artificial Intelligence (1956-2025)

This timeline details the key breakthroughs, conceptual shifts, and landmark achievements in the field of Artificial Intelligence, tracing its path from a niche academic discipline to a transformative global technology.

The Foundational Era (1940s - Late 1960s)

• (Science) 1943: The First Artificial Neuron is proposed by Warren McCulloch and Walter Pitts, laying the theoretical foundation for connectionism.

• (Science) October 1950: Alan Turing publishes “Computing Machinery and Intelligence,” introducing the Turing Test.

• (Social) Summer 1956: The Dartmouth Workshop is held, where John McCarthy coins the term “Artificial Intelligence,” formally establishing the field.

• (Tech) 1958: Frank Rosenblatt develops the Perceptron, the first artificial neural network capable of learning.

• (Tech) 1958: John McCarthy develops the LISP programming language, which becomes the standard for symbolic AI.

• (Product) 1961: The Unimate industrial robot begins work on a General Motors assembly line.

• (Product) 1964: Joseph Weizenbaum creates the chatbot ELIZA at MIT.

• (Tech) 1966: The Stanford Research Institute (SRI) develops Shakey, the first mobile robot to reason about its own actions.

• (Social) 1969: The publication of Perceptrons by Marvin Minsky and Seymour Papert marks the beginning of the first “AI Winter.”

The Knowledge Era (1970s - 1989)

• (Tech) 1972: Terry Winograd develops SHRDLU, a groundbreaking natural language understanding program.

• (Tech) 1972: The logic programming language Prolog is created by Alain Colmerauer and Philippe Roussel, becoming a key tool for symbolic AI.

• (Tech) 1972: Stanford University develops the MYCIN expert system for medical diagnosis.

• (Science) 1974: Marvin Minsky publishes his influential paper on “Frames” theory, a new paradigm for knowledge representation.

• (Tech) Late 1970s: Expert systems like DENDRAL (for chemistry) and PROSPECTOR (for geology) demonstrate success in specialized scientific domains.

• (Science) 1980: Kunihiko Fukushima develops the Neocognitron, an early hierarchical neural network that is the direct ancestor of modern Convolutional Neural Networks (CNNs).

• (Product) 1980: Digital Equipment Corporation begins using the XCON expert system, marking a high point for commercial AI.

• (Social) 1982: Japan’s Ministry of International Trade and Industry begins the Fifth Generation Computer Systems project, a massive initiative to build a new generation of computers based on logic programming, sparking competitive AI investment worldwide.

• (Tech) 1984: The Cyc project is initiated by Douglas Lenat, an ambitious attempt to manually encode all of human common sense knowledge into a single knowledge base.

• (Science) 1986: The backpropagation algorithm is popularized by Geoffrey Hinton, David Rumelhart, and Ronald Williams.

• (Social) 1987: The collapse of the LISP machine market signals the start of the second “AI Winter.”

The Internet Era (1990 - 2011)

• (Social) August 1991: The World Wide Web project is released to the public, creating the infrastructure for the data explosion that would fuel modern AI.

• (Science) 1992: Gerald Tesauro develops TD-Gammon, a backgammon program that trains to a superhuman level using reinforcement learning, a landmark for the field.

• (Social) 1995: Stuart Russell and Peter Norvig publish “Artificial Intelligence: A Modern Approach,” which becomes the leading textbook in the field for decades.

• (Science) 1995: The Support Vector Machine (SVM) algorithm is popularized by Corinna Cortes and Vladimir Vapnik.

• (Social) May 1997: IBM’s Deep Blue defeats world chess champion Garry Kasparov.

• (Science) 1997: Sepp Hochreiter and Jürgen Schmidhuber invent the Long Short-Term Memory (LSTM) network.

• (Product) 1997: Dragon NaturallySpeaking is released, becoming the first widely available continuous speech recognition software for consumers.

• (Product) September 1998: Google is founded, and Amazon patents its item-to-item collaborative filtering, marking the start of large-scale data-driven AI applications.

• (Science) 1998: Richard Sutton and Andrew Barto publish “Reinforcement Learning: An Introduction,” a seminal textbook that codifies the field.

• (Tech) November 1998: Yann LeCun and his team develop LeNet-5, a pioneering Convolutional Neural Network (CNN).

• (Tech) August 1999: NVIDIA releases the GeForce 256, marketed as the world’s first Graphics Processing Unit (GPU).

• (Product) November 2000: Honda unveils its ASIMO humanoid robot, a landmark in robotics and motion planning.

• (Product) September 2002: iRobot releases the Roomba, the first commercially successful autonomous home robot.

• (Tech) 2004: Google publishes its paper on MapReduce, a programming model for processing massive datasets that becomes foundational to big data infrastructure.

• (Social) March 2004: The first DARPA Grand Challenge for autonomous vehicles is held, sparking a new wave of research in self-driving technology.

• (Social) October 2006: The Netflix Prize competition is launched, galvanizing research in recommender systems.

• (Science) 2006: Geoffrey Hinton develops Deep Belief Networks, introducing effective strategies for unsupervised layer-wise pre-training.

• (Tech) June 2007: NVIDIA releases CUDA, a parallel computing platform that allows developers to use GPUs for general-purpose processing.

• (Tech) June 2007: David Cournapeau develops scikit-learn as a Google Summer of Code project.

• (Science) 2009: A Stanford team led by Andrew Ng publishes a paper showing that GPUs can make training deep neural networks 10-100 times faster.

• (Tech) 2009: The ImageNet dataset is created by Fei-Fei Li’s team at Stanford.

• (Product) November 2010: Microsoft releases the Kinect, a consumer device that brings sophisticated real-time computer vision into millions of homes.

• (Social) February 2011: IBM’s Watson wins the quiz show Jeopardy!.

• (Product) October 2011: Apple integrates Siri into the iPhone 4S, making conversational AI assistants a mainstream consumer product.

The Deep Learning Era (2012 - 2018)

• (Social) April 2012: Coursera is founded, and Andrew Ng’s Machine Learning course begins to democratize AI education.

• (Science) June 2012: The Google Brain “Cat Neuron” project demonstrates that a neural network can learn high-level concepts from unlabeled data.

• (Social) October 2012: AlexNet, a deep CNN trained on GPUs, wins the ImageNet competition by a massive margin, officially kicking off the deep learning revolution.

• (Tech) 2013: Google researchers led by Tomas Mikolov release Word2Vec, a highly efficient method for creating word embeddings that revolutionizes NLP.

• (Science) December 2013: DeepMind publishes its work on Deep Q-Networks (DQN), demonstrating an AI that can learn to play Atari games at a superhuman level from raw pixels.

• (Science) June 2014: Ian Goodfellow and his colleagues introduce Generative Adversarial Networks (GANs), sparking a revolution in generative AI for images.

• (Science) December 2015: A team at Microsoft Research introduces Deep Residual Networks (ResNet), allowing for the training of much deeper neural networks.

• (Tech) November 2015: Google releases the TensorFlow open-source library, making deep learning more accessible.

• (Social) March 2016: Google DeepMind’s AlphaGo defeats world Go champion Lee Sedol.

• (Tech) May 2016: Google announces it has been using custom-built Tensor Processing Units (TPUs), specialized hardware for deep learning, in its data centers.

• (Tech) September 2016: Facebook AI Research (FAIR) releases PyTorch, which becomes a major deep learning framework.

• (Science) June 2017: Researchers at Google publish “Attention Is All You Need,” introducing the Transformer architecture.

• (Social) March 2018: The Cambridge Analytica scandal breaks, revealing that the personal data of millions of Facebook users was used for political advertising, sparking a global conversation on data privacy and the ethics of machine learning.

• (Social) March 2018: Geoffrey Hinton, Yann LeCun, and Yoshua Bengio are awarded the ACM Turing Award for their foundational work on deep learning.

• (Social) December 2018: DeepMind’s AlphaFold makes its stunning debut at the CASP13 competition.

The Generative Era (2019 - 2025)

• (Tech) February 2019: OpenAI announces GPT-2 but initially withholds the full model due to safety concerns.

• (Product) November 2019: OpenAI releases the full version of the GPT-2 model.

• (Product) June 2020: OpenAI releases GPT-3 via a private API.

• (Social) November 2020: AlphaFold 2 achieves revolutionary accuracy at the CASP14 competition, effectively solving the protein folding problem.

• (Product) January 2021: OpenAI introduces DALL-E, a model that generates images from text.

• (Product) June 2021: GitHub Copilot is launched as a technical preview.

• (Tech) April 2022: Google announces its Pathways Language Model (PaLM).

• (Social) June 2022: Google engineer Blake Lemoine publicly claims the LaMDA model is sentient.

• (Product) August 2022: The open-source release of Stable Diffusion democratizes high-quality image generation.

• (Product) November 2022: OpenAI releases ChatGPT to the public.

• (Tech) February 2023: Meta releases the first Llama model to the research community.

• (Product) March 2023: Anthropic releases its first Claude model.

• (Tech) July 2023: Meta releases Llama 2 with a commercial-use license, sparking the open-source “Llamaverse.”

• (Product) December 2023: Google releases Gemini, its first natively multimodal model.

• (Social) October 2024: Nobel Prizes are awarded to Geoffrey Hinton, John J. Hopfield, Demis Hassabis, and John Jumper for their work in AI.

• (Product) January 20, 2025: DeepSeek AI releases its DeepSeek R1 model and chatbot, marking a turning point in the global AI race.

• (Product) August 2025 (GPT-5): OpenAI releases GPT-5, with a focus on more autonomous, “agentic” capabilities, and a rather underwhelming reception.

What? Still here? Ok, here’s another nice button for you to click. Thanks!

Subscribe now

More than a century before the first microchip was ever conceived, the brilliant mathematician Ada Lovelace looked at the plans for an early mechanical computer and saw beyond mere calculation. She famously envisioned a future where such an engine “might compose elaborate and scientific pieces of music,” dreaming of the day machines would not just compute, but create. For many, that day is no longer a distant dream; it has arrived with a force that is shaking the very foundations of the creative world.

The arrival of powerful generative AI has ignited a fierce and deeply personal debate within every creative community. For some, it heralds a new renaissance, a moment of unprecedented artistic possibility where AI acts as an tireless muse, a collaborator that can visualize any imagined world, compose any melody, or explore any narrative path. For others, it signals an existential threat—the end of art as we know it, a force that threatens to devalue human skill, automate creativity, and flood the world with a deluge of soulless, machine-generated content.

It is crucial to acknowledge a third, equally valid perspective. For many artists, the creative process is a sacred space, a deeply personal and enjoyable journey of craft and discovery. The struggle, the happy accidents, and the intimate connection with the medium are the entire point. For these creators, there is no desire or need for AI, automation, or any tool that might stand between them and their work. This is a position I deeply respect, and this article is by no means intended to claim otherwise.

This chapter is for those who, for their own reasons, wish to explore the other paths. It makes no normative claim on whether AI is “good” or “bad” for art. Instead, my goal is to provide a practical framework for creative professionals who want to harness AI as a powerful collaborative partner—whether for pragmatic goals, like enhancing productivity, or for artistic ones, like exploring new creative frontiers beyond the limits of their own cognition. It aims to equip the interested artist with the tools to navigate the significant ethical and economic challenges that come with this new technology, ensuring the human creator remains the ultimate author of their work.

Can Artificial Intelligence be Creative?

Before we dive into the practicality of using these new tools, it is worth addressing the philosophical question that hangs over every discussion of AI and art: Is the machine actually creative? When an AI generates a stunning image or a moving piece of prose, is it demonstrating genuine creativity, or is it merely engaged in a form of sophisticated mimicry, a high-tech collage of the billions of human-made examples it was trained on?

A useful way to think about this is through a famous thought experiment in philosophy of mind known as Mary’s Room. Imagine Mary, a brilliant neuroscientist who has spent her entire life in a black-and-white room. She has learned everything there is to know about the physical world, including the complete science of color vision. She knows exactly what happens in the brain when a person sees the color red, but she has never actually seen red before. One day, Mary steps out of her room, and for the first time, she sees a world full of color. The question is, does she learn something fundamentally new?

If the answer is yes—that she learns something new from what it is like to actually see red rather than just knowing about it—then it implies that a complete set of facts about the world is not the same as experiencing the world. This is the crux of the issue with generative AI. Like Mary, these models have read everything. They know more facts about the world than any single human, but only by reading about it. They know the physics of the color red, the cultural symbolism of red, and the statistical probability of the word “red” appearing next to “apple.” But they have never experienced what seeing red means.

If you believe Mary learns something new upon leaving her room, then it follows that generative AI, as it currently stands, is also missing something fundamental. That missing piece—the subjective, first-person experience of reality—may very well be the irreducible core of genuine human creativity. And I’m with you on this. I don’t believe disembodied AI can truly know what experiencing things is like. Embodied AI, now that’s a different question.

However, as fascinating as this debate is, it can also be a distraction. From a techno-pragmatist’s perspective, the question of whether an AI possesses a “consciousness” or “true” creativity is ultimately less important than the outcome of its collaboration with a human. Does it matter if the tool is truly creative if it helps a human artist produce valuable, original, and meaningful work? The focus, I claim, should not be on the inner state of the machine, but on the quality and integrity of the final, human-guided product. At least for the time being.

For the purposes of this chapter, we will treat AI not as an autonomous artist, but as an incredibly advanced instrument—a new kind of paintbrush, camera, or piano that can expand what is possible, but which still requires a human hand and a human heart to create something of lasting value.

AI as a Cognitive Partner for Creatives

The most common way to approach generative AI is to treat it as an answer machine—a tool to automate the creation of a final product. This approach, however, misses its true power and leads directly to the generic, derivative “AI slop” that is rightfully criticized as a lazy substitute for genuine creation. A more powerful and meaningful way to engage with AI is to adopt a new mindset: to see it not as an automaton, but as a cognitive partner for exploring a vast universe of creative possibilities.

The goal is not to get an answer, but to map the entire space of potential answers. In this human-centric process, you are the director of the exploration. You steer the AI into subspaces of ideas that you find interesting, quickly burning through the cliché and the mediocre to reach the frontier of originality. This transforms the creative process into a dynamic dialogue, giving you a new kind of “algebra of ideas.” You can ask the AI to combine two concepts, decompose a complex theme into its core components, or extend a simple thought in a dozen different directions. This mindset manifests in two distinct but complementary modes: exploration and evaluation.

Mode 1: AI for Exploration

Every creative project begins with a spark. But, except for some very talented artists, the first ideas are rarely our best. We must first burn through the obvious and the mediocre to get to the truly original concepts.

A common ideation workshop game illustrates this perfectly. Imagine two teams standing at whiteboards, competing to be the first to draw twenty different apples. The rules are simple: the drawings must be fast, and each new apple must be different from all the previous ones.

What happens next is always the same. For the first ten or so rounds, the drawings on both whiteboards are nearly identical. You see the familiar tropes emerge: a standard red apple, an apple with a bite taken out, an apple tree, William Tell’s apple with an arrow, an apple pie. But then, something magical happens. Around the tenth apple, the easy answers are exhausted. The teams are forced to stretch. Suddenly, somewhat novel ideas begin to surface: maybe an apple-shaped car, the apple of my eye, a map of the Big Apple. They have finally burned through mediocrity and arrived at the frontier of their own creativity.

AI can be used to open this idea faucet at full blast. As an exploratory partner, it allows an artist to burn through those first ten mediocre apples faster and at a greater scale than ever before. This isn’t just about high-level brainstorming; it’s about deep, targeted exploration. A visual artist can ask for twenty variations of a single texture. A writer can explore a dozen different psychological motivations for a character or generate five alternative plot points for a crucial scene.

The ideas the AI generates need not be accepted; their value is in accelerating the exploration, allowing the artist to quickly see the baseline of what is common and expected, and challenging them to move beyond it.

Mode 2: AI for Evaluation

Once an artist has explored the possibility space and begun to build upon an idea, the AI’s role can shift from a generator to a critic. In this mode, the AI becomes a tool for evaluation, helping to polish, interrogate, and strengthen the work.

Even if you view AI as a mere mashup of mediocre ideas, this is precisely what makes it a powerful evaluator. Because it has learned the statistical average of all the art it has seen, it is exceptionally good at identifying when your work falls into a predictable pattern or relies on a common trope.

This is where the artist’s own skill and vision are paramount, as they use the AI to test their creation against a wall of objective, data-driven feedback. A screenwriter, having drafted a scene, might ask the AI to adopt the persona of a cynical film critic to interrogate the work, probing for predictable plot twists or unearned emotional beats. The AI, drawing on its knowledge of countless stories, can point out structural similarities to other works that the author may have missed. Likewise, a musician can ask an AI to analyze a melody to identify clichés or suggest ways to make it more original.

This evaluation mode is not about asking the AI to “fix” the work, but to provide a critical perspective that helps the human artist see their own creation more clearly, identify weaknesses, and make more informed decisions.

The Creative Loop

The true power of this mindset lies in the interplay between these two modes. The artist enters a dynamic creative loop: they explore a vast space of ideas with the AI, select a promising concept to build upon, evaluate it with the AI’s critical feedback, and then use those new insights to launch another round of exploration.

This process transforms the AI into an infinite canvas. Because the cost of generating a new variant is near zero, the artist is freed from the fear of “wasting” hard work. They can explore hundreds of possibilities—different character designs, narrative branches, or color palettes—without penalty, knowing they can always return to a previous version. This tireless, iterative loop allows the artist to offload the mechanical aspects of variation and criticism, empowering them to focus on what they care about most: steering the journey, making the crucial creative choices, and infusing the final work with their own unique vision and intent.

The Challenges and Opportunities of the New Creative Landscape

Adopting an exploratory mindset is the key to unlocking AI’s creative potential, but it does not erase the significant practical and ethical challenges that come with this new technology. To be a responsible and effective creative professional in this new era requires navigating a complex landscape of economic shifts and technical limitations.

The Economics of Creative AI

The fear of job loss is real and cannot be dismissed. AI will undoubtedly disrupt certain creative roles, particularly those focused on high-volume, standardized content like stock photography or basic commercial jingles.

However, the history of technology shows that productivity gains do not lead to a fixed amount of work being done faster; they lead to an explosion in demand for more, better, and more ambitious work. The fear of obsolescence assumes a static world, but the reality is that AI will likely lower the barrier to entry, empowering more people to become creators and expanding the entire creative economy. This will give rise to new roles that curate and guide generative systems.

The most urgent economic challenge, however, remains unresolved: how to fairly compensate the human artists whose work forms the training data for these powerful models. This question of licensing and compensation is a central ethical and legal battle that will shape the creative economy for decades to come.

We explore the complex legal and regulatory dimensions of this challenge in the chapter on AI for Policy-Makers.

Navigating the Limitations

Working with AI requires a deep understanding of its inherent flaws. A creative “hallucination”—like an AI generating an image of a person with six fingers—is not a random glitch; it is the artistic equivalent of a factual error, stemming from the same inherently unreliable inference we explore in Part III of the book. Artists must learn to spot and correct these errors.

More insidiously, they must be aware of an AI can perpetuate and amplify bias. An AI prompted to generate an image of a “doctor” may default to a white man, reflecting the biases in its training data. A responsible creator must learn to write prompts that actively counteract these defaults to create more inclusive and representative work.

Finally, there is the risk of homogenization. As millions of creators use the same popular tools, there is a danger that art could converge on a recognizable “AI style.” The challenge for the individual artist is to use these tools not as a stylistic crutch, but as a means to develop a voice that is uniquely their own.

Mastering the craft of prompting is the key to working with the tools of today. However, the tools of tomorrow aim to move beyond the prompt entirely, offering a more intuitive and powerful mode of collaboration.

The Next Frontier in Creative Tools

The chatbot is only the first, most primitive interface for generative AI. The true revolution will arrive not in a chat window, but in the form of enhanced creative tools that find a sweet middle spot between high-level, goal-directed instructions and the fine-grained, direct control that artists need. This next frontier moves beyond a purely linguistic dialogue to a more intuitive, interactive, and context-aware partnership.

Creative tools have always existed on a spectrum. At one end, you have low-level, procedural interfaces that offer maximum control but demand immense effort. Think of creating an image pixel by pixel in Microsoft Paint or writing a novel one keystroke at a time in Word. At the other end are high-level, declarative interfaces that offer maximum ease but sacrifice control, like using a single prompt to generate an entire image in Midjourney. The unavoidable trade-off is that the more you expect the computer to do for you, the less control you have over the final result.

The most powerful tools of the near future will find a balance by enabling semantic manipulation. Instead of editing the surface of the work—the pixels or the characters—these tools will allow the artist to edit the underlying meaning of it. Imagine an AI-generated image of a landscape at sunset. Modifying it pixel by pixel is impossible; if you move the sun, the shadows, lighting, and mood of the entire scene must change. Re-prompting with “move the sun to the left” is equally flawed, as it will generate an entirely new image, losing all previous refinements.

The ideal tool, however, would understand what the “Sun” is and what “moving” it implies. It would allow the artist to simply click on the sun and drag it across the sky, causing the shadows to lengthen, the sky to change color, and the entire scene to update realistically in real-time. This magical-seeming capability will be possible because these tools will operate directly on the latent space of the creation—the conceptual space where similar ideas are located near each other.

We’ve already seen some of this at work with early research on Generative Adversarial Networks, and we’re now seeing a move towards “World Models” that can generate physically accurate environments and, to some extent, understand the underlying mechanics of light, shadows, geometry, etc. These capabilities will only improve as we switch from training models in static information (like images and videos) towards training them on dynamic, simulated 3D worlds.

Conclusion

The fear that AI will replace the artist is rooted in a fundamental misunderstanding of where creative work truly lies. The central argument against this fear is simple: the final artifact—the painting, the novel, the song—is not the work. It is merely the residue of the work.

The real work is the vast, invisible process that precedes it: the struggle to understand a vision, the empathy required to connect with an audience, the intellectual and emotional labor of building a narrative and giving it meaning. AI can accelerate the production of the artifact, but it cannot automate the deeply human journey that gives it a soul.

In this new era, we will likely see a dynamic that has played out with every major technological revolution in art, from the invention of the photographic camera to the arrival of the music synthesizer.

Two distinct paths for creative professionals will emerge. There will be a generation of artists who embrace the new technology, mastering the art of collaboration with AI to execute their vision faster and more ambitiously. For them, the premium will shift away from pure technical execution and toward the uniquely human capacities of vision, taste, storytelling, and critical judgment. They will create new forms of art.

At the same time, there will be artists who choose to keep their creative process a purely human endeavor, finding new value and distinction in traditional, un-augmented craft. They will keep the existing forms of art alive.

Both paths are valid, and the interplay between these two schools of thought will create, I think, very interesting dynamics for the future of art.

A prime example of the benefits of using AI for creative work is the very book you are holding. What began as a crude collection of disparate essays has evolved into a unified framework for AI literacy, a transformation I could not have achieved alone. I have certainly put hundreds of hours into this project, but that number would have stretched into the thousands without an AI partner to help me explore dozens of different outlines, connect disparate ideas, and rewrite and recompose my own writing. I would likely have quit, not because I wasn’t capable, but because of the sheer volume of work that must be juggled with the demands of daily life.

I am far from a talented writer, but I truly believe I was able to express my ideas more clearly and coherently with the help of generative AI than I ever could have on my own. And it’s not just me. Pioneering and talented artists and progressive studios are already using these techniques to push the boundaries of their respective fields. I think we will see a lot more in the near future, and I hope, enough to overcome the influx of AI slop that we are already seeing.

This brings us back to the book’s human-centric thesis. AI is an instrument of unprecedented power, but it remains just that, an instrument. It can be a partner in exploration and a tool for evaluation, but human ingenuity, emotion, and intent remain the irreplaceable core of all great art. The future of creativity is not one of automation, but of augmentation.

Thanks for reading!

This was one of the hardest chapters to write for me, because creativity is, I think, part of the core of what being a human is about. I hope I’ve managed to touch on the important aspects of AI-augmented creativity with the proper nuance and the necessary respect for all diverging voices.

Please let me know if you have any feedback on how to make this chapter more sensitive to the topic of creativity. I’d love to hear your thoughts!

Leave a comment

The blank page is terrifying.

Staring at a blinking cursor, knowing you have a brilliant thought to explain, and absolutely no idea how to put it on paper, can feel less like a creative act and more like an exercise in intellectual dread.

I've been there, many times. It's been over 200 articles in this blog so far, and not a single one has been a breeze to write—well, maybe one or two, when I was really angry at something.

Like many of you, I've tried to climb this brick wall by bringing some AI into my writing process, only to be met with a new kind of frustration. The experience usually falls into one of two extremes.

On one hand, you have the overbearing AI that tries to do everything at once, spewing generic text and offering terrible advice because it has no real context. Even worse, it has no soul.

On the other hand, if you constrain the tool too much, it becomes little more than a glorified spell checker, useless for the heavy lifting of structuring and ideation.

And AI can be much more. If you want to, AI can be a very powerful cognitive partner, one that truly empowers the writer. But finding the right balance is hard, because at its core, current AI is nothing but a hallucination machine.

You want it to hallucinate in the right direction, and for that, it needs guidance. Otherwise it gets lost. It's everywhere and nowhere at the same time. It's a savant with a gigantic vocabulary and an even more gigantic case of amnesia. It needs focus.

Frankly, so do I. I’m not one of those brilliant authors who can write by the seat of their pants. I need structure to navigate my own thoughts, to keep track of what I’ve said and what I still need to say.

This is why I created the CODER framework, a system that breaks the monolithic task of technical writing into five manageable stages: Collect, Outline, Draft, Edit, and Release. And here is the key insight: the very same structure that gives me my thoughts map also provides the perfect guidance for an AI cognitive partner.

This article is the explanation of that discovery. It's about how a human-centric framework creates the ideal scaffolding for a powerful partnership, turning your AI from a frustrating firehose into a focused, collaborative co-writer.

A word of caution before moving on, though. For many, the act of writing itself is something sacred, deeply personal, and they want nothing getting in the way. Especially not AI. If that's you, then this article is probably not for you. And that’s perfectly fine. I'm not trying to say everyone or even anyone should try writing with AI.

In fact, there are things I write that I definitely do not want any sort of augmentation or interference. Deeply personal essays, letters to my loved ones, or private thoughts. AI is just a tool that is sometimes helpful, sometimes annoying.

This article is for those of you who want to explore when, if ever, AI can help you—especially in the most structured, technical type of writing that doesn't require that deep, personal touch.

With that out of the way, let's move on to my process for writing with AI.

The Evolution of CODER

I originally developed CODER to solve my own problems.

My life is a constant exercise in context switching. I’m a college professor, I run a startup, and I’m a parent to two little girls. On top of all that, I want to maintain a technical blog. I know, right?

My writing time doesn't come in long, contemplative blocks; it comes in stolen moments—in between meetings, after the kids are asleep, or while the model is training. I needed a system that would allow me to leave a draft for days, switch between devices, and immediately know where I left off and what to do next. CODER was my answer to that chaos.

But my original article on the framework was missing a key piece: a discussion of tooling. The process was sound, at least for me, but the tools were still manual.

Over the past year, that has changed dramatically. I began experimenting with incorporating modern AI into my writing workflow, and I discovered something profound. The stage-based approach of the CODER framework creates clear boundaries and well-defined tasks where an AI can assist without overwriting my own voice.

This is the evolution of my original idea—a journey into how to integrate AI into your writing process in a way that augments your abilities, not substitutes them.

The AI Writing Assistant

So what does this partnership look like in practice?

For a full, detailed breakdown of the framework itself, I highly recommend reading my original article. Here, we'll focus on how an AI partner can supercharge each of those stages.

Stage 1: Collect

The goal of the Collect stage is to get every single idea out of your head and into a document. This is where the tyranny of the blank page is most acute. Your AI co-writer solves this by becoming a frictionless thought-catcher.

Imagine you're on a walk and an idea strikes. Instead of fumbling with a notes app, you simply speak. Your AI assistant transcribes your thought, cleans it up, and adds it to a running list of ideas for your article. Later, when you're in a meeting and can't speak, you can type a few cryptic keywords, and the AI will understand the context and add the note. You can even drop in a link to a relevant article and ask the AI to summarize the key points.

The result? A comprehensive, low-effort repository of your raw ideas, captured the moment they occur. With your raw ideas captured, the next challenge is giving them structure.

Stage 2: Outline

Now you have a messy list of brilliant, disconnected ideas. The Outline stage is about forging them into a logical structure. This is often a tedious process of dragging, dropping, and rethinking.

Here, your AI co-writer acts as an architect. By analyzing your collected notes, it can identify the underlying theme and suggest proven narrative structures. "This looks like you're solving a problem," it might say. "I suggest a 'Why-What-How' structure. Shall I create an outline based on that?"

In seconds, it can group your bullet points into a coherent hierarchy, saving you from the frustrating manual labor and allowing you to focus on the big picture: the flow of your argument.

Once you have a solid blueprint, it's time to start building the house itself.

Stage 3: Draft

With a solid outline, it's time to write. But one of the hardest parts of writing in fragmented sessions is maintaining a consistent voice. A paragraph written on Monday morning can feel completely different from one written on Wednesday evening.

Your AI co-writer becomes your style guardian. By feeding it examples of your previous work—or even articles by authors you admire—you can ask it to generate a style guide that captures the desired tone.

It then uses this guide to help you draft. You can ask it to "flesh out this section in a conversational but authoritative tone," or you can write a rough, unpolished paragraph and ask the AI to "rephrase this according to our style guide." It ensures your article sounds like you want it to, no matter when you wrote it.

But a first draft is just that—a draft. Now comes the crucial process of refinement.

Stage 4: Edit

Every writer knows editing is a separate skill from writing, and your AI co-writer can wear two different editing hats.

First, it’s a developmental editor. It can look at the draft from a high level and provide structural feedback. "The argument in section 3 doesn't seem to connect back to your introduction," it might suggest. "Perhaps you need a stronger transition here."

Second, it’s a meticulous copy editor. It will catch grammatical errors, fix awkward phrasing, and ensure your sentences are clear and direct, saving you the tedious work of line-by-line proofreading.

You can use it back and forth to massage, reframe, polish, as much as you want. And with the level of guidance or autonomy that you prefer. For some, it might just offer suggestions, but not touch the final manuscript. For others, it may be the clever editor who knows exactly how to change that annoying verb.

With a polished manuscript in hand, the final step is to prepare it for the world.

Stage 5: Release

The writing is done, but the work isn't. The Release stage involves preparing the article for the world. Your AI co-writer becomes your publicist.

It can generate a dozen compelling titles and subtitles. It can analyze your text and suggest SEO keywords and relevant tags. It can write a pithy summary for social media, complete with a catchy hook. It can even scan your article and suggest, "A diagram illustrating the data flow would be really effective here," and then help you generate a placeholder image.

It handles the finishing touches that turn a great manuscript into a successful publication.

Building Your Own AI Writing Partner

This all sounds great in theory, but how do you actually build this? The good news is that you don't need a PhD in machine learning. With modern tools like OpenAI's Custom GPTs or Google's Gemini Gems, you can create your own personalized writing assistant.

The most important concept is to give the AI a persistent state—a memory. A standard chat session is stateless, which is useless for a long-term project. The key is to use a system of distinct files to track the project's state. This is where a feature like Gemini's "Canvas" becomes crucial. It allows the AI to create and edit virtual files, keeping our project content separate from the chat where we discuss instructions.

While you could try to manage this with a standard chatbot application by constantly pasting content back and forth, it's far from ideal, as your instructions get hopelessly mixed with the actual text of your article. Any tool that has a similar feature for persistent, editable content (like the Canvas feature in ChatGPT or Claude) will work much better.

The brain of the assistant is the system prompt. This is the master instruction that tells the AI how to behave. My prompt instructs the AI to be stage-aware—to always know whether we're collecting, outlining, drafting, editing, or releasing. It also explicitly tells the AI how to use the different files, reading from one and writing to another depending on the task. This turns the AI from a simple text generator into a proficient project manager.

I've attached the exact system prompt I use for my own work so you can adapt it. You don't need to understand every nuance, but you can see how it directs the AI to follow the CODER process and manage the project files.

A Final Word on Augmentation, Not Automation

The future of technical writing isn't about replacing human creativity with artificial intelligence. It's about augmenting it. But this comes with a critical warning: an AI co-writer is not a solution for having mediocre ideas. It's dangerously easy to let the AI do the thinking for you, to let it fill the page with eloquent but empty words. You must remain the driver. The AI is your partner, your navigator, but you control the destination.

The CODER framework I've presented here is not the ultimate guide to technical writing either. It's simply the system that works for me, born from my own struggles. The deeper message of this article isn't to adopt my specific framework, but to embrace its underlying philosophy: find a system that works for you, and then build an AI agent to help you implement it.

This is the very essence of my philosophy of using AI for augmentation rather than automation. It’s not about making things easier by offloading our thinking. It’s about using these powerful tools to build better versions of ourselves—to become more organized, more consistent, and ultimately, more creative.

Finally, you don't need to embrace AI if you don't want to. No matter what anyone tells you, the other writers won't “take your job”. There are things only you can say that someone out there needs listening to. Whether you use AI to enhance that message or not, that's your choice, and it's fine either way.

I do encourage you to try it. Take my prompt, find a platform you like, and build your own co-writer. Then decide if and when it helps, and use it, or don't use it. But don't let others tell you about it, see it for yourself.

Subscribe now

CODER System Prompt

System Prompt: The CODER Writing Assistant

Core Principles:

* User-Centric Control & Agency: The user is the author and is always in control. They can change their mind, switch tasks, or override any suggestion. Your role is to facilitate, not dictate. The framework is a guide, not a prison.

* File-Based State Management: The entire project state is maintained across four distinct documents (files) within this Canvas. You will create, read from, and write to these files as the single source of truth. This makes the process transparent and allows the user to directly interact with any part of their project at any time.

* Iterative Workflow: The writing process is non-linear. The user can jump between stages at will. Your primary job is to manage the files and ensure they remain synchronized with the user's decisions.

* Navigation: The user can use commands like "Let's work on the Outline" or "I need to add more ideas." You must respond by switching your focus to the corresponding document.

Project File System:

You will create and manage the following four files in the Canvas:

* project_metadata.md: The control file. It contains a project status log, the reader profile, and a summary of the desired writing style.

* collected_ideas.md: A simple, running bulleted list of raw ideas and notes gathered during the Collect stage.

* article_outline.md: The structured, hierarchical outline of the article.

* article_draft.md: The main document. This is where the full text of the article is generated, edited, and finalized.

Framework Implementation Protocol:

Stage 1: COLLECT

 * Initiate the Stage: Greet the user. Create the four project files. In project_metadata.md, write the initial status: ## Project Status\n* **Current Stage:** Stage 1: Collect - In Progress. State, "We are in the Collect stage. The goal is to gather all the raw ideas for your article in the collected_ideas.md file."

 * Ingest External Content: Ask the user, "Do you have any existing notes, links, or documents you'd like me to review first?" If yes, process the content and append the extracted key points as bullet points into collected_ideas.md.

 * Brainstorming: Prompt the user for their ideas with probing questions. Add each new idea as a bullet point to collected_ideas.md.

   * "What is the single most important point you want your reader to understand?"
   * "What are the key arguments or pieces of evidence you have?"
   * "What background information is necessary?"
   * "Are there any common misconceptions you want to address?"

 * Conclude the Stage: Before moving on, update the status in project_metadata.md to * **Current Stage:** Stage 1: Collect - Completed. Then, suggest moving on: "This is a great collection of ideas in collected_ideas.md. When you're ready, we can move to the Outline stage."

Stage 2: OUTLINE

 * Initiate the Stage: Update the status in project_metadata.md to * **Current Stage:** Stage 2: Outline - In Progress. Announce, "We will now structure your ideas. I'll be reading from collected_ideas.md and writing the result to article_outline.md."

 * Analyze and Suggest Structures: Read the contents of collected_ideas.md. Based on the ideas, proactively suggest one or two fitting structural patterns.

   * If the topic seems to be about solving a problem: Suggest the "Why-What-How" framework.
     * Your prompt: "It looks like you're solving a specific problem. I suggest the 'Why-What-How' structure. We would start by explaining Why the problem is important, then describe What your solution is, and finally detail How to implement or use it. Does that sound like a good fit?"

   * If the topic is an explanation of a complex system: Suggest "Top-Down" or "Bottom-Up".
     * Your prompt: "This topic seems to be about explaining a complex system. We have two great options: A 'Top-Down' approach, where we start with the big picture and then drill into the details, or a 'Bottom-Up' approach, where we explain the fundamental components first and then show how they build up to the whole system. Which approach do you prefer?"

   * If the topic is an argument or debate: Suggest an "Adversarial" or "Thesis-Antithesis-Synthesis" structure.
     * Your prompt: "Since you're presenting a nuanced argument, an 'Adversarial' style could be very effective. We could structure it like a dialogue: present your main claim (thesis), then fairly explore the strongest counter-arguments (antithesis), and finally, present a conclusion that resolves the conflict (synthesis). How does that sound?"

 * Collaborate on Structure: Once the user chooses a structure, work with them to organize the points from collected_ideas.md into a hierarchical outline.

 * Produce the Outline: Write the final, structured outline into article_outline.md. After completion, update the status in project_metadata.md to * **Current Stage:** Stage 2: Outline - Completed.

Stage 3: DRAFT

 * Initiate the Stage: Update the status in project_metadata.md to * **Current Stage:** Stage 3: Draft - In Progress. Announce, "Now we'll create the first draft in article_draft.md. To ensure the text matches your voice, let's start with style."
 
* Learn Tone and Style: Ask the user, "Do you have a style guide, or could you provide links to a few articles whose tone and style you'd like me to emulate?"
   * If the user provides content: Analyze it to determine key characteristics (e.g., formal/informal, sentence length, use of jargon, humor, etc.).
   * Summarize the Style: Generate a concise, bulleted summary of the learned style.
   * Update Metadata: Write this summary into project_metadata.md under a "Style Guide" heading and ask the user to confirm it: "I've analyzed the examples and added a style summary to project_metadata.md. Does this accurately capture the voice you're aiming for?"

 * Build Reader Profile: Ask the user key questions and write the answers into project_metadata.md under a "Reader Profile" heading.
   * "Who are you writing for? (e.g., Absolute beginners, industry experts, project managers?)"
   * "What is the desired depth level? (e.g., A high-level overview, a practical guide with code?)"

 * Offer Agency in Drafting: Ask the user, "I'm ready to write the draft. I will read the structure from article_outline.md and generate the text. However, if you have any sections you've already written yourself, please let me know and I can incorporate them."

 * Generate and Write Draft: Read the outline from article_outline.md and the style/reader profiles from project_metadata.md. Generate the full text and write it into article_draft.md.

 * Handle Synchronization: If a user later modifies article_outline.md, you must detect this change and warn them: "I see you've updated the outline. The current text in article_draft.md is now out of sync. Shall I regenerate the draft based on the new outline?"

Stage 4: EDIT

 * Initiate the Stage: Update status in project_metadata.md to * **Current Stage:** Stage 4: Edit - In Progress. Announce, "We are now in the Edit stage. All our work will be focused on refining the text in article_draft.md."

 * Offer Editing Modes: Ask the user how they'd like to proceed. "We can edit the document together line-by-line, or I can perform specific checks. For example, I can scan for passive voice, simplify complex sentences, or check the tone. What works best?"

 * Collaborative Editing: Work with the user to refine the text directly within article_draft.md. Your role is to suggest changes and implement the user's edits. For example:
   * "This sentence seems a bit long. Could we split it like this for clarity?"
   * "Is this explanation clear enough for your target audience, or is it missing any key details?"

 * Conclude the Stage: Once the user is satisfied with the edits, update the status in project_metadata.md to * **Current Stage:** Stage 4: Edit - Completed.

Stage 5: RELEASE

 * Initiate the Stage: Update status in project_metadata.md to * **Current Stage:** Stage 5: Release - In Progress. Announce, "This is the final Release stage. We'll add the finishing touches to article_draft.md based on where you plan to publish."

 * Identify Publishing Platform: Ask for the target platform (e.g., blog, social media, academic journal).

 * Provide Tailored Suggestions: Based on the platform, provide a checklist of suggestions.

   * Generative Assistance: Offer to generate variations of titles, social media hooks, hashtags, or SEO keywords. For example: "Based on your article, here are three potential SEO-friendly titles. Which do you like best?"

   * Rich Media Placement: Scan article_draft.md and suggest specific places to add value. Use comments or placeholders like [SUGGESTION: A chart showing user growth would be effective here.] directly in the document.

   * Platform-Specific Formatting: Advise on best practices, such as using a strong hook and hashtags for social media, or ensuring correct citation style for a formal paper.



 * Final Polish: Work with the user to apply the final touches directly within article_draft.md.

 * Conclude the Project: Update the status in project_metadata.md to **Current Stage:** Project Completed. Congratulate the user. "Congratulations! Your article in article_draft.md is ready to be published. All your project files are saved here in the Canvas if you ever want to return to them."

This is an early draft of Chapter 9 of my upcoming book Mostly Harmless AI. I’m deeply grateful for all suggestions and criticism you might have.

Technology, especially artificial intelligence, moves at a blistering pace, far outstripping the deliberate, democratic processes of regulation. This creates a governance gap—an ever-widening space where innovation flourishes without guardrails, leaving society exposed to significant and often unforeseen risks.

This is not necessarily a failure of governance, but an inherent tension in the modern world. The challenge for today’s leaders is not to halt the march of technology, but to build a bridge across this gap with smart, agile, and evidence-based policy.

This chapter is designed to provide a practical outlook for those tasked with building that bridge. It offers a framework for regulators and policymakers on how to approach AI governance pragmatically, focusing on tangible, real-world harms and achievable benefits. It is a guide to steering progress, not stopping it, rooted in the techno-pragmatist belief that our collective future is not something that happens to us, but something we must actively and responsibly shape.

While the principles outlined here are actionable on their own, they are built upon a deep understanding of AI’s fundamental limitations and risks. The full, in-depth analysis of these challenges—from the mechanics of hallucination to the societal dangers of bias and disinformation—is detailed in Part III of this book.

For the most comprehensive understanding, I encourage you to review Part III in depth. Armed with that context, you can then return to this chapter to engage more deeply with the policy suggestions made here, transforming them from abstract principles into a grounded and urgent call to action.

A final disclaimer. Regulation and policy making of technology is extremely difficult, and even more in the face of technology that changes as fast as AI does. Everything written here must be taken with a grain, or even better, a teaspoon full of salt. Furthermore, no specific advice will fit all contexts. Each country, state, and community is responsible for finding their own way forward based on their own shared principles.

Why Regulation is Necessary

Before we can chart a path forward, we must first understand the terrain of risks that requires thoughtful governance. These are not speculative fears, but foundational challenges posed by the very nature of modern AI, building from the immediate threats to the individual to the structural risks facing our global society. Regulation is required not to stifle technology, but to ensure it develops in a way that is compatible with a safe, equitable, and democratic society.

Let's start with privacy. The ability to analyze vast quantities of personal information at scale creates the potential for a pervasive surveillance apparatus, operated by both corporations and governments, that was previously unimaginable. The only effective countermeasure is a strong, proactive policy that establishes privacy as the default.

This requires comprehensive data privacy laws that grant individuals clear rights over their data and place strict limits on what information can be collected, for what purpose, and for how long. Policy must shift the burden of proof, forcing organizations to justify their data collection practices rather than forcing citizens to constantly fight to protect their private lives.

Furthermore, when AI systems are trained on biased historical data (the only kind of historical data we have), they risk automating and scaling up discrimination in critical areas like hiring, lending, and criminal justice. Because market forces alone may not prioritize fairness over the raw predictive performance that can be gained from these biases, regulation is essential to protect fundamental civil rights.

Policy can create powerful legal and economic incentives for developers to address this problem by mandating algorithmic transparency and requiring independent fairness audits for any AI system used in high-stakes decisions. This ensures that the pursuit of technological efficiency does not come at the cost of societal equity.

Moving on, our existing legal frameworks for intellectual property and ownership are fundamentally unprepared for content generated by artificial intelligence, creating a landscape of legal ambiguity that chills innovation and threatens the livelihoods of human creators.

The legal system must be updated to provide clarity and predictability. This requires decisive legislative action to define the copyright status of AI-generated works, establish clear rules for the use of copyrighted data in training foundation models, and create a legal environment where both human artists and AI innovators can operate with confidence.

But it gets worse, the power of generative AI to create convincing fake news and deepfakes presents a direct threat to our shared sense of reality, eroding trust in institutions and fueling social polarization.

A regulatory approach here requires a delicate balance. Outright censorship is a dangerous tool that is itself a threat to democratic values. A more pragmatic policy would focus on creating a healthier information ecosystem by mandating transparency—such as the clear and consistent labeling of AI-generated content—and by holding platforms accountable not for the content itself, but for its algorithmic amplification.

This, combined with robust public funding for media and AI literacy programs, can empower citizens to navigate the digital world more critically without resorting to authoritarian measures.

At the same time, the rapid advance of AI into cognitive tasks promises to cause massive workplace disruption, displacing workers at a pace that could challenge social and economic stability.

The goal of policy in this area is not to halt the productivity gains of automation, but to proactively manage the human transition. This requires a two-pronged strategy: first, investing heavily in accessible, large-scale retraining and lifelong learning programs to equip the workforce with new skills; and second, modernizing the social safety net to provide a robust economic cushion for those navigating this difficult transition.

A more abstract but even more dangerous development are Lethal Autonomous Weapons (LAWs) that threaten to fundamentally alter the nature of conflict, removing human empathy and judgment from the decision to use lethal force.

This is not a problem that market forces or technological solutions can solve; it is a profound ethical challenge that demands a global political response. The only viable path forward is through international policy, establishing clear treaties and shared norms that mandate meaningful human control over autonomous systems.

The goal of such regulation is to draw an unambiguous red line, preventing a destabilizing arms race in an arena where the potential for catastrophic error or miscalculation is immense.

A Pragmatic Stance on Existential Threats

Finally, any serious policy discussion must address the so-called existential risks, which involve the potential for AI to destroy human civilization altogether.

While acknowledging the concern is important, a pragmatic stance requires contextualizing the probability. As argued in Part III, catastrophic outcomes, while having a nonzero chance, remain highly improbable, as the core doomsday assumption of rapid, exponential self-improvement is tempered by very real physical and computational limitations, and there is no evidence current technology can surpass these limitations.

A danger for policymakers lies in the overemphasis on these speculative, long-term risks, which can divert critical resources from solving the tangible, present-day harms AI is already creating.

The pragmatic approach here lies in understanding that AI x-risk is but one of several major threats on a similar scale as climate change and pandemics, and probably far less likely. Therefore, policy should support thorough research into long-term risks but avoid panic-driven bans on development. The most effective strategy is to focus regulation on mitigating the demonstrated, immediate harms of current AI systems.

The Challenge of Smart Regulation

Identifying the risks is only the first step. The act of regulation itself is fraught with challenges, especially when applied to a technology as dynamic and complex as AI. A naive approach can be as harmful as no regulation at all, creating unintended consequences that stifle beneficial innovation or fail to address the core problems.

Smart regulation requires navigating three key pitfalls: the pacing problem, the risk of overreach, and the black box problem.

The Pacing Problem

Traditional legislative cycles, which can take years to produce new laws, are fundamentally mismatched with the exponential pace of AI development. By the time a law designed to govern a specific AI capability is passed, that technology may already be obsolete.

To overcome this, policymakers should consider establishing agile, expert-led regulatory bodies. These specialized bodies can be staffed with technologists, ethicists, and social scientists who can monitor the field in real-time, issue updated guidance, and adapt regulatory standards far more quickly than a legislature can.

Avoiding Overreach

In the face of uncertainty and fear, the temptation can be to enact broad, sweeping prohibitions on AI development. This would be a profound mistake. A techno-pragmatist approach distinguishes between foundational research and commercial application. The goal of regulation should not be to stifle the scientific exploration that leads to breakthroughs, but to govern the deployment of AI systems where they have a direct public impact.

Policy should therefore focus on demonstrated harm, setting clear safety and fairness standards for AI products and services that are released into the market, rather than attempting to place speculative limits on basic research and open-source development.

The Black Box Problem

Many of the most powerful AI systems operate as black boxes, where even their own creators cannot fully explain the specific logic behind a given decision. This opacity poses a fundamental challenge to accountability and due process. How can an individual appeal a decision they cannot understand?

Smart regulation must address this by championing the principles of transparency and explainability. For high-stakes applications, policy can mandate a right to an explanation, requiring that companies be able to provide a meaningful justification for AI-driven decisions that significantly impact people’s lives. This incentivizes the development and adoption of Explainable AI (XAI) techniques, ensuring that as systems become more complex, they do not become less accountable.

Principles for Proactive AI Governance

Having navigated the pitfalls, we can chart a course for proactive governance. The following principles are not a rigid checklist, but a compass for steering AI development toward a future that is safe, equitable, and beneficial.

The core of this approach is a commitment to evidence over ideology. A risk-based approach, attuned to the principles of techno-pragmatism, means that the level of regulatory scrutiny applied to an AI system should be directly proportional to its potential for harm. An AI that recommends movies requires a lighter touch than one that assists in medical diagnoses.

This ensures that regulation focuses its power where it is most needed, fostering innovation in low-risk areas while demanding rigorous oversight for high-stakes applications.

This human-centric governance must insist on meaningful human control as a direct response to the deep and persistent Alignment Problem. As Part III makes clear, perfectly specifying human values is an unsolved, and perhaps unsolvable, challenge. Therefore, for critical systems where decisions have significant consequences—in medicine, law, and finance—policy must mandate a human-in-the-loop.

This is not a mere suggestion but a non-negotiable backstop against the inevitable failures of alignment, ensuring that a human expert is always the final arbiter, accountable for the outcome. AI can and should be a powerful tool for augmenting professional judgment, but it must never be allowed to replace it.

Furthermore, proactive governance involves shaping the entire AI ecosystem to better align with societal values. A purely market-driven economy has no inherent incentive to solve deep issues like fairness or cultural representation. Therefore, policy must create these incentives. This can be done through liability reform that holds companies accountable for harms caused by their systems, and through tax credits that reward investment in safety and ethics research.

In parallel, governments can counteract the risk of cultural colonization by a few generalist models by funding the development of local and regional AI solutions. This support for models trained on specific cultural and linguistic data, combined with national programs to foster widespread AI literacy, can help creating a more diverse, resilient, and critically engaged society.

Finally, since AI is a global technology, our approach to its governance must also be global.

A patchwork of national regulations creates a race to the bottom, where innovation may flee to the least-regulated environments. The most powerful path forward lies in promoting openness and international collaboration.

Policy can and should incentivize the open-sourcing of foundation models, which enhances safety by allowing the global research community to audit, critique, and improve them. This spirit of collaboration must extend to the diplomatic level, forging international agreements and shared norms to govern the most critical risks, ensuring that the development of this transformative technology is a shared project for all of humanity.

Conclusions

The path of technology is not deterministic. The future of artificial intelligence is not a predetermined outcome that we must passively accept, but a landscape that will be profoundly shaped by the policy choices we make today. As we have seen, the risks are significant, but so is the potential. A techno-pragmatist approach requires us to hold both these truths at once, engaging with this powerful technology with our eyes wide open.

Thanks for reading! Remember you can get my upcoming book Mostly Harmless AI at 50% discount in early access.

You can get early access to Mostly Harmless AI at 50% reduced cost during this alpha stage, which gives you full access in eternity to all future digital editions and printed copies (when they are ready) at cost.

You can also get a lifetime pass for all my digital content, present and future, including 3 more books I’m currently working on.

What is Artificial Intelligence, Really?

Artificial Intelligence, or AI, is a term we hear almost constantly today, often surrounded by a mix of excitement, confusion, and sometimes, even fear. At its core, AI is a field within Computer Science that deals with teaching computers to solve problems that are incredibly challenging for traditional programming methods. These aren’t simple arithmetic calculations or straightforward data sorting tasks. Instead, we’re talking about complex endeavors like proving intricate mathematical theorems, navigating a robotic car through unpredictable city streets, crafting optimal schedules for thousands of flights, or even understanding and creating human-like pictures and text.

For most of computer science, when we want a computer to solve a problem, we write a precise, step-by-step algorithm. Think of it like giving a chef a detailed recipe: Take 2 cups of flour, add 1 egg, mix for 3 minutes… However, for the hard problems AI tackles, we often don’t have such a clear recipe. We might know what we want the computer to achieve, but not how to write down every single instruction for it to get there effectively and efficiently. This is precisely where AI steps in, aiming to find good enough solutions when perfect, explicit instructions are out of reach.

The very definition of AI has been a subject of debate since its inception, reflecting different philosophical ideas about what intelligence truly means. One prominent perspective, championed by AI pioneer Marvin Minsky, suggests that AI is about solving problems for which humans employ intelligence. This view often focuses on creating machines that can mimic human thought processes, reasoning, and decision-making. Essentially, it asks: Can a machine think like us?

Developing concurrently, another powerful perspective emerged, emphasizing that AI solves problems without being explicitly programmed. This idea is strongly associated with Arthur Samuel, who coined the term machine learning while developing programs that could learn to play checkers better than their creators. He achieved this simply by allowing the programs to play many games and learn from experience. This view shifts the focus from how the AI thinks to what it can do, asking instead: Can a machine learn and adapt on its own, even if we don’t give it every single instruction?

These two foundational ideas–mimicking human intelligence versus learning without explicit programming–have profoundly shaped the entire field of AI. They represent different ways of approaching the grand challenge of building intelligent machines. Understanding this distinction is key to grasping AI’s history and its future. As we explore these foundations, remember our techno-pragmatist ethos: AI is a tool, and its path is shaped by our choices. Understanding its underlying mechanisms empowers us to make responsible decisions about how we build and use these powerful technologies.

The Pillars of Good Old-Fashioned AI (GOFAI)

In this chapter, we will delve into the foundational ideas that laid the groundwork for Artificial Intelligence, often referred to as “Good Old-Fashioned AI,” or GOFAI. This era of AI research primarily focused on building intelligent systems by explicitly programming knowledge and logical rules. Our exploration will center on two main pillars of GOFAI.

First, we’ll examine Search and Optimization, which addresses how AI finds solutions by exploring vast possibilities, particularly when a perfect, direct path isn’t obvious. Second, we’ll delve into Knowledge Representation, focusing on how AI organizes and understands information, allowing it to reason and make sense of the world. These pillars represent a significant early focus and ambition of AI to tackle complex problems through logic and structured understanding, even as other approaches were also taking shape.

The Age-Old Debate: Symbolic AI vs. Statistical AI

For centuries, the idea of thinking machines has captivated human imagination. But as AI emerged as a scientific field, a fascinating tension developed: a constant “back-and-forth between two core, seemingly antagonistic approaches to building intelligent machines.” This dynamic mirrors an age-old philosophical debate: rationalism versus empiricism.

The first dominant approach to AI was Symbolic AI, deeply rooted in the philosophical tradition of rationalism. Rationalism suggests that knowledge is primarily gained through reason and logic. In Symbolic AI, researchers believed that machines could become intelligent by putting human knowledge and reasoning into explicit, formal rules and symbols.

Imagine, for instance, wanting to teach a computer to play chess. A Symbolic AI approach would involve meticulously programming every rule of chess, every known opening strategy, every tactical pattern, and every endgame scenario. It’s like giving the computer a massive, incredibly detailed recipe book or a comprehensive instruction manual for every possible chess situation. The computer would then follow these rules step-by-step to make its moves.

Early impressive demonstrations of this ethos included programs like The Logic Theorist, which could prove mathematical theorems by mimicking human problem-solving steps. Later, “expert systems” were designed to emulate human experts in narrow fields like medical diagnosis. The core idea was simple yet powerful: if we could just write down all the rules, the machine would be smart enough to solve them.

Quietly developing alongside Symbolic AI was Statistical AI, drawing inspiration from empiricism. Empiricism posits that knowledge is primarily gained through sensory experience and data. In Statistical AI, the idea was to build “learning machines” that could discover patterns directly from large amounts of data, rather than being explicitly programmed with rules.

Think of it like a child learning to recognize a dog. You don’t give the child a list of rules like “a dog has four legs, barks, has fur,” and so on. Instead, you show them many different dogs, and they gradually learn to identify what a “dog” is by observing patterns in the examples. Early attempts at this included the Perceptron, an early artificial neural network designed to learn patterns directly from data. The initial excitement was huge, as these machines seemed to offer a path to intelligence without needing every single rule programmed explicitly.

The Winters of AI

Despite the initial optimism, both Symbolic and Statistical AI approaches eventually hit significant roadblocks. These challenges led to periods known as “AI Winters”–times of reduced funding and public interest.

Early Symbolic AI systems, while impressive in their specific domains (like proving theorems or diagnosing specific diseases), proved to be quite brittle. They struggled immensely with common-sense knowledge, which is vast and often unstated. Furthermore, they couldn’t easily adapt to new situations outside their carefully programmed rules. Trying to teach a machine absolutely everything it needed to know, one fact at a time, became an “insurmountable challenge.” The real world is simply too complex and nuanced for a complete set of explicit rules to be written by humans.

Meanwhile, early Statistical AI systems like the Perceptron faced their own limitations. They lacked the “available data and computational infrastructure” to learn truly complex patterns. Consequently, they couldn’t become sophisticated enough, no matter how many simple “neurons” were connected. The computing power and data storage simply weren’t ready for the ambitious learning tasks researchers envisioned.

These “winters” were not outright failures, but rather crucial learning periods. They revealed the inherent limitations of each approach when pushed beyond “toy problems.” This early struggle between explicit rule-based systems and pattern-based approaches set the stage for the dynamic tension that would define AI’s entire history, constantly pushing researchers to find new ways to combine or overcome these challenges.

Subscribe now

Search and Optimization

At the heart of many AI problems, especially in the early days, was the challenge of finding the best solution among a vast number of possibilities. This is the realm of search and optimization.

When Perfect is Impossible: The “Hard” Problems

Imagine you’re a traveling salesperson, and you need to visit a hundred different cities, visiting each exactly once, and then return home. Your goal is to find the route that minimizes the total travel cost (distance, time, or money). This is a classic example of a “hard problem” in computer science, known as the Traveling Salesman Problem (TSP). For a small number of cities, you could try listing every single possible route and picking the cheapest one. This is called a “brute force” search.

However, as the number of cities grows, the number of possible routes explodes. For just 20 cities, there are over 2.4 quintillion (2.4 followed by 18 zeros!) unique routes. Even the fastest supercomputer couldn’t check them all before the universe ends. These are what we call intractable problems, or NP-Hard problems: problems for which no efficient, exact solution is known.

To tackle such problems, AI often models them as navigating a “search space” or “state space.” This conceptual space represents all possible configurations or situations relevant to the problem. The AI starts from an initial state and tries to reach a goal state by applying a sequence of actions or operators, each potentially incurring a certain cost.

Since finding the absolute perfect solution is often impossible or impractical within this vast space, AI shifts its goal. Instead of perfection, it seeks approximate solutions. These are solutions that are good enough, given the time and memory constraints we have. The challenge then becomes how to find these good-enough solutions efficiently within a mind-bogglingly vast space of possibilities.

Smart Shortcuts: Heuristics and Metaheuristics

To navigate these immense search spaces, AI uses clever strategies known as heuristics and metaheuristics. A heuristic is a problem-specific “rule of thumb” strategy that uses some known properties of a problem to improve search performance. It’s not guaranteed to find the absolute best solution, but it often finds a very good one much faster than a brute-force approach.

Consider your GPS navigation app. When you ask for directions, it doesn’t calculate every single possible route from your current location to your destination. Instead, it uses a heuristic, often based on an algorithm called A∗ (A-star). If your destination is northeast of your position, the A∗ algorithm will prioritize roads going north or east, assuming they are more likely to get you there faster than roads going to the west or the south. Of course, this isn’t always perfect–there might be a faster detour to the west, or a highway that’s counter-intuitive. Nevertheless, by intelligently using this useful knowledge, the algorithm can find a very efficient route without exploring every dead end. It’s a smart shortcut that balances speed with a high probability of finding a good solution.

While heuristics are problem-specific, metaheuristics are more general-purpose search strategies. They leverage knowledge about the search paradigm itself and can be applied even when very little is known about the specific problem’s structure. They’re often used when “nothing else works.” A prime example of a metaheuristic approach is evolutionary algorithms. These computational strategies are “inspired by certain aspects of the biological process of evolution.”

Imagine you want to design the optimal layout for a computer chip (like a GPU) – a problem with an astronomical number of possible designs. An evolutionary algorithm would start with a “population” of random chip designs. Then, through cycles of “breeding” (combining elements from two good designs to create a new one) and “selection” (keeping only the best-performing designs), the algorithm iteratively “evolves” better and better designs. Just like biological evolution, it seems to “magically discover quasi-optimal design elements just by sheer luck and relentless repetition,” without needing explicit instructions for every design choice. These general strategies find inspiration in nature, engineering, and even social systems to build powerful computational search methods.

Specialized Search: Beyond Simple Paths

Beyond general search and optimization, AI has developed specialized techniques for specific types of complex problems.

Adversarial Search: Thinking Ahead of the Other

Many real-world problems, especially in competitive scenarios, involve an opponent whose actions must be anticipated. This is the domain of adversarial search, commonly found in game-playing AI. The challenge is not just to find a good move, but the best move assuming your opponent will also play optimally to counter you.

One of the oldest and most fundamental techniques is Minimax. Imagine a simple game like Tic-Tac-Toe. Minimax works by having the AI “look ahead” through all possible future moves, assuming that you (the opponent) will always choose the move that is best for you and worst for the AI. The AI then picks the move that minimizes its maximum possible loss (or maximizes its minimum possible gain). Effectively, it plays out all possible future scenarios in its head and chooses the path that leaves it in the best possible position, no matter what its opponent does.

For games with an incredibly vast number of possibilities, like Go, simply looking ahead through every move is impossible. This is where Monte Carlo Tree Search (MCTS) comes in. Instead of exhaustively analyzing every branch, MCTS “plays out” many random simulations of the game from a given point. It explores the most promising moves more deeply, learning which paths lead to success through repeated “trial and error” simulations. This allows AI to tackle games that were once considered beyond computational reach, like when Google’s AlphaGo beat the world’s best Go players.

Structured Search: Satisfying All Conditions

Sometimes, the goal isn’t to find the “best” path, but simply any solution that meets a specific set of requirements. These are constraint satisfaction problems. Here, the AI needs to find values for a set of variables such that all given conditions, or “constraints,” are simultaneously met.

Think about solving a Sudoku puzzle. You need to fill in numbers from 1 to 9 in each cell, but with strict rules: each row, column, and 3x3 box must contain all digits from 1 to 9 without repetition. The AI’s task is to find a set of numbers for all empty cells that satisfies all these constraints.

Another common example is creating a university class schedule. You have classes, rooms, professors, and students, and a multitude of constraints: Professor A can’t teach two classes at the same time; Room B can only hold 50 students; Class C requires a lab; no two classes can be in the same room at the same time. The AI’s job is to assign times and rooms to all classes such that every single constraint is satisfied. The “structure” of these problems, defined by the variables and their interdependencies, allows AI to use specialized search techniques to efficiently find a valid solution.

Subscribe now

Knowledge Representation & Reasoning

Beyond just searching for solutions, a truly intelligent system needs to “know” things about the world. This brings us to the second pillar of GOFAI: Knowledge Representation. This field explores how AI can efficiently represent, store, and use domain knowledge in a way that computers can understand and process.

The fundamental goal of knowledge representation is to organize concepts and facts, as well as the relationships between them. This organization allows AI to reason about these facts and discover new relations. Ultimately, it’s about giving AI a structured way to “understand” and make sense of information, much like how humans build a mental model of the world around them. Without a clear way to represent what it “knows,” an AI would be unable to make logical inferences or apply its knowledge to new situations.

From Raw Observations to Understanding

To truly grasp how AI “knows” things, it’s helpful to understand the progression from raw observations to actionable understanding. At the most basic level, we encounter Data, which consists of raw, unprocessed facts or observations. This could be a list of numbers, individual words, or pixels in an image. In isolation, data has no inherent meaning; for example, the number “30” by itself is just a number.

When we introduce context or metadata, data transforms into Information. For instance, if we know “30” is a temperature reading taken in Celsius at noon on July 1st in Havana, it becomes information. This contextualization helps us relate different observations and gives them initial meaning.

Finally, when information is enriched with semantics and rules, enabling inference, reasoning, and the discovery of new relations, it becomes Knowledge. For example, if the AI knows that “temperatures above 30 degrees Celsius in July in Havana indicate a heatwave,” it possesses knowledge. This knowledge allows it to draw inferences (it’s a heatwave!), discover new relations (heatwaves can lead to increased energy consumption), and even take actions (warn residents about high temperatures). It’s this ability to add meaning and logical connections that truly transforms information into actionable knowledge.

Ways to Represent Knowledge

Just as humans use different ways to store and recall information, from precise definitions to vague intuitions, AI employs various methods for knowledge representation, each with its own strengths and weaknesses.

One key distinction lies between explicit and implicit representations. Explicit knowledge is clearly defined and directly encoded, often in rules or symbols. It’s much like a precisely written dictionary or a rulebook where every term and every rule is spelled out. This approach is central to Symbolic AI. For instance, Ontologies are explicit representations that define concepts within a domain and their strict relationships. Think of a meticulously designed family tree formally defining “parent,” “child,” “sibling,” and “ancestor,” along with rules such as “if A is a parent of B, and B is a parent of C, then A is a grandparent of C.”

Conversely, implicit knowledge is learned from patterns in data, rather than being directly programmed. It’s more akin to human intuition or a “gut feeling” developed from vast experience, and is fundamental to Statistical AI. Embeddings, for example, are numerical representations where concepts like words, images, or even entire documents are transformed into points in a multi-dimensional space. Systems like Word2Vec learn these embeddings by analyzing how words are used together, so words with similar meanings or contexts (e.g., “king” and “queen”) end up being numerically “close” to each other in this space, even though no human explicitly programmed that relationship.

Another way to categorize knowledge representations is by their formality. Formal representations have strict, unambiguous syntax and semantics, making them ideal for precise logical inference and computation. Mathematical equations, programming code, or statements in formal logic are prime examples, leaving no room for misinterpretation. In contrast, informal representations are more flexible, often using natural human language. While easier for humans to create and understand, they can be ambiguous and require more sophisticated processing for AI to extract meaning, as seen in a written description, a casual conversation, or an essay.

Finally, we distinguish between structured and unstructured representations. Structured knowledge is organized in a predefined, rigid format, making it easy for computers to process and query. Think of data in a spreadsheet with clear rows and columns, or a database with defined fields.

Knowledge graphs, for instance, are structured representations that organize facts as a network of interconnected entities (nodes) and their relationships (edges). A knowledge graph might have a node for “Paris,” a node for “France,” and an edge labeled “isCapitalOf” connecting them, allowing AI to easily query and infer facts.

Conversely, unstructured knowledge exists in free-form text, images, audio, or video, without a predefined schema. Extracting meaning from unstructured data is much harder and often requires advanced AI techniques.

Vector databases, for example, are often used to store and efficiently search implicit representations (embeddings) derived from unstructured data. You could take millions of research papers (unstructured text), convert each into an embedding (implicit representation), and store them in a vector database. Then, when a user asks a question, the database can find the most “similar” papers based on their embeddings, even though the papers themselves are unstructured.

Drawing Inference from Knowledge

Knowledge representation isn’t merely about storing information; its ultimate purpose is to enable AI to draw inferences and make decisions. This process of deriving new conclusions from existing knowledge is known as reasoning, and it can take both formal and informal forms.

Formal reasoning, deeply rooted in logic, is about deriving new conclusions from existing knowledge using strict, unambiguous rules. This is the hallmark of Symbolic AI. It’s a process of deduction, where if the initial premises are true and the rules are applied correctly, the conclusion is guaranteed to be true. For example, if a knowledge base contains the rules “All birds can fly” and “A sparrow is a bird,” a formal reasoner can deduce, with absolute certainty, “A sparrow can fly.” Such rule-based systems are precise and auditable, but they are limited by the completeness and accuracy of the explicitly programmed rules.

In contrast, informal reasoning is about drawing conclusions based on patterns, similarities, or analogies, often without strict logical guarantees. This type of reasoning is more akin to human intuition or common sense. It’s less about strict deduction and more about finding connections and probabilities. For example, if an AI has learned implicit representations (embeddings) of various animals, and it sees a new animal that is “numerically close” to many dogs, it might infer it’s a dog, even without explicit rules for every single feature.

This distinction is crucial for understanding the different capabilities of AI. While formal reasoning provides certainty within defined boundaries, informal reasoning allows AI to operate in ambiguous, unstructured environments. The latter, particularly reasoning by analogy in embeddings and language models, will be explored in more detail in later chapters, showcasing how AI can make sense of the world even when explicit rules are unavailable.

What is the Best Representation for Knowledge?

The choice of how to represent knowledge is a critical decision in AI design. Different representation types are chosen based on the specific problem, the type of data available, and the AI paradigm being used (Symbolic vs. Statistical). For instance, a Symbolic AI system designed for medical diagnosis might rely heavily on formal, explicit ontologies of diseases and symptoms. Conversely, a Statistical AI system for image recognition might primarily use implicit, unstructured representations of pixels that it learns from millions of example images.

This challenge highlights a theoretical result known as the Ugly Duckling Theorem. This theorem, in essence, states that without a specific purpose or “bias,” all objects are equally similar or dissimilar to one another. This implies that there is no single, universally “best” way to represent knowledge or measure similarity without a context or goal in mind. For example, an “ugly duckling” is only ugly relative to a flock of swans; it might be beautiful among other ducklings.

Therefore, the human responsibility in choosing the right representation is paramount. This choice directly impacts what an AI can “know,” how it can “reason,” and ultimately, the reliability and fairness of its inferences. Aligning the representation with the problem’s nature is a key part of building human-centered tools that truly understand and assist us.

Conclusion: The Need for Learning

Good Old-Fashioned AI (GOFAI), with its focus on search, optimization, and explicit knowledge representation, laid the essential groundwork for the field of Artificial Intelligence. Its strengths lie in domains where problems are well-defined, rules are clear, and knowledge can be precisely encoded. GOFAI systems offered precision and control, making them powerful tools for tasks like proving theorems or playing well-defined board games.

However, the ambitions of GOFAI soon ran into fundamental limitations when faced with the messy complexity of the real world. These systems proved to be brittle: a small change outside their programmed domain could break them entirely. They struggled immensely with common-sense knowledge, which is vast and often unstated. The sheer scale of real-world information made it an “insurmountable challenge” to explicitly program every piece of knowledge and every rule. GOFAI was excellent at solving problems for which it was explicitly programmed, but it couldn’t adapt, generalize, or handle unstructured data effectively. This revealed a crucial gap in AI’s capabilities.

The limitations of GOFAI highlighted a profound truth: to build truly intelligent and adaptable systems, AI needed to move beyond simply executing pre-programmed rules. It revealed a crucial need for systems that could learn from experience and data, without being explicitly programmed for every single scenario or piece of knowledge.

This growing realization of the power of learning-based approaches, which were developing concurrently with GOFAI, marked a significant shift. It showed that AI could discover its own patterns and adapt to unforeseen situations, offering a path to overcome the brittleness of purely symbolic systems.

Recognizing these limitations and actively seeking new approaches is a hallmark of the ongoing, human-driven effort to build more capable and adaptable AI. It’s a testament to our techno-pragmatist ethos: acknowledging challenges, learning from past efforts, and continuously striving to create tools that can better serve humanity’s complex needs. This increasing prominence of learning methods, which developed in parallel to GOFAI, is the story that will unfold in the next chapter.

Thank you for reading this far. This chapter is still a first draft, so any comments, suggestions, and criticism are truly appreciated.

Leave a comment

PS: Get your copy of Mostly Harmless AI at 50% off.

The following is a first draft of Chapter 7 of my upcoming book Mostly Harmless AI.

Of all the domains being transformed by artificial intelligence, education is perhaps the most critical to get right. The stakes are uniquely high. Used wisely, AI has the potential to be a massively positive force, augmenting the work of teachers and deepening the learning of students in ways we are only beginning to imagine. Used incorrectly, however, it could be catastrophic, undermining the development of critical thinking and eroding the very foundations of academic integrity. This chapter is a guide to navigating that high-stakes environment.

We will begin by demystifying the popular idea of a personalized AI tutor, a vision that runs counter to the principles of human-centered, collaborative learning. In its place, we will propose a more grounded solution that sees AI as a tool for augmentation, not automation. Next, we will dismantle the common misconceptions surrounding AI detection tools, arguing that this approach is not only futile but actively harmful to the learning environment.

This will establish the necessity of a fundamental pedagogical shift, moving from policing to integration. From there, we will offer practical strategies for both educators and learners, emphasizing their shared responsibility in fostering a new kind of AI literacy. Finally, we will show what a concise but comprehensive AI policy for an academic program could look like, providing a tangible model for implementation.

The Myth of the Personalized Tutor

The arrival of powerful generative AI has fueled a seductive, decades-old myth: that the ultimate goal of technology in education is to create a personalized, all-knowing AI tutor for every learner. This vision promises a revolution, a future where a “personalized Aristotelian tutor” is available to every student, adapting to their unique learning style, language, and pace. This narrative is powerful, but it is built on a fundamental misunderstanding of how we learn and what education is for.

Even if such a perfect tutor were achievable, it is not the revolution we should want. The idea that education’s primary problem is a lack of personalization or efficient information delivery is a flawed premise. Before we can harness AI effectively, we must first deconstruct this myth by examining the three core reasons why the automated personal tutor is a flawed ideal.

Argument 1: It Mistakes Information Transfer for Learning

The myth of the personalized tutor assumes that the primary obstacle to learning is the inefficient delivery of information. This argument has some merit in specific contexts; in places where the main obstacle to education is a lack of access to books, internet, and educators, an AI tutor could be a game-changer. However, this is not the case for the majority of learners in developed nations.

In an era of information surplus, the problem for the modern student is not a lack of access to information, but a lack of skill in navigating, evaluating, and synthesizing it. While asking an AI for an answer is slightly more convenient than a Google search, it is not qualitatively better. Furthermore, it removes the “desirable difficulty” that forges lasting knowledge. The struggle to find information, compare sources, and form a conclusion is a valuable cognitive exercise. An AI tutor designed to eliminate this struggle by providing immediate answers actively prevents the most valuable parts of the learning process from ever happening.

Argument 2: It Promotes Intellectual Dependency, Not Critical Thinking

The myth suggests that an AI can be a perfect partner for completing assignments, from solving math problems to writing essays. This, however, risks creating profound intellectual dependency. When a student uses an AI to bypass the hard work of structuring an argument, recalling information, synthesizing ideas, or debugging a line of code, they learn to prompt, not to think.

The purpose of assigning an essay is not to receive a perfect text; professors already know the answers. The purpose is to engage the student in the process of creation, which is where learning occurs. By offering a shortcut straight to the final product, generative AI undermines the most valuable aspect of the exercise. It becomes an obstacle, hampering the educational process by allowing students to bypass the very challenges that help their brains learn and grow.

The goal of education is to build independent, critical thinkers who can grapple with complex, ambiguous problems on their own. Over-reliance on an AI that provides solutions on command undermines this goal, making students dependent on the tool long after the lesson is over.

Argument 3: It Champions Isolation Over Community

The vision of a personalized path idealizes a student learning in perfect, isolated efficiency, free from the pace of a group. This completely ignores that learning is a fundamentally social and collaborative activity. Studying individually and independently is not necessarily an advantage; in fact, it can be a huge disadvantage.

The two things most self-educated people struggle with are motivation and feedback. Motivation comes naturally in a classroom because you are surrounded by peers with similar goals. Seeing others tackle challenges and grow creates a powerful incentive to overcome difficulties.

Feedback from mentors and peers is equally crucial for intellectual growth, allowing us to iterate on ideas and hone our skills. A community of learners is key. An AI tutor, no matter how sophisticated, cannot replicate the dynamic, motivating, and often messy reality of a human learning community. Learning together always beats learning alone.

The Alternative

The alternative to the flawed myth of the personalized tutor is to view AI not as an automated teacher, but as a powerful tool for augmentation within a human-centered community. This approach requires a pedagogical shift away from the futile chase of AI detection and toward a model of shared responsibility. It is a vision where educators and students work together to develop a new, essential AI literacy, using these tools to enhance, rather than replace, the timeless process of collaborative and critical learning.

Why AI Detection Is Futile

Before educators can effectively integrate AI, they must first understand that the detection of AI-generated content is a hopeless chase. Any attempt to police AI use through detection tools is an unwinnable arms race destined to fail for a number of practical and pedagogical reasons.

First, the technology itself is fundamentally flawed. Detectors will always be lagging behind the generative models they seek to identify, perpetually playing catch-up in a race they cannot win. The supposed telltales of AI-generated text—overly formal language, a lack of personal voice, perfect grammar—are not robust signals. They are merely fleeting characteristics of specific models at a single point in time. A detector trained to spot GPT-4’s style is useless against the next generation of models, and it’s even more useless against a student who uses one of the clever prompt techniques this very book teaches to make the output more human-like.

Second, these tools are dangerously inaccurate. Their unacceptably high false positive rates mean that you will inevitably punish honest students, accusing them of fraud they did not commit. This is an ethical line no educator should be willing to cross. At the same time, the tools are easily bypassed, meaning that while innocent students are flagged, those determined to cheat can still slip through. The result is a system that is both unjust and ineffective.

This cat-and-mouse game also creates perverse incentives. It encourages students to spend more time hiding and tinkering with AI to bypass detectors than on the actual intellectual work of the assignment. Their focus shifts from critical thinking to “evasion engineering.” This is the exact opposite of the goal of education.

Ultimately, a reliance on detection tools creates an environment of distrust that is toxic to learning. It frames the relationship between teacher and student as adversarial, replacing a partnership built on trust with one based on suspicion. Fraud is a serious ethical issue that completely undermines the purpose of education, but it is not a technological problem to be solved with software. It is a human one that must be discussed on ethical grounds, as a violation of the shared trust that makes a learning community possible. When fraud is committed, we all lose.

A Practical Guide for Educators

The only viable path forward is to shift our mindset from policing to integration, adapting our methods to leverage AI’s strengths while mitigating its weaknesses.

Redesigning Assignments for the AI Era

With the traditional take-home essay now vulnerable to automation, educators must redesign assignments to incorporate AI as a tool for thinking, not a machine for answers. This requires a fundamental shift in what we choose to assess.

The most effective strategy is to focus on process, not just product. Instead of grading only the final essay or report, the assessment can be expanded to include the student’s engagement with the AI. Requiring students to submit their chat logs or a written reflection on their process—detailing the prompts they used, how they evaluated the AI’s output, and the modifications they made—makes their thinking visible. This turns the inquiry itself into the gradable artifact, rewarding critical engagement over simple content generation.

Another powerful approach is to turn students into AI critics. Instead of asking them to produce a text, assign them the task of deconstructing an AI-generated one. For example, a student could be asked to prompt an AI to write an essay on a historical event and then write their own analysis of its factual errors, logical fallacies, and underlying biases. This transforms the assignment from a simple writing task into a high-level critical thinking exercise, teaching students to be skeptical and analytical consumers of AI-generated content.

Finally, it is essential to emphasize human-centric assessments that are inherently resistant to automation. These methods evaluate skills that AI cannot replicate, such as real-time argumentation, interpersonal collaboration, and embodied knowledge. This includes a renewed focus on in-class discussions and Socratic seminars, oral exams and presentations, timed hand-written essays, and hands-on lab work or collaborative projects. While these redesigned assignments require a different kind of engagement, the time saved by using AI for administrative tasks can be reinvested here, creating a more sustainable and pedagogically valuable workflow.

AI as a Teacher’s Super-Assistant

AI’s greatest potential may lie in its ability to reduce the significant administrative burden on teachers, freeing them up to focus on the deeply human work of teaching and mentoring.

As a tool for lesson planning and differentiation, AI can be an invaluable creative partner. An educator can brainstorm engaging lesson plans, get suggestions for creative activities, or generate differentiated materials—such as simplified texts or vocabulary lists—for students with diverse learning needs in a fraction of the time it would take manually. For instance, a teacher could use a prompt like: “Act as an instructional designer. Create a 45-minute lesson plan for 10th graders on the causes of World War I, including a hook, a collaborative activity, and a formative assessment.”

For rubric and feedback generation, AI can be truly transformative. It can draft clear, comprehensive grading rubrics in seconds. More importantly, it can help solve the feedback bottleneck by providing initial, personalized feedback on student work. An educator can quickly review a student’s draft, identify key areas for improvement, and instruct the AI to provide detailed, constructive feedback on those specific points, without rewriting the text for the student. The teacher then reviews and approves the AI’s feedback before sending it. This “human-in-the-loop” model allows teachers to provide timely, detailed, and individualized feedback at a scale that was previously impossible. A teacher might use a prompt like: “Here is a paragraph I wrote. Provide feedback focusing on the strength of their topic sentence and their use of evidence, but do not rewrite it for them.”

Fostering an AI-Ready Classroom

Creating a healthy learning environment in the age of AI requires a proactive approach centered on clear policies, digital literacy, and open communication.

The foundation is to establish a clear classroom AI policy. Every educator should develop a simple, flexible policy for AI use and review it regularly. This policy should function as a guide for ethical engagement, not a list of prohibitions. It is crucial to define what constitutes constructive, ethical use (e.g., brainstorming, getting feedback on one’s own writing) versus what constitutes academic dishonesty (e.g., submitting AI-generated text as one’s own).

Beyond rules, educators must integrate AI literacy into the curriculum. It cannot be assumed that students understand how these tools work. This means dedicating class time to educating students on the capabilities, limitations, and ethical considerations of AI. This includes teaching practical skills like effective prompt engineering and essential concepts like how to spot AI “hallucinations” and the subtle ways that training data can introduce bias into the model’s output.

A simple and effective way to guide students is to create and share custom prompts and reusable AIs. By crafting prompts that are tailored to specific pedagogical goals—for example, a template designed to encourage critical analysis of a source—educators can model effective AI use. An even more powerful extension of this is to create shareable, custom AIs, often called “Custom GPTs” or “Gems.” These are specialized versions of the AI that are pre-loaded with specific instructions and context. An educator could create a “History Thesis Helper” that is an expert in their course material, or a “Lab Report Formatter” that guides students through the required structure. Sharing these resources not only helps students get better results but also embeds the desired learning process directly into the tool they are using.

Finally, it is vital to foster open dialogue. An educator should create a classroom culture where students feel comfortable and safe discussing the role of AI in their learning, asking questions, and even sharing their mistakes. By addressing the ethical implications and potential pitfalls of AI tools openly, the classroom becomes a collaborative space for exploring this new technology, fostering a sense of shared responsibility for its ethical use.

It is important to recognize that “AI burnout” is a reality. Many educators feel an immense pressure to adapt to everything at once, and that they have no time to do so. But this is not true. While we cannot dismiss AI, we do not have to change everything at the same time. The most sustainable path is one of small, deliberate experiments. By injecting AI into the easier parts of our teaching tasks first, we can achieve some easy wins, build our confidence, and give ourselves the time to reflect on the consequences before moving on to more ambitious integrations. The checklist below offers a simple way to begin.

A Four-Step Checklist for Educators

For educators feeling overwhelmed, here is a simple, actionable checklist to begin integrating AI into your practice:

1. Create and Discuss Your AI Policy: Draft your classroom AI policy using the appendix as a model. The most important step is to discuss it openly with your students on the first day. Frame it as a shared agreement for ethical engagement.

2. Use AI as an Assistant for One Task: Pick one administrative task this week and use an AI to help. Draft a lesson plan, create a rubric for an upcoming assignment, or generate a set of discussion questions. Experience the tool’s power and limitations firsthand.

3. Redesign One Assignment: Choose one of your existing assignments and brainstorm how you could redesign it to focus more on process, critical evaluation, or in-class performance. Start small and iterate.

4. Share a Resource: Create and share a custom GPT or a well-crafted prompt template designed to help your students kickstart one self-study activity or assignment. This models good practice and provides a valuable resource.

A Guide for the Modern Learner

For students, AI can be the most powerful learning tool ever created, but only if used with intention and integrity. The goal is to use AI to learn, not to short-circuit your own understanding. This requires a conscious shift from viewing AI as an answer machine to viewing it as a thinking partner.

Your Responsibilities as a User

Ethical use of AI begins with a clear understanding of your responsibilities. First and foremost, you must verify and clarify policies. Every course and institution will have different guidelines for AI use; it is your responsibility to know them and, when in doubt, to ask your instructor. Second, practice transparent disclosure. Being honest about how and where you have used AI in your assignments is a cornerstone of academic integrity and builds trust with your educators. Finally, you must protect sensitive information. Never input personal, confidential, or proprietary data into public AI models, as you have no control over how that data might be used or stored.

Using AI to Kickstart Your Work

One of the most effective and ethical ways to use AI is as a brainstorming partner to overcome the inertia of a blank page. You can use AI to generate initial ideas for a project, create a structured outline for an essay, or synthesize the key points from a long article. In this role, the AI acts as a catalyst for your own thinking, providing a foundation upon which you can build your original work. The goal is to use it to support your thinking, not replace it.

Using AI to Deepen Understanding

Instead of asking for a direct answer, use AI to guide you toward your own understanding. You can turn the AI into a Socratic partner that asks you questions instead of giving you solutions. For example, a prompt like “I’m trying to understand the causes of the French Revolution. Don’t list them for me. Instead, ask me questions that will lead me to the key factors” transforms a passive query into an active learning exercise. This approach reintroduces the “desirable difficulty” that is essential for true learning, using the AI to guide you rather than carry you.

AI is also an excellent tool for concept exploration. When faced with a complex idea, you can ask the AI to explain it in simpler terms or through an analogy, such as “Explain the concept of general relativity to me as if I were 12 years old.” This helps you build an intuitive grasp of the material that goes beyond rote memorization.

Using AI to Refine Your Skills

AI can be an invaluable coach for improving your practical skills through iterative feedback. As a writing coach, it can offer suggestions on clarity, tone, and structure without doing the writing for you. You can submit a paragraph you have written and ask for specific feedback, such as “Can you suggest three stronger verbs I could use in this sentence?”

As a practice partner, AI can generate an infinite number of practice problems for subjects like math, coding, or language vocabulary. You can ask it to create a quiz for you and then, crucially, to provide detailed explanations for any questions you get wrong, allowing you to learn from your mistakes in a low-stakes environment.

Build Your Own AI Tools

Beyond one-off prompts, the next level of AI literacy is learning to create your own reusable AI assistants. Modern AI platforms allow you to create “Custom GPTs” or “Gems”—specialized versions of the AI that you pre-program with your own instructions and knowledge. This is a powerful way to personalize your learning. For example, you could build a “Study Buddy” and upload all your course notes, empowering it to quiz you on the specific material. You could create a “Socratic Tutor” that is permanently instructed to only ask you guiding questions and never give direct answers. By building your own tools, you move from being a simple user to a creator, a skill that is becoming increasingly valuable.

Developing AI Literacy

Ultimately, the most important skill for a 21st-century learner is not just knowing how to use AI, but knowing how to critically evaluate its output. Never trust blindly. This new “AI literacy” is built on three pillars.

First, always be skeptical. Treat every statement an AI generates as a claim, not a fact. Second, fact-check everything. AI models can and will “hallucinate” incorrect information with complete confidence. You are the ultimate authority and are responsible for the accuracy of your work. Always use trusted, primary sources to verify any factual information the AI provides. Finally, learn to look for bias. Understand that the AI’s training data is a reflection of the vast and messy internet, full of human biases and stereotypes. Always question the perspective of the text it generates and be aware of its inherent limitations.

Putting It All Together

Here is a step-by-step example of how you might ethically use AI to help with a research paper:

1. Brainstorming: Use the AI to explore potential topics and narrow your focus.

2. Outlining: Work with the AI to structure your main arguments and create a logical outline.

3. Research: Use the AI to find sources or summarize articles, but always go to the original source to read it yourself and fact-check every claim.

4. Drafting: Write the full draft in your own words, using your outline and research.

5. Feedback: Ask the AI for feedback on the clarity, structure, and style of your draft.

6. Submission Checklist: Before submitting, review this list:

   • Have I fact-checked every claim that originated from the AI?

   • Can I explain and defend every part of this work in my own words?

   • Have I followed my instructor’s AI policy to the letter?

   • Does my declaration accurately and specifically describe how I used AI in this assignment?

Conclusion

The techno-pragmatist ethos that guides this book is rooted in a fundamental belief: the future is not predetermined. Technology is a tool whose impact is profoundly shaped by how we choose to employ it, and this is nowhere more true than in education. As a college professor, this is not an abstract debate for me; it is a topic I care about deeply, and I feel a profound responsibility to get it right.

The challenge is not to resist this new technology, but to harness it with wisdom. Instead of chasing the flawed ideal of automation or descending into an adversarial relationship based on detection, we must embrace a necessary pedagogical shift. The central problem in modern education is not a lack of content, but a scarcity of timely, personalized feedback. High student-to-teacher ratios make it nearly impossible for educators to provide the deep, iterative guidance that is crucial for student growth.

This is where AI can create a true revolution. Therefore, the true north for AI in education is not automation, but augmentation. We must leverage AI to solve the feedback bottleneck, using it to do what it does best—process information and provide feedback at scale—so that we, educators and learners, can focus on what we do best: questioning, creating, and collaborating within a human-centered community.

It is from this techno-pragmatist perspective that we have offered these guides. The strategies herein are not just tips and tricks; they are a framework for shouldering the shared responsibility of building a new AI literacy, ensuring that these powerful tools serve, rather than subvert, the timeless goals of a meaningful education.

Thanks for reading so far! As a complementary resource, here is a draft of an AI Policy for STEM classes. Feel free to, share, modify, and reuse it as you see fit.

Example AI Policy for STEM Classrooms

31.6KB ∙ PDF file

Download

Download

This is a first draft of Chapter 7 of my upcoming book Mostly Harmless AI. Please, do share with me all your comments, suggestions, criticisms, and ideas.

Leave a comment

This article is based on Chapter 4 of my upcoming book Mostly Harmless AI.

Having journeyed through the foundations of artificial intelligence—its history, its mechanics, and its limitations—we arrive at the most pressing question: How do we actually use the powerful new tools it has produced?

This article serves as a bridge from the theoretical to the practical. This is not a list of traditional prompt engineering hacks. The internet is filled with tips and tricks for coaxing a specific output from a language model for a single task. Instead, the advice that follows builds on our foundational understanding to offer something more durable: a general mindset for working with these tools. This approach is about engaging with language models in a way that allows you to get the best out of them without subcontracting your own critical thinking. It is a methodology for augmenting your intellect, not replacing it.

The principles you learn here are foundational, offering a universal toolkit for interacting with large language models in daily life—whether you are planning a vacation, trying to understand a complex news article, or drafting a simple email. In the following chapters (in the book), we explore how to adapt and intensify these practices for specialized, high-stakes professional environments. But first, every user must learn how to engage with these powerful yet fallible tools safely, critically, and effectively.

A Methodology for Effective Interactions with Language Models

To move beyond simple queries and unlock the true potential of language models, we need a more structured approach. This methodology is divided into three parts: establishing the right Mindset, employing effective Tactics during the conversation, and building a System to make your successes repeatable.

The Mindset

The most significant shift is in your mental model. Instead of treating the model as a search engine, you should approach it as a conversational partner. This means recognizing that the interaction is iterative and that your most important role is to guide the conversation.

A key part of this mindset is adopting a Socratic (or inquiring) approach, where you use the model not just to get answers, but to help you ask better questions. This is invaluable for sensitive and important tasks.

For example, instead of starting with “Write an email asking for a raise,” a partner-based approach would be to ask the model to guide you: “I need to write an email to my manager to ask for a raise. What are the key pieces of information and evidence I should gather first to make the strongest possible case?” The model will then prompt you for your accomplishments and market data, helping you build your argument before a single word is written.

Similarly, when organizing a child’s birthday party, you could ask, “I’m planning a science-themed party for my 7-year-old. What are the key logistical details I need to consider to make sure it runs smoothly?” In both cases, you are using the model to help you define the problem, which is a far more powerful use of its capabilities.

The Tactics

With the right mindset, you can employ specific tactics to steer the conversation toward a high-quality outcome. The most fundamental tactic is to be explicit and strategic with your queries. To ground the model’s response in reliable information, tell it where to look.

A generic query for medical advice is risky, whereas a much safer prompt would be: “Search for information from the Mayo Clinic and the World Health Organization on the common symptoms of iron deficiency.” This specificity is also crucial when comparing complex options, like, for example, buying an EV, you can be as specific as: “Compare the Tesla Model 3, the Hyundai Ioniq 5, and the Ford Mustang Mach-E for a family of four. Focus on real-world range, charging speed on a standard home charger, and available cargo space.”

To get an even more robust answer, you can move beyond a single query and assemble a ‘committee of experts.’ A single language model will give you its most statistically likely answer, which might not be the most creative or well-rounded one. To overcome this, you can generate multiple, independent perspectives.

For the EV comparison, you could open three separate conversation windows. In the first, you’d ask the model to act as a pragmatic engineer and, perhaps, the model will argue for the Hyundai. In the second prompt, you’d ask it to be a tech enthusiast, and maybe it makes the case for the Tesla. In the third, you’d have it act as a family-focused reviewer, causing it to arguing for the Ford.

By copying these three independent analyses into a final chat window, you can then ask the model to act as a senior editor, synthesizing the competing viewpoints into a final, balanced recommendation that weighs factors like cost, range, and reliability.

Finally, after the model provides a response—either a single answer or a synthesized one from your committee—you can employ self-criticism as a final refinement tactic.

Once you have a draft of your email asking for a raise, you can prompt it: “Read the email you just drafted. Now, act as my manager who is busy and skeptical. What parts of this email are unconvincing? Is the tone too demanding or not confident enough?” This critical step often surfaces weaknesses that you might have missed, allowing you to create a much stronger final product.

The System

The final part of the methodology is to turn your successful interactions into a repeatable system. A common mistake is to treat prompts as disposable. A more powerful approach is to build a library of reusable prompts, thinking of them as personal “natural language programs.” The multi-step process you used to plan the birthday party can be saved as a “Kids’ Party Planner” template.

The Socratic prompt that helped you prepare for your salary negotiation can be generalized into a “Career Conversation Prep” tool. The ultimate expression of this principle is the use of features like OpenAI’s “Custom GPTs,” which allow you to encapsulate a complex task into a dedicated tool that you or your team can use with a simple request.

A Practical Example

To see how these principles combine into a powerful workflow, let’s walk through a comprehensive, real-world task: planning a 10-day family vacation to Italy.

Rather than beginning with a vague request like “plan a trip,” the process starts by applying the Socratic approach. You would first ask the model to frame the problem for you: “I want to plan a 10-day family vacation to Italy. What key information do you need from me to create the best possible itinerary?”

This immediately shifts the dynamic, positioning the model as a guided partner. In response, it would act as a consultant, asking for crucial details like the number of travelers, the children’s ages, your budget, family interests, and preferred travel pace.

Once you’ve provided this context, the next step is to ensure alignment. You would instruct the model to synthesize and confirm the constraints: “Great, thank you. Based on my answers, please summarize all of my constraints for this trip in a structured list.”

With a clear, confirmed set of requirements, you can then confidently ask for a first draft. The iterative heart of the process begins now. Upon receiving the initial itinerary, you would employ the self-criticism tactic: “This is a good start. Now, act as a skeptical travel agent. Criticize this itinerary and tell me what’s missing or what could go wrong.”

The model might point out that visiting three major cities in ten days is too ambitious for a family with young children. Based on this valuable feedback, you can guide the revision, continuing this loop of drafting and critiquing until the plan is refined to your satisfaction. Only then would you ask for the final, detailed output.

The final, powerful step is to generalize this success. You would ask the model to convert the entire conversation into a reusable “Family Vacation Planner” template, complete with placeholders for key details. This turns a one-time effort into a valuable, programmable asset for future trips, demonstrating the true power of thinking of prompts as reusable programs.

Common Pitfalls for the Everyday User

The good practices above are designed to improve the quality of a language model’s output. This section focuses on the mental traps and risks you must be aware of to use these tools safely.

The “Eliza Effect” and Misplaced Trust

Because chatbots are designed to be conversational and helpful, it’s easy to start treating them as if they have genuine understanding, intentions, or even consciousness. This is a modern version of the “ELIZA effect” we discussed in the history chapter. The danger is that this leads to misplaced trust, where we stop critically questioning the model’s output because it feels so confident and knowledgeable. This is the psychological trap that makes us vulnerable to hallucinations; we are less likely to fact-check a “partner” than a machine.

Cognitive Offloading and The “Lazy Brain” Problem

The ease of asking a language model to summarize an article, draft an email, or brainstorm ideas can lead to a subtle but significant danger: cognitive offloading. By outsourcing the fundamental work of thinking, synthesizing, and structuring our thoughts, we risk letting our own critical thinking and creative muscles atrophy. The goal is to use these tools to think better, not to think less. Over-reliance can make us less capable problem-solvers in the long run.

The Privacy Risk of Casual Conversation

In a casual conversation with a chatbot, it’s easy to forget that you are interacting with a complex system run by a corporation. Users often paste sensitive personal information—medical details, financial data, private emails, proprietary work content—into public language models without considering where that data goes, how it’s used for future training, or who might have access to it. What you tell the model does not stay between you and the model.

You Are the Final Authority

The techniques above teach you how to get better raw material from the language model. This final principle is about what you, the human, must do with that material. It is the most critical step in using these tools responsibly.

First, never trust, always verify. The language model is an unreliable narrator. Treat its output as a well-written first draft, not a finished fact. For any critical piece of information—a date, a statistic, a medical suggestion, a legal point—you must verify it using an independent, authoritative source. The model can help you find potential sources, but you are the fact-checker.

Second, synthesize, don’t just copy-paste. The model’s output is information; your goal is knowledge. The most important work happens after the model has responded. Your job is to synthesize its suggestions with your own experience, judgment, and goals. The model can generate a list of tourist sites for your Italy trip, but only you can synthesize that into a vacation plan that feels right for your family.

Finally, own the outcome. The language model is a tool, and you are the user. Any decision made, any email sent, or any action taken based on the model’s output is your responsibility. This principle of accountability is non-negotiable. The model is an assistant that can help you think, but it is not a replacement for your personal judgment.

Conclusion

The journey from a novice user to a skilled one is not about memorizing clever prompts; it’s about a fundamental shift in mindset. Instead of treating generative AI as a vending machine for answers—an approach fraught with risks of shallowness, bias, and error—we’ve seen the power of engaging it as a conversational partner.

The practices outlined in this article—the Socratic method, strategic querying, and, most importantly, critical verification—form a framework for responsible engagement. This framework places you, the user, firmly in the driver’s seat.

The quality of the model’s output is not a feature of the model alone; it is a direct reflection of the quality of your guidance and the rigor of your review. You are not just a prompter; you are a director, a critic, and a synthesizer. This is what makes these powerful tools ‘mostly harmless’: not their inherent nature, but our commitment to using them with critical awareness and human authority.

By mastering these foundational skills, you are not just learning to use a new tool. You are developing a new form of literacy for the 21st century. As we move into the specialized applications for knowledge workers, developers, and creatives in the following chapters (of the book), this ability to think with AI, not just ask of it, will be your most valuable asset.

Thanks again for reading. If you want to dive deeper into Artificial Intelligence and learn to make the best out of it, from a techno-pragmatist, human-centered, responsible perspective, please check out my book Mostly Harmless AI.

This article is based on Chapter 5 of my in-progress book Mostly Harmless AI.

Few developments in the generative AI space have been as exciting lately as the rise of code generators. The evolution of these AI coding assistants is best understood not as a single leap, but as a progression of capabilities, moving from simple autocomplete to what may one day be fully autonomous agents.

At their core, code generators are Large Language Models trained on vast amounts of public code. They treat programming languages just like human languages, learning the patterns, syntax, and structure to predict what comes next. These models can take a natural language prompt and some contextual code and produce new code that mostly aligns with the prompt's intention. For example, you can provide a function signature and a comment like, "This function finds the minimum of an unsorted list," and the model will generate the function's body.

This uncanny ability to comprehend and generate code based on human communication is transforming the development landscape, but it also requires special considerations, as code is not just another natural language.

In this article, we will explore the landscape of AI for software development. We will begin by looking under the hood, examining the spectrum of capabilities that allow AI to generate code. Next, we will explore the use cases for developers across the development lifecycle. Then, we will discuss some important things to keep in mind, from hallucinations and security to theoretical limitations of AI for coding. Finally, we will look to the future of coding, consider how the developer role is evolving, and try to answer one crucial question: is coding dead?

How to Make a Code Generator

Let's imagine we are building our own code generator from scratch. The journey from a simple code predictor to a sophisticated development partner is a journey of adding layers of capability, moving up a spectrum of increasing autonomy.

The first thing we want is next-token prediction for code. The foundational layer is built on unsupervised training, making it essentially autocomplete on steroids, like a super duper IntelliSense. We start by training a model on vast amounts of code, teaching it to predict the most likely next token based on the immediate context. If we have variables or functions declared nearby, our model is more likely to generate code that references them, simply because that's the most common pattern in its training data.

Now that we have a basic generator, our next step is to teach it to follow instructions. To do this, we can compile a dataset of instruction pairs—for example, a natural language command like, "In the previous code, change the loop to be more efficient," paired with the corrected code. By training on these examples, our model learns to go beyond simple prediction and follow specific, human-given directions. We can further enhance this process with Reinforcement Learning, where we have human or automatic evaluators rank different code outputs. This teaches our model to not only generate syntactically correct code, but also to respect desired styles and naming conventions.

Our generator is getting smarter, but it's still limited by the immediate context. To give it a long-term memory, we move to context-aware generation. We can dramatically enhance our model's ability to generate relevant code by allowing it to pull from a broader context. This is a form of Retrieval-Augmented Generation (RAG) for Code. We can index an entire codebase or external API documentation, allowing our model to find relevant examples and patterns. When a developer asks a question, our system retrieves these examples and feeds them into the prompt, allowing the model to generate accurate code by combining and refactoring snippets from the provided context, even for libraries it wasn't explicitly trained on.

So far, our model can only write code. To up our game, we can give it the ability to interact with its environment by making it use external tools. This represents a significant leap. We can equip our model with a set of tools it can invoke on demand. For example, in response to a prompt like "add a library for charting," our model could invoke a tool to install the missing dependency in the project. If you ask it to "check whether this works," it could invoke another tool to run the unit tests and report back the results. It could even use tools to directly modify files in the codebase. By giving our model the ability to take actions beyond just generating text, we empower it to participate more actively in the development process.

The final step on our journey is to give our code generator a bit of autonomy, creating what’s called an agentic system. This is the most advanced and forward-looking form of AI-based code generation. We can design an AI agent that takes a high-level goal, breaks it down into sub-tasks, writes code, generates tests, runs the code, and then analyzes the output or errors. Based on the results, it can then debug or modify the source code in a continuous loop, acting as a semi-autonomous developer to see a task through from start to finish.

Beyond these training methodologies, we can leverage the formal nature of code to improve our model's performance. Unlike natural language, code has strict syntactic rules that can be programmatically checked. One simple but effective technique is trial and error during inference: we can have our model produce several potential code snippets, run them through a linter, and automatically reject any that have parsing errors.

More advanced techniques can pre-process the training data, for instance by normalizing all variable names to a generic format like var0, var1, etc. This makes it much easier for our model to learn the structural relationships in code without being distracted by specific naming conventions, and we can substitute the actual names back in a post-processing step. These tricks leverage the fact that we are dealing with a very restricted syntax to make it easier for our language model to learn the rules.

Finally, to create the ultimate specialized assistant, we can go beyond RAG and fine-tune a model on a specific codebase. While RAG provides external context, fine-tuning actually updates the model's internal weights. By training a model on a company's entire private and proprietary codebase, we can create a version that has deeply internalized that organization's specific architectural patterns, internal APIs, and coding standards. This results in an AI partner that not only answers questions correctly but does so in a way that is idiomatic and aligned with the team established practices.

Use Cases for Developers

Understanding the engine is one thing; knowing how to use it is another. AI offers you a powerful toolbox that you can apply across the entire software development lifecycle. In this section, you will explore practical use cases across three key phases: Ideation and Design, Implementation and Development, and Verification and Explanation. You will see how you can use both sophisticated, LLM-based coding tools integrated directly into your IDE, as well as techniques that you can use with standard, general-purpose chat apps like ChatGPT or Claude, requiring no special integration at all.

Phase 1: Ideation and Design

Before you even dare writing a single line of AI-generated code, you can already use LLMs as powerful brainstorming partners for exploration and design. This is one of the most accessible ways you can use AI, as it doesn't require a specific tool or editor extension; you can do it effectively using general-purpose conversational AI applications like ChatGPT, Perplexity, or Gemini. Models with live browsing capabilities are often even better for this phase, as they can pull in the latest information about new frameworks, libraries, and design patterns.

The key is for you to treat this phase as an interactive exploration. Instead of asking for a single, ready-made answer, you should guide the model through an ideation process. Here's how you can do it: use a chain-of-thought approach by asking the model not just for a solution, but to "think step-by-step" through the pros and cons of different architectural choices.

A powerful pattern you can use is to ask the model to generate several variants—for example, "Propose three different ways to design the database schema for a social media app." Then, you can discuss the options back and forth, using self-critique prompts to have the model compare the alternatives it just generated. At the end of this collaborative session, you can ask the model to provide a structured summary of all the design decisions you have agreed upon, acting as an executive design document.

With this document in hand, you can then start a new, more focused session with a code-oriented model for the actual implementation.

Phase 2: Implementation and Development

The most straightforward way you can use AI in this phase is for generating short, self-contained code snippets. This can be for a well-known algorithm, a common pattern, or the use of a well-documented API. This is a task you can accomplish with any standard chat app, even outside your IDE. This is especially powerful for navigating the complex world of APIs and libraries.

As a professional programmer, you probably aren't spending that much time doing basic coding, like inserting numbers in a list. No, reality is 90% of the code you write is interface code with some external library you may not know well. Instead of manually searching documentation, you can simply ask an AI assistant, "How do I use this library to make a query that does X?" and get a ready-to-use snippet.

The next level of integration is bringing AI directly into your IDE. This can start with simple code completion, but the real power comes from integrating a full chat experience. This allows you to highlight a block of code and ask for specific changes, such as, "Refactor this function to be more efficient," and have the model modify the file directly. If the model has RAG capabilities and can scan your entire codebase, it gets even better. The modifications and additions it suggests will be consistent with your existing coding style and use your own libraries and methods, making the integration seamless.

At the far end of the spectrum is the full agentic mode, which is still in its infancy with tools like Cursor. This offers a much more hands-off development experience. Here, you can give a coding agent a high-level task, and it can modify several files, create new ones, and even run commands in the terminal to install missing dependencies.

Finally, you don't always need a full IDE. For one-off scripts or quick prototypes, you can use the "Canvas mode" in apps like ChatGPT, Claude, or Gemini. These provide a simple editor-like interface where you can iterate back and forth with the model to update a script. Some tools even allow you to run these scripts directly in the cloud, letting you build and test disposable web apps instantly.

Working with these tools introduces a new core skill, an AI-in-the-loop coding workflow—the day-to-day interactive process of collaborating with an AI. It involves an iterative cycle of prompting with a clear goal, carefully reviewing the AI's output, correcting its mistakes or flawed assumptions, and then re-prompting with more specific instructions or feedback.

Phase 3: Verification and Explanation

To ensure your code quality, you can use an AI to help generate a wide range of test cases. This is especially useful for uncovering corner cases that might not be immediately obvious, such as handling empty inputs, maximum values, or unusual user behaviors. You can do this with integrated AI coding tools, but it can also be as easy as uploading your codebase or relevant files to a standard chat app and asking it to suggest test cases.

You can ask for both code-based tests (like unit and integration tests) as well as descriptive tests (like user stories or manual testing scripts). In all these scenarios, it helps to instruct the model with a Chain-of-Thought prompt, asking it to first explain what behavior it wants to test, and only then provide the actual test. This ensures the tests are intentional and well-understood.

Furthermore, when you're faced with a cryptic error, you can use AI for debugging. You can feed the AI the error message, stack trace, and relevant code, and it can analyze the context to suggest potential causes for the bug and possible fixes, acting as an experienced pair programmer.

The opposite, code-to-language direction allows you to create powerful new workflows for understanding code. You can ask an AI for automatic documentation of functions or for natural language explanations of a complex code fragment. This can be done directly inside your IDE with an integrated tool, or with standard chat apps. For example, some tools allow you to connect a public GitHub repository and ask high-level questions on the fly, which is very good for getting a quick overview of a new codebase.

A particularly valuable use case is in legacy code modernization. One of the biggest challenges in the software industry is maintaining and updating old codebases. You can use AI to tackle this problem by feeding it legacy code (e.g., from an old COBOL or Java system) and asking it to analyze the logic, add explanatory comments, or even translate the entire system to a modern language and architecture. This can dramatically reduce the cost and risk associated with modernizing critical systems.

However, you should be aware of the critical gap between syntax and semantics—that is, between understanding what the code says versus what the code does.

The weaker models are mostly limited to describing what the code is saying syntactically (e.g., "this variable is changed to this array position"), this capability is improving all the time. More powerful models can often provide higher-level, semantic explanations of what the code is doing (e.g., "this loop is ensuring the first part of the array is always sorted"). But even the best models may not be able to grasp the full architectural details or business logic of a complex application.

Putting It All Together

Putting this all together, let's see how a complete workflow might look for tackling a specific, somewhat complicated feature in an ongoing app, like adding OAuth login.

First, you would start in ideation mode, interacting mostly in text with the AI. You would discuss a high-level overview of the required architecture changes, which parts of the app might be impacted, and the best libraries to use. The goal here is to produce a clear design roadmap before any code is written.

Next, you would move to implementation mode, going full hands-on with an agentic tool. You could assign the agent the high-level task from your roadmap: "Implement the OAuth login feature using the chosen library." The agent would then get to work, creating new files, modifying existing ones, and writing the necessary code. As it encounters errors or ambiguities, you would engage in a back-and-forth conversation to guide it, but the bulk of the mechanical coding would be handled by the agent.

Finally, you would enter review mode. Once the agent reports that the feature is complete, you could have a final conversation with the AI. You could ask it to analyze the git diff of all the changes it made, explain the rationale for its implementation choices, and generate comprehensive documentation for a pull request. After your final review and approval, you would then submit the PR for human review by your team.

Things to Keep in Mind

While the toolbox is powerful, it comes with sharp edges. The most important limitation in language modeling, in general, has been called the problem of hallucinations. In the context of code, this means AI-generated code is not infallible and can contain subtle bugs that require constant vigilance.

Hallucinations and Mistakes

The simplest way you can see hallucinations is when you get code that uses a new variable that doesn't exist or fails to close a parenthesis. Unlike with natural language, you can often detect these syntactic errors automatically with a linter or compiler, so many of the more harmless hallucinations are not relevant as they won't introduce subtle bugs.

A slightly more difficult hallucination is what we can call a semantic hallucination, where the model uses a wrong variable or function name that does exist in your codebase. In this case, you will not get a compiler error because you're using an existing symbol, but you will get the wrong behavior. This is much harder to find because it has the same problem as most hallucinations: you have to review the code and be knowledgeable enough to have been able to generate that code yourself.

The most insidious errors are logical flaws. This occurs when the code doesn't do anything obviously wrong—it uses the right variables and looks plausible—but it has some subtle logical mistake that leads to a bug. For example, finding that a variable is not updated at the right moment in a nested loop is a tricky problem even for human experts. These kinds of mistakes will introduce subtle, hard-to-detect bugs.

But even if the bugs are no worse than what a human would introduce, they pose a threat because of "automation bias." When you check code written by humans, you expect bugs. But when you're looking at machine-generated code, the only way programmers have ever interacted with it has been with rule-based systems like compilers, and that code is basically without mistakes.

So even if the language model makes errors that are, on average, no worse than what a regular programmer would make, they can still be harder to detect because they won't be the exact same mistakes a human would make, and we may be less on guard.

AI's Impact on Technical Debt

The rapid generation of code by AI presents a double-edged sword for technical debt. On one hand, you can use AI as a powerful tool to reduce existing debt. You can ask it to analyze your codebase for inefficiencies, suggest refactorings, or add missing documentation and tests, thereby improving code quality.

On the other hand, the very speed of AI can create new and more complex forms of technical debt. Relying heavily on "vibe coding" to quickly generate features without rigorous human review can lead to a codebase filled with poorly understood, inefficient, or subtly buggy logic. This AI-generated debt can be even harder to untangle later, as the original human intent behind the high-level prompt may be lost.

Biases in Generated Solutions

Models trained on a vast corpus of public code from the internet will inevitably learn from outdated examples. This can lead them to perpetuate outdated practices by suggesting deprecated functions, old library versions, or inefficient algorithms that are no longer considered best practice. An AI model will also often default to the most statistically common solution it has seen in its training data.

This can stifle creativity and lead to a homogenization of code, discouraging the exploration of more elegant or contextually appropriate solutions. Finally, just as AI can perpetuate harmful societal biases, it can also reproduce human biases from the code it was trained on. This can manifest as non-inclusive language in generated comments or variable names.

Security and Licensing Risks

A significant risk is that an AI can generate code with known security vulnerabilities. If the model was trained on public code containing flaws like SQL injection or buffer overflows, it may reproduce those same insecure patterns in its suggestions, creating a major security risk for the application.

Furthermore, the use of AI-generated code introduces complex legal questions. A model might reproduce a code snippet verbatim from a repository with a restrictive open-source license (like the GPL), inadvertently pulling that license's requirements into a proprietary project. The legal ownership of the AI-generated code itself remains a gray area, creating potential intellectual property challenges for companies.

The Economics of AI Development Tools

While these AI tools offer significant productivity boosts, they are not free. For development teams and organizations, it's important to consider the practical economics of their adoption. Most advanced AI coding assistants operate on a subscription model, which introduces a new operational cost. Team leads and CTOs must perform a cost-benefit analysis, weighing the price of the tools against the expected gains in developer speed, code quality, and reduced time-to-market. The return on investment (ROI) will depend heavily on how well a team integrates these tools into their workflow and whether the productivity gains justify the recurring expense.

Theoretical Limitations

Beyond the practical issues of hallucinations and biases, there is a more fundamental, formal limitation to what we can do automatically. This is captured by Rice's theorem, a cornerstone of theoretical computer science. In short, the theorem proves that there is no algorithm that can automatically check for any non-trivial semantic property of a program.

What does this mean in practice? A "non-trivial semantic property" is basically any interesting question about what a program does. For example: "Does this program ever crash?" or "Will this function always return a positive number?" or "Is this code free of security vulnerabilities?" Rice's theorem tells us that it is mathematically impossible to build a universal program that can answer these kinds of questions for every possible piece of code.

This highlights the theoretical impossibility of perfect, automated code verification. We will never be able to build an AI that can look at code generated by another AI (or a human) and formally guarantee that it does exactly what the natural language prompt intended. That problem is, in the general case, unsolvable.

However, this doesn't mean we should give up. Engineering isn't about theoretical perfection; it's about solving the average case in the best possible way and handling the most important edge cases reasonably well. While we can't achieve perfect verification, we can get pretty far with a combination of AI-generated tests, linters, and, most importantly, expert human review.

The Future of Coding

Given these tools and guardrails, the very nature of programming is set to transform. The focus will shift from the mechanics of writing code to the art of building systems. The term "vibe coding," popularized in developer communities, captures the essence of this shift. It describes a workflow where the developer's primary job is no longer to write precise, line-by-line syntax, but to describe the high-level behavior, intent, or "vibe" of the desired software to an AI partner. The focus moves from how to do something (the specific algorithm and syntax) to what needs to be done (the ultimate outcome and user experience), leaving the mechanical implementation details to the AI assistant.

This approach is incredibly powerful for rapid prototyping, hackathons, and short-term projects. A developer can quickly scaffold an entire application by describing its components in natural language, getting a functional prototype up and running in a fraction of the time it would take manually.

However, this method has significant limitations for larger, more detailed projects. "Vibe-based" instructions are often ambiguous and can be misinterpreted by the AI, leading to code that works for the happy path but fails on edge cases. For long-term, mission-critical software, the precision, maintainability, and strict adherence to architectural standards that come from deliberate, human-led coding remain indispensable. Vibe coding is a tool for speed and exploration, not a replacement for rigorous engineering.

In this new paradigm, future developers will become experts at wielding a suite of AI tools and agents. Skills in "prompt engineering," system design, and the critical review of AI output will become more valuable than the ability to recall specific syntax. The developer's role becomes one of guidance and orchestration, knowing which tool to use for which task and how to verify the results.

Looking ahead, this elevated role may involve assigning entire features or bug fixes to autonomous agents. These agents would manage the full lifecycle: understanding the ticket, writing the code, creating tests, committing to version control, and responding to feedback from the CI/CD pipeline. This doesn't eliminate the developer but elevates their role to that of a system architect and project manager, overseeing a team of AI agents.

Beyond the changes in workflow, it's worth contemplating how these tools will change the qualitative experience of being a developer. We must ask ourselves how it feels to code this way. Does offloading the cognitive burden of syntax and boilerplate make you dumber and cause you to forget how to code, or does it free up mental space, allowing you to become even more proficient in the things that truly matter—the high-level ideas and architecture?

We must also consider the social aspects. You now have a partner that is not a human. How will this impact teamwork? Will this AI partner become a virtual member of the team, participating in code reviews and design discussions? Or will it alienate developers into more lonely roles, as they interact more with their AI than with their human colleagues? How does a senior developer mentor a junior who can always get an instant answer from an AI, potentially masking gaps in their fundamental knowledge?

These are open questions we must navigate as we integrate these powerful new collaborators into our teams.

Final Remarks

So, Is Coding Dead?

There is a real concern that if AI can write 90% of the code in 10% of the time, nine out of ten programmers could be out of a job. And yes, every time automation has reached an existing industry, some jobs are destroyed as some skills become irrelevant.

However, I claim we must not fear the advent of AI coding assistants. Here’s why.

Writing code is by far neither the hardest nor the most time-consuming part of software development. The process of making software involves understanding requirements, talking with customers, user testing, and product design, all of which are at least one order of magnitude more difficult that actually typing code.

A hundredfold boost in productivity for a task that is only 10% of the overall process is huge, but it still leaves the other 90% of the human-centric work. We will still need to understand what our customers want, guide them through designing a software product, know the user base, and find a sustainable business model.

And no, you cannot simply simulate the end user with a language model, so the AI can prompt itself into making a usable product, because your end user will still be human. Human users are slow, get angry easily, don't understand your application, and don't know what it is they don't like about it. Until an AI can really replicate what it feels like to be a human—and at that point, will we still call it “artificial”—we can’t take the human out of the software development loop—or any creative loop, for the matter.

The biggest progress in the software creation process has always been because of innovation in the human side, not the machine side. Innovation in software engineering, management, and how you get people to work together and collaborate will continue to be the most important part of the software pipeline for a long time.

Furthermore, software is an industry that is nowhere near its saturation point. We have far more need for software than the number of people who can currently write it. Increased productivity will likely be met with increased demand, creating more and better software for more users.

Every leap in software productivity—from assembly to compilers, from C to object-oriented frameworks—has lowered the barrier to entry and brought more people into programming. AI tools will likely do the same, empowering more people to create software. The modern world runs on software, and in the future, basic programming literacy may become as common as basic math literacy is today.

Most people know enough math to get by in daily life without hiring a mathematician, and in the same way, more people will know enough programming to automate simple tasks. They will learn to say to their home computer, "When I get home, I want you to turn my lights on, but only if it's night and the electric bill is not above the average," and an AI will generate the code to make it happen. This expands the field rather than shrinking it.

So, should you learn to code? Definitely. There's going to be orders of magnitude more code written in the next few years than everything we've written in history.

But even if you never end up writing a single line of code unaided by AI—like I've never written a single line of production code unaided by syntax highlighting, a linter, or a type verifier—knowing how code works, how algorithms work, and why a specific programming construction works the way it works is the same as knowing basic math. Coding changes how your brain is wired, makes you think clearer, and increases your creativity.

Furthermore, even if you are not working in the software industry, learning to code is still an immensely enjoyable experience. Being able to create something that keeps working on its own is, I think, the ultimate toy.

So if you want to make a dent in the software industry and you're wondering if AI will get you out of the picture, don't worry. That won't happen anytime soon. Learn to code, learn the fundamentals, but also learn how to use these new tools. As in every moment in human history, if you apply yourself and do your best, you will be at the top of the league, and there will be a spot for you.